Run Llama 3.1 70B on a Locally GPU server in 10 minutes

A 70B-class model is where local AI stops feeling like a toy. Reasoning improves, instructions stick, and long documents stay coherent. The catch is hardware: you need roughly 40 GB of memory for a well-quantized 70B, which is exactly why our GPU 4090 instance (24 GB VRAM + 64 GB RAM) is the sweet spot. Ollama splits the model between GPU and system memory automatically, and the result is fast enough for real-time chat.

This guide takes you from an empty server to a private, authenticated LLM endpoint your apps can call. Total time is about 10 minutes, most of it spent waiting for the model download.

1. Deploy the instance

From your Locally panel, deploy a GPU 4090 with the Ubuntu 24.04 + CUDA template. Drivers, CUDA 12 and Docker come preinstalled, so there is nothing to configure at the OS level. Pick the region closest to where your requests will originate.

When the instance shows Active, connect over SSH:

ssh root@YOUR_SERVER_IP

Verify the GPU is visible before anything else:

nvidia-smi

You should see the RTX 4090 with 24576 MiB of memory. If you do, the hard part is already over.

2. Install Ollama

One command:

curl -fsSL https://ollama.com/install.sh | sh

The installer detects the NVIDIA driver and registers Ollama as a systemd service, so it survives reboots and starts automatically. Confirm it is running:

systemctl status ollama

3. Pull and run the model

Llama 3.1 70B in its default 4-bit quantization is about 40 GB. On our network this downloads in a few minutes:

ollama pull llama3.1:70b
ollama run llama3.1:70b

That second command drops you into an interactive chat. Ask it something you would normally send to a cloud API and watch the tokens stream. On the 4090 instance you can expect roughly 15 to 25 tokens per second depending on context length, comfortably conversational.

4. Expose a private API

Ollama serves an OpenAI-compatible API on port 11434, bound to localhost by default. The clean way to reach it from your laptop is an SSH tunnel, which keeps the port closed to the internet:

ssh -N -L 11434:127.0.0.1:11434 root@YOUR_SERVER_IP

Now any tool on your machine that speaks the OpenAI format can point at http://localhost:11434/v1:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:70b",
    "messages": [{"role": "user", "content": "Summarize KVM in one line."}]
  }'

If you prefer a public endpoint with authentication, put Caddy or Nginx in front with an API key header and enable the Locally edge firewall to allowlist only your IPs. Both setups are covered in our hardening checklist.

5. Keep it healthy

• Watch VRAM. nvidia-smi -l 2 gives you a live view. If you load two large models at once, the second spills to RAM and slows down.
• Snapshot after setup. One click in the panel. If an experiment breaks the box, restore in minutes instead of reinstalling.
• Pin your model versions. Tags like llama3.1:70b-instruct-q4_K_M guarantee the exact quantization survives upgrades.

The math that makes this worth it

Moderate usage of a frontier cloud API lands between $150 and $400 per month once real context lengths are involved. The GPU 4090 instance is a flat $129, runs any open model you want, keeps every prompt on your hardware and doubles as a full server for whatever else you host. Once inference is a fixed cost, you stop rationing your own product.

Your models. Your machine. Your rules. That is the entire pitch, and after ten minutes on a fresh instance, it is also just how your stack works now.

Questions about sizing, quantization or a specific model? Talk to us, we run these workloads daily and will point you at the right plan for free.