Sweet Spot on a Cheap Cloud GPU: MoE, Vision Ceilings, and VRAM Budgeting
117 tokens per second from a “35-billion-parameter” model on a $0.27/hr GPU. The counterintuitive part: a smaller 27B model on the same GPU manages 33 tok/s — about 3.5× slower. The difference is not the parameter count on the spec sheet. It is how many of those parameters actually fire on every token.
This is the story of an afternoon spent learning to read a VRAM budget — and why “it fits” is the wrong question to ask.
This is Part 3 of a series. In Part 2 — From Local to Cloud GPU, the rented 80 GB datacenter GPU vanished overnight and the cheap replacement turned out to be the more interesting machine. Part 3 picks up where that left off: now that the pod exists, what do you actually run on it?
The goal was a single self-hosted LLM doing three things at once — chat in English and a second European language, read screenshots, and drive an agentic code editor — on the cheapest GPU the marketplace would rent (~24 GB VRAM, $0.25–0.40/hr). The answer turned out to be less about which model and more about understanding why a model fits.
The pod, briefly
A rented GPU pod is just an SSH box with a GPU bolted on. Nothing about LLM serving is pre-installed. The unglamorous 80% of this project was packaging: a small CLI that streams a single provisioning script over SSH and stands up the full stack — model server, web UI, search backend, HTTPS tunnel — idempotently.
Two infra details matter for the rest of the story:
- A GPU pod is an unprivileged container. No Docker-in-Docker, no
systemd in the usual sense. Every long-lived process (LLM server, web
UI, search) runs as a native process under
tmux, so a dropped SSH session does not kill the stack. - The container disk is ephemeral. Anything you want to survive a
stop/start — model weights, chat DB, TLS certs, package caches — has to
live on the mounted volume (
/workspace) or be symlinked there.
And one painful discovery: not all datacenters are equal. Two regions
on the same marketplace, same GPU class, produced wildly different
provisioning times. The bottleneck was egress to the Python package CDN —
one region pulled wheels at ~220 KB/s (a full provision took hours),
another at ~90 MB/s (minutes). Lesson: before you blame disk or CPU
for a slow build, curl a known wheel and time it. Network egress to
package mirrors varies by something like 400×.
The model search: MoE vs dense
The obvious opening question: what is the most capable model that fits 24 GB VRAM? The counterintuitive answer is a 35B Mixture-of-Experts model. Not a 27B dense model. The MoE wins on speed and capability.
The reason is the part of the spec sheet most people skip. A 35B MoE is not “a 35B model” at inference time. Only the routed experts — roughly 3B parameters — fire per token. The rest sit idle. So you get 35B-class quality at ~3B-class compute cost.
On a 24 GB GPU, this means:
- The full Q4-quantized 35B MoE (~22 GB of weights) fits entirely in VRAM at 96–100% GPU utilization.
- Throughput is ~117 tokens/sec — faster than many models a fraction of its size on paper.
The dense alternative — a 27B dense model where all 27.8B parameters are active on every token — manages only ~33 tok/s on the same GPU.
Throughput, same GPU, same coding prompt
| Model | Architecture | Active params/token | tokens/sec |
|---|---|---|---|
| 35B MoE coder (hand-tuned prompt) | MoE | ~3B | ~118 |
| 35B MoE coder (RL-tuned, SOTA coding bench) | MoE | ~3B | ~114 |
| 35B MoE base | MoE | ~3B | ~116 |
| 27B dense coder | dense | 27.8B | ~33 |
The three MoE rows are essentially the same family — they differ mostly in system prompt and fine-tuning. The dense row is a different architecture entirely. Dense ≠ better. It just means every token pays the full parameter bill.
The vision ceiling rabbit hole
This is where the post gets interesting, and where “it fits” starts lying to you.
Multimodal models bundle a CLIP vision encoder. To process an image, that encoder needs a ~250 MB VRAM spike — a transient allocation on top of the weights, the KV cache, and the running context. On a GPU already at 99% capacity from weights, that spike has nowhere to go. The result is a silent CUDA OOM at inference time, even though the model nominally “fits.”
That kicked off a systematic sweep: how does context-window size trade off against vision headroom?
Free VRAM vs context window (Q4 MoE 35B, 24 GB GPU)
| Context | Free VRAM | Image upload |
|---|---|---|
| 16k | 268 MB | ✅ safe |
| 24k | 92 MB | ✅ small images only |
| 26k | 50 MB | ❌ CUDA OOM |
| 32k | 170 MB | ❌ (vision spike > free) |
| 64k | 104 MB | ❌ |
So there is a vision ceiling somewhere around 25–40k context on 24 GB — not the 256k context length the model advertises on its spec sheet. The binding constraint is not the weights. It is the encoder spike plus the KV cache growing with context.
Two ways to push the ceiling
KV-cache quantization. Switching the attention KV cache from the
default f16 to q8_0 halves that allocation. On this particular hybrid
architecture (recurrent state plus attention), the saving is modest —
about 38 MB at 32k context — because most of the “memory” is a
non-quantizable recurrent state. But it was enough to cross the vision
threshold and lift the ceiling from ~25k to ~40k context, with no
throughput penalty.
Text-only mode. If you never send an image, the vision spike never happens, and context can grow all the way to the model’s training max (256k). The trade-off is progressive CPU offload as the KV cache outgrows VRAM: 256k context runs at about 28 tok/s, 128k at ~48 tok/s. Usable for long-document reasoning, painful for chat.
Dead ends worth flagging
So you do not repeat them.
The documented “reserve VRAM” knobs that do nothing. The model
server exposes two environment variables — call them *_GPU_OVERHEAD
and *_FIT_TARGET — documented as “reserve this much VRAM before
allocating model layers.” We verified they were set in the process
environment at 0, 1 GB, and 2 GB. The memory-fit logic ignored them
entirely. Layer allocation was byte-identical across all three runs.
Do not waste an afternoon on them.
Two models in VRAM silently offloads the loser. Loading a small 1.2B model alongside the big one on a 24 GB GPU sounded harmless. Measured throughput on the small model dropped from ~260 tok/s (100% GPU) to ~5 tok/s (92% CPU) — a 50–60× regression — purely because the bigger model was resident. Worse, the server does not rebalance when VRAM frees up. You have to unload and reload the small model to get it back on the GPU.
Agentic editors that override context length. Some agentic code
editors default to the model’s maximum training context (e.g.
262144), silently reloading the model at that size and triggering
heavy CPU offload. Your “fast” config suddenly gets slow and the only
symptom is throughput. Set the context length explicitly per-model, in
the model config, not the client.
The sweet spot
Two model variants, one $0.27/hr 24 GB GPU, no per-token billing, data never leaves the pod.
| Use case | Model | ctx | Notes |
|---|---|---|---|
| Chat + images | 35B MoE, Q4, system-tuned, q8_0 KV cache | 40k | Multilingual (EN + a second EU language), vision, light coding. ~83 tok/s. The q8_0 KV cache is what makes vision viable at this context. |
| Coding (agentic, in-editor) | 27B dense, /nothink prompt | 156000 | All params active → deeper per-token reasoning. ~33 tok/s but produced a ~90%-working build of Prompt-Vault on first try. ~5.7 GB VRAM headroom. |
Total cost: under $0.30/hour, fully self-hosted.
The coding track running live on the pod, captured straight from ollama ps:
$ ollama ps
NAME ID SIZE PROCESSOR CONTEXT UNTIL
qwen3.6:27b-coder 9099e03d8e7a 22 GB 100% GPU 156000 Forever
22 GB resident on a 24 GB GPU, fully in VRAM (100% GPU, no CPU offload), holding a 156k context indefinitely. That is the headroom budget paying off — and the reason the dense coder gets the large context window while the MoE chat+vision track does not.
Lessons learned
- MoE changes the “what fits” calculus. A 35B MoE is a small model wearing a big model’s coat. Do not dismiss high-parameter models on small GPUs without checking the active parameter count.
- “Fits in VRAM” is a lie without headroom. A model whose weights sum to “24 GB on a 24 GB GPU” is unusable for anything that spikes — vision, a second resident model, a growing KV cache. Always budget 1–2 GB of slack.
- Vision has a hidden cost. The encoder’s ~250 MB spike, not the weight total, is the binding constraint for multimodal inference. Measure free VRAM, not weights-summed.
- Measure, do not assume. Documented knobs that do nothing, “free VRAM” that is not contiguous, throughput that varies 3.5× for “the same” model class — the only reliable answers come from a quick sweep on your actual hardware.
Final thoughts
You do not need an 80 GB GPU to get a genuinely useful multi-tool assistant. You need an afternoon with a VRAM budget spreadsheet and the willingness to distrust the spec sheet. The cheap GPU was always enough — it just took some time to understand exactly why.
Credit: This walkthrough was inspired by this video, which explores the practical differences between Qwen 3.5 and 3.6 — highlighting how architecture choices like Mixture-of-Experts versus dense models affect performance. For the agentic-coding track, this video demonstrates configuring the Zed editor to interact with local LLMs.