Three Gemma 4 Deployments on One T4G for Under $3: What the Runtime Changes, and What It Doesn't

When you strip a machine learning serving experiment down to its cheapest possible form, something interesting happens: you start catching mistakes that expensive infrastructure lets you ignore. That is the lesson buried inside a recent run comparing three different inference runtimes serving the same Gemma 4 E2B checkpoint on a single AWS G5g instance armed with a T4G GPU.

The Setup

Three runtimes, one model, one machine. The contenders were vLLM, JAX, and PyTorch — each tasked with serving Google's Gemma 4 E2B checkpoint through the same benchmark harness and measured against the same evaluation statistic. The goal was not to crown a winner but to observe what stays consistent across backends and what diverges when the underlying runtime changes.

The G5g instance was the right fit here. Its T4G GPU offers a balance of cost and capability that makes this kind of three-way comparison practical at a budget that would make most cloud bills blush. The entire experiment consumed 19 instance-hours across 4.5 instance-hours of active benchmarking — total spend coming in comfortably under three dollars.

What the Runtime Actually Changes

The most immediate difference between vLLM, JAX, and PyTorch in a serving context is how they handle memory and batch scheduling during inference. vLLM's PagedAttention mechanism reshapes how KV cache memory gets allocated, which in practice changes how throughput scales under concurrent requests. JAX takes a different path entirely, leaning on XLA compilation to fuse operations, which can tighten decode latency for certain model shapes. PyTorch sits somewhere in between, offering flexibility without the specialized inference kernels of the other two.

These are not cosmetic differences. Under load, the three runtimes produced meaningfully different throughput curves. But here is the catch — the ranking of their decode performance was not stable. The decode leaderboard reversed depending on how long the system had been running at startup. A runtime that led at boot time would sometimes trail hours later, and vice versa, driven by subtle interactions between memory fragmentation, cache warming, and JIT compilation phases that differ across backends.

What It Does Not Change

Despite the runtime variance, some things held rock solid across all three deployments. The model outputs themselves remained consistent — the Gemma 4 E2B checkpoint produces the same semantic results regardless of whether vLLM, JAX, or PyTorch is orchestrating the inference pass. This is the sanity check that matters most for anyone building on top of these runtimes. The serving backend is an implementation detail, not a correctness variable.

The evaluation harness and the benchmark statistic also remained constant, which is what made the comparison meaningful. Changing the measurement tool while changing the runtime would have turned this into an apples-to-oranges mess. Keeping the harness identical meant that observed differences could be confidently attributed to the runtime's behavior rather than to measurement artifacts.

Five Wrong Claims That Did Not Survive the Test

That under-three-dollar price tag was not incidental — it was deliberate. Running cheap and repeatable made it feasible to run the comparison multiple times, vary the warmup sequences, and catch five claims about relative performance that did not hold up under repeated measurement. Common culprits were assumptions about which runtime handles short prompts better, which one benefits more from longer context windows, and which backend scales most gracefully under burst traffic. All five looked plausible based on single-run data. None survived cross-runtime verification.

Expensive infrastructure has a way of encouraging single-run conclusions. When each experiment costs hundreds of dollars, there is pressure to trust the first number and move on. Budget constraints, in this case, forced the discipline of verification.

Takeaways for Practitioners

• Runtime choice shapes performance characteristics, not model behavior. The checkpoint is the checkpoint. The serving stack is a delivery mechanism.
• Decode rankings can flip between boots. Warmup and memory state matter. Single measurements are fragile across any backend.
• Cheap, repeatable experiments catch systematic errors that expensive ones paper over. The constraint of a sub-three-dollar run forced rigor.

This kind of comparison is easy to dismiss as trivial when the hardware is modest and the cost is low. But the patterns it surfaces — runtime-dependent decode ordering, memory behavior divergence, the stability of model outputs across backends — apply just as much to expensive deployments. The only difference is that the cheap run makes you look twice.

Further reading: https://dev.to/gde/three-gemma-4-deployments-on-one-t4g-for-under-3-what-the-runtime-changes-and-what-it-doesnt-jo3