Running Gemma 4 on EC2 G5g: Graviton2 AMD with NVIDIA GPU

If you've ever tried to serve a modern transformer on genuinely weird silicon, you already know the drill: half the stack assumes x86, the other half assumes NVIDIA, and the corner where both expectations break is where the real field report begins. Running Gemma 4 E2B with vLLM on an AWS G5g instance — an Arm Graviton2 host paired with an NVIDIA T4G (SM 7.5) — sits squarely in that corner. There is no published build that targets this exact combination, which is part of what makes it interesting.

The headline problem isn't CUDA. It's not driver compatibility, and it isn't the absence of prebuilt aarch64 wheels for PyTorch or vLLM — those exist, and AWS's own documentation quietly walks you through bootstrapping them on Graviton. What actually blocks a working deployment is a far more mundane detail: the default 64 KiB shared-memory limit that vLLM's multiprocessing engine assumes is available for tensor-parallel coordination. On a vanilla G5g, the engine starts, briefly appears healthy, and then deadlocks or fails to spawn as soon as the rank workers try to exchange state.

The fix is unromantic. Bumping /dev/shm — either by mounting a tmpfs sized to a few gigabytes or by passing --shm-size through whatever container runtime is in play — is the single change that turns the setup from a stack trace into a model that responds. Once that's in place, the rest of the path is surprisingly conventional: a CUDA 11.8 aarch64 wheel, a matching vLLM build, the Gemma 4 E2B weights pulled from Hugging Face, and a launch command that looks identical to what you'd run on any other GPU box. The T4G's 16 GB of VRAM is more than enough for the E2B variant at reasonable batch sizes, and Graviton2's memory bandwidth keeps token generation from feeling sluggish.

What makes the G5g combination worth knowing about

• Price-to-FLOP. G5g instances are positioned as a cost-efficient GPU tier, and pairing them with a smaller E-class model like Gemma 4 E2B is a reasonable fit for inference workloads that don't need an A10G or larger.
• Arm + NVIDIA is a real niche. Most published vLLM guidance implicitly assumes an x86 host. Documenting a working aarch64 + SM 7.5 path closes a small but persistent hole.
• The bottleneck is plumbing, not silicon. The shared-memory limit is a Linux container default, not a GPU or CPU limitation, which is exactly the kind of failure mode that's easy to miss in benchmarks and obvious only in production logs.

For teams already standardized on Graviton — or simply curious whether an Arm host can serve a transformer without giving up GPU acceleration — the G5g is a quietly capable target. Just don't be surprised when the thing that finally lets traffic flow isn't a new driver, but a docker run flag.

Further reading: https://dev.to/gde/running-gemma-4-on-ec2-g5g-graviton2-amd-with-nvidia-gpu-25ci