Affiliate disclosure: Some links on this page may be affiliate or sponsored links. We may earn a commission at no extra cost to you. This does not change our editorial criteria.

GPU rental for LLM inference

This page is a beginner guide for teams who already have a language model and need computers to answer questions with it. A large language model (LLM) is a computer program that predicts text. Inference means using a trained model, not teaching it. You send in a prompt. The model writes a reply. That is the job this page is about.

A GPU is a graphics processing unit: a chip that is very good at the math in generating those replies. GPU cloud means you rent those chips by the hour. Teaching the model from scratch lives on rent GPUs for LLM training. Teaching an existing model a smaller job lives on GPU rental for model fine-tuning.

Who it is for: product, platform, and procurement people who want a chatbot, an API, or an internal assistant without buying a server room yet. You do not need to be an ML engineer to follow this page.

You rent GPUs when you want to serve answers this week, traffic is still a guess, and you can shut the machines off if the product dies. If the same GPUs answer questions all day every day, and turning them off would break the product, a dedicated site starts to beat a rental meter. That is powered land.

What inference actually is

Inference is using the model. The heavy teaching already happened. What remains is: load the model into GPU memory, accept questions, and generate tokens. A token is a piece of a word. Latency is how long a user waits for the first pieces of the answer. Throughput is how many answers you can finish per hour.

The model has to fit in the GPU's own memory (often called VRAM), or be split across several GPUs. A model that does not fit will not start, no matter how cheap the hourly rate looks.

NVIDIA TensorRT is documented as a library for running neural networks fast on NVIDIA GPUs. vLLM is a widely used open-source engine for serving language models. NVIDIA also publishes NIM, which packages models as services. You do not have to use any one of those. You do need a cloud image with NVIDIA drivers and CUDA, which is NVIDIA's software layer for NVIDIA chips.

You rent GPUs when traffic is still a guess

You rent GPUs when you are launching an assistant, trying a new model size, or covering a spike. Elastic capacity (machines you can add and remove) beats a room of idle chips.

A typical rental inference job looks like this: pick a GPU with enough memory to hold the model, start a serving engine, put a private network in front of it, watch latency, and shut extra machines off at night if you can.

Ask whether the provider can keep the same GPU type. Serving code is picky about memory size. Ask where the users are. A cheap GPU on another continent can add enough delay to make the product feel broken. Ask what happens if the provider takes the machine back. Spot (interruptible) GPUs are a poor place for the only copy of a customer-facing assistant.

You rent GPUs when batching can keep the chip busy

Batching means grouping several questions so the GPU works on them together. A GPU sitting idle between single questions is wasted rent. This is why "cheap GPU" and "cheap inference" are not the same. Price the answers you actually serve, plus idle time. A chatbot that is quiet for 20 hours a day may still be a good rental. A chatbot that is busy 24 hours a day is how inference becomes work that never turns off. Keep the model weights (the saved numbers that are the model) next to the GPUs.

Inference is not training

Training updates the model. Inference does not. Do not buy training-shaped multi-GPU nodes for a small assistant that fits on one high-memory GPU. If you are still teaching the model, go to training or fine-tuning. Customer text and internal documents are still sensitive. Shared public notebooks are the wrong shape for prompts you would not paste into a random website. Use a private network slice (a VPC) or single-tenant nodes when the prompts or weights are restricted.

What to check before you rent

Memory first: does the model fit? Region second: if a person is waiting, the GPUs should live near those people. Tenancy third: who else shares the machine? Spot is fine for tests and a bad fit for a production endpoint. Confirm the serving engine (vLLM, TensorRT, or whatever you actually run) and CUDA drivers. Egress is the fee for copying data out of the cloud. Streaming answers is usually small. Copying model weights every day is not.

GPU cloud partners (placeholder)

Partner rows below are placeholders until terms are signed. No live rates. No outbound URLs. When links go live, they will use rel="sponsored".

This table is not a ranking. It will reflect partner relationships once partners exist.

GPU cloud pricing varies. Verify live rates.

There is no honest single dollar-per-hour number on this page. List prices move with GPU type, region, and whether you pay only when the machine is on. Quoted "from" rates often skip storage, data transfer, idle time, and the serving software. See the GPU cloud pricing index for how we talk about rates. We will not invent hourly prices here. Treat every public figure as stale until you re-check the provider.

Compare GPU-hours you will actually keep on, not a perfect-month utilization story. Always-on inference is a different math problem from a two-week training run.

When GPU rental stops working

Rental stops working when the endpoint is busy most hours, the model size is stable, and the next constraint is megawatts and a site, not another API key. A megawatt is a large unit of electrical power. You are then running an inference cluster that happens to live on someone else's floor.

The honest alternative is powered land: a development site with a credible path to large-load electricity. Browse the U.S. data center directory, then read when GPU clusters need powered land. Hybrid is normal. Rent the experiments and the spikes. Site the baseline that never turns off.

Find powered land when you need a site. Rent GPUs when you need capacity now.

Related

• Rent GPUs for LLM training and fine-tuning
• GPU rental for model fine-tuning
• When GPU clusters need powered land

FAQ

What is LLM inference in plain English?

It is using a language model that already exists to generate answers. You are not teaching it. You are running it.

How is inference different from training?

Training updates the model and usually needs many GPUs for a limited time. Inference serves answers and often stays on. A small assistant may fit on one GPU. A large training run will not.

Can I run inference on a cheaper spot GPU?

Only if a restart in the middle of a user's question is acceptable. Production endpoints usually want on-demand or dedicated machines. Spot is for tests.

How much GPU memory do I need?

Enough to hold the model (and some working space) in the GPU's own memory. If it does not fit, you need a bigger GPU or several GPUs that can share the model. Ask the provider for VRAM, not just a product name.

When is dedicated hardware cheaper than renting for inference?

When the same GPUs would be busy most hours for a planning horizon in years. Count real hours, including nights. We do not publish dollar-per-hour rates. Use the pricing index, then re-check the provider.

What if the limiter is power and a site, not another GPU?

Then rental is the wrong product. Use the powered-land GPU guide and what powered land is.

Sources

1. NVIDIA TensorRT
2. vLLM documentation
3. NVIDIA NIM
4. NVIDIA CUDA Toolkit
5. What is powered land
6. U.S. data center directory
7. Powered Lands data center map

Need land or power instead? See the map or contact.