GPU cloud pricing is where most self-hosting decisions go wrong. The headline rate on a provider’s front page is the price of the GPU, not the price of running your service. Published rates from 20 providers on 17 October 2026 put an NVIDIA H100 SXM 80GB on-demand at $2.69/hr on RunPod’s Community Cloud and $12.29/GPU-hr on an Azure ND96isr_H100_v5. A 4.6x gap before you count anything else.
Add persistent storage for your weights, egress for your responses, the idle minutes you spend debugging a crashed pod, and the minimum purchasable unit, and the same one-GPU model costs $2,017 on RunPod and $72,622 on Azure. A 36x spread. That is what this article is about.
The short answer: three different markets priced three different ways
There is no single GPU cloud market. There are three, priced on completely different logic.
The neocloud / independent tier (RunPod, Lambda, Vast.ai, CoreWeave, Crusoe, Together AI) sells dedicated GPUs directly and generally gives you free egress. The hyperscaler tier (AWS, Google Cloud, Azure, Oracle) sells eight-GPU nodes, bundles storage and networking into the bill, and charges $0.09/GB or more for data leaving the building. The serverless inference tier (RunPod Serverless, Replicate, Hugging Face Inference Endpoints, Modal, Baseten) bills by the second or minute but constrains which models you may run.
For self-hosting an arbitrary open-weight model, only the first two tiers are really open. The serverless tier generally cannot run your fine-tune.
“A Llama 3.1 70B Instruct checkpoint is 141.1 GB of safetensors. At RunPod’s network volume rate of $0.07/GB-month that is $9.87 every month: billed whether your GPU is running or not. On AWS S3 Standard at $0.023/GB-month it is $3.25. On top of that, moving 2 TB of responses out of AWS costs $175.32 and out of RunPod costs nothing.” All figures verified 17 October 2026.
What the cost model includes
Rates are from provider price lists and pricing APIs checked on 17 October 2026. Monthly totals are calculations under the stated workload assumptions, not measured invoices. Throughput figures come from MLCommons’ MLPerf Inference results or vendor specifications; no GPUs were benchmarked for this comparison. Unconfirmed storage and egress rates are marked in the tables.
Most models discussed here are open-weight. Llama 3.1, for example, uses a bespoke community licence rather than an OSI-approved open-source licence.
GPU cloud pricing: what each provider actually charges per hour
All figures are the provider’s own published rate, checked 17 October 2026. Prices are per GPU per hour unless stated. “Bookable” means the provider publishes both a price and a self-serve deploy path.
| Provider | GPU | VRAM | Rate | Tier | Smallest unit |
|---|---|---|---|---|---|
| RunPod | H100 SXM | 80 GB | $2.69/hr | Community Cloud, on-demand | 1 GPU |
| RunPod | H100 SXM | 80 GB | $3.99/hr | Secure Cloud, on-demand | 1 GPU |
| RunPod | H200 | 141 GB | $3.59/hr | Community Cloud, on-demand | 1 GPU |
| RunPod | H200 | 141 GB | $5.29/hr | Secure Cloud, on-demand | 1 GPU |
| RunPod | B200 | 180 GB | $5.98/hr | Community Cloud, on-demand | 1 GPU |
| RunPod | B200 | 180 GB | $7.99/hr | Secure Cloud, on-demand | 1 GPU |
| RunPod | RTX 4090 | 24 GB | $0.34/hr | Community Cloud, on-demand | 1 GPU |
| Lambda Cloud | H100 SXM | 80 GB | $3.99/hr | Self-serve, first-come | 1 GPU |
| Lambda Cloud | B200 SXM | 180 GB | $6.69/hr | Self-serve, first-come | 1 GPU |
| Lambda Cloud | A100 SXM | 80 GB | $2.79/hr | Self-serve, first-come | 1 GPU |
| Vast.ai | H100 NVL | 94 GB | from $1.98/hr | Host-set marketplace | 1 GPU |
| Vast.ai | L40S | 48 GB | from $0.47/hr | Host-set marketplace | 1 GPU |
| Vast.ai | RTX 4090 | 24 GB | from $0.12/hr | Host-set marketplace | 1 GPU |
| CoreWeave | HGX H100 | 80 GB | $6.16/GPU-hr ($49.24/hr node) | On-demand / $2.46 spot | 8 GPU |
| CoreWeave | HGX H200 | 141 GB | $6.31/GPU-hr ($50.44/hr node) | On-demand / $2.62 spot | 8 GPU |
| CoreWeave | HGX B200 | 180 GB | $8.60/GPU-hr ($68.80/hr node) | On-demand / $4.26 spot | 8 GPU |
| CoreWeave | GB200 NVL72 | 186 GB | $10.50/GPU-hr ($42.00/hr unit) | On-demand | 4 GPU |
| Crusoe | H100 80GB HGX | 80 GB | $4.50/GPU-hr | On-demand | 1 GPU |
| Crusoe | H200 141GB HGX | 141 GB | $5.40/GPU-hr | On-demand | 1 GPU |
| Crusoe | B200 180GB HGX | 180 GB | $8.50/GPU-hr | On-demand | 1 GPU |
| Crusoe | AMD MI300X | 192 GB | $3.45/GPU-hr | On-demand | 1 GPU |
| Together AI | HGX H100 | 80 GB | $3.99/hr on-demand, $1.99/hr preemptible | Dedicated endpoints | 1 GPU |
| Together AI | HGX H200 | 141 GB | $5.99/hr on-demand, $2.99/hr preemptible | Dedicated endpoints | 1 GPU |
| Together AI | HGX B200 | 180 GB | $8.19/hr on-demand, $4.09/hr preemptible | Dedicated endpoints | 1 GPU |
| Hugging Face | H200 (AWS) | 141 GB | $5.00/hr | Inference Endpoints, per minute | 1 GPU |
| Hugging Face | H100 (GCP) | 80 GB | $10.00/hr | Inference Endpoints, per minute | 1 GPU |
| AWS | p5.48xlarge (8x H100) | 80 GB | $55.04/hr node = $6.88/GPU-hr | On-demand | 8 GPU |
| AWS | p5en.48xlarge (8x H200) | 141 GB | $63.30/hr node = $7.91/GPU-hr | On-demand | 8 GPU |
| AWS | p6-b200.48xlarge (8x B200) | 180 GB | $113.93/hr node = $14.24/GPU-hr | On-demand | 8 GPU |
| Google Cloud | a3-highgpu-8g (8x H100) | 80 GB | $88.49/hr node = $11.06/GPU-hr | On-demand / $6.60 spot | 8 GPU |
| Google Cloud | a3-ultragpu-8g (8x H200) | 141 GB | $84.81/hr node = $10.60/GPU-hr | On-demand / $6.35 spot | 8 GPU |
| Google Cloud | a4-highgpu-8g (8x B200) | 180 GB | Flex-start $8.06/GPU-hr; on-demand N/A | Flex-start / spot | 8 GPU |
| Azure | ND96isr_H100_v5 (8x H100) | 80 GB | $98.32/hr node = $12.29/GPU-hr | On-demand / $2.27 spot | 8 GPU |
| Oracle Cloud | BM.GPU.H100.8 | 80 GB | $10.00/GPU-hr | On-demand | 8 GPU |
AWS per-GPU figures divide the node price in the AWS Price List by eight. Vast.ai’s figures are marketplace snapshots, not fixed tariffs: its documentation states that “pricing is set by the host and is specific to each offer.” Two samples of the cheapest H100 NVL offer ranged from $1.98/hr to $2.27/hr.
AMD’s MI300X is the wildcard. Crusoe lists it at $3.45/GPU-hr with 192 GB of HBM3: more memory than an H200 at under a third of Azure’s H100 rate. MI355X (288 GB) is contact-sales on Crusoe, so available but not self-serve. Google Cloud’s accelerator-optimized family does not include AMD GPUs at all.
The tier-naming trap
Every provider names its tiers differently, and the names hide real price differences. RunPod splits pods into Community Cloud (third-party data centre operators) and Secure Cloud (RunPod’s own facilities). For an H100 SXM that is $2.69 versus $3.99: a 48% premium. That tier difference matters more than the headline price.
Vast.ai has no tiers at all: pricing is per-host and per-offer, with host-set rates for rental, storage ($0.04-$0.67/GB-hour in the offers I sampled) and bandwidth ($0.0013-$0.0378/GiB). Together AI labels the discount preemptible, CoreWeave and AWS call it spot. AWS adds on-demand capacity reservations; Google Cloud offers Flex-start plus three flavours of committed use discount.
One genuine conflict worth flagging: Together AI’s pricing page lists an H100 at both $5.49/hr and $3.99/hr. These are different products. The $5.49 figure belongs to “Single-tenant GPU instances”; the $3.99 figure belongs to “On-demand hourly rates and reserved capacity” for dedicated endpoints, which their documentation confirms “bills per minute by hardware while a deployment runs, regardless of model or request volume.”
Minimum billing increments decide how much idle time you pay for
Billing granularity is the difference between paying for a 20-second model load and a 20-minute one.
- RunPod bills pods and serverless “by the second for compute and storage,” with serverless billed “rounded up to the nearest second.”
- Replicate publishes per-second rates: H100 $0.001525/sec ($5.49/hr), A100 80GB $0.0014/sec ($5.04/hr), L40S $0.000975/sec ($3.51/hr), T4 $0.000225/sec ($0.81/hr).
- Vast.ai charges base rental “for every second your instance is in the active/connected state.”
- Lambda Cloud is the outlier: on-demand instances are “billed in one-minute increments.”
- AWS EC2 bills per second with a 60-second minimum; Google Cloud bills a minimum of one minute, then per second.
- Hugging Face Inference Endpoints shows hourly rates but “the actual cost is calculated by the minute,” and adds scale-up hours for extra replicas.
If your workload runs for hours at a time this barely matters. If you are spinning up short jobs or debugging, a one-minute floor means a 15-second CUDA warm-up costs a full minute.
What breaks the headline $/hr number
Four cost lines do most of the damage. All rates verified 17 October 2026.
Egress. AWS charges $0.09/GB for the first 10 TB of internet transfer out of us-east-1, then $0.085, $0.07 and $0.05, with 100 GB free per month. Azure charges $0.087/GB for the first 100 GB then $0.083/GB in East US, plus $0.01/GB for inter-availability-zone traffic each way and $0.02/GB inter-region. RunPod states “no fees for data ingress or egress.” Lambda states “you are not charged for ingress or egress.” CoreWeave’s pricing table lists “Data transfer between CoreWeave and internet” as Free. Crusoe states it “does not charge for network ingress or egress.”
Persistent storage. A Llama 3.1 70B Instruct checkpoint is 141.1 GB of safetensors: confirmed from the Hugging Face model API on 17 October 2026. For comparison, Llama 3.1 405B Instruct is 811.7 GB and Qwen3-235B-A22B-Instruct-2507 is 470.2 GB. Published rates: RunPod container disk $0.10/GB-month, network volume $0.07/GB-month below 1 TB and $0.05 above. CoreWeave’s distributed file storage is $0.070/GB-month, object hot tier $0.06. Crusoe’s shared disks are $0.07/GiB-month. AWS S3 Standard in us-east-1 is $0.023/GB-month. Hugging Face Hub storage is listed at $8-$12 per TB-month with egress and CDN “included at no extra cost,” versus AWS S3 at $23 per TB-month.
Idle GPU time after a crash. No provider publishes this rate, because it is the same rate: that is the point. While your pod runs and you read a stack trace, RunPod and Lambda keep billing. RunPod is explicit that you pay for a “cold start: from when the worker starts until your handler is ready, mostly your handler loading models into GPU memory,” though container pulls and cached-model downloads happen while scaled down and are not billed.
Cold starts. RunPod’s serverless workers wait up to an idle timeout (default 5 seconds) after the last job before scaling down, billing through that window. Modal’s default scaledown_window is 60 seconds, and Modal warns that increasing it reduces cold starts but “you will be billed for any resources used while the container is idle.” Replicate splits the difference: for public models “setup and idle time for the model is free,” but for private models and deployments “you pay for all the time instances of the model are online.”
Worked example: the real monthly cost of a 70B endpoint
Assumptions, stated so you can change them: one H100 SXM 80GB; Llama 3.1 70B Instruct in FP8 so weights occupy roughly 70 GB and leave room for KV cache; 730 hours of continuous uptime; 500 GB of persistent storage; 2 TB of egress; and 6.9 hours of billed-but-idle time (one 15-minute model load, eight incidents averaging 35 minutes of debugging, plus eight 15-minute post-incident reloads).
Where a provider only sells eight-GPU nodes, I charge the whole node, because that is what you would actually buy.
| Provider | Compute (730 h) | Billed idle (6.9 h) | Storage (500 GB) | Egress (2 TB) | Monthly total | vs RunPod Community |
|---|---|---|---|---|---|---|
| RunPod Community Cloud | $1,963.70 | $18.56 | $35.00 | $0.00 | $2,017.26 | 1.0x |
| RunPod Secure Cloud | $2,912.70 | $27.53 | $35.00 | $0.00 | $2,975.23 | 1.5x |
| Together AI dedicated | $2,912.70 | $27.53 | $80.00 | $0.00 | $3,020.23 | 1.5x |
| Lambda Cloud | $2,912.70 | $27.53 | see note | $0.00 | ~$3,040.23 | 1.5x |
| Crusoe | $3,285.00 | $31.05 | $35.00 | $0.00 | $3,351.05 | 1.7x |
| CoreWeave (8x H100 node) | $35,945.20 | $339.76 | $35.00 | $0.00 | $36,319.96 | 18.0x |
| AWS (p5.48xlarge) | $40,179.20 | $379.78 | $11.50 | $175.32 | $40,745.80 | 20.2x |
| Oracle (BM.GPU.H100.8) | $58,400.00 | $552.00 | unverified | unverified | $58,952.00+ | 29.2x |
| Google Cloud (a3-highgpu-8g) | $64,597.99 | $610.58 | unverified | unverified | $65,208.58+ | 32.3x |
| Azure (ND96isr_H100_v5) | $71,773.60 | $678.41 | unverified | $170.38 | $72,622.39+ | 36.0x |
Lambda’s storage figure uses the $0.20/GiB-month rate Lambda’s billing documentation uses as an illustrative example. Lambda states the rate “might not reflect current pricing.”
Why the headline comparison is wrong by far more than 2-3x. The common version of this comparison is “$3.99/hr versus $12.29/hr, so the hyperscaler is 3x more expensive.” Arithmetically correct, practically useless, because it ignores packaging. Compare totals: $2,975.23 on RunPod Secure Cloud versus $72,622.39 on Azure is 24x. On AWS it is 13.7x. The reason is not that Azure’s H100s are slower. It is that a single 70B model does not need eight of them, and Azure will not sell you one.
The effect reverses when your workload genuinely needs eight GPUs resident: at which point the hyperscaler premium is worth checking against your own utilisation numbers. And one more hidden line: RunPod’s default spend limit is $80 per hour across all resources, with pods stopping automatically at $0 balance. On a multi-GPU prepaid plan, that limit is what wakes someone up at 3am.
GPU availability in October 2026: announced versus bookable
“Bookable” below means a published price and a self-serve deploy path. “Priced” means the hardware appears in a price list, while immediate availability remains unconfirmed.
Bookable at a single-GPU unit: RunPod (B300, B200, H200, H100 SXM/PCIe/NVL, A100 SXM/PCIe, L40S, RTX 4090, RTX 5090, L4, A40, in both tiers), Lambda Cloud (B200 SXM, H100 SXM, A100 SXM 80 GB and 40 GB, V100), Crusoe (B200, H200, H100, A100 SXM/PCIe, L40S, MI300X, B300), and Together AI dedicated endpoints (H100, H200, B200, B300 on-demand and preemptible).
Priced but not self-serve: Crusoe lists GB200 NVL72 and AMD MI355X as “Contact sales” for both on-demand and spot. Nebius publishes no per-GPU hourly price list at all: its compute pricing pages 404, and its homepage carries a vendor claim of “43% better TCO for fine-tuning vs. AWS” and “112% better TCO for inference vs. AWS” with no published methodology. Baseten’s dedicated deployment pricing is “Talk to Sales.”
On-demand availability in the published tables: Google Cloud’s table is the most explicit. For a4-highgpu-8g (B200) the on-demand column reads N/A, with only Flex-start ($8.06/GPU-hr) and Spot ($4.95/GPU-hr) populated. For A4X Max (GB300) and A4X (GB200), every column is No except “Future reservations in AI Hypercomputer.” A3 Ultra (H200) is likewise No for on-demand and on-demand reservations.
Marketplace reality: Vast.ai’s public bundles API returned 64 offers across 21 GPU models on 17 October 2026. H200 SXM, B200 and A100 SXM4 did not appear. The cheapest H100 NVL offer was $1.98/GPU-hr, but a second query minutes later returned $2.27 for the same model, which is why I would not put a Vast.ai number in a budget without checking it the morning you sign.
Queue times and “out of capacity” rates are not confirmed here. If capacity is your binding constraint, test availability before committing to a provider.
Performance: what MLPerf Inference v5.1 actually measures
Throughput claims on vendor pages are marketing. The audited dataset used here is MLCommons’ MLPerf Inference results. The latest published closed-division release is v5.1, released 28 October 2025: about twelve months old; the repository was last updated in March 2026. There is no v6.0 result set.
Methodology: 586 datacenter results from 32 submitters, all audited for compliance. I filtered to compliance == 1 with no errors, took the Llama 3.1 405B model, and grouped by accelerator, weight data type and scenario. Offline reports peak throughput, Server enforces a per-query latency target, Interactive enforces a per-token latency target. Results are whole-system, so they include CPU and interconnect, not just the GPU.
Llama 3.1 405B Server scenario, FP4 weights, eight-GPU B200 nodes, TensorRT 10.11 with CUDA 12.8-12.9:
| Submitter | System | Throughput (tokens/s) | vs NVIDIA reference |
|---|---|---|---|
| Nebius | 8x B200 SXM 180GB | 1,279.5 | +3.2% |
| Lenovo | 8x B200 SXM 180GB | 1,249.0 | +0.7% |
| NVIDIA DGX B200 | 1,246.8 | +0.5% | |
| Lambda | 8x B200 SXM 180GB | 1,246.8 | +0.5% |
| Dell | PowerEdge XE9680L | 1,246.6 | +0.5% |
| Oracle | BM.GPU.B200.8 | 1,243.7 | +0.3% |
| NVIDIA | NVIDIA DGX B200 | 1,240.4 | reference |
| NVIDIA | GB200 NVL72 (72 GPU) | 11,614.3 | : |
| NVIDIA | GB300 NVL72 (72 GPU) | 12,248.4 | : |
One conclusion dominates. Ten submitters (Nebius, Lenovo, Google, Lambda, Dell, Supermicro, GigaComputing, Broadcom, Oracle, NVIDIA) land between 1,240 and 1,280 tokens/s on identical hardware and software, a 3.2% spread. Which provider’s B200 you rent matters roughly one-thirtieth as much as which GPU you rent. Lambda and Google report byte-identical figures because both ran on a DGX B200, which shows the “provider” is often just the hardware vendor.
For smaller models the gap widens. On Llama 2 70B Offline in FP8, an 8x MI300X node reached 53,572 tokens/s and an 8x MI325X node 65,320, against 124,879 for a 32-GPU H100 configuration. Different node sizes and precisions, so read those as orders of magnitude, not a bake-off.
Two limitations. MLPerf fixes batch size and token lengths per model in the benchmark rules; those settings are not reproduced in this table, so check the scenario before comparing throughput. And these are 405B numbers: an 8B model on the same B200 is bound by memory bandwidth, not compute. Modal puts it plainly: small-batch language-model work “results in a bottleneck on memory, not arithmetic,” so a faster GPU “may not be worth the extra cost.”
Inference-specialised hardware and the supported-model trap
Groq’s LPU, Cerebras’ WSE, SambaNova, AWS Inferentia and Trainium, and Google TPU are genuinely faster per dollar on the models they support. The constraint is sharper than most comparisons admit.
Cerebras documents it explicitly: a “Model architecture” is “the model structure that an endpoint supports, such as qwen-3.8-27b. Custom weights must use the same architecture.” Dedicated Inference (where you would bring your own fine-tune) is “available to enterprise customers.” The self-serve developer tier publishes per-token pricing instead: GPT OSS 120B at $0.35/M input and $0.75/M output, Qwen 3.8 27B at $0.99/M and $1.49/M, with speeds of ~3,000 and ~1,850 tokens/s that are vendor-reported.
Groq publishes a fixed catalogue. The production list was Llama 3.1 8B, Llama 3.3 70B, GPT OSS 120B, GPT OSS 20B and Whisper variants. Llama 3.1 8B and Llama 3.3 70B list price as “Contact Sales.” GPT OSS 120B is $0.15/M input and $0.60/M output. Note the “Max File Size” column: several models are capped well below a 70B checkpoint.
AWS Inferentia and Trainium are cheapest per chip on paper: an inf2.xlarge is $0.7582/hr, trn1.32xlarge is $21.50/hr for sixteen chips, but both require compiling for the Neuron SDK and lock you to architectures with a Neuron implementation.
The rule: if you serve a stock model everyone runs, specialised hardware can win on cost per million tokens. If you serve a fine-tune nobody else runs, you need general-purpose GPU rental, and the $/hr table is the only comparison that matters.
How to choose: decision rules by workload
- Eliminate anyone who cannot run your weights. Check VRAM first, then whether the provider sells your architecture. A provider that will not deploy a fine-tune is not an option regardless of price.
- Establish true GPU-hours, not wall-clock hours. Measure idle fraction over a representative week. If it is under about 5%, serverless wins; above 20%, you want a dedicated instance.
- Multiply by the minimum purchasable unit before comparing. If your model fits on one GPU and the provider only sells eight, the effective rate is eight times the sticker price. This eliminates AWS, Azure, Google Cloud and Oracle for most 7B-70B serving.
- Add egress at your real volume. At $0.09/GB, a service returning 10 TB a month pays $900 for the privilege of leaving AWS. If your product returns meaningful volumes of generated content, free-egress providers are structurally cheaper and no amount of compute optimisation closes that gap.
- Price the tier, not the headline. RunPod Community versus Secure Cloud is a 48% difference on an H100 SXM. Together AI’s reserved tiers step H100 from $3.99 on-demand to $3.19 on a 181-day commitment.
- Use spot only for restartable batch work. AWS gives two minutes of warning; Google Cloud’s default notice is zero seconds with a best-effort 30-second termination, and neither is covered by a compute SLA. Together AI’s preemptible H100 at $1.99/hr is useless for an interactive endpoint.
- Do not upgrade hardware on schedule. Upgrade when you are memory-bound or throughput-bound, measured. MLPerf’s 3.2% vendor spread means the GPU model decides performance; the price gap decides cost.
- Get a quote for anything multi-node. Lambda’s 1-Click Clusters price 16 B200 GPUs at $9.86/GPU-hr for a two-week-to-one-year term, rising to $8.87 at 256 GPUs. Reserved capacity is where neoclouds actually compete with hyperscalers.
What breaks: failure modes and gotchas
Spot interruption is not a discount, it is a bet. Google Cloud is blunt: Spot VMs “are not covered by any SLA and are excluded from the Compute Engine SLA,” the default notice is zero seconds, and termination is “best-effort and up to 30 seconds.” If it lands mid-inference your user sees an error. Budget checkpoint frequency against restart cost.
Idle GPU time after a crash is the most under-priced line in cloud AI. A 141 GB checkpoint does not reload in seconds. Modal: “for models in the tens of gigabytes, this can reduce boot times from minutes to seconds”: meaning without the optimisation, minutes. Multiply by your incident rate.
Balance exhaustion deletes your data. Vast.ai: when credits reach $0.00 “your instances, storage volumes, and data will be scheduled for deletion,” and without a card on file “your instances and stored data will be destroyed.” RunPod stops pods at $0 balance; pods with a network volume keep their data, pods without one are terminated and “their data cannot be recovered.”
Storage outlives the GPU, and keeps billing. RunPod’s volume disk costs $0.10/GB-month running and $0.20/GB-month stopped. Lambda: “billing continues as long as a filesystem exists, even if it’s not mounted to an instance.” Google Cloud notes persistent disks on a preempted VM “will incur charges for storage until you delete them.”
Community Cloud is not Secure Cloud. RunPod’s cheaper tier runs on third-party hosts. If you handle regulated data that is a compliance decision, not a cost decision, and it belongs before the bill rather than after an incident.
Minimum billing increments punish short jobs. A 15-second container start on a one-minute floor is four times what you planned. AWS’s 60-second and Google Cloud’s one-minute minimums behave the same way.
Watch for price-list drift. Paperspace’s pricing URL now resolves to DigitalOcean’s pricing page, where GPU Cloud lists H100 at $3.09/hr on-demand and A100 at $2.24/hr with a footnote stating that rate is “for a 3-year commitment.” Always check whether the front-page number is the number you will be charged.
Licensing and compliance when you self-host
The hosting choice is rarely the compliance decision. The licence is.
Llama 3.1 ships under the Llama 3.1 Community License, a custom commercial licence, not an OSI-approved one. The licence text adds a threshold most self-hosters have never read: if, on the version’s release date, your products had more than 700 million monthly active users in the preceding calendar month, “you must request a license from Meta, which Meta may grant to you in its sole discretion.” It also incorporates Meta’s Acceptable Use Policy by reference. The model cards date to 23 July 2024, more than two years old. Check compatibility with your weight format and quantisation tools.
Contrast with permissive licences: Qwen3 and Mistral families ship under Apache 2.0 or MIT. If your bottleneck is legal review time rather than GPU time, that choice may be worth more than the hourly rate difference.
Review the licence before commercial use. Anything commercial or touching regulated data warrants review of the licence text, the Acceptable Use Policy, and any commitments you have made to customers about model provenance.
The bottom line
GPU cloud pricing rewards three behaviours and punishes three others.
Reward knowing your GPU-hours rather than your calendar hours. Reward free egress, which is the entire neocloud tier and none of the hyperscaler tier. Reward buying the right GPU rather than the fashionable GPU, because MLPerf’s 3.2% vendor spread is real and the 4.6x price spread is bigger.
Punish comparing sticker rates without checking the minimum purchasable unit. Punish ignoring that a 141 GB checkpoint costs real money to store and slow minutes to load. Punish putting an interactive endpoint on spot.
If I were starting today: RunPod Community Cloud on an H100 SXM for development, moving to Secure Cloud the moment real data touches it. Lambda if I want one vendor and predictable hardware, at the same $3.99/hr. Together AI’s preemptible tier for batch evaluation at $1.99/hr. And a real quote on a multi-node reservation before assuming AWS or Azure is the only option.
FAQ
What is the cheapest way to rent an H100 for LLM inference right now?
On 17 October 2026 the lowest publicly listed H100 SXM 80GB rate in this comparison was $2.69/hr on RunPod’s Community Cloud, against $3.99/hr on RunPod Secure Cloud, Lambda Cloud and Together AI’s dedicated endpoints, and $4.50/hr on Crusoe.
The tier split is the thing to understand. Community Cloud hosts run on third-party data centre operators; Secure Cloud runs on RunPod’s own facilities. That 48% premium buys a different physical security and compliance posture, and it is the cheapest real saving available in this comparison if your data allows it.
Vast.ai’s marketplace showed cheaper H100 NVL offers from $1.98/hr, but those are host-set prices that moved between two queries in the same session. A lead, not a budget line.
How much does it really cost to serve a 70B model for a month?
For one H100 SXM running continuously for 730 hours, I calculated $2,017.26 on RunPod Community Cloud and $2,975.23 on RunPod Secure Cloud, including 500 GB of persistent storage and roughly 6.9 hours of billed-but-idle time.
On AWS the same single-GPU workload costs $40,745.80, because the smallest purchasable unit is an eight-GPU p5.48xlarge at $55.04/hr. That gap is a packaging difference, not a hardware quality difference: you are paying for seven GPUs you are not using.
Do GPU clouds charge for data transfer?
It depends entirely on who you pick, and this is the widest divergence in the comparison.
RunPod states there are no fees for data ingress or egress. Lambda states you are not charged for ingress or egress. CoreWeave’s pricing table lists “Data transfer between CoreWeave and internet” as Free. Crusoe states it “does not charge for network ingress or egress, either within a VPC or to/from the public internet.”
AWS charges $0.09/GB for the first 10 TB of internet transfer out of us-east-1. Azure charges $0.087/GB for the first 100 GB then $0.083/GB, plus $0.01/GB inter-availability-zone each way.
For 2 TB a month that is $175.32 on AWS versus $0 on the four independent clouds. If your service returns generated content at volume, this line can outweigh every optimisation you have left to do.
Is spot pricing worth it for inference?
Only for batch work that can restart cleanly.
CoreWeave lists HGX H100 spot at $19.71/hr against $49.24/hr on-demand. Together AI lists preemptible H100 at $1.99/hr against $3.99/hr on-demand. Azure’s ND96isr_H100_v5 spot rate is $2.27/GPU-hr against $12.29 on-demand.
The catch is interruption. AWS provides a two-minute Spot interruption notice. Google Cloud’s default notice duration is zero seconds, with a best-effort termination period of up to 30 seconds, and Spot VMs are excluded from the Compute Engine SLA entirely. An endpoint that dies mid-request cannot use spot.
Why is my bill higher than the advertised price per GPU-hour?
Four usual reasons, in order of how often they bite.
First, persistent storage. A Llama 3.1 70B Instruct checkpoint is 141.1 GB of safetensors, which at RunPod’s $0.07/GB-month network volume rate is $9.87 every month: billed whether your GPU is running or not. A 405B model at 811.7 GB is $56.82.
Second, egress. Third, billed idle time: RunPod charges through the serverless cold-start phase while your handler loads weights into VRAM. Fourth, and usually by far the largest, the minimum purchasable unit.
Can I serve my own fine-tune on Groq or Cerebras?
Usually not, and the documentation is unusually direct.
Cerebras states that custom weights “must use the same architecture” the endpoint supports, and Dedicated Inference is enterprise-only, contact sales. Groq’s supported-model list is a fixed catalogue; Llama 3.1 8B and Llama 3.3 70B list price as “Contact Sales,” and several models carry an explicit max file size well below a 70B checkpoint.
If your fine-tune is on a supported architecture, great. If it is a custom mixture-of-experts or an unusual fine-tune, you need general-purpose GPU rental.
Is a faster GPU always worth the higher hourly price?
Compare tokens per dollar, not tokens per second.
MLPerf Inference v5.1 shows an 8xB200 node reaching 1,279.5 tokens/s on the Llama 3.1 405B Server scenario for Nebius versus 1,240.4 tokens/s for NVIDIA’s own DGX B200. Twelve submitters fall within 3.2% on identical hardware and software. The GPU model dominates performance almost entirely.
But prices differ by 4.6x per GPU-hour between RunPod and Azure. The cheaper instance of the same hardware class is almost always the bigger win.
What is the cheapest way to run a 7B or 8B model?
Consumer and workstation cards beat data-centre GPUs at this size, because the workload is memory-bandwidth-bound rather than compute-bound.
RunPod lists RTX 4090 at $0.34/hr on Community Cloud and $0.89/hr on Secure Cloud, and RTX 5090 at $0.69/hr and $1.19/hr. Vast.ai’s public marketplace listing on 17 October 2026 showed RTX 4090 offers from $0.1215/hr and RTX 5090 from $0.2129/hr.
At 8B parameters a 24 GB card is usually enough, and nobody needs an H100. Watch the no-NVLink caveat on multi-GPU consumer boxes above one card.
How do I know whether B200 or GB200 capacity is real?
Check whether the provider publishes a price and a self-serve deploy path.
RunPod lists B200 at $5.98/hr Community and $7.99/hr Secure with working deploy links. Lambda lists B200 SXM at $6.69/GPU-hr as self-serve first-come access. Together AI lists B200 on-demand at $8.19/GPU-hr and preemptible at $4.09/GPU-hr.
By contrast, Google Cloud’s table lists a4-highgpu-8g on-demand as N/A, with only Flex-start and Spot populated, and lists A4X (GB200) and A4X Max (GB300) as unavailable in every column except future reservations.
For rack-scale parts, expect to talk to sales. Crusoe lists GB200 NVL72 as contact-sales for both on-demand and spot.
Do I need legal advice to self-host an open-weight model?
It depends on the licence, and for anything commercially significant you should get advice.
Llama 3.1 is not open source in the OSI sense. It ships under the Llama 3.1 Community License, which requires you to request a separate licence from Meta if your products had more than 700 million monthly active users in the month preceding the version’s release date, and which incorporates Meta’s Acceptable Use Policy by reference.
Qwen and Mistral families ship under Apache 2.0 or MIT, with far fewer constraints. If you have a choice of base model and your bottleneck is legal review rather than GPU time, that choice may be worth more than the hourly rate difference.
Sources
- RunPod GPU cloud pricing: per-GPU hourly rates for Pods (Community vs Secure Cloud) and Serverless workers, verified 17 Oct 2026
- RunPod Pods pricing documentation: per-second billing, storage rates, no egress fees, $80/hr default spend limit
- RunPod Serverless pricing documentation: cold-start, execution and idle-time billing phases
- RunPod storage types: container, volume, network and global volume pricing
- Lambda GPU cloud pricing: instance and 1-Click Cluster per-GPU hourly rates
- Lambda billing overview: one-minute billing increments, filesystem billing, no ingress/egress charge
- Lambda filesystems: filesystem capacity limits and billing behaviour
- CoreWeave cloud pricing: on-demand, spot and inference rates; free internet data transfer; storage tiers
- Together AI pricing: dedicated endpoint on-demand, preemptible and reserved GPU rates; per-token pricing
- Together AI dedicated model inference: per-minute billing regardless of request volume
- Replicate pricing: per-second hardware rates for CPU, T4, L40S, A100, H100, H200
- Replicate billing documentation: setup and idle billing for private models and deployments
- Hugging Face pricing: Spaces hardware rates and Hub storage tiers with included egress
- Hugging Face Inference Endpoints pricing: per-minute instance rates across AWS and GCP
- Vast.ai billing documentation: host-set pricing, per-second rental, storage and bandwidth charges
- Vast.ai instances overview: per-second billing and marketplace model
- Crusoe pricing: per-GPU-hr on-demand and self-serve deployment rates for NVIDIA and AMD; free egress; storage rates
- AWS EC2 on-demand pricing: per-second billing with 60-second minimum; 100 GB free egress per month
- AWS EC2 Spot Instances: up to 90% discount claim
- Amazon EC2 spot instances user guide: two-minute Spot interruption notice
- AWS Price List API: AmazonEC2 on-demand: p5, p5en, p6-b200, p6-b300, p4d, p4de, g6, g6e, inf2 and trn1 rates in us-east-1
- AWS Price List API: AWSDataTransfer: internet data transfer out tiers in us-east-1
- AWS Price List API: AmazonS3: S3 Standard storage rate in us-east-1
- Azure Retail Prices API: ND96isr_H100_v5, NC96ads_A100_v4 and East US bandwidth tiers
- Google Cloud accelerator-optimized pricing: a2, a3 and a4 machine type on-demand, Flex-start, Spot and CUD rates
- Google Compute Engine VM instance pricing: one-minute minimum billing, then per second
- Google Cloud accelerator-optimized machines: GPU-to-machine-series mapping and which series support on-demand
- Google Cloud Spot VMs: up to 91% discount, zero-second default notice, 30-second termination, no SLA
- Oracle Cloud Infrastructure price list: BM.GPU.H100.8, BM.GPU.H200.8, BM.GPU.A100-v2.8 and BM.GPU.L40S-NC.4 per-GPU-hour rates
- Modal pricing: plan tiers, free egress allowances, 1.15-1.75x region multiplier, 3x non-preemptible multiplier
- Modal GPU acceleration: supported GPU list and automatic H100-to-H200 upgrade behaviour
- Modal cold start performance: 60-second default
scaledown_windowand idle billing - Cerebras inference pricing: developer tier per-token rates and vendor-reported throughput
- Cerebras dedicated inference: architecture constraint on custom weights
- GroqCloud supported models: hosted model catalogue, per-token prices and max file sizes
- MLCommons inference_results_v5.1: audited MLPerf Inference v5.1 closed-division results (released 28 Oct 2025)
- MLPerf Inference v5.1 summary results: throughput data used for the performance tables
- Hugging Face model API: meta-llama/Llama-3.1-70B-Instruct: 141.1 GB of safetensors
- Llama 3.1 70B Instruct model card: licence type, release date and training disclosure
- Llama 3.1 Community License: 700 million MAU threshold and Acceptable Use Policy incorporation
- Paperspace / DigitalOcean pricing: GPU Cloud H100 and A100 rates and commitment footnotes
- Baseten cloud pricing: dedicated deployment billing down to the minute, pricing via sales
- Nebius AI Studio: Token Factory serverless and dedicated endpoints; no published per-GPU price list
- Blaxel pricing: repositioning to agent sandboxes billed per GB RAM-second
- Nscale H100 GPU nodes: hardware specifications without a published per-GPU hourly price list
