Baseten

What Is Baseten?

Baseten puts AI models into production and keeps them there. The pitch is reliability rather than novelty: optimised runtimes, high availability across clouds, and a claim of 99,99% uptime out of the box — which is roughly four minutes of downtime a month, and an unusual thing for an inference vendor to put in writing.

It sits between two things people already understand. Above it are token APIs where somebody else chooses the models. Below it is raw compute where you build the serving layer yourself. Baseten is for teams that have a specific model they need served properly and would rather not spend a quarter building the infrastructure to serve it.

The customer list reads accordingly — Zed Industries, Wispr, OpenEvidence, ClickUp and Writer, across code completion, transcription, medical AI and domain-specific language models. Those are all cases where the model is the product and latency is a feature users notice.

How It Works

Two Ways In

Model APIs are pre-optimised open models you can call immediately — GLM, DeepSeek and Kimi variants among them. That is the low-commitment entry point, and it works like any other token API.

Dedicated deployments are the real product. You bring a proprietary or fine-tuned model and Baseten serves it on infrastructure sized for it, with the runtime optimisation that is otherwise a specialist job. Training via the Loops SDK connects to the same path, so a model can go from tuning to production without changing platform.

Performance as the Product

The published claims are specific enough to check: over twice the throughput and 10% lower latency on embeddings than competitors, sub-300ms transcription latency without unpredictable spikes, and 6× better GPU utilisation on compound workloads.

The phrase worth noticing is no unpredictable spikes. In production inference, tail latency matters more than average latency, because the ninety-ninth percentile is what users complain about. Optimising for consistency rather than for a good median is a mature engineering position and a difficult one to market, which is probably why few vendors bother.

Cross-Cloud Availability

High availability spans clouds rather than regions within one. That is a meaningful distinction: a single provider's outage takes down every region you were spread across, and the only real defence is being somewhere else entirely.

There is also Baseten for Model Labs, aimed at model creators who want distribution and monetisation rather than deployment. It is a different business from the main platform, and worth knowing exists if you are on the producing side of the model economy.

What It Costs

The Basic plan has no monthly fee and bills usage. Pro and Enterprise add volume discounts and support commitments rather than a different product.

Cached input pricing is the detail to build around. At a tenth of the standard input rate, an application with a long stable system prompt and short varying user input can cut its bill dramatically — but only if prompts are structured so the shared portion is genuinely reusable. That is an architectural decision, not a setting.

New accounts receive complimentary credits for experimentation. The Basic plan includes email and in-app chat support; higher tiers add Slack, Zoom and dedicated engineering contact, which is the part enterprises are actually buying.

Who Gets the Most From It

Companies Whose Model Is the Product

If you trained it, fine-tuned it, or depend on serving it faster than a general vendor would, this is the category Baseten was built for. The named customers are all in exactly this position.

Latency-Sensitive Production Work

Code completion, live transcription, anything typed into. Consistent tail latency is worth more here than a slightly better average, and it is what the platform optimises for.

Workloads That Cannot Go Down

Cross-cloud high availability and a stated 99,99% figure are the reason to pay a platform rather than run your own serving. Medical and financial applications are the obvious cases.

Heavy Embedding Pipelines

Twice the throughput on embeddings compounds quickly when you are indexing continuously. For large retrieval systems, throughput on the embedding step often decides the whole architecture.

What to Watch Out For

• Performance claims are the vendor's own benchmarks against unnamed competitors. Reproduce them on your model before you plan around them.
• Dedicated GPUs bill by the hour regardless of traffic. An idle H100 is $6.50 an hour whether or not anybody calls it.
• 99,99% uptime out of the box is a statement of capability, not automatically a contractual SLA on the entry plan. Check what your tier actually commits to.
• Cached input pricing only helps if prompts are structured for it. The tenfold saving is real and it is conditional.
• Pro and Enterprise pricing is not published. Volume discounts mean a conversation rather than a calculator.
• If you only need to call somebody else's model, this is more platform than the job requires. Its value is in serving yours.

Frequently Asked Questions

Is Baseten free to try?

New accounts get complimentary credits for experimentation, and the Basic plan has no monthly fee — you pay for usage. There is no permanent free tier.

Can I deploy my own model?

Yes, and that is the main purpose. Dedicated deployments serve proprietary and fine-tuned models on infrastructure sized for them, with runtime optimisation handled by the platform.

How is it different from Modal?

Modal is general serverless compute where you write the code that runs. Baseten is specifically about serving models reliably, with the optimisation and availability work already done. Modal is more flexible; Baseten does more for you within a narrower scope.

What does cached input pricing mean?

Repeated prompt prefixes are billed at roughly a tenth of the normal input rate. If every request shares a long system prompt, restructuring so that portion is cacheable is the single largest cost saving available.

Does it support training?

Yes, through the Loops SDK, with deployment straight into production inference on the same platform. That continuity is part of why teams choose it over assembling training and serving separately.

Which GPUs can I get?

T4 at $0.63 an hour through L4, A10G, A100 and H100 to B200 at $9.98. An H100 MIG partition at $3.75 is a useful middle option when a full H100 is more than the model needs.

The Bottom Line

Baseten is for the stage after you have decided which model you are using and discovered that serving it well is its own discipline. Optimised runtimes, cross-cloud availability and consistent tail latency are the things teams build for themselves over months, and they are the things being sold here.

The model APIs are a reasonable way in, but they are not the reason to be here — plenty of vendors serve open models for similar money. Come for the dedicated deployments, structure your prompts to exploit cached input, and verify the performance claims against your own model before they become a line in your plan.