Modal

What Is Modal?

Modal is not a model API. It is somewhere to run your own code on GPUs, without becoming the person on your team who understands Kubernetes. You write Python, add decorators saying what hardware the function needs, and it runs in the cloud while still feeling like it runs locally.

That distinction places it in a different category from the token vendors. OpenRouter, Groq and Fireworks sell you inference on models they host. Modal sells you the machine. If your work is calling a model, you do not need this; if your work is training something, processing a million files, or serving a model nobody else hosts, this is the layer where that happens.

The engineering claims are specific: sub-second cold starts for containers, autoscaling from zero to more than a thousand GPUs, and sub-10ms latency for globally distributed inference. The first of those is the one that makes the model work — serverless GPUs are only useful if starting one is fast enough that you can afford to have none running.

How It Works

Hardware as a Line of Python

Application logic and hardware requirements live in the same codebase. A function is annotated with what it needs — a GPU type, memory, a container image — and Modal provisions that when the function is called and releases it when the call ends.

The consequence is that infrastructure stops being a separate artefact. There is no cluster definition drifting out of sync with the code it runs, because the requirement is written next to the thing that requires it. For small teams without a platform engineer, this is the difference between shipping a GPU workload and postponing it.

Scaling From Zero

Because cold starts are sub-second, keeping zero capacity idle is a viable default rather than a compromise. Traffic arrives, containers start, work happens, containers stop, and you are billed for the seconds in between.

This suits the actual shape of most AI workloads, which are spiky rather than steady. A batch job that runs for twenty minutes each night costs twenty minutes. On an hourly-billed GPU it would cost an hour, and on a reserved instance it would cost the whole day.

Three Kinds of Work

Inference covers serving models, including multi-modal ones, online or in batches. Training covers fine-tuning, reinforcement learning, multi-node distributed runs and hyperparameter sweeps — the last of which is a natural fit, since a sweep is many short independent jobs and that is exactly what per-second billing rewards.

The third is sandboxes: isolated environments for coding agents and other autonomous systems to execute in. As agents increasingly write and run code, somewhere disposable to run it becomes infrastructure rather than a nicety, and this has quietly become one of Modal's more important uses.

What It Costs

Compute is billed per second, which is the detail that changes the arithmetic against hourly vendors. The plan fee is separate and small relative to what serious compute costs.

The $30 monthly credit on the free plan is worth noting because it recurs rather than being a one-off. At T4 rates that is around fifty hours of GPU time a month, permanently, which is enough to run a small personal service without ever paying.

Note that sandboxes and notebooks are billed at higher CPU and memory rates than ordinary functions — three times the CPU rate. If agent sandboxes are your main use, model them separately rather than assuming the headline numbers.

Who Gets the Most From It

Spiky and Batch Workloads

Nightly jobs, periodic processing, anything that runs hard for minutes and then stops. Per-second billing with sub-second starts is precisely the shape these workloads want, and it is where Modal beats an hourly vendor outright.

Serving Models Nobody Else Hosts

A fine-tuned model, an unusual architecture, something you built. Token APIs cannot help; this is the layer that can, without you writing deployment infrastructure first.

Teams Without a Platform Engineer

The decorator model exists so that Python developers can run GPU work without learning orchestration. For a small team that is not a convenience but the difference between the project happening and not.

Sandboxes for Coding Agents

Isolated, disposable environments where generated code can execute without touching anything that matters. As agents move from suggesting code to running it, this stops being optional.

What to Watch Out For

• It is Python. The programming model is the product, and if your stack is elsewhere this is not the tool for you.
• You still have to know what you are deploying. Modal removes the orchestration, not the machine learning.
• Sandbox and notebook compute costs about three times the standard CPU rate. Easy to miss, and it is the fastest-growing use case.
• Per-second billing is only cheaper if your work is intermittent. Continuous serving at scale will cost less on reserved capacity elsewhere.
• The Team plan is $250 a month before any compute. For a small team that is a real threshold to cross.
• Sub-second cold starts describe the container. Loading a large model into GPU memory is a separate delay you design around.

Frequently Asked Questions

Is Modal an AI model API?

No. It runs your code on cloud hardware. If you want to send a prompt and get an answer, use an inference provider. If you want to run something you wrote on a GPU, this is the right layer.

Is there a free tier?

The Starter plan has no monthly fee and includes $30 of compute credit every month. That is a recurring allowance rather than a one-time trial, which makes small hobby deployments genuinely free.

How does per-second billing compare with hourly?

It wins decisively on intermittent work. A twelve-minute job costs twelve minutes here and an hour elsewhere. For continuously saturated GPUs the difference largely disappears and reserved pricing from a cluster vendor may be cheaper.

Can I train models on it?

Yes — fine-tuning, reinforcement learning, multi-node distributed training and hyperparameter sweeps are all supported workloads. Sweeps in particular suit the billing model, since they are many short parallel jobs.

What GPUs are available?

From T4 and L4 at the inexpensive end through A10, L40S and A100 to H100, H200, B200 and B300. The spread means you can match hardware to the job rather than paying for a large GPU because it was the only option.

Do I need to write Dockerfiles?

Container images are defined in Python alongside the rest of the configuration, which is the point of the design — one language for logic, dependencies and hardware.

The Bottom Line

Modal is what you use when the model is not the product — when you have your own code, your own weights, or a job that needs a thousand GPUs for four minutes. Per-second billing and fast cold starts make intermittent GPU work economical in a way hourly rental never has.

It is Python-shaped and unapologetic about it, and it removes orchestration rather than expertise. If those suit you, the recurring $30 monthly credit means finding out costs nothing at all.