Groq

What Is Groq?

Groq runs open language models very fast. Not incrementally faster — several times faster, at hundreds of tokens per second, quickly enough that a long answer finishes rendering before you have finished reading the first line of it.

It achieves this by not using GPUs. The company designed its own chip, the Language Processing Unit, around the specific shape of running a language model rather than around the general matrix mathematics graphics hardware was built for. Owning the silicon is why the speed figures look the way they do, and it is also why the model catalogue is smaller than everyone else's.

That trade is the whole product. You get open-weight models — Llama, GPT-OSS, Whisper and Groq's own agentic systems — served faster than anywhere else, and you do not get the frontier commercial models, because those do not run on hardware their owners do not control.

How It Works

Speed You Can Design Around

The published throughput figures are specific: roughly 560 tokens per second on Llama 3.1 8B, about 280 on Llama 3.3 70B, around 500 on GPT-OSS 120B and near 1,000 on GPT-OSS 20B. Context windows reach 131,072 tokens.

Numbers like these change what you are willing to build. Below a certain latency a language model stops feeling like a request you wait for and starts feeling like a function you call, and interfaces that would be unpleasant at two seconds become natural at two hundred milliseconds. Voice, live editing, anything with a human waiting — those are the cases where this is not a benchmark boast but the difference between shipping and not.

Agentic Systems With Tools Built In

Groq Compound and Compound Mini are models packaged with web search, code execution and computation already wired in, running at around 450 tokens per second. Rather than assembling tool use yourself, you call a system that already has it.

This pairs naturally with the speed. Agents are loops, and a loop that takes ten steps at half a second each is a different product from one that takes ten steps at three seconds each. Fast inference is worth more to agents than to chat, which is an underappreciated point in most comparisons.

A Free Tier That Is Actually Usable

Every model is available on the free tier with no card required, bounded by rate limits rather than by feature restrictions: on the order of 30 requests a minute, 6,000 tokens a minute and 14,400 requests a day per organisation.

That is enough to build and demonstrate a real application, not just to run a hello world. Adding a payment method switches the account to pay-as-you-go with no monthly base fee — the free tier is a smaller version of the same thing rather than a different product.

What It Costs

Pricing is per token with no subscription. Because the models are open weights served on unusual hardware, the rates sit at the cheap end of the market rather than the expensive one.

The stacking discount is the detail worth acting on. Any workload that does not need an answer this second — overnight classification, bulk summarisation, dataset preparation — can run at about a quarter of the on-demand rate by combining batching with caching. That is a larger saving than most people get from switching provider.

Who Gets the Most From It

Anything With a Person Waiting

Voice assistants, live suggestions, interactive tools. When response time is part of the interface rather than a technical detail, the hardware advantage stops being abstract.

Agents That Take Many Steps

Every extra step multiplies latency. A fast model turns a chain that felt sluggish into one that feels responsive, and the Compound systems remove the work of wiring tools up yourself.

High-Volume, Low-Difficulty Work

Tagging, routing, extraction, moderation. A small open model does these well, the per-token price is already low, and batching with caching takes it lower still.

Developers Who Want to Start Without a Card

The free tier reaches every model and is bounded by rate limits rather than crippled features. For prototyping, teaching, and hackathons that is a meaningfully different offer from a small credit balance that runs out.

What to Watch Out For

• No frontier commercial models. If your work needs the strongest available reasoning, this is not where it happens — the catalogue is open weights.
• Speed does not raise quality. A fast 8B model is still an 8B model, and hearing it be wrong quickly is not an improvement.
• Rate limits, not credits, bound the free tier. That is generally better, but a burst of traffic hits the ceiling immediately rather than gradually.
• Owning the hardware means capacity is theirs to allocate. Availability of a specific model is a supply decision, not something you can route around.
• Published throughput figures are ideal conditions. Your prompt length and output length will move them.
• The batch discount requires you to actually batch. It is a real 50% and it is not automatic.

Frequently Asked Questions

Is Groq free?

There is a free tier covering every model with no card required, limited by requests and tokens per minute rather than by features. Beyond that it is pay as you go with no monthly base fee.

Why is it so much faster?

Custom silicon. The Language Processing Unit is designed for running language models rather than adapted from graphics hardware, and that architectural difference is where the throughput comes from.

Can I run GPT or Claude on it?

No. Groq serves open-weight models — Llama, GPT-OSS, Whisper and its own Compound systems. Closed commercial models run only on infrastructure their owners control.

How do I actually get the cheapest rate?

Use the Batch API for anything that can tolerate delay and structure prompts so the shared portion can be cached. Each halves the rate and they stack, landing at roughly a quarter of on-demand pricing.

Is it suitable for production?

Yes, with the usual caveat about single-provider dependency. Teams that cannot tolerate an outage often front it with a router so they can fall back elsewhere while keeping Groq as the fast default.

Does it do speech?

Yes — Whisper models are served alongside the language models, which matters because voice applications are precisely the ones where the latency advantage is most noticeable.

The Bottom Line

Groq sells one thing extremely well: speed on open models, at prices that start at five cents a million tokens and fall to about a quarter of that with batching and caching. The free tier is genuinely usable, and the hardware advantage is real rather than a benchmark selected to flatter.

The limit is equally clear. There are no frontier commercial models here, and no amount of throughput compensates when the task genuinely needs the strongest reasoning available. Use it where latency is the constraint and an open model is good enough — which, for a surprising share of production work, it is.