Why Everyone Suddenly Wants an Inference Engineer
For most of the last decade, the prestige job in machine learning was training. You built the model, you ran the giant job, you wrote the paper. Serving the thing afterwards was someone else's problem, usually a platform team with a Flask app and a prayer.
That has flipped. In 2026 most AI compute dollars go to inference rather than training, and companies have discovered that "make this model fast and cheap in production" is a specialty in its own right. I wanted to know how big that demand is, what the job involves day to day, and who is paying for it, so I went through the numbers.
The money moved to inference
Three independent estimates land on the same story. Deloitte's Tech Trends 2026 puts inference at roughly a third of all AI compute in 2023, half in 2025, and about two thirds in 2026. Gartner, measuring cloud infrastructure spend rather than compute cycles, marks 2026 as the first year inference spending beats training. Looking further out, several analysts expect around 70% of data center demand to come from inference by 2030.
Gartner's numbers are worth spelling out because they are dollars, not percentages of a fuzzy denominator:
| AI-optimized IaaS spend | 2025 | 2026 (forecast) |
|---|---|---|
| Total | $21.5B | $42B |
| Inference | $23.3B (55%) | |
| Training | $19B (45%) | |
| Inference share by 2029 | over 65% |
The forcing function is agents. A request that used to be one model call is now a plan, several tool calls, a critique and a rewrite, each with its own prefill and decode. Goldman Sachs projects global token volume growing roughly 24x between 2026 and 2030. NVIDIA's Jensen Huang put it more bluntly on the August 2026 earnings call: "most AI are now agentic."
Cheaper per token, bigger in total
The strange part of the inference story is that unit prices are collapsing at the same time as spend explodes. Epoch AI tracked the cheapest model that could match GPT-4's original MMLU score. In March 2023 that was GPT-4 itself at about $37.50 per million tokens (blended). By February 2025 it was Gemini 2.0 Flash at $0.18. Across the benchmarks Epoch looked at, the annual price decline ranged from 9x to 900x depending on the capability threshold.
This is the classic Jevons setup. Every time serving gets cheaper, someone finds a use for ten times more of it. Menlo Ventures reported enterprise LLM API spend going from $3.5B to $8.4B in six months. Your per-token price fell and your bill went up anyway.
Two things follow for the labor market. First, the people who drive those price drops are extremely valuable, because a 2x efficiency gain on a fleet that costs hundreds of millions a year is a lot of money. Second, once a company is spending enough on vendor APIs, it starts asking whether it should run open models itself, and that question cannot be answered without someone who understands serving.
What an inference engineer actually does
The clearest definition I have found comes from Philip Kiely's Inference Engineering (adapted in The Pragmatic Engineer, March 2026): people who "work across the stack from CUDA to Kubernetes in pursuit of faster, less expensive, and more reliable serving." In practice the job comes down to three numbers:
- TTFT, time to first token. This is set by prefill, which is compute bound.
- ITL and TPS, inter-token latency and tokens per second. These are set by decode, which is bound by memory bandwidth.
- Cost per token, which really means how many requests you can pack onto one GPU before the first two numbers fall apart.
Here is where the time goes in a single request, and why the two phases need different treatment:
Prefill runs the whole prompt through the model once and writes the KV cache. Decode then reads that cache on every step to produce one token. Nearly every optimization in the field targets one of these two boxes.
The day-to-day toolkit is a small set of levers that all interact with each other:
| Lever | What it does | The catch |
|---|---|---|
| Continuous batching | Interleaves requests token by token so the GPU never idles | Trades per-user latency for throughput |
| Prefix / KV caching | Reuses attention state across requests that share a prefix | Zero benefit if the first token differs |
| Quantization | Lower precision weights (FP8, INT4) to cut memory and boost throughput | Attention and the KV cache are the most sensitive parts |
| Speculative decoding | A draft model proposes tokens, the target verifies several per pass | Helps TPS, not TTFT; disabled at high batch sizes |
| Parallelism | Tensor or expert parallelism across GPUs | Interconnect becomes the bottleneck |
| Disaggregation | Separate prefill and decode onto independently scaled workers | You now ship KV cache over the network |
The reason this is a real specialty rather than "read the vLLM docs" is that the levers fight each other. Quantizing the KV cache makes disaggregation cheaper. Bigger batches eat the spare compute that speculative decoding needs. One Baseten engineer reportedly tried 77 configurations before doubling the throughput of a code-completion model. Working through that search space, with a real traffic pattern and a real budget, is most of what the job is.
Who is hiring, and what they pay
Postings for these roles are unusually specific about tools. NVIDIA's LLM inference roles ask for hands-on work with vLLM, SGLang or TensorRT-LLM, CUDA and kernel experience (CUTLASS, Triton), profiling with Nsight, and scale-out orchestration with NCCL and Kubernetes. Startups building on vLLM want people who understand KV-cache memory management and prefix caching well enough to have shipped features into the engine itself. Open-source contributions to vLLM, SGLang, llama.cpp or ExecuTorch show up as a plus in almost every listing.
Public salary bands from 2026 postings:
| Role | Base salary (USD) |
|---|---|
| NVIDIA, DL Performance Engineer, LLM Inference (Toronto) | CA$94K to CA$220K |
| LM Studio, Software Engineer, Inference Runtime (NYC) | $170K to $220K |
| Inferact (vLLM core), Member of Technical Staff | $200K to $300K |
| NVIDIA, Manager LLM Inference, L2 to L3 | $184K to $357K |
| NVIDIA, Eng Manager LLM Inference at Scale, L3 to L4 | $224K to $431K |
Equity sits on top of all of these. A senior engineer who halves a company's GPU bill pays for themselves within the year, and hiring managers know it.
How rare is this skill set?
Very. When ChatGPT launched at the end of 2022 there were perhaps a few hundred inference engineers in the world, concentrated at OpenAI, Anthropic, Google, NVIDIA and Midjourney. Gergely Orosz's estimate for 2026 is a few thousand people globally with real production experience serving closed models, and the open-model side is younger still.
That scarcity sits inside a broader market that is not friendly. Around 170,000 tech jobs were cut in 2026 while the four largest tech companies committed a combined $725B to AI infrastructure. Stanford's 2026 AI Index found employment for software developers aged 22 to 25 down about 20% since 2024. The Bay Area median time-to-hire for senior engineers stretched from 38 days to 67 days over the same period. So demand is not up across the board. It is pooling in a few narrow specialties while the general market stays cold.
The counter-argument
A few caveats before you rewrite your résumé.
Most jobs are still at the application layer. Frontier labs and GPU clouds hire small inference teams and are picky about who gets in. The thousands of companies building on top of those models in legal, healthcare, finance and coding are where the volume of openings is, and those roles care more about RAG, evals and agents than about CUDA.
Build versus buy also cuts both ways. Orosz's own take is that self-hosting only makes sense once you are already spending heavily on vendor inference. A company at $20K a month on API calls should not hire an inference engineer. A company at $2M a month probably should.
And the price collapse could slow. Several analysts expect the 10x-per-year declines of 2021 to 2025 to taper toward 3x to 5x through 2027. If that happens, efficiency work gets harder and therefore more valuable, but the easy wins from swapping in a newer model get smaller.
If you want in
The same roadmap keeps showing up across postings, the Kiely book and practitioner write-ups, and it is short:
- Learn the memory math. Be able to size a KV cache for a given model, context length and batch size by hand, and know why a 70B model in FP16 needs 140GB but fits on an H100 at FP8.
- Run vLLM or SGLang for real. Serve an open model, load test it, watch TTFT p99 and the prefix cache hit rate, then change one thing at a time.
- Read one engine's source. Start with the scheduler, block manager and sampler. Fix a small issue and send the PR. On a résumé for these roles, that one line carries more weight than anything else you can put on it.
- Profile before you optimize. Nsight Systems, the PyTorch profiler, roofline plots. Every posting mentions profiling, and very few candidates can actually do it.
- Ship something end to end, with autoscaling and a cost dashboard, so you can talk about the infrastructure layer and not just the kernel layer.
Kiely closes his book by telling readers they are early, and I think that is right. The field is a few years old, there are only a few thousand people in it, and the compute bill it sits on top of is doubling every year. That is an unusual combination to find in a job market this cold.
Sources
- Gartner via TechTimes, inference beats training in AI-optimized IaaS spend (Aug 2026)
- Deloitte Tech Trends 2026, summarized in Introl, inference vs training infrastructure economics
- Epoch AI, LLM inference prices have fallen rapidly but unequally across tasks
- The Pragmatic Engineer, What is inference engineering? (Mar 2026), adapted from Philip Kiely's Inference Engineering
- The Pragmatic Engineer, State of the software engineering job market in 2026
- HeroHunt, How to recruit AI inference engineers, 2026 guide
- NVIDIA Q2 FY2027 results, press release on sec.gov and CNBC coverage
- Salary bands from public NVIDIA, LM Studio and Inferact postings via Dreamwork, Why Not Remote and freehire
- KORE1 LLM engineer salary guide 2026
- Spheron, What is inference engineering? The 2026 GPU cloud guide