Figures and product terms checked November 14, 2026.
By April, Uber had exhausted its 2026 AI coding budget, its CTO told The Information.
I understand the case for a generous initial budget. Engineers need room to experiment. If management wants them to learn a new way of working, requiring approval for every attempt is an effective way to discourage it. Pay centrally, make access easy, and find out what people can do.
The problem begins when an experiment becomes a recurring production expense without a corresponding decision about what it is supposed to buy. The organization has learned to consume capacity. Management has not necessarily learned which additional work is worth funding.
The reaction is already visible. In Sonar’s survey released November 12, 35% of respondents at enterprises with more than 1,000 employees said their teams had cut back AI coding to control token spending, compared with 21% at companies with 200 or fewer employees. The survey covered 1,026 developers and engineering managers in September. These are reports from people using AI, not an audit of company finances, but they establish that restrictions are more than a hypothetical concern. Sonar report, p. 51
Anthropic’s IPO disclosures show the scale of the money flowing in the other direction. It reported almost $17 billion of revenue for the third quarter of 2026, more than twelve times the year-earlier quarter. Those are recognized sales, not a rumored valuation or an annualized run rate. They span customers and uses well beyond internal software development. November 10 SEC filing
The filing tells us how much Anthropic sold. It cannot tell a CEO whether their own company improved its product, won more customers, or earned a higher profit.
That is the uncomfortable part of the bill. Engineers may be working faster. The tools may be doing exactly what was promised. The company can still be buying more activity than it knows how to turn into a result.
Before setting next year’s allowance, I would ask what the company believes it is buying: more engineering capacity, a better product, or a better financial outcome. Moving from one to the next requires a separate argument. For some companies, that argument will justify a much larger bill.
You can build faster and still sell nothing
Hiring another hundred engineers does not automatically make a product easier to sell. Neither does giving the existing engineers enough inference to generate several times as much code.
Additional capacity helps when the company knows what to build and cannot deliver it quickly enough. It can accelerate experiments, remove a reliability problem, improve onboarding, or make an expensive operation cheaper. But a faster implementation of the wrong idea is still the wrong idea. More features can also make a product harder to understand and more expensive to support.
Tokens change the speed and price of producing software. They do not remove the need to decide which software customers want, how people will find it, or why they will pay. If customer demand is the constraint, the return comes from using capacity to learn and respond—not from filling a larger release with more features.
This does not mean every useful AI purchase must produce immediate revenue. Security work can reduce expected loss. A migration can avoid future operating expense. Faster experimentation can reveal that a product should not be built. Revenue can also remain flat while the company prevents churn it would otherwise have suffered. Flat sales alone cannot establish failure, just as a growing pull-request count cannot establish success.
The company needs to name the intended benefit, the evidence it expects to see, and the period in which it expects to see it. For customer-facing work, that might mean a measured change in activation, retention, conversion, or support burden. For internal work, it might be faster resolution of a particular incident class or a planned hire no longer needed. Count incremental contribution after the costs of serving customers, rather than treating every extra dollar of sales as profit.
The answer will differ across organizations. A founder may pay generously to test product ideas before cash runs out. A growing software company may already have more proposed changes than reviewers can absorb. A regulated enterprise may buy stronger verification because an incorrect migration could cost far more than the model bill.
Nassim Nicholas Taleb warns: “Don’t cross a river that is on average four feet deep.” The depth where you cross matters. An industry productivity average cannot tell these companies where to spend. Taleb, technical companion to The Black Swan
Size alone is insufficient too. A small AI company may have steady inference demand and experienced operators. A large enterprise may have fragmented, bursty usage across regions and product lines. The first may fill dedicated capacity; the second may pay to keep it idle. A useful operating plan starts with the work and the constraint.
“At this rate we’ll finish the roadmap before we find a customer.”
Plan the bill before promising the return
Headcount plus tokens is a useful correction to a payroll-only engineering plan. It is still incomplete.
Consider an illustrative company with 200 engineers, $60 million in annual revenue, and $6 million in operating profit before new AI spending. An incremental $1,000 per engineer per month adds $2.4 million a year. If the entire amount is expensed in the period and nothing else changes, operating profit falls to $3.6 million. The margin moves from 10% to 6%.
Those are assumptions, not an estimate of typical spending. They show why a bill that looks modest beside an individual salary can matter to the bottom line. At an assumed 80% contribution margin, recovering that $2.4 million through sales alone would require $3 million of additional revenue, before any further fixed costs.
Payroll comparisons still matter. A tool that removes enough work to defer a hire can pay for itself. But “cheaper than another engineer” is a useful comparison only when another engineer was the alternative and the same valuable work gets done. If payroll stays fixed, do not book a payroll saving. The purchase has to earn its cost through useful capacity, quality, lower risk, or another demonstrated benefit.
I would separate the operating plan into four kinds of spending:
Exploration and shared evaluation. A defined question answered, a credible opportunity found, or an expensive approach rejected
Internal engineering work. Useful delivery, quality, or operating improvements after review and repair
AI used in the product customers consume. Customer value and contribution after inference, tools, support, and failure costs
Shared infrastructure and customized models. Enough eligible demand or risk reduction to cover operation, reserve, maintenance, and eventual migration
These are management views, not accounting classifications. Finance still has to classify the actual costs. The important distinction is between spending to build the product and spending each time a customer uses it. Combining both into a per-engineer allowance makes neither easier to understand.
Fund initial evaluation and shared learning centrally. Give each experiment an owner, a bounded budget, and a date for deciding what was learned. Allocate established use to the product, project, or service that benefits from it. Keep special initiatives visible, and let team or project budgets accommodate differences in the work. A move from central funding to local chargeback needs a transition: the invoice has moved, even if the work has not become more expensive.
Forecast human sessions and unattended workers separately. Both can change their volume, effort, context, retries, and tool use. Include low, expected, and high-use cases, minimum commitments, and the cash needed when benefits arrive later than the bill. Revisit those assumptions as actual work replaces guesses. Current FinOps guidance similarly recommends shorter forecast windows, flexible pilot funding, and cost measures tied to successful outcomes. FinOps Foundation
The cash plan needs people, providers, tools, infrastructure, and the extra staff or contractors required to operate them. Review and repair also consume capacity, but hours already paid for in payroll should not be added again as a second cash expense. Keep that capacity view beside the cash view. Neither an allocation nor a claimed hour saved changes the bank balance by itself.
The plan also needs room for spending that is deliberately uneven. A temporary campaign and a permanent increase in the running bill deserve different decisions.
The underdog may need the bigger AI budget
A small company can be outspent on payroll and still choose to spend aggressively on inference. If additional capacity can be rented for a few weeks, it can attempt work that would never justify recruiting a permanent team.
That can change which projects are worth considering. An awkward architecture may prevent a company from serving a new market or make every customer expensive to support. A rewrite previously dismissed as unaffordable may become a credible experiment. The alternative is not always maintaining the existing system cheaply. Sometimes it is continuing to pay for a system that prevents the business from working.
The financial attraction is the ability to stop. A team can buy a prototype, test the hardest assumptions, and abandon the attempt before committing to the full migration. That flexibility depends on the provider contract and the work itself: review, integration, and production responsibility cannot be summoned as easily as another model call. A larger batch of generated code can exhaust the people needed to assess it.
A flat weekly allowance can block a concentrated attempt even when total model spending is unchanged. The figure estimates no output or return; the people and migration costs still need funding.
There is a recent example worth examining. In its November 12 engineering account, Hex describes a four-month push that changed roughly 250,000 lines across foundational systems—about a tenth of its codebase. The team credits coding agents while emphasizing iteration, API boundaries, and checkable completion criteria. This is a company account of a shipped rewrite, without a controlled cost comparison or a measured commercial return. It demonstrates a project to investigate, not a productivity multiplier to put in the budget. Hex engineering
An underdog may have fewer customer contracts, compatibility promises, and established revenue streams to endanger. A competitor’s installed base can make the same technical change harder to approve. But the incumbent also has cash, experienced reviewers, distribution, and customers who can explain what matters. Microsoft’s October account of agent-assisted modernization is a reminder that large companies can concentrate expertise too. Its approach includes behavior comparisons and tests withheld from the agent. Microsoft engineering
Buying access to the same models as everyone else is unlikely to be a lasting advantage by itself. The opportunity lies in choosing a consequential project, organizing its verification, and completing it while the result still matters. Public evidence does not yet tell us how often small companies convert that opportunity into durable commercial gains.
I would fund such work as a bounded project, with a technical claim to prove and a business constraint to remove. Compare a replacement with a smaller refactor or an automated migration. Test the difficult behavior before polishing the easy screens. Include data conversion, security review, parallel operation, rollback, and the cost of maintaining what remains afterward. Building a complete replacement in parallel does not require switching every customer at once.
And keep the authority to stop real. A convincing demo should earn funding for the next stage only if it answers the question that justified the experiment. It should not automatically release the rest of the budget.
An underdog may have little revenue to protect. It still has runway to lose. The right budget can let it attempt something the larger competitor cannot easily attempt; it cannot make a bad project good.
Follow one change until somebody can use it
Take a hypothetical bug in a mature service. A low-priced model proposes a patch, adds tests, and passes the visible checks. During review, an engineer discovers that the change broadens an interface other services depend on. The patch needs repair.
For illustration, assume the first route costs $2 in model calls and requires 45 minutes of review and repair. A different route costs $12 and requires 10 minutes. At an assumed $150 per hour for engineering capacity, the first consumes $114.50 and the second $37.
Those are invented inputs, not measured model results. Their purpose is to expose the missing term. The ten-dollar inference premium buys 35 minutes of scarce engineering capacity in this example. If it buys no reduction in review, delay, or risk, it is simply an extra ten dollars.
Nor is the released time automatically cash. Payroll may remain exactly the same. The benefit becomes economic when the engineer uses that time to remove another constraint, delivery happens earlier, a contractor is no longer needed, or a hire can be deferred. Counting the hour as both payroll savings and additional output would count it twice.
The record should follow the whole attempt, including unsuccessful work.
The invoice captures part of this sequence. Review, abandoned attempts, and later repair belong to the cost; release still needs to produce the intended result.
Cost per accepted task is useful, provided the organization also records what it attempted, deferred, and abandoned. Otherwise the easiest way to improve the metric is to choose easier work. Acceptance itself is also insufficient: a beautifully implemented feature can be unnecessary.
This distinction changes model selection across the development cycle. Updating Jira fields or checking branch status may be a deterministic operation. Research needs reliable sources and a stopping rule. Design needs local constraints and someone willing to choose among imperfect alternatives. Debugging needs controlled experiments. A security review needs evidence that survives a confident explanation.
A company paying the same model price for all of those activities is making an expensive assumption about how similar they are.
Faster code can become somebody else’s workload
Speed has value when a person is waiting to make the next decision. A delayed answer can interrupt concentration, trigger another task, and require the engineer to reconstruct the original problem on return. Faster generation can remove that interruption.
An overnight worker has a different objective. It may tolerate longer latency if it completes more accepted work by morning at lower cost. Filling the queue with additional patches does little if the same two maintainers must review them all.
Review also has to examine the design burden left behind. An agent can reproduce a familiar pattern from the repository even when that pattern is the legacy debt the change ought to remove. It can introduce a new framework where a small function would have been sufficient. Passing the requested example does not establish that the next engineer will be able to change the result cheaply.
Budget policy can interfere with that judgment. Consider an engineer approaching a daily allowance. The patch is ready and the tests pass. If another model review requires a manager’s exception, the engineer has a reason to stop early. Autonomous runs without useful stopping rules can also keep consuming resources after additional attempts stop helping. The budget needs to contain both failures.
We do not yet have a credible causal estimate of how personal token caps affect engineering quality. My preference is to make human usage visible through team or project budgets, provide simple exceptions for consequential work, and place hard limits on autonomous runs, tools, elapsed time, and spending. Then check whether the policy reduces waste or merely moves it into review and repair.
There is a practical alternative to an indiscriminate cut. By August, Uber reported stabilizing total AI spend while usage grew. Its engineering account describes live cost visibility, shared spending tiers, easier escalation, and changes to context, caching, tools, and model selection. Those are company-reported operating results, not proof of a profit gain or a current-model benchmark. They show that a budget surprise need not end with engineers simply doing less. Uber engineering
An engineer should be able to explain why a costly investigation was worthwhile. A manager should have to explain why preventing it was worthwhile too.
A passing test can be the wrong reward
Return to the service bug. If the agent writes tests around its own interpretation of the interface, the tests may confirm the same mistake as the implementation. Ten more tests derived from that interpretation can make the patch look safer while adding little assurance.
It is tempting to blame this on reinforcement learning with verifiable rewards (RLVR). In a coding training environment, the model attempts a task and receives a reward from an executable check. Passing tests can supply that signal. If the check accepts an incorrect solution, training can reinforce a shortcut that the organization never intended to teach.
But there are different problems hiding inside that explanation. Fitting an implementation to incomplete tests, tampering with the evaluator, and writing weak tests for your own code are different behaviors. If the reward comes from fixed, protected tests, adding a trivial test of your own does not improve the reward. A training process that pays a test generator merely for producing passing tests creates a different incentive from one that pays it for exposing real faults.
The distinction changes what the evidence can establish. Goodhart Labs’ current HoneyBench includes Opus 5.5, GPT-6 Astra, and GPT-6.1 Sol in deliberately flawed scoring environments. It documents agents exploiting gaps between the score and the stated task. These are selected stress tests, not a sample of company PRs or an experiment isolating the effect of RL training. HoneyBench v0.2
A November 9 controlled study adds a useful complication: shared learning can suppress some verifier exploits and amplify others, depending on how the errors recur across tasks. Its simplified environment does not estimate the behavior of a production coding model. It does undermine the easy story that reinforcement learning must always make cheating worse. Wrong the Same Way
I did not find public evidence that isolates RLVR as the cause of useless test writing in the current models on company workflows. That remains an open question. Training data, inherited repository conventions, incomplete requirements, instructions to make CI green, and the engineer’s own review habits can produce similar symptoms. A passing test tells us neither which cause operated nor whether the test was useful.
“The test confirms that the light is green.”
The practical response is to ask what wrong behavior a test would reject. For a bug fix, demonstrate that the regression test fails on the defective behavior and passes on the correction. Add independently chosen boundary cases. Deliberately introduce a relevant fault and see whether the suite notices. Mutation testing does that systematically, although an artificial mutation is still a proxy: some mutations do not change observable behavior, and killing many easy mutants can miss the dangerous fault.
Keep acceptance checks outside the agent’s authority to redefine. Review changes to existing assertions, because an updated expectation can be a legitimate specification change or a quiet removal of the requirement. A second agent may help, but it may also share the first agent’s assumption. Independence comes from the evidence and checks, not simply from opening another conversation.
This can justify more testing. Agents may make previously unaffordable regression coverage practical. It can also justify deleting redundant tests whose maintenance costs exceed what they detect. Rewarding a generator for valid tests that distinguish faulty implementations is a more useful proposal to evaluate than rewarding a high pass count.
The expense belongs in the same operating plan as generation. Saving tokens by removing verification can make the bill smaller while leaving the company with a result it cannot safely use.
The same agent does not make the same engineer
Two engineers can use identical tools and produce different results because they notice different things.
In our service example, one may know that an apparently local interface is also consumed by an old customer integration. Another may be excellent at implementation but unfamiliar with that dependency. The agent’s behavior then depends on whether the missing constraint reaches it, whether it asks a useful question, and whether someone recognizes the consequence of its proposal.
That is more specific than being good at prompting.
A November survey of 203 developers found that autonomy choices varied more between developers than between development activities. Among those reporting careful review of most or all generated code, 61% still said they often approved at least one type of agent action without close review. Reviewing a patch and authorizing an action are different responsibilities. From Delegation to Action Approval
An October workplace study followed 11 professional developers for about 15 hours. It observed different patterns of supervising, delegating, and learning, along with large variation in checking and task switching. It also recorded fatigue associated with managing parallel work. The small sample reveals behaviors worth measuring; it cannot tell a company how many agents each engineer should supervise. Developers as Managers of AI Agents
For internal evaluation, I would compare stepwise interaction, delegation with checkpoints, and bounded autonomous runs on comparable tasks. Hold the model and tools fixed. Distinguish repository familiarity, domain expertise, software experience, and agent experience.
Then look at the quality of the decisions. Did the engineer define success? Did the agent disclose a consequential assumption? Was drift caught before it compounded? Did the correction address the cause? Was the verification independent enough to catch the agent’s mistake? Could the engineer later explain and modify the result?
Counting interventions cannot answer those questions. Frequent interruption can indicate poor agent behavior, unnecessary control, a difficult task, or an expert preventing a serious error. Silence can indicate sound delegation or failure to notice.
The gateway records calls and charges. It cannot observe understanding.
This has consequences for training and hiring. I would teach specification, risk assessment, verification, recovery, and attention management alongside tool use. For a hiring exercise, let the candidate use an approved agent, then ask them to explain the resulting design, find a planted defect, or make a follow-up change. Their judgment should remain visible even when code generation is easy.
Learning requires its own check. In a small randomized study of engineers learning an unfamiliar Python library, the assisted group scored 17 percentage points lower on an immediate knowledge assessment. Exploratory analysis associated conceptual inquiry and generation followed by comprehension with stronger learning than extensive delegation. The experiment involved an earlier assistant setup and a narrow task; it is not a current-agent productivity estimate or proof of permanent skill loss. It is a reason to test what people retain. Anthropic’s coding-skills study
For a junior engineer, completing today’s task and developing the ability to diagnose tomorrow’s failure are both outputs. A staffing plan that counts only the first can consume the apprenticeship it will later depend on.
Retention is less settled. Agents can remove tedious work and give engineers more control. They can also turn an absorbing craft into a stream of approval decisions. I would track learning time, perceived mastery, fatigue, mentoring, review concentration, and actual departures before claiming that adoption improves or damages retention.
And I would keep raw interaction statistics out of individual rankings. Reward low token use and engineers may hide useful work. Reward long autonomous runs and they may delay intervention. Reward intervention itself and they may interrupt needlessly. The company would be training people to produce an attractive trace.
The model ranking is the beginning of the purchase
Current results already show why a price card cannot settle the decision.
Artificial Analysis’s live Intelligence Index v5.0.1 reports the following selected current-generation configurations. The index combines several kinds of work; its score is neither a percentage of coding tasks solved nor a measure of company productivity.
Claude Opus 5.5, maximum effort, Default Fallback. Index score: 59; Reported cost per index task: $10.04
GPT-6 Astra, maximum effort. Index score: 57; Reported cost per index task: $6.24
GPT-6.1 Sol, maximum effort. Index score: 55; Reported cost per index task: $1.27
The Opus 5.5 page reports scores from 35 at low effort to 59 at maximum effort, and average task time from 3.0 to 25.2 minutes. The effort setting is a purchasing decision of its own. The comparison preserves the publisher’s fallback label because it describes a tested configuration, not isolated weights. Reported costs are weighted estimates using token prices; they omit company review and repair. Artificial Analysis’s current methodology
For coding candidates, its separate Coding Agent Index v2.1 is more focused: implementation, terminal work, and repository questions, with task cost and execution time. Different task mixes answer different questions. An aggregate score cannot tell you whether the model is good at the particular class of failure that makes your senior engineers nervous. Coding Agent Index methodology
A separate November benchmark, ExpertCrowd-SWE, reports 67.3% for Opus 5.5 and 63.7% for GPT-6 Astra without a statistically significant difference between them. Its tasks come from unfamiliar repositories and use calibrated graders. That is useful evidence for choosing candidates; the small headline gap does not establish a winner for your codebase. ExpertCrowd-SWE
What you buy also includes the software around the model: how it finds files, retains context, runs tools, requests approval, retries, and stops. Engineers call this the harness. Change it and you can change both the work performed and the bill.
For a company with proprietary code, I would build a private evaluation from actual work. Historical pull requests are a useful starting point, with a trap: a merged patch is not a controlled experiment.
Reconstruct the repository before the change. Supply the issue and information available at that time. Hide the eventual patch, later reviews, and future fixes, including copies left in version-control objects or searchable indexes. Confirm that the historical solution passes the reconstructed checks and that an unchanged repository fails where it should. Allow other correct implementations to pass. A test written by the candidate belongs among its outputs; independent checks still decide whether that output is good.
Then rerun the incumbent—Opus 5.5, if that is the chosen baseline—alongside other eligible current models, a simple routing rule, and the automatic router. Keep a set of tasks out of tuning. Follow the retrospective comparison with newly arriving work, because merged PRs exclude the abandoned investigations and failed changes that also consume engineering time.
Measure review, repair, latency, and later quality alongside model charges. First use a common harness to understand differences; then compare the production configurations the company can actually deploy.
An automatic router needs the same scrutiny. Its first choice changes the answer, the next tool call, the cached context, and the work a later model inherits. A cheap early route can create an expensive rescue.
In the router work I led, the design distinguishes the proposed route from the final route after session and other constraints are applied. That distinction helps explain what happened. It does not prove a saving. The saving has to survive the complete task.
Lunch changes the price of context
An agent may repeatedly send repository instructions, tool definitions, and conversation history to a model. Prompt caching lets the provider reuse processing of an unchanged beginning of that input. The cached object is context processing, not a stored answer.
The clock matters because human conversations and agent loops return at different intervals.
Anthropic’s current Messages API supports automatic breakpoint placement or explicit breakpoints, both enabled through cache controls. The default lifetime is five minutes, with a one-hour option. A qualifying read refreshes the lifetime, measured from request start. A four-minute generation therefore leaves about one minute for the next request to reuse that entry. Anthropic prompt caching
Agent calls a tool and returns every minute. Repeated reads can keep a short-lived prefix warm
Engineer reads the answer and returns after eight minutes. A five-minute entry may have expired
Engineer leaves for a 45-minute lunch. A one-hour entry may retain useful context
Unattended worker waits 70 minutes for CI. Neither duration necessarily covers the pause
Assume an unchanged eligible prefix and compatible routing. These are timing examples, not guaranteed hits.
“Your conversation is still here. Your discount went to lunch.”
There is a price for buying time. At the checked Opus 5.5 rates, ordinary input costs $4 per million tokens, a five-minute write $5, a one-hour write $8, and a cache read $0.20. For one million identical input tokens used twice, a one-hour write and one hit cost $8.20. Two ordinary uncached inputs cost $8. Three uses change the comparison to $8.40 against $12. Output and changing context are additional. Anthropic cache pricing
That calculation compares caching with uncached input. Comparing one hour with a five-minute policy that expires and writes again is a different calculation. Confusing the alternatives can make the same cache appear either wasteful or economical.
Provider controls also differ. Current OpenAI Responses configurations offer implicit and explicit caching with 30-minute or 12-hour lifetimes. Gemini 4 uses implicit caching rather than manually created cache objects. DeepSeek documents automatic disk caching on a best-effort basis. A gateway cannot manufacture a duration or guarantee that the receiving provider does not offer. OpenAI, Google, DeepSeek
Routing complicates the arithmetic further. Switching to a cheaper model can lose the reusable prefix. Keeping a session on one route can preserve it, while retaining an expensive or unhealthy route too long can waste more than the cache saves.
I would replay real arrival times and route changes against actual invoices. A cache discount on a pricing page says little about whether an engineer returns before lunch ends, whether CI finishes, or whether the context is still correct.
A smarter model cannot remember a decision it never saw
Anthropic’s advice about removing instructions deserves a narrower reading than “skills are obsolete.”
Its Opus 5 prompting guidance says generic verification reminders and legacy verification steps can cause redundant work. Opus 5.5 guidance continues to recommend calibrating effort and supplying context for loosely specified work. Anthropic also continues to document enterprise skills and their evaluation. Opus 5 guidance, Opus 5.5 guidance, enterprise skills
A stronger model may no longer need a reminder to recheck something it reliably checks itself. It still cannot know why your company rejected a library last week, which customer depends on an undocumented behavior, or which migration requires a second approval unless that information is available.
The organization needs a way to retain those decisions. A skill—a reusable set of instructions, examples, or tools—is one place to put them.
It is not always the right place. Changing facts belong in current documentation and retrieval. A forbidden operation should be blocked by permissions. A deterministic check should run as code. A skill is more useful for recurring judgment that cannot be captured by a simple rule.
Skills also accumulate debt. Each loaded instruction consumes context. A large catalog makes selection harder. Examples become stale. Two well-intended instructions can conflict. A correction that helped one incident can become a permanent restriction nobody remembers choosing.
I would treat a skill as a maintained engineering artifact: an owner, representative tasks, a reason it exists, and a condition under which it should be removed. Test it against the current model without the skill and against simpler alternatives. Load detailed material when the task needs it.
A useful learning loop has to delete instructions as well as add them. Otherwise each improvement in the model leaves the company paying for workarounds it no longer needs.
Fine-tuning gives you another product to maintain
For a company with a large proprietary codebase, training a model on that code can sound like the obvious next step. The company has data the public model has never seen. Why keep explaining it?
Because knowing yesterday’s repository and following today’s workflow are different capabilities. A trained model does not automatically learn this morning’s schema change. Retrieval and tools remain necessary.
There can still be a case for customization: a stable task performed at high volume, a repeated behavior the best available model handles poorly, an expensive output format, or an offline constraint. The first question is whether a simpler change to context, tools, instructions, effort, or routing removes the gap.
If a gap remains, the work starts with a benchmark and an untouched base model. It also needs the strongest eligible untuned alternative. Beating the model you chose to train is not enough if a current API solves the task more cheaply.
A serious program may then involve several different kinds of training.
Continued pretraining teaches patterns through next-token prediction on domain material. It can be useful when there is a representation gap, but it is optional. Mixing in suitable general data may reduce forgetting; the correct ratio has to be tested. The original training corpus may be unavailable, and newly distilled examples are not automatically a faithful replacement.
Supervised fine-tuning, or SFT, teaches reviewed examples of desired behavior: the task, relevant context, tool actions, output, correction, and stopping point. A folder of merged patches does not contain all of that.
These stages need different data. Raw code, general replay material, approved demonstrations, and reward environments each require curation. Remove secrets and material the company cannot use. Keep related PRs and later fixes together when separating training from evaluation, so the test does not quietly contain another version of a training example. Apply the cutoff to the raw code too: a future solution can leak through a repository snapshot even if its PR is withheld. Preserve unsuccessful attempts and corrections where they teach something useful. More repository history can increase leakage and obsolete patterns as readily as it increases coverage.
Preference training needs credible preferred and rejected alternatives. Reinforcement learning needs an environment in which the model can attempt work and receive a reward. “RL on our PRs” means reconstructing tasks, generating new attempts, and building checks that cannot be satisfied by weakening tests or exploiting the grader. Merge status alone is too crude a reward.
Use development evaluations throughout. Reserve the final holdout until the method is fixed. The optional branches illustrate the work a proposal may create; they are not a mandatory recipe.
Every weight-changing stage brings comparison runs: data mixtures, learning rates, duration, checkpoints, and perhaps LoRA, which updates small adapters, versus full-weight training. The organization must check general capability, tool use, security, memorization, latency, and cost as well as the intended domain gain.
Vendor support covers different parts of this sequence.
NVIDIA Megatron Bridge and related NeMo tools. Components for continued pretraining and supervised adaptation on supported open models; the company still integrates and operates the program
Together AI. Managed adapter or full fine-tuning on supported open models, with model and serving constraints
Microsoft Foundry Programmed Training. A preview interface for custom post-training loops using LoRA; access approval and production limitations apply
Google managed reinforcement tuning. Technically relevant reward workflows, but its current preview terms exclude proprietary or confidential data and commercial production
These are support distinctions, not quality rankings. NVIDIA, Together, Microsoft, Google
A vendor offering fine-tuning does not imply that its strongest current model is available for it. That gap belongs in the comparison from the beginning.
The direct frontier APIs illustrate the distinction. OpenAI says its fine-tuning platform is winding down and closed to new users. Anthropic says the Claude API does not currently offer fine-tuning and directs interested customers to their contact. Neither statement rules out a separately negotiated program, but neither supports assuming public access to the full pipeline. OpenAI, Anthropic
The router design I led includes a proposed path for collecting training candidates. Such capture can help organize examples. It cannot supply missing local tool state, establish that the result was good, construct a reward environment, or prove that training pays.
The recurring bill includes data curation, experiments, serving, evaluation, refresh, and migration when the base model ages. A narrow tune may earn all of it. I would require that case before giving the company another model to maintain.
Owning the GPUs means owning the idle hours
There are several ways to buy inference: directly from a lab; through AWS, Microsoft, or Google; from a specialist managed provider; on dedicated rented capacity; or on hardware the company owns.
“Neocloud” does not tell you which product you are buying. A managed model endpoint and a rack of rented GPUs leave very different amounts of work with the customer.
The same model name can also hide different snapshots, numerical precision, context limits, tool support, cache behavior, and scheduling. Quantization uses less precise representations to reduce memory or computation. Whether it preserves the behavior your task needs has to be tested. A low price is not proof that a provider secretly quantizes, and an identical model label is not proof of an identical service.
Architecture affects the cost without deciding it. A smaller model that takes more reasoning steps or retries can lose to a larger model that finishes quickly. A sparse mixture-of-experts model activates only part of its parameters for each token, but the rest of the weights still have to be stored or made accessible. Moving information between experts, memory pressure, and uneven work can erase part of the arithmetic advantage.
DeepSeek is a useful reason to examine sparse serving, not a reason to infer that sparse models are inherently slower, cheaper, or worse. Data, training, tools, precision, hardware, and workload change the comparison.
At the provider level, GPUs and power are only part of the economics. Capital cost, depreciation, utilization, networking, memory, operations, research spending, and overhead matter too. Batching can spread cost across more requests while increasing waiting time. Price then adds competition, scarcity, contracts, subsidies, and margin. An API price is not a transparent statement of physical production cost.
Self-hosting makes those tradeoffs your responsibility. vLLM and SGLang offer serving machinery, but selecting a framework does not settle model compatibility, parallelism, cache capacity, concurrency, or recovery. Their current documentation exposes those as deployment choices. vLLM, SGLang
The model fitting in memory is only the first test. Active conversations need additional memory. Peak demand and failure recovery require spare capacity. Search, retrieval, tool execution, secrets, approvals, monitoring, and upgrades remain part of the product.
An illustrative break-even shows the sensitivity. Suppose a rented deployment costs $20,000 per month after staffing and reserve, with $1 of variable cost per accepted task. Suppose the comparable API task costs $3. At equal quality and service, the simple break-even is 10,000 accepted tasks a month. If API caching lowers the comparable task to $1.50, the threshold becomes 40,000.
Those are scenario inputs, not cloud quotes. The result also fails if the deployment cannot serve that volume within the required latency or needs another replica to do so.
Owned hardware adds purchase cost, financing, useful life, residual value, power, cooling, and replacement. A desktop already purchased for an engineer can have a low marginal cost for offline work, while a company-wide deployment has a very different full cost.
I would test on rented capacity before buying. A startup can rationally pay a provider margin to preserve runway and attention. A large company with steady eligible demand and existing operators may earn a saving from ownership. Either may choose local inference for privacy or offline availability even when it costs more. That is a valid purchase when the reason is stated honestly.
A gateway helps when used carefully
An engineer may spend the morning tidying Jira tickets, analyzing a routine usage report, and reviewing a risky change. Those jobs do not require the same model or the same amount of reasoning. A gateway can make the switch automatically, within rules the organization has tested, instead of asking every engineer to keep a model price list in their head.
Routine housekeeping may fit a cheaper route. A deep review may justify the strongest eligible model. Repeated calculations may belong in a script, with the model interpreting the result. The saving can be modest on each task and still worth collecting across recurring work. It earns its place when quality holds and the saving survives classification, retries, review, and any cache lost in the switch.
The router design I led combines that selection with capability floors and session continuity. It can raise the model requirement for demanding work while resisting a cheaper switch that would disrupt an ongoing task or waste a warm cache. The useful question is which work can move safely, and when. Sending every turn to the cheapest model would miss the point.
The same records can show where working habits deserve attention. Is a premium model doing routine data analysis that a script could handle? Is a conversation carrying a large amount of old history into a new task? Are large tool results repeatedly consuming context without helping the next decision? These are more useful questions than whether an engineer spent more tokens than a colleague.
Our design includes sampled, content-free signals for large or history-dominated context, low cache reuse, raw-data analysis without an observed deterministic tool, and strongest-tier use on a simple request shape. These are candidates for investigation. A long context may be necessary; sending full history is normal for some APIs; a tool may have run locally where the gateway cannot see it. Detailed workflow review requires a separately authorized capture path. I would use the findings to improve the harness and teach better practices, with the actual task in view before judging the engineer.
That visibility also makes the operating plan usable. The documented controls let team administrators allocate additional allowance within a team budget. An administrator can grant a user or team an expiring temporary budget for a migration, evaluation, or other one-off effort. A temporary team grant is a shared pool; the regular team budget funds individual allocations. Managers can delegate within a boundary, and a short project need not permanently raise everybody’s allowance.
Organizational workload tracking needs the same care. Usage by owner, team, model, and routing policy can explain where the bill is accumulating. Authorized task analysis can add a view of coding, research, data analysis, and operations. Proper project accounting also needs a trusted project identity and the team that owned the work when it occurred. The documented team reports use current membership, so they should not be mistaken for a historical project ledger.
The gateway can also keep provider credentials out of individual clients, restrict access to approved routes, and record which policy governed a request. These controls give procurement, security, and finance a common record to work from. Compliance still depends on the actual contracts, permissions, and data rules behind those routes.
One accounting distinction matters throughout: provider spend and the allowance shown to a user are different numbers. A credit for a failed request can restore the user’s allowance without erasing the company’s provider charge. A discounted internal profile can encourage an approved route without changing the upstream price. Finance should be able to reconcile both views, so an internal subsidy does not appear to be a technical saving.
A qualified fallback can also keep work moving when a provider fails. Recovery needs to know what has already happened: emitted tool arguments and an executed tool action are different states. The execution record must prevent duplicate work. Two provider names may also share a cloud, region, or identity service, so the alternative has to be tested against the failure it is meant to cover.
I would scale these controls to the organization: basic tagging and worker limits for a small team; shared routing, evaluation, and delegated budgets for a growing company; additional access and audit controls where the work requires them. Account for the gateway’s own latency, maintenance, and recovery. Used this way, it gives management practical ways to fund useful work and correct avoidable expense without renegotiating every request.
Your exit may depend on somebody else’s permission
Infrastructure decisions outlive model rankings. A capacity commitment, a fine-tune, or a provider-specific integration can still be on the books when the assumptions that justified it change.
The current competitive debate makes this concrete. OpenAI announced that its November 20, 2026 usage-policy update will require review and approval for services providing direct access to its models, including routers and gateways. That effective date is still ahead at the time of writing. The company’s stated rationale is consistent safeguards across services. For a business offering such access, it also creates a dependency that must be understood before promising customers provider choice. OpenAI’s announcement
That is a commercial consequence. It does not, by itself, prove an intent to exclude competitors.
The same distinction matters in regulation. The White House’s March legislative framework calls for a uniform national approach. California signed additional AI audit and assessment legislation in September. A proposed federal approach and enacted state laws have different legal status; neither automatically imposes the same obligations on a frontier developer and a company using an API. White House framework, California announcement
A rule with large fixed compliance costs can favor incumbents even when its safety rationale is sincere. A uniform rule can also reduce the cost of complying with incompatible local requirements. Claims of regulatory capture need evidence about the text, who sought it, its costs, and the alternatives. Suspicion is a reason to investigate.
U.S.–China competition changes the available options too. BIS’s January revision moved certain advanced-chip export applications, including H200 and MI325X-class products, to conditional case-by-case review. That illustrates how hardware access can change through policy, rather than through a model’s technical merit. BIS
Open weights can give a company more control over serving, modification, and retirement. The license, hardware, staff, and a model that meets the task still have to exist. An exit option is more credible after a migration rehearsal than after a slide showing a second provider.
NVIDIA’s November leadership transition is a fresh reason to examine these dependencies. The latest discussions raise scenarios about cadence, financing, and export restrictions. Those are scenarios from investors and political participants.
For a company outside the labs, the useful response is to preserve evaluation cases, task state, and a practical ability to switch. Accept provider-specific features when their value warrants the dependency. Record what would be required to leave.
Before approving the next increase
I would expect an AI budget proposal to explain which work the company intends to attempt, what currently prevents that work from paying off, and why additional inference is the best next purchase.
The answer may justify a sharp, temporary increase for a rewrite that removes an expensive constraint. It may justify verification capacity before more generation, or a premium for an eligible enterprise route. Repetitive housekeeping may need ordinary code. The spending profile should follow the opportunity and its risks.
What I would not accept is a larger budget justified only by growing usage—or a cut justified only by the size of the invoice. Both avoid deciding what the work is worth.
Management encouraged engineers to learn these tools. It now owes them a coherent decision about which work to continue, what quality to preserve, and what the business expects to gain. A smaller competitor may make a costly, well-chosen attempt while we debate an allowance. We may also spend freely and build something nobody needs. Choosing between those possibilities remains our job.








