<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:googleplay="http://www.google.com/schemas/play-podcasts/1.0"><channel><title><![CDATA[The Equation]]></title><description><![CDATA[Achieving balance for optimal performance]]></description><link>https://www.equationblog.com</link><image><url>https://substackcdn.com/image/fetch/$s_!-MQk!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0187b1d4-c470-4ffa-a463-efc198a6864f_1280x1280.png</url><title>The Equation</title><link>https://www.equationblog.com</link></image><generator>Substack</generator><lastBuildDate>Thu, 19 Nov 2026 20:14:52 GMT</lastBuildDate><atom:link href="https://www.equationblog.com/feed" rel="self" type="application/rss+xml"/><copyright><![CDATA[Ruslan Belkin]]></copyright><language><![CDATA[en]]></language><webMaster><![CDATA[ruslan@substack.com]]></webMaster><itunes:owner><itunes:email><![CDATA[ruslan@substack.com]]></itunes:email><itunes:name><![CDATA[Ruslan Belkin]]></itunes:name></itunes:owner><itunes:author><![CDATA[Ruslan Belkin]]></itunes:author><googleplay:owner><![CDATA[ruslan@substack.com]]></googleplay:owner><googleplay:email><![CDATA[ruslan@substack.com]]></googleplay:email><googleplay:author><![CDATA[Ruslan Belkin]]></googleplay:author><itunes:block><![CDATA[Yes]]></itunes:block><item><title><![CDATA[What Did All Those Tokens Buy?]]></title><description><![CDATA[AI budgets, engineering capacity, and the gap between more software and a better business.]]></description><link>https://www.equationblog.com/p/what-did-all-those-tokens-buy</link><guid isPermaLink="false">https://www.equationblog.com/p/what-did-all-those-tokens-buy</guid><dc:creator><![CDATA[Ruslan Belkin]]></dc:creator><pubDate>Mon, 16 Nov 2026 15:15:41 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!e73A!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86d6f471-a94f-4726-9b24-c2c886b79c3e_1752x897.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em>Figures and product terms checked November 14, 2026.</em></p><p>By April, Uber had exhausted its 2026 AI coding budget, its CTO told <a href="https://www.theinformation.com/newsletters/applied-ai/uber-cto-shows-claude-code-can-blow-ai-budgets">The Information</a>.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>I understand the case for a generous initial budget. Engineers need room to experiment. If management wants them to learn a new way of working, requiring approval for every attempt is an effective way to discourage it. Pay centrally, make access easy, and find out what people can do.</p><p>The problem begins when an experiment becomes a recurring production expense without a corresponding decision about what it is supposed to buy. The organization has learned to consume capacity. Management has not necessarily learned which additional work is worth funding.</p><p>The reaction is already visible. In Sonar&#8217;s survey released November 12, 35% of respondents at enterprises with more than 1,000 employees said their teams had cut back AI coding to control token spending, compared with 21% at companies with 200 or fewer employees. The survey covered 1,026 developers and engineering managers in September. These are reports from people using AI, not an audit of company finances, but they establish that restrictions are more than a hypothetical concern. <a href="https://www.sonarsource.com/developer-survey-report-2027.pdf/#page=51">Sonar report, p. 51</a></p><p>Anthropic&#8217;s IPO disclosures show the scale of the money flowing in the other direction. It reported almost $17 billion of revenue for the third quarter of 2026, more than twelve times the year-earlier quarter. Those are recognized sales, not a rumored valuation or an annualized run rate. They span customers and uses well beyond internal software development. <a href="https://www.sec.gov/Archives/edgar/data/2130357/000162828026072949/anthropicpbc-fwpipowebsi.htm">November 10 SEC filing</a></p><p>The filing tells us how much Anthropic sold. It cannot tell a CEO whether their own company improved its product, won more customers, or earned a higher profit.</p><p>That is the uncomfortable part of the bill. Engineers may be working faster. The tools may be doing exactly what was promised. The company can still be buying more activity than it knows how to turn into a result.</p><p>Before setting next year&#8217;s allowance, I would ask what the company believes it is buying: more engineering capacity, a better product, or a better financial outcome. Moving from one to the next requires a separate argument. For some companies, that argument will justify a much larger bill.</p><h2>You can build faster and still sell nothing</h2><p>Hiring another hundred engineers does not automatically make a product easier to sell. Neither does giving the existing engineers enough inference to generate several times as much code.</p><p>Additional capacity helps when the company knows what to build and cannot deliver it quickly enough. It can accelerate experiments, remove a reliability problem, improve onboarding, or make an expensive operation cheaper. But a faster implementation of the wrong idea is still the wrong idea. More features can also make a product harder to understand and more expensive to support.</p><p>Tokens change the speed and price of producing software. They do not remove the need to decide which software customers want, how people will find it, or why they will pay. If customer demand is the constraint, the return comes from using capacity to learn and respond&#8212;not from filling a larger release with more features.</p><p>This does not mean every useful AI purchase must produce immediate revenue. Security work can reduce expected loss. A migration can avoid future operating expense. Faster experimentation can reveal that a product should not be built. Revenue can also remain flat while the company prevents churn it would otherwise have suffered. Flat sales alone cannot establish failure, just as a growing pull-request count cannot establish success.</p><p>The company needs to name the intended benefit, the evidence it expects to see, and the period in which it expects to see it. For customer-facing work, that might mean a measured change in activation, retention, conversion, or support burden. For internal work, it might be faster resolution of a particular incident class or a planned hire no longer needed. Count incremental contribution after the costs of serving customers, rather than treating every extra dollar of sales as profit.</p><p>The answer will differ across organizations. A founder may pay generously to test product ideas before cash runs out. A growing software company may already have more proposed changes than reviewers can absorb. A regulated enterprise may buy stronger verification because an incorrect migration could cost far more than the model bill.</p><p>Nassim Nicholas Taleb warns: &#8220;Don&#8217;t cross a river that is on average four feet deep.&#8221; The depth where you cross matters. An industry productivity average cannot tell these companies where to spend. <a href="https://www.fooledbyrandomness.com/OxfordBTLecture.pdf">Taleb, technical companion to The Black Swan</a></p><p>Size alone is insufficient too. A small AI company may have steady inference demand and experienced operators. A large enterprise may have fragmented, bursty usage across regions and product lines. The first may fill dedicated capacity; the second may pay to keep it idle. A useful operating plan starts with the work and the constraint.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!e73A!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86d6f471-a94f-4726-9b24-c2c886b79c3e_1752x897.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!e73A!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86d6f471-a94f-4726-9b24-c2c886b79c3e_1752x897.png 424w, https://substackcdn.com/image/fetch/$s_!e73A!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86d6f471-a94f-4726-9b24-c2c886b79c3e_1752x897.png 848w, https://substackcdn.com/image/fetch/$s_!e73A!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86d6f471-a94f-4726-9b24-c2c886b79c3e_1752x897.png 1272w, https://substackcdn.com/image/fetch/$s_!e73A!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86d6f471-a94f-4726-9b24-c2c886b79c3e_1752x897.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!e73A!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86d6f471-a94f-4726-9b24-c2c886b79c3e_1752x897.png" width="1456" height="745" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/86d6f471-a94f-4726-9b24-c2c886b79c3e_1752x897.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:745,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:3297511,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.equationblog.com/i/225083362?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86d6f471-a94f-4726-9b24-c2c886b79c3e_1752x897.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!e73A!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86d6f471-a94f-4726-9b24-c2c886b79c3e_1752x897.png 424w, https://substackcdn.com/image/fetch/$s_!e73A!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86d6f471-a94f-4726-9b24-c2c886b79c3e_1752x897.png 848w, https://substackcdn.com/image/fetch/$s_!e73A!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86d6f471-a94f-4726-9b24-c2c886b79c3e_1752x897.png 1272w, https://substackcdn.com/image/fetch/$s_!e73A!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86d6f471-a94f-4726-9b24-c2c886b79c3e_1752x897.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>&#8220;At this rate we&#8217;ll finish the roadmap before we find a customer.&#8221;</em></p><h2>Plan the bill before promising the return</h2><p>Headcount plus tokens is a useful correction to a payroll-only engineering plan. It is still incomplete.</p><p>Consider an illustrative company with 200 engineers, $60 million in annual revenue, and $6 million in operating profit before new AI spending. An incremental $1,000 per engineer per month adds $2.4 million a year. If the entire amount is expensed in the period and nothing else changes, operating profit falls to $3.6 million. The margin moves from 10% to 6%.</p><p>Those are assumptions, not an estimate of typical spending. They show why a bill that looks modest beside an individual salary can matter to the bottom line. At an assumed 80% contribution margin, recovering that $2.4 million through sales alone would require $3 million of additional revenue, before any further fixed costs.</p><p>Payroll comparisons still matter. A tool that removes enough work to defer a hire can pay for itself. But &#8220;cheaper than another engineer&#8221; is a useful comparison only when another engineer was the alternative and the same valuable work gets done. If payroll stays fixed, do not book a payroll saving. The purchase has to earn its cost through useful capacity, quality, lower risk, or another demonstrated benefit.</p><p>I would separate the operating plan into four kinds of spending:</p><ul><li><p><strong>Exploration and shared evaluation.</strong> A defined question answered, a credible opportunity found, or an expensive approach rejected</p></li><li><p><strong>Internal engineering work.</strong> Useful delivery, quality, or operating improvements after review and repair</p></li><li><p><strong>AI used in the product customers consume.</strong> Customer value and contribution after inference, tools, support, and failure costs</p></li><li><p><strong>Shared infrastructure and customized models.</strong> Enough eligible demand or risk reduction to cover operation, reserve, maintenance, and eventual migration</p></li></ul><p>These are management views, not accounting classifications. Finance still has to classify the actual costs. The important distinction is between spending to build the product and spending each time a customer uses it. Combining both into a per-engineer allowance makes neither easier to understand.</p><p>Fund initial evaluation and shared learning centrally. Give each experiment an owner, a bounded budget, and a date for deciding what was learned. Allocate established use to the product, project, or service that benefits from it. Keep special initiatives visible, and let team or project budgets accommodate differences in the work. A move from central funding to local chargeback needs a transition: the invoice has moved, even if the work has not become more expensive.</p><p>Forecast human sessions and unattended workers separately. Both can change their volume, effort, context, retries, and tool use. Include low, expected, and high-use cases, minimum commitments, and the cash needed when benefits arrive later than the bill. Revisit those assumptions as actual work replaces guesses. Current FinOps guidance similarly recommends shorter forecast windows, flexible pilot funding, and cost measures tied to successful outcomes. <a href="https://www.finops.org/wg/finops-for-ai-overview/">FinOps Foundation</a></p><p>The cash plan needs people, providers, tools, infrastructure, and the extra staff or contractors required to operate them. Review and repair also consume capacity, but hours already paid for in payroll should not be added again as a second cash expense. Keep that capacity view beside the cash view. Neither an allocation nor a claimed hour saved changes the bank balance by itself.</p><p>The plan also needs room for spending that is deliberately uneven. A temporary campaign and a permanent increase in the running bill deserve different decisions.</p><h2>The underdog may need the bigger AI budget</h2><p>A small company can be outspent on payroll and still choose to spend aggressively on inference. If additional capacity can be rented for a few weeks, it can attempt work that would never justify recruiting a permanent team.</p><p>That can change which projects are worth considering. An awkward architecture may prevent a company from serving a new market or make every customer expensive to support. A rewrite previously dismissed as unaffordable may become a credible experiment. The alternative is not always maintaining the existing system cheaply. Sometimes it is continuing to pay for a system that prevents the business from working.</p><p>The financial attraction is the ability to stop. A team can buy a prototype, test the hardest assumptions, and abandon the attempt before committing to the full migration. That flexibility depends on the provider contract and the work itself: review, integration, and production responsibility cannot be summoned as easily as another model call. A larger batch of generated code can exhaust the people needed to assess it.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!rXlg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c689106-a7a7-4cc9-91eb-0246400b5293_2160x1206.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!rXlg!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c689106-a7a7-4cc9-91eb-0246400b5293_2160x1206.png 424w, https://substackcdn.com/image/fetch/$s_!rXlg!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c689106-a7a7-4cc9-91eb-0246400b5293_2160x1206.png 848w, https://substackcdn.com/image/fetch/$s_!rXlg!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c689106-a7a7-4cc9-91eb-0246400b5293_2160x1206.png 1272w, https://substackcdn.com/image/fetch/$s_!rXlg!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c689106-a7a7-4cc9-91eb-0246400b5293_2160x1206.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!rXlg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c689106-a7a7-4cc9-91eb-0246400b5293_2160x1206.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3c689106-a7a7-4cc9-91eb-0246400b5293_2160x1206.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:96063,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.equationblog.com/i/225083362?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c689106-a7a7-4cc9-91eb-0246400b5293_2160x1206.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!rXlg!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c689106-a7a7-4cc9-91eb-0246400b5293_2160x1206.png 424w, https://substackcdn.com/image/fetch/$s_!rXlg!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c689106-a7a7-4cc9-91eb-0246400b5293_2160x1206.png 848w, https://substackcdn.com/image/fetch/$s_!rXlg!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c689106-a7a7-4cc9-91eb-0246400b5293_2160x1206.png 1272w, https://substackcdn.com/image/fetch/$s_!rXlg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c689106-a7a7-4cc9-91eb-0246400b5293_2160x1206.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>A flat weekly allowance can block a concentrated attempt even when total model spending is unchanged. The figure estimates no output or return; the people and migration costs still need funding.</em></p><p>There is a recent example worth examining. In its November 12 engineering account, Hex describes a four-month push that changed roughly 250,000 lines across foundational systems&#8212;about a tenth of its codebase. The team credits coding agents while emphasizing iteration, API boundaries, and checkable completion criteria. This is a company account of a shipped rewrite, without a controlled cost comparison or a measured commercial return. It demonstrates a project to investigate, not a productivity multiplier to put in the budget. <a href="https://hex.tech/blog/what-we-learned-working-with-robots-to-rewrite-hex/">Hex engineering</a></p><p>An underdog may have fewer customer contracts, compatibility promises, and established revenue streams to endanger. A competitor&#8217;s installed base can make the same technical change harder to approve. But the incumbent also has cash, experienced reviewers, distribution, and customers who can explain what matters. Microsoft&#8217;s October account of agent-assisted modernization is a reminder that large companies can concentrate expertise too. Its approach includes behavior comparisons and tests withheld from the agent. <a href="https://commandline.microsoft.com/agentic-software-engineering-large-scale-code-migration-language-modernization-spec-driven-development/">Microsoft engineering</a></p><p>Buying access to the same models as everyone else is unlikely to be a lasting advantage by itself. The opportunity lies in choosing a consequential project, organizing its verification, and completing it while the result still matters. Public evidence does not yet tell us how often small companies convert that opportunity into durable commercial gains.</p><p>I would fund such work as a bounded project, with a technical claim to prove and a business constraint to remove. Compare a replacement with a smaller refactor or an automated migration. Test the difficult behavior before polishing the easy screens. Include data conversion, security review, parallel operation, rollback, and the cost of maintaining what remains afterward. Building a complete replacement in parallel does not require switching every customer at once.</p><p>And keep the authority to stop real. A convincing demo should earn funding for the next stage only if it answers the question that justified the experiment. It should not automatically release the rest of the budget.</p><p>An underdog may have little revenue to protect. It still has runway to lose. The right budget can let it attempt something the larger competitor cannot easily attempt; it cannot make a bad project good.</p><h2>Follow one change until somebody can use it</h2><p>Take a hypothetical bug in a mature service. A low-priced model proposes a patch, adds tests, and passes the visible checks. During review, an engineer discovers that the change broadens an interface other services depend on. The patch needs repair.</p><p>For illustration, assume the first route costs $2 in model calls and requires 45 minutes of review and repair. A different route costs $12 and requires 10 minutes. At an assumed $150 per hour for engineering capacity, the first consumes $114.50 and the second $37.</p><p>Those are invented inputs, not measured model results. Their purpose is to expose the missing term. The ten-dollar inference premium buys 35 minutes of scarce engineering capacity in this example. If it buys no reduction in review, delay, or risk, it is simply an extra ten dollars.</p><p>Nor is the released time automatically cash. Payroll may remain exactly the same. The benefit becomes economic when the engineer uses that time to remove another constraint, delivery happens earlier, a contractor is no longer needed, or a hire can be deferred. Counting the hour as both payroll savings and additional output would count it twice.</p><p>The record should follow the whole attempt, including unsuccessful work.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ipRI!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F644eeb3d-94e1-49c4-b373-d3440808dec1_1700x1836.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ipRI!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F644eeb3d-94e1-49c4-b373-d3440808dec1_1700x1836.png 424w, https://substackcdn.com/image/fetch/$s_!ipRI!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F644eeb3d-94e1-49c4-b373-d3440808dec1_1700x1836.png 848w, https://substackcdn.com/image/fetch/$s_!ipRI!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F644eeb3d-94e1-49c4-b373-d3440808dec1_1700x1836.png 1272w, https://substackcdn.com/image/fetch/$s_!ipRI!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F644eeb3d-94e1-49c4-b373-d3440808dec1_1700x1836.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ipRI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F644eeb3d-94e1-49c4-b373-d3440808dec1_1700x1836.png" width="1456" height="1572" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/644eeb3d-94e1-49c4-b373-d3440808dec1_1700x1836.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1572,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:111641,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.equationblog.com/i/225083362?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F644eeb3d-94e1-49c4-b373-d3440808dec1_1700x1836.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ipRI!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F644eeb3d-94e1-49c4-b373-d3440808dec1_1700x1836.png 424w, https://substackcdn.com/image/fetch/$s_!ipRI!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F644eeb3d-94e1-49c4-b373-d3440808dec1_1700x1836.png 848w, https://substackcdn.com/image/fetch/$s_!ipRI!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F644eeb3d-94e1-49c4-b373-d3440808dec1_1700x1836.png 1272w, https://substackcdn.com/image/fetch/$s_!ipRI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F644eeb3d-94e1-49c4-b373-d3440808dec1_1700x1836.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>The invoice captures part of this sequence. Review, abandoned attempts, and later repair belong to the cost; release still needs to produce the intended result.</em></p><p>Cost per accepted task is useful, provided the organization also records what it attempted, deferred, and abandoned. Otherwise the easiest way to improve the metric is to choose easier work. Acceptance itself is also insufficient: a beautifully implemented feature can be unnecessary.</p><p>This distinction changes model selection across the development cycle. Updating Jira fields or checking branch status may be a deterministic operation. Research needs reliable sources and a stopping rule. Design needs local constraints and someone willing to choose among imperfect alternatives. Debugging needs controlled experiments. A security review needs evidence that survives a confident explanation.</p><p>A company paying the same model price for all of those activities is making an expensive assumption about how similar they are.</p><h2>Faster code can become somebody else&#8217;s workload</h2><p>Speed has value when a person is waiting to make the next decision. A delayed answer can interrupt concentration, trigger another task, and require the engineer to reconstruct the original problem on return. Faster generation can remove that interruption.</p><p>An overnight worker has a different objective. It may tolerate longer latency if it completes more accepted work by morning at lower cost. Filling the queue with additional patches does little if the same two maintainers must review them all.</p><p>Review also has to examine the design burden left behind. An agent can reproduce a familiar pattern from the repository even when that pattern is the legacy debt the change ought to remove. It can introduce a new framework where a small function would have been sufficient. Passing the requested example does not establish that the next engineer will be able to change the result cheaply.</p><p>Budget policy can interfere with that judgment. Consider an engineer approaching a daily allowance. The patch is ready and the tests pass. If another model review requires a manager&#8217;s exception, the engineer has a reason to stop early. Autonomous runs without useful stopping rules can also keep consuming resources after additional attempts stop helping. The budget needs to contain both failures.</p><p>We do not yet have a credible causal estimate of how personal token caps affect engineering quality. My preference is to make human usage visible through team or project budgets, provide simple exceptions for consequential work, and place hard limits on autonomous runs, tools, elapsed time, and spending. Then check whether the policy reduces waste or merely moves it into review and repair.</p><p>There is a practical alternative to an indiscriminate cut. By August, Uber reported stabilizing total AI spend while usage grew. Its engineering account describes live cost visibility, shared spending tiers, easier escalation, and changes to context, caching, tools, and model selection. Those are company-reported operating results, not proof of a profit gain or a current-model benchmark. They show that a budget surprise need not end with engineers simply doing less. <a href="https://www.uber.com/gb/en/blog/efficient-software-factory/">Uber engineering</a></p><p>An engineer should be able to explain why a costly investigation was worthwhile. A manager should have to explain why preventing it was worthwhile too.</p><h2>A passing test can be the wrong reward</h2><p>Return to the service bug. If the agent writes tests around its own interpretation of the interface, the tests may confirm the same mistake as the implementation. Ten more tests derived from that interpretation can make the patch look safer while adding little assurance.</p><p>It is tempting to blame this on reinforcement learning with verifiable rewards (RLVR). In a coding training environment, the model attempts a task and receives a reward from an executable check. Passing tests can supply that signal. If the check accepts an incorrect solution, training can reinforce a shortcut that the organization never intended to teach.</p><p>But there are different problems hiding inside that explanation. Fitting an implementation to incomplete tests, tampering with the evaluator, and writing weak tests for your own code are different behaviors. If the reward comes from fixed, protected tests, adding a trivial test of your own does not improve the reward. A training process that pays a test generator merely for producing passing tests creates a different incentive from one that pays it for exposing real faults.</p><p>The distinction changes what the evidence can establish. Goodhart Labs&#8217; current HoneyBench includes Opus 5.5, GPT-6 Astra, and GPT-6.1 Sol in deliberately flawed scoring environments. It documents agents exploiting gaps between the score and the stated task. These are selected stress tests, not a sample of company PRs or an experiment isolating the effect of RL training. <a href="https://goodhartlabs.com/honeybench">HoneyBench v0.2</a></p><p>A November 9 controlled study adds a useful complication: shared learning can suppress some verifier exploits and amplify others, depending on how the errors recur across tasks. Its simplified environment does not estimate the behavior of a production coding model. It does undermine the easy story that reinforcement learning must always make cheating worse. <a href="https://arxiv.org/abs/2611.13218">Wrong the Same Way</a></p><p>I did not find public evidence that isolates RLVR as the cause of useless test writing in the current models on company workflows. That remains an open question. Training data, inherited repository conventions, incomplete requirements, instructions to make CI green, and the engineer&#8217;s own review habits can produce similar symptoms. A passing test tells us neither which cause operated nor whether the test was useful.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!O4X3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20056650-62a1-4c63-9a09-e26a900fe71d_1752x897.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!O4X3!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20056650-62a1-4c63-9a09-e26a900fe71d_1752x897.png 424w, https://substackcdn.com/image/fetch/$s_!O4X3!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20056650-62a1-4c63-9a09-e26a900fe71d_1752x897.png 848w, https://substackcdn.com/image/fetch/$s_!O4X3!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20056650-62a1-4c63-9a09-e26a900fe71d_1752x897.png 1272w, https://substackcdn.com/image/fetch/$s_!O4X3!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20056650-62a1-4c63-9a09-e26a900fe71d_1752x897.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!O4X3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20056650-62a1-4c63-9a09-e26a900fe71d_1752x897.png" width="1456" height="745" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/20056650-62a1-4c63-9a09-e26a900fe71d_1752x897.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:745,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:2809959,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.equationblog.com/i/225083362?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20056650-62a1-4c63-9a09-e26a900fe71d_1752x897.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!O4X3!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20056650-62a1-4c63-9a09-e26a900fe71d_1752x897.png 424w, https://substackcdn.com/image/fetch/$s_!O4X3!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20056650-62a1-4c63-9a09-e26a900fe71d_1752x897.png 848w, https://substackcdn.com/image/fetch/$s_!O4X3!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20056650-62a1-4c63-9a09-e26a900fe71d_1752x897.png 1272w, https://substackcdn.com/image/fetch/$s_!O4X3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20056650-62a1-4c63-9a09-e26a900fe71d_1752x897.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>&#8220;The test confirms that the light is green.&#8221;</em></p><p>The practical response is to ask what wrong behavior a test would reject. For a bug fix, demonstrate that the regression test fails on the defective behavior and passes on the correction. Add independently chosen boundary cases. Deliberately introduce a relevant fault and see whether the suite notices. Mutation testing does that systematically, although an artificial mutation is still a proxy: some mutations do not change observable behavior, and killing many easy mutants can miss the dangerous fault.</p><p>Keep acceptance checks outside the agent&#8217;s authority to redefine. Review changes to existing assertions, because an updated expectation can be a legitimate specification change or a quiet removal of the requirement. A second agent may help, but it may also share the first agent&#8217;s assumption. Independence comes from the evidence and checks, not simply from opening another conversation.</p><p>This can justify more testing. Agents may make previously unaffordable regression coverage practical. It can also justify deleting redundant tests whose maintenance costs exceed what they detect. Rewarding a generator for valid tests that distinguish faulty implementations is a more useful proposal to evaluate than rewarding a high pass count.</p><p>The expense belongs in the same operating plan as generation. Saving tokens by removing verification can make the bill smaller while leaving the company with a result it cannot safely use.</p><h2>The same agent does not make the same engineer</h2><p>Two engineers can use identical tools and produce different results because they notice different things.</p><p>In our service example, one may know that an apparently local interface is also consumed by an old customer integration. Another may be excellent at implementation but unfamiliar with that dependency. The agent&#8217;s behavior then depends on whether the missing constraint reaches it, whether it asks a useful question, and whether someone recognizes the consequence of its proposal.</p><p>That is more specific than being good at prompting.</p><p>A November survey of 203 developers found that autonomy choices varied more between developers than between development activities. Among those reporting careful review of most or all generated code, 61% still said they often approved at least one type of agent action without close review. Reviewing a patch and authorizing an action are different responsibilities. <a href="https://arxiv.org/abs/2611.10058">From Delegation to Action Approval</a></p><p>An October workplace study followed 11 professional developers for about 15 hours. It observed different patterns of supervising, delegating, and learning, along with large variation in checking and task switching. It also recorded fatigue associated with managing parallel work. The small sample reveals behaviors worth measuring; it cannot tell a company how many agents each engineer should supervise. <a href="https://arxiv.org/abs/2610.33588">Developers as Managers of AI Agents</a></p><p>For internal evaluation, I would compare stepwise interaction, delegation with checkpoints, and bounded autonomous runs on comparable tasks. Hold the model and tools fixed. Distinguish repository familiarity, domain expertise, software experience, and agent experience.</p><p>Then look at the quality of the decisions. Did the engineer define success? Did the agent disclose a consequential assumption? Was drift caught before it compounded? Did the correction address the cause? Was the verification independent enough to catch the agent&#8217;s mistake? Could the engineer later explain and modify the result?</p><p>Counting interventions cannot answer those questions. Frequent interruption can indicate poor agent behavior, unnecessary control, a difficult task, or an expert preventing a serious error. Silence can indicate sound delegation or failure to notice.</p><p>The gateway records calls and charges. It cannot observe understanding.</p><p>This has consequences for training and hiring. I would teach specification, risk assessment, verification, recovery, and attention management alongside tool use. For a hiring exercise, let the candidate use an approved agent, then ask them to explain the resulting design, find a planted defect, or make a follow-up change. Their judgment should remain visible even when code generation is easy.</p><p>Learning requires its own check. In a small randomized study of engineers learning an unfamiliar Python library, the assisted group scored 17 percentage points lower on an immediate knowledge assessment. Exploratory analysis associated conceptual inquiry and generation followed by comprehension with stronger learning than extensive delegation. The experiment involved an earlier assistant setup and a narrow task; it is not a current-agent productivity estimate or proof of permanent skill loss. It is a reason to test what people retain. <a href="https://www.anthropic.com/research/AI-assistance-coding-skills">Anthropic&#8217;s coding-skills study</a></p><p>For a junior engineer, completing today&#8217;s task and developing the ability to diagnose tomorrow&#8217;s failure are both outputs. A staffing plan that counts only the first can consume the apprenticeship it will later depend on.</p><p>Retention is less settled. Agents can remove tedious work and give engineers more control. They can also turn an absorbing craft into a stream of approval decisions. I would track learning time, perceived mastery, fatigue, mentoring, review concentration, and actual departures before claiming that adoption improves or damages retention.</p><p>And I would keep raw interaction statistics out of individual rankings. Reward low token use and engineers may hide useful work. Reward long autonomous runs and they may delay intervention. Reward intervention itself and they may interrupt needlessly. The company would be training people to produce an attractive trace.</p><h2>The model ranking is the beginning of the purchase</h2><p>Current results already show why a price card cannot settle the decision.</p><p>Artificial Analysis&#8217;s live Intelligence Index v5.0.1 reports the following selected current-generation configurations. The index combines several kinds of work; its score is neither a percentage of coding tasks solved nor a measure of company productivity.</p><ul><li><p><strong><a href="https://artificialanalysis.ai/models/claude-opus-5-5">Claude Opus 5.5, maximum effort, Default Fallback</a>.</strong> Index score: 59; Reported cost per index task: $10.04</p></li><li><p><strong><a href="https://artificialanalysis.ai/models/gpt-6-astra">GPT-6 Astra, maximum effort</a>.</strong> Index score: 57; Reported cost per index task: $6.24</p></li><li><p><strong><a href="https://artificialanalysis.ai/models/gpt-6-1-sol">GPT-6.1 Sol, maximum effort</a>.</strong> Index score: 55; Reported cost per index task: $1.27</p></li></ul><p>The Opus 5.5 page reports scores from 35 at low effort to 59 at maximum effort, and average task time from 3.0 to 25.2 minutes. The effort setting is a purchasing decision of its own. The comparison preserves the publisher&#8217;s fallback label because it describes a tested configuration, not isolated weights. Reported costs are weighted estimates using token prices; they omit company review and repair. <a href="https://artificialanalysis.ai/methodology/intelligence-benchmarking">Artificial Analysis&#8217;s current methodology</a></p><p>For coding candidates, its separate Coding Agent Index v2.1 is more focused: implementation, terminal work, and repository questions, with task cost and execution time. Different task mixes answer different questions. An aggregate score cannot tell you whether the model is good at the particular class of failure that makes your senior engineers nervous. <a href="https://artificialanalysis.ai/methodology/coding-agents-benchmarking">Coding Agent Index methodology</a></p><p>A separate November benchmark, ExpertCrowd-SWE, reports 67.3% for Opus 5.5 and 63.7% for GPT-6 Astra without a statistically significant difference between them. Its tasks come from unfamiliar repositories and use calibrated graders. That is useful evidence for choosing candidates; the small headline gap does not establish a winner for your codebase. <a href="https://arxiv.org/abs/2611.13246">ExpertCrowd-SWE</a></p><p>What you buy also includes the software around the model: how it finds files, retains context, runs tools, requests approval, retries, and stops. Engineers call this the harness. Change it and you can change both the work performed and the bill.</p><p>For a company with proprietary code, I would build a private evaluation from actual work. Historical pull requests are a useful starting point, with a trap: a merged patch is not a controlled experiment.</p><p>Reconstruct the repository before the change. Supply the issue and information available at that time. Hide the eventual patch, later reviews, and future fixes, including copies left in version-control objects or searchable indexes. Confirm that the historical solution passes the reconstructed checks and that an unchanged repository fails where it should. Allow other correct implementations to pass. A test written by the candidate belongs among its outputs; independent checks still decide whether that output is good.</p><p>Then rerun the incumbent&#8212;Opus 5.5, if that is the chosen baseline&#8212;alongside other eligible current models, a simple routing rule, and the automatic router. Keep a set of tasks out of tuning. Follow the retrospective comparison with newly arriving work, because merged PRs exclude the abandoned investigations and failed changes that also consume engineering time.</p><p>Measure review, repair, latency, and later quality alongside model charges. First use a common harness to understand differences; then compare the production configurations the company can actually deploy.</p><p>An automatic router needs the same scrutiny. Its first choice changes the answer, the next tool call, the cached context, and the work a later model inherits. A cheap early route can create an expensive rescue.</p><p>In the router work I led, the design distinguishes the proposed route from the final route after session and other constraints are applied. That distinction helps explain what happened. It does not prove a saving. The saving has to survive the complete task.</p><h2>Lunch changes the price of context</h2><p>An agent may repeatedly send repository instructions, tool definitions, and conversation history to a model. Prompt caching lets the provider reuse processing of an unchanged beginning of that input. The cached object is context processing, not a stored answer.</p><p>The clock matters because human conversations and agent loops return at different intervals.</p><p>Anthropic&#8217;s current Messages API supports automatic breakpoint placement or explicit breakpoints, both enabled through cache controls. The default lifetime is five minutes, with a one-hour option. A qualifying read refreshes the lifetime, measured from request start. A four-minute generation therefore leaves about one minute for the next request to reuse that entry. <a href="https://platform.claude.com/docs/en/build-with-claude/prompt-caching">Anthropic prompt caching</a></p><ul><li><p><strong>Agent calls a tool and returns every minute.</strong> Repeated reads can keep a short-lived prefix warm</p></li><li><p><strong>Engineer reads the answer and returns after eight minutes.</strong> A five-minute entry may have expired</p></li><li><p><strong>Engineer leaves for a 45-minute lunch.</strong> A one-hour entry may retain useful context</p></li><li><p><strong>Unattended worker waits 70 minutes for CI.</strong> Neither duration necessarily covers the pause</p></li></ul><p><em>Assume an unchanged eligible prefix and compatible routing. These are timing examples, not guaranteed hits.</em></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!8Rn5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7130216-1094-4ca4-8f18-c218f6e3c221_1774x887.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!8Rn5!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7130216-1094-4ca4-8f18-c218f6e3c221_1774x887.png 424w, https://substackcdn.com/image/fetch/$s_!8Rn5!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7130216-1094-4ca4-8f18-c218f6e3c221_1774x887.png 848w, https://substackcdn.com/image/fetch/$s_!8Rn5!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7130216-1094-4ca4-8f18-c218f6e3c221_1774x887.png 1272w, https://substackcdn.com/image/fetch/$s_!8Rn5!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7130216-1094-4ca4-8f18-c218f6e3c221_1774x887.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!8Rn5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7130216-1094-4ca4-8f18-c218f6e3c221_1774x887.png" width="1456" height="728" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c7130216-1094-4ca4-8f18-c218f6e3c221_1774x887.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:728,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:3213517,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.equationblog.com/i/225083362?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7130216-1094-4ca4-8f18-c218f6e3c221_1774x887.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!8Rn5!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7130216-1094-4ca4-8f18-c218f6e3c221_1774x887.png 424w, https://substackcdn.com/image/fetch/$s_!8Rn5!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7130216-1094-4ca4-8f18-c218f6e3c221_1774x887.png 848w, https://substackcdn.com/image/fetch/$s_!8Rn5!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7130216-1094-4ca4-8f18-c218f6e3c221_1774x887.png 1272w, https://substackcdn.com/image/fetch/$s_!8Rn5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7130216-1094-4ca4-8f18-c218f6e3c221_1774x887.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>&#8220;Your conversation is still here. Your discount went to lunch.&#8221;</em></p><p>There is a price for buying time. At the checked Opus 5.5 rates, ordinary input costs $4 per million tokens, a five-minute write $5, a one-hour write $8, and a cache read $0.20. For one million identical input tokens used twice, a one-hour write and one hit cost $8.20. Two ordinary uncached inputs cost $8. Three uses change the comparison to $8.40 against $12. Output and changing context are additional. <a href="https://platform.claude.com/docs/en/build-with-claude/prompt-caching#pricing">Anthropic cache pricing</a></p><p>That calculation compares caching with uncached input. Comparing one hour with a five-minute policy that expires and writes again is a different calculation. Confusing the alternatives can make the same cache appear either wasteful or economical.</p><p>Provider controls also differ. Current OpenAI Responses configurations offer implicit and explicit caching with 30-minute or 12-hour lifetimes. Gemini 4 uses implicit caching rather than manually created cache objects. DeepSeek documents automatic disk caching on a best-effort basis. A gateway cannot manufacture a duration or guarantee that the receiving provider does not offer. <a href="https://developers.openai.com/api/docs/guides/prompt-caching">OpenAI</a>, <a href="https://aistudio.google.com/docs/caching">Google</a>, <a href="https://api-docs.deepseek.com/guides/kv_cache/">DeepSeek</a></p><p>Routing complicates the arithmetic further. Switching to a cheaper model can lose the reusable prefix. Keeping a session on one route can preserve it, while retaining an expensive or unhealthy route too long can waste more than the cache saves.</p><p>I would replay real arrival times and route changes against actual invoices. A cache discount on a pricing page says little about whether an engineer returns before lunch ends, whether CI finishes, or whether the context is still correct.</p><h2>A smarter model cannot remember a decision it never saw</h2><p>Anthropic&#8217;s advice about removing instructions deserves a narrower reading than &#8220;skills are obsolete.&#8221;</p><p>Its Opus 5 prompting guidance says generic verification reminders and legacy verification steps can cause redundant work. Opus 5.5 guidance continues to recommend calibrating effort and supplying context for loosely specified work. Anthropic also continues to document enterprise skills and their evaluation. <a href="https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5">Opus 5 guidance</a>, <a href="https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5-5">Opus 5.5 guidance</a>, <a href="https://platform.claude.com/docs/en/agents-and-tools/agent-skills/enterprise">enterprise skills</a></p><p>A stronger model may no longer need a reminder to recheck something it reliably checks itself. It still cannot know why your company rejected a library last week, which customer depends on an undocumented behavior, or which migration requires a second approval unless that information is available.</p><p>The organization needs a way to retain those decisions. A skill&#8212;a reusable set of instructions, examples, or tools&#8212;is one place to put them.</p><p>It is not always the right place. Changing facts belong in current documentation and retrieval. A forbidden operation should be blocked by permissions. A deterministic check should run as code. A skill is more useful for recurring judgment that cannot be captured by a simple rule.</p><p>Skills also accumulate debt. Each loaded instruction consumes context. A large catalog makes selection harder. Examples become stale. Two well-intended instructions can conflict. A correction that helped one incident can become a permanent restriction nobody remembers choosing.</p><p>I would treat a skill as a maintained engineering artifact: an owner, representative tasks, a reason it exists, and a condition under which it should be removed. Test it against the current model without the skill and against simpler alternatives. Load detailed material when the task needs it.</p><p>A useful learning loop has to delete instructions as well as add them. Otherwise each improvement in the model leaves the company paying for workarounds it no longer needs.</p><h2>Fine-tuning gives you another product to maintain</h2><p>For a company with a large proprietary codebase, training a model on that code can sound like the obvious next step. The company has data the public model has never seen. Why keep explaining it?</p><p>Because knowing yesterday&#8217;s repository and following today&#8217;s workflow are different capabilities. A trained model does not automatically learn this morning&#8217;s schema change. Retrieval and tools remain necessary.</p><p>There can still be a case for customization: a stable task performed at high volume, a repeated behavior the best available model handles poorly, an expensive output format, or an offline constraint. The first question is whether a simpler change to context, tools, instructions, effort, or routing removes the gap.</p><p>If a gap remains, the work starts with a benchmark and an untouched base model. It also needs the strongest eligible untuned alternative. Beating the model you chose to train is not enough if a current API solves the task more cheaply.</p><p>A serious program may then involve several different kinds of training.</p><p>Continued pretraining teaches patterns through next-token prediction on domain material. It can be useful when there is a representation gap, but it is optional. Mixing in suitable general data may reduce forgetting; the correct ratio has to be tested. The original training corpus may be unavailable, and newly distilled examples are not automatically a faithful replacement.</p><p>Supervised fine-tuning, or SFT, teaches reviewed examples of desired behavior: the task, relevant context, tool actions, output, correction, and stopping point. A folder of merged patches does not contain all of that.</p><p>These stages need different data. Raw code, general replay material, approved demonstrations, and reward environments each require curation. Remove secrets and material the company cannot use. Keep related PRs and later fixes together when separating training from evaluation, so the test does not quietly contain another version of a training example. Apply the cutoff to the raw code too: a future solution can leak through a repository snapshot even if its PR is withheld. Preserve unsuccessful attempts and corrections where they teach something useful. More repository history can increase leakage and obsolete patterns as readily as it increases coverage.</p><p>Preference training needs credible preferred and rejected alternatives. Reinforcement learning needs an environment in which the model can attempt work and receive a reward. &#8220;RL on our PRs&#8221; means reconstructing tasks, generating new attempts, and building checks that cannot be satisfied by weakening tests or exploiting the grader. Merge status alone is too crude a reward.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!kRaQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e66e1c6-9122-46e9-973a-fdaae01edb5e_1700x2550.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!kRaQ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e66e1c6-9122-46e9-973a-fdaae01edb5e_1700x2550.png 424w, https://substackcdn.com/image/fetch/$s_!kRaQ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e66e1c6-9122-46e9-973a-fdaae01edb5e_1700x2550.png 848w, https://substackcdn.com/image/fetch/$s_!kRaQ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e66e1c6-9122-46e9-973a-fdaae01edb5e_1700x2550.png 1272w, https://substackcdn.com/image/fetch/$s_!kRaQ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e66e1c6-9122-46e9-973a-fdaae01edb5e_1700x2550.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!kRaQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e66e1c6-9122-46e9-973a-fdaae01edb5e_1700x2550.png" width="1456" height="2184" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1e66e1c6-9122-46e9-973a-fdaae01edb5e_1700x2550.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:2184,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:195942,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.equationblog.com/i/225083362?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e66e1c6-9122-46e9-973a-fdaae01edb5e_1700x2550.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!kRaQ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e66e1c6-9122-46e9-973a-fdaae01edb5e_1700x2550.png 424w, https://substackcdn.com/image/fetch/$s_!kRaQ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e66e1c6-9122-46e9-973a-fdaae01edb5e_1700x2550.png 848w, https://substackcdn.com/image/fetch/$s_!kRaQ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e66e1c6-9122-46e9-973a-fdaae01edb5e_1700x2550.png 1272w, https://substackcdn.com/image/fetch/$s_!kRaQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e66e1c6-9122-46e9-973a-fdaae01edb5e_1700x2550.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>Use development evaluations throughout. Reserve the final holdout until the method is fixed. The optional branches illustrate the work a proposal may create; they are not a mandatory recipe.</em></p><p>Every weight-changing stage brings comparison runs: data mixtures, learning rates, duration, checkpoints, and perhaps LoRA, which updates small adapters, versus full-weight training. The organization must check general capability, tool use, security, memorization, latency, and cost as well as the intended domain gain.</p><p>Vendor support covers different parts of this sequence.</p><ul><li><p><strong>NVIDIA Megatron Bridge and related NeMo tools.</strong> Components for continued pretraining and supervised adaptation on supported open models; the company still integrates and operates the program</p></li><li><p><strong>Together AI.</strong> Managed adapter or full fine-tuning on supported open models, with model and serving constraints</p></li><li><p><strong>Microsoft Foundry Programmed Training.</strong> A preview interface for custom post-training loops using LoRA; access approval and production limitations apply</p></li><li><p><strong>Google managed reinforcement tuning.</strong> Technically relevant reward workflows, but its current preview terms exclude proprietary or confidential data and commercial production</p></li></ul><p>These are support distinctions, not quality rankings. <a href="https://docs.nvidia.com/nemo/megatron-bridge/latest/training/entry-points.html">NVIDIA</a>, <a href="https://docs.together.ai/docs/fine-tuning/overview">Together</a>, <a href="https://learn.microsoft.com/en-us/azure/foundry/fine-tuning/programmed-training/overview">Microsoft</a>, <a href="https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/tuning/reinforcement-tuning/quick-start">Google</a></p><p>A vendor offering fine-tuning does not imply that its strongest current model is available for it. That gap belongs in the comparison from the beginning.</p><p>The direct frontier APIs illustrate the distinction. OpenAI says its fine-tuning platform is winding down and closed to new users. Anthropic says the Claude API does not currently offer fine-tuning and directs interested customers to their contact. Neither statement rules out a separately negotiated program, but neither supports assuming public access to the full pipeline. <a href="https://developers.openai.com/api/docs/guides/model-optimization">OpenAI</a>, <a href="https://platform.claude.com/docs/en/about-claude/glossary">Anthropic</a></p><p>The router design I led includes a proposed path for collecting training candidates. Such capture can help organize examples. It cannot supply missing local tool state, establish that the result was good, construct a reward environment, or prove that training pays.</p><p>The recurring bill includes data curation, experiments, serving, evaluation, refresh, and migration when the base model ages. A narrow tune may earn all of it. I would require that case before giving the company another model to maintain.</p><h2>Owning the GPUs means owning the idle hours</h2><p>There are several ways to buy inference: directly from a lab; through AWS, Microsoft, or Google; from a specialist managed provider; on dedicated rented capacity; or on hardware the company owns.</p><p>&#8220;Neocloud&#8221; does not tell you which product you are buying. A managed model endpoint and a rack of rented GPUs leave very different amounts of work with the customer.</p><p>The same model name can also hide different snapshots, numerical precision, context limits, tool support, cache behavior, and scheduling. Quantization uses less precise representations to reduce memory or computation. Whether it preserves the behavior your task needs has to be tested. A low price is not proof that a provider secretly quantizes, and an identical model label is not proof of an identical service.</p><p>Architecture affects the cost without deciding it. A smaller model that takes more reasoning steps or retries can lose to a larger model that finishes quickly. A sparse mixture-of-experts model activates only part of its parameters for each token, but the rest of the weights still have to be stored or made accessible. Moving information between experts, memory pressure, and uneven work can erase part of the arithmetic advantage.</p><p>DeepSeek is a useful reason to examine sparse serving, not a reason to infer that sparse models are inherently slower, cheaper, or worse. Data, training, tools, precision, hardware, and workload change the comparison.</p><p>At the provider level, GPUs and power are only part of the economics. Capital cost, depreciation, utilization, networking, memory, operations, research spending, and overhead matter too. Batching can spread cost across more requests while increasing waiting time. Price then adds competition, scarcity, contracts, subsidies, and margin. An API price is not a transparent statement of physical production cost.</p><p>Self-hosting makes those tradeoffs your responsibility. vLLM and SGLang offer serving machinery, but selecting a framework does not settle model compatibility, parallelism, cache capacity, concurrency, or recovery. Their current documentation exposes those as deployment choices. <a href="https://docs.vllm.ai/en/latest/serving/parallelism_scaling/">vLLM</a>, <a href="https://docs.sglang.io/docs/advanced_features/pd_disaggregation">SGLang</a></p><p>The model fitting in memory is only the first test. Active conversations need additional memory. Peak demand and failure recovery require spare capacity. Search, retrieval, tool execution, secrets, approvals, monitoring, and upgrades remain part of the product.</p><p>An illustrative break-even shows the sensitivity. Suppose a rented deployment costs $20,000 per month after staffing and reserve, with $1 of variable cost per accepted task. Suppose the comparable API task costs $3. At equal quality and service, the simple break-even is 10,000 accepted tasks a month. If API caching lowers the comparable task to $1.50, the threshold becomes 40,000.</p><p>Those are scenario inputs, not cloud quotes. The result also fails if the deployment cannot serve that volume within the required latency or needs another replica to do so.</p><p>Owned hardware adds purchase cost, financing, useful life, residual value, power, cooling, and replacement. A desktop already purchased for an engineer can have a low marginal cost for offline work, while a company-wide deployment has a very different full cost.</p><p>I would test on rented capacity before buying. A startup can rationally pay a provider margin to preserve runway and attention. A large company with steady eligible demand and existing operators may earn a saving from ownership. Either may choose local inference for privacy or offline availability even when it costs more. That is a valid purchase when the reason is stated honestly.</p><h2>A gateway helps when used carefully</h2><p>An engineer may spend the morning tidying Jira tickets, analyzing a routine usage report, and reviewing a risky change. Those jobs do not require the same model or the same amount of reasoning. A gateway can make the switch automatically, within rules the organization has tested, instead of asking every engineer to keep a model price list in their head.</p><p>Routine housekeeping may fit a cheaper route. A deep review may justify the strongest eligible model. Repeated calculations may belong in a script, with the model interpreting the result. The saving can be modest on each task and still worth collecting across recurring work. It earns its place when quality holds and the saving survives classification, retries, review, and any cache lost in the switch.</p><p>The router design I led combines that selection with capability floors and session continuity. It can raise the model requirement for demanding work while resisting a cheaper switch that would disrupt an ongoing task or waste a warm cache. The useful question is which work can move safely, and when. Sending every turn to the cheapest model would miss the point.</p><p>The same records can show where working habits deserve attention. Is a premium model doing routine data analysis that a script could handle? Is a conversation carrying a large amount of old history into a new task? Are large tool results repeatedly consuming context without helping the next decision? These are more useful questions than whether an engineer spent more tokens than a colleague.</p><p>Our design includes sampled, content-free signals for large or history-dominated context, low cache reuse, raw-data analysis without an observed deterministic tool, and strongest-tier use on a simple request shape. These are candidates for investigation. A long context may be necessary; sending full history is normal for some APIs; a tool may have run locally where the gateway cannot see it. Detailed workflow review requires a separately authorized capture path. I would use the findings to improve the harness and teach better practices, with the actual task in view before judging the engineer.</p><p>That visibility also makes the operating plan usable. The documented controls let team administrators allocate additional allowance within a team budget. An administrator can grant a user or team an expiring temporary budget for a migration, evaluation, or other one-off effort. A temporary team grant is a shared pool; the regular team budget funds individual allocations. Managers can delegate within a boundary, and a short project need not permanently raise everybody&#8217;s allowance.</p><p>Organizational workload tracking needs the same care. Usage by owner, team, model, and routing policy can explain where the bill is accumulating. Authorized task analysis can add a view of coding, research, data analysis, and operations. Proper project accounting also needs a trusted project identity and the team that owned the work when it occurred. The documented team reports use current membership, so they should not be mistaken for a historical project ledger.</p><p>The gateway can also keep provider credentials out of individual clients, restrict access to approved routes, and record which policy governed a request. These controls give procurement, security, and finance a common record to work from. Compliance still depends on the actual contracts, permissions, and data rules behind those routes.</p><p>One accounting distinction matters throughout: provider spend and the allowance shown to a user are different numbers. A credit for a failed request can restore the user&#8217;s allowance without erasing the company&#8217;s provider charge. A discounted internal profile can encourage an approved route without changing the upstream price. Finance should be able to reconcile both views, so an internal subsidy does not appear to be a technical saving.</p><p>A qualified fallback can also keep work moving when a provider fails. Recovery needs to know what has already happened: emitted tool arguments and an executed tool action are different states. The execution record must prevent duplicate work. Two provider names may also share a cloud, region, or identity service, so the alternative has to be tested against the failure it is meant to cover.</p><p>I would scale these controls to the organization: basic tagging and worker limits for a small team; shared routing, evaluation, and delegated budgets for a growing company; additional access and audit controls where the work requires them. Account for the gateway&#8217;s own latency, maintenance, and recovery. Used this way, it gives management practical ways to fund useful work and correct avoidable expense without renegotiating every request.</p><h2>Your exit may depend on somebody else&#8217;s permission</h2><p>Infrastructure decisions outlive model rankings. A capacity commitment, a fine-tune, or a provider-specific integration can still be on the books when the assumptions that justified it change.</p><p>The current competitive debate makes this concrete. OpenAI announced that its November 20, 2026 usage-policy update will require review and approval for services providing direct access to its models, including routers and gateways. That effective date is still ahead at the time of writing. The company&#8217;s stated rationale is consistent safeguards across services. For a business offering such access, it also creates a dependency that must be understood before promising customers provider choice. <a href="https://openai.com/index/broad-access-through-consistent-standards/">OpenAI&#8217;s announcement</a></p><p>That is a commercial consequence. It does not, by itself, prove an intent to exclude competitors.</p><p>The same distinction matters in regulation. The White House&#8217;s March legislative framework calls for a uniform national approach. California signed additional AI audit and assessment legislation in September. A proposed federal approach and enacted state laws have different legal status; neither automatically imposes the same obligations on a frontier developer and a company using an API. <a href="https://www.whitehouse.gov/releases/2026/03/president-donald-j-trump-unveils-national-ai-legislative-framework/">White House framework</a>, <a href="https://www.gov.ca.gov/2026/09/09/governor-newsom-signs-first-in-the-nation-ai-safeguards-to-protect-californians-calls-on-the-federal-government-to-do-its-part/">California announcement</a></p><p>A rule with large fixed compliance costs can favor incumbents even when its safety rationale is sincere. A uniform rule can also reduce the cost of complying with incompatible local requirements. Claims of regulatory capture need evidence about the text, who sought it, its costs, and the alternatives. Suspicion is a reason to investigate.</p><p>U.S.&#8211;China competition changes the available options too. BIS&#8217;s January revision moved certain advanced-chip export applications, including H200 and MI325X-class products, to conditional case-by-case review. That illustrates how hardware access can change through policy, rather than through a model&#8217;s technical merit. <a href="https://www.bis.gov/press-release/department-commerce-revises-license-review-policy-semiconductors-exported-china">BIS</a></p><p>Open weights can give a company more control over serving, modification, and retirement. The license, hardware, staff, and a model that meets the task still have to exist. An exit option is more credible after a migration rehearsal than after a slide showing a second provider.</p><p>NVIDIA&#8217;s November leadership transition is a fresh reason to examine these dependencies. The latest discussions raise scenarios about cadence, financing, and export restrictions. Those are scenarios from investors and political participants.</p><p>For a company outside the labs, the useful response is to preserve evaluation cases, task state, and a practical ability to switch. Accept provider-specific features when their value warrants the dependency. Record what would be required to leave.</p><h2>Before approving the next increase</h2><p>I would expect an AI budget proposal to explain which work the company intends to attempt, what currently prevents that work from paying off, and why additional inference is the best next purchase.</p><p>The answer may justify a sharp, temporary increase for a rewrite that removes an expensive constraint. It may justify verification capacity before more generation, or a premium for an eligible enterprise route. Repetitive housekeeping may need ordinary code. The spending profile should follow the opportunity and its risks.</p><p>What I would not accept is a larger budget justified only by growing usage&#8212;or a cut justified only by the size of the invoice. Both avoid deciding what the work is worth.</p><p>Management encouraged engineers to learn these tools. It now owes them a coherent decision about which work to continue, what quality to preserve, and what the business expects to gain. A smaller competitor may make a costly, well-chosen attempt while we debate an allowance. We may also spend freely and build something nobody needs. Choosing between those possibilities remains our job.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[Performance Reviews and the Cost of Avoiding Judgment]]></title><description><![CDATA[Performance-review season is approaching.]]></description><link>https://www.equationblog.com/p/performance-reviews-and-the-cost</link><guid isPermaLink="false">https://www.equationblog.com/p/performance-reviews-and-the-cost</guid><dc:creator><![CDATA[Ruslan Belkin]]></dc:creator><pubDate>Sat, 07 Nov 2026 15:15:30 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!gk94!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f635c7d-5779-475e-b96f-d8fc03c07441_2400x1460.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Performance-review season is approaching. Across tech companies, the annual or twice-yearly cycle will soon absorb weeks of attention from the people responsible for getting work done.</p><p>Employees will reconstruct their accomplishments. Managers will collect peer feedback, write narratives, defend ratings, and negotiate promotions. Leadership teams will spend hours discussing people whose work they have never directly observed. Then the manager will return to the employee and explain a decision shaped by some combination of performance, budget, salary bands, and the strength of the argument made in the room.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>We call this feedback.</p><p>For the person receiving it, the immediate questions are usually more concrete: What happened to my compensation? Am I being promoted? Does my manager still want me here?</p><p>If you lead a company, the period before the cycle begins is a useful time to ask what all this work actually buys. Once the calendar fills and the forms open, the process becomes surprisingly difficult to question.</p><p>I think we have allowed the performance review to become a substitute for several things managers should already be doing. The cost goes well beyond the hours spent writing it.</p><h2>What the review is trying to do</h2><p>There are at least four different jobs inside a conventional review: help someone improve, establish a record of their contribution, allocate compensation and promotions, and manage employment risk.</p><p>Each needs a different conversation.</p><p>Coaching requires enough trust for someone to admit uncertainty. Compensation discussions invite people to present the strongest possible case for themselves. Documentation requires factual accuracy. Promotion requires a judgment about sustained scope and readiness. Putting all of this into one document does not resolve the conflicts. It makes the employee responsible for navigating them.</p><p>Consider the request for a candid self-assessment. The company asks an employee to describe weaknesses in a document that may influence their compensation. A rational employee learns to disclose the weaknesses that sound like strengths, or the ones already visible to everyone.</p><p>We should not be surprised when the result is uninformative.</p><p>Useful feedback can be a gift, particularly from someone whose judgment you trust. A delayed verdict, delivered after the consequential decisions have already been made, is something else. Especially when the words have no clear connection to the outcome.</p><p>A manager who has withheld a concern for six months has already failed at the feedback part. Writing it carefully in December does not recover the lost time.</p><p>This is also why I do not believe in scheduled one-on-ones as the default. A concern waits for the meeting, or the meeting arrives with nothing worth discussing. Both people then feel obliged to fill the time.</p><p>I prefer frequent check-ins and conversations around common work. Obstacles become visible there, decisions can be examined, and other concerns surface naturally. A standing one-on-one can create an artificial separation between doing the work together and talking about it.</p><p>This requires more of the manager. You have to initiate conversations, stay accessible, and make time when something matters. Sensitive issues need privacy. Development needs deliberate attention. Neither should wait for a scheduled slot.</p><p>The compensation conversation has its own obligation to honesty. A strong performer near the top of a salary band may receive a smaller increase than someone further down the band. A promotion may be justified while the budget is constrained. The company may simply be unable to pay what it would like.</p><p>Those are real constraints. Describing them as a deficiency in the employee&#8217;s performance is where the damage begins.</p><h2>The cost is management attention</h2><p>The software invoice is easy to find. The time consumed across the organization is usually not.</p><p>Take an illustrative company with 500 employees. Suppose each review consumes five hours of manager research and writing, two hours of self-assessment, ninety minutes of peer input, a 45-minute conversation counted for both participants (ninety person-minutes), and two hours of calibration and administration allocated per employee.</p><p>That is twelve person-hours per employee. Six thousand hours per cycle. At an assumed fully loaded cost of $150 an hour, the labor cost is $900,000. A second comparable cycle would repeat it.</p><p>These are assumptions, not an industry benchmark. Replace them with your own. Include the 360 requests, the promotion packets, the revisions after calibration, and the conversations after the official conversation. Count meeting participants, not just meeting duration.</p><p>Then consider what is missing from the calculation: decisions delayed, difficult issues left untouched until the cycle, and relationships between managers strained by negotiating against one another for a fixed pool of recognition.</p><p>The question is whether those hours produce decisions good enough to justify what they displace.</p><p>Tools like Lattice hardly address that question. They organize the work, enforce the deadlines, and make completion visible. They can also make a questionable process more rigid: a field becomes mandatory because it exists; a reminder becomes an obligation; the dashboard becomes the definition of success.</p><p>As of November 2026, Lattice lists its Foundations package at $13 per seat per month. At that list price, 500 seats would cost $78,000 a year before any of the human work begins. The same pricing page highlights a customer&#8217;s increase in review participation from 62 percent to 100 percent. That establishes participation. It does not, by itself, establish better management. <a href="https://lattice.com/pricing">Lattice pricing</a></p><p>The most revealing example comes from Lattice itself. In 2023, it replaced its own traditional biannual review cycle after finding the process too burdensome and insufficiently useful for feedback. It continued using Lattice, but changed the management design. <a href="https://lattice.com/articles/how-lattice-built-its-continuous-performance-model-using-lattice">Lattice&#8217;s account</a></p><p>I think that is the right order of operations. Decide what management needs to accomplish, remove unnecessary work, and then choose tooling. Buying a more complete workflow before answering those questions can leave the company paying to preserve the problem.</p><h2>When the curve overrides the work</h2><p>Calibration has a legitimate purpose. Managers have different standards. Some avoid difficult conversations. Others consistently underestimate people whose work is less visible. Comparing evidence can expose both.</p><p>The problem starts when the room has to produce a predetermined distribution.</p><p>Suppose the template is 10 percent at the top, 70 percent in the middle, and 20 percent at the bottom. Other companies use different proportions. The exact percentages are less important than the assumption that every team should approximate them.</p><p>Teams are not random samples drawn from the company.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!gk94!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f635c7d-5779-475e-b96f-d8fc03c07441_2400x1460.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!gk94!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f635c7d-5779-475e-b96f-d8fc03c07441_2400x1460.png 424w, https://substackcdn.com/image/fetch/$s_!gk94!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f635c7d-5779-475e-b96f-d8fc03c07441_2400x1460.png 848w, https://substackcdn.com/image/fetch/$s_!gk94!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f635c7d-5779-475e-b96f-d8fc03c07441_2400x1460.png 1272w, https://substackcdn.com/image/fetch/$s_!gk94!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f635c7d-5779-475e-b96f-d8fc03c07441_2400x1460.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!gk94!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f635c7d-5779-475e-b96f-d8fc03c07441_2400x1460.png" width="1456" height="886" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5f635c7d-5779-475e-b96f-d8fc03c07441_2400x1460.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:886,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Illustrative comparison of two teams against a shared performance standard, both forced into identical ratings&quot;,&quot;title&quot;:&quot;Illustrative comparison of two teams against a shared performance standard, both forced into identical ratings&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Illustrative comparison of two teams against a shared performance standard, both forced into identical ratings" title="Illustrative comparison of two teams against a shared performance standard, both forced into identical ratings" srcset="https://substackcdn.com/image/fetch/$s_!gk94!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f635c7d-5779-475e-b96f-d8fc03c07441_2400x1460.png 424w, https://substackcdn.com/image/fetch/$s_!gk94!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f635c7d-5779-475e-b96f-d8fc03c07441_2400x1460.png 848w, https://substackcdn.com/image/fetch/$s_!gk94!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f635c7d-5779-475e-b96f-d8fc03c07441_2400x1460.png 1272w, https://substackcdn.com/image/fetch/$s_!gk94!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f635c7d-5779-475e-b96f-d8fc03c07441_2400x1460.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>The teams are different. The required distribution is identical.</em></p><p>People select managers. Managers select people. Some teams develop better judgment, stronger working relationships, and more effective ways of delivering. Others accumulate problems. These differences can persist and compound.</p><p>Even if the company as a whole followed a normal distribution, it would not follow that each team should reproduce it. Nor should we assume normality in the first place. Bradley and Aguinis examined 274 performance distributions covering 200,825 teams across several kinds of work. Only 11 percent were normal. Their research concerns performance across teams, rather than ratings within a particular team, but it directly challenges the assumption that exceptional teams should be rare anomalies. <a href="https://pubsonline.informs.org/doi/10.1287/orsc.2022.1619">Organization Science</a></p><p>I have seen the practical consequences several times in my career: stronger and weaker teams are leveled by the process until their ratings obscure the difference between them. The strong team has too many people above the expected distribution. The weaker team receives recognition that looks comparable on paper.</p><p>The curve is satisfied. The people who know the work are not.</p><p>Taleb makes a related point about averages: &#8220;The more nonlinearity in the response, the less informational the average.&#8221; His example is water: receiving an adequate daily supply on average is little comfort if it all arrives on one day and nothing arrives for the next nine. The variation changes the outcome. <a href="https://nassimtaleb.org/2016/09/cannot-generalize-knowledge-parts/">Taleb&#8217;s essay</a>.</p><p>The narrower error in calibration does not require complicated mathematics. A description of a population cannot tell you what each selected subgroup must look like. Using it that way erases precisely the differences management should understand.</p><p>There is a moral cost here as well as a statistical one. A manager may have to explain to someone who did excellent work why that work was insufficient, when the actual insufficiency was the number of available slots. The employee is left searching for a performance lesson in a budget or quota decision.</p><p>That can change the relationship permanently.</p><p>In a study of a multinational firm, Brittany Bond found that employees downgraded because of a recognition cutoff were at least 34 percent more likely to leave voluntarily. Larger bonuses and reassurance about their prospects did not eliminate the effect. Recognition mattered independently of the money. <a href="https://pubsonline.informs.org/doi/10.1287/mnsc.2023.01204">Management Science</a></p><p>There is an important distinction: I always stack-rank my own reports. I consider the exercise crucial. Comparing people forces me to examine my reasons, confront inconsistencies, and challenge preferences that might otherwise remain invisible. It is particularly valuable when unexpected layoffs put decisions under severe time pressure.</p><p>That does not make my ranking automatically unbiased. It makes my reasoning available for scrutiny.</p><p>I can rank a strong team without declaring that a fixed percentage of it is weak. Relative position does not establish whether someone meets an absolute performance bar. Nor does it settle every layoff decision, where the work the company will retain also matters.</p><p>Calibration should test the evidence and the standard. Once it requires the evidence to fit the allocation, it becomes horse trading with a personnel record attached.</p><h2>What scale actually requires</h2><p>A growing company needs more explicit management. Expectations must survive beyond the founder&#8217;s memory. Compensation must be explainable across teams. Underperformance must be addressed consistently. Employees need a way to correct factual errors.</p><p>The annual ceremony is one way of organizing some of that work. It is not the requirement itself.</p><p>California, despite its reputation, does not generally require private employers to conduct annual performance reviews or place an at-will employee on a formal performance improvement plan before termination. At-will employment is subject to important exceptions, including unlawful discrimination and retaliation, contractual obligations, and protected rights. Company policies and agreements matter too. <a href="https://www.dir.ca.gov/dlse/TerminationOfEmployment.pdf">California termination guidance</a></p><p>The useful discipline is concrete: communicate standards, apply them consistently, record relevant facts, and explain the actual reasons for decisions. That is also the emphasis of the EEOC&#8217;s guidance on evaluations. <a href="https://www.eeoc.gov/employers/small-business/5-im-conducting-performance-evaluations">EEOC guidance</a></p><p>An inaccurate positive review can be as troublesome as an unjustified negative one. If the record says someone performed well for years, a sudden assertion of longstanding poor performance deserves scrutiny. More paperwork does not necessarily create a better record.</p><p>This is a management argument, not legal advice. Counsel should help establish the requirements for the company&#8217;s circumstances. Leadership should then distinguish those requirements from the habits built around them.</p><p>The same distinction applies to PIPs. An improvement plan is useful when it identifies an observable gap, explains the required change, provides appropriate support, and gives the employee a real chance to succeed. If the decision to terminate has already been made, calling the intervening paperwork an improvement opportunity is dishonest.</p><p>&#8220;Fire fast&#8221; should mean confronting the problem early enough to make a clear decision. It should not mean surprising someone with a concern the manager never communicated.</p><p>Startups have an intrinsic advantage here. The people making decisions are often close enough to see the work. Scope can change, feedback can arrive, and a decision can follow without passing through several levels of interpretation.</p><p>As companies grow, review cycles arrive alongside hiring approvals, headcount controls, freezes, and pay bands. Reviews do not cause all those constraints. But together they can produce a company in which changing the team takes much longer than recognizing the need to change it.</p><p>Some of that friction buys fairness and financial discipline. Some buys only delay. Leaders need to know which is which.</p><p>Jack Dorsey&#8217;s reported 2023 memo at Block challenged the delay directly:</p><blockquote><p>&#8220;Performance should be continuously evaluated, and feedback should not be queued up for later.&#8221;</p></blockquote><p>He proposed ending annual reviews and PIPs, using milestones to prompt decisions, and simplifying promotions. The reported memo retained calibration and introduced visible performance ratings. It was a change in how management operated, not an end to evaluation. <a href="https://www.sfgate.com/tech/article/jack-dorsey-block-layoff-easier-18509249.php">Reporting on the memo</a>, <a href="https://www.levels.fyi/community/thread/jOWeeE/block-removes-pips-and-performance-reviews-in-favor-of-immediate-termination">reproduced memo</a></p><p>The principle is useful. A promotion should recognize a sustained change in responsibility when the evidence supports it, rather than wait for an annual negotiation.</p><p>But removing the calendar creates another obligation. People still need to know the bar, where they stand, and how a decision can be challenged. Otherwise continuous evaluation becomes a permanent threat. A smaller process can still be arbitrary.</p><h2>How many OKRs survived the quarter?</h2><p>How many times have your OKRs truly survived the quarter?</p><p>I mean remained the best use of the team&#8217;s time, with the same assumptions and priorities, after contact with customers, competitors, technical reality, and new information.</p><p>The fact that they remained in the tool does not answer the question.</p><p>Objectives and targets have uses. Boards and investors need to understand what management intends to accomplish. A team improving a stable process can benefit from a clear, measurable goal. The problem is the growing precision of commitments as they cascade further from the strategy.</p><p>A company objective becomes a departmental objective, then a team objective, then an individual score. Dependencies are shared, but accountability is divided. Revenue and retention arrive late and reflect many causes. Convenient activity metrics arrive sooner, but may say little about value.</p><p>A team can do the right thing and miss its key result. It can also hit the number while making the business worse.</p><p>Imagine a team committed to increasing activation. Midway through the quarter, it discovers that the fastest way to raise the metric also increases costly support cases and early churn. Abandoning the original approach may be the best decision it makes. A review system that penalizes the missed target teaches the team to defend the original plan next time.</p><p>Marty Cagan, once a vocal advocate of OKRs, wrote that he had stopped recommending them in the majority of companies he encountered. His criticism was specific: objectives imposed on teams without meaningful autonomy, competing functional goals, and leadership that failed to supply strategy. The framework could not compensate for the missing management. <a href="https://www.svpg.com/team-objectives-overview/">Silicon Valley Product Group</a></p><p>Deming&#8217;s formulation was more severe:</p><blockquote><p>&#8220;Eliminate management by objective. Eliminate management by numbers, numerical goals. Substitute leadership.&#8221; <a href="https://deming.org/explore/fourteen-points/">The Deming Institute</a></p></blockquote><p>I would keep useful measurements. I would stop treating a quarterly hypothesis as a contract with reality.</p><p>The military offers a better organizing idea: commander&#8217;s intent. The leader communicates the purpose, priorities, desired end state, and boundaries, then gives people authority to adapt their actions as circumstances change. The Army&#8217;s account emphasizes trust, judgment, and accepting risk so others can act without constant intervention. <a href="https://www.army.mil/article/215297/mission_command_requires_sharp_commanders_intent">U.S. Army</a></p><p>For a company, that means explaining what must improve for the customer, which constraints cannot be violated, what resources are available, and what the team can change without asking permission. Measures help test whether the approach is working. Review triggers make it clear when assumptions need to be reconsidered.</p><p>Targets still matter. Some are aspirations; some are forecasts; some are actual commitments to customers or other teams. Leaders should say which. A contractual deadline cannot be quietly reclassified as aspirational because it became inconvenient.</p><p>A team that changes direction should explain the evidence and consequences. But changing direction in service of the objective can be evidence of leadership. We should be able to recognize that without negotiating an exception to a scoring system.</p><h2>Leadership, leverage, and results</h2><p>At LinkedIn, I learned a simpler way of evaluating people from Jeff Weiner: leadership, leverage, and results. I have used it since.</p><p>The difference from forced calibration is fundamental. Leadership, leverage, and results ask what the person contributed. A quota asks how many places are available.</p><p>The framework is simple enough to remember, but each word requires real judgment.</p><p><strong>Leadership</strong> starts with the word lead. How much did the person anticipate, and how much did they merely react? Did they identify a problem before it became an escalation? Create clarity where ownership was ambiguous? Help others act with better judgment?</p><p>This applies to individual contributors as much as managers. Title does not establish leadership, and visibility can be a poor substitute for it.</p><p><strong>Leverage</strong> asks what was created per unit of effort, and what other people can now accomplish because of it. A reusable system, a better hire, a simpler architecture, a decision that prevents months of wasted work, or someone who teaches others to solve a recurring problem can all create it.</p><p>Repeated heroics deserve appreciation. They also deserve a question: why does the organization still need the same rescue?</p><p><strong>Results</strong> asks what actually changed. What shipped, what improved for customers, what became more reliable, what revenue or cost consequence followed? Did the improvement last?</p><p>Results combine judgment, process, hard work, collaboration, and luck. Hard work matters enormously; it is one of the inputs a person can control. But hours alone cannot tell us whether the effort was directed well. Nor should a lucky outcome erase a reckless decision.</p><p>These dimensions should remain visible separately. I would not average them into a decimal. Someone who produces strong short-term results while damaging everyone around them has created a problem that arithmetic should not conceal.</p><p>The discipline is to attach concrete examples to each judgment, appropriate to the person&#8217;s role and opportunity. It should be possible for the employee to understand the assessment and dispute a factual error.</p><p>This still requires a manager to decide. That is a feature.</p><h2>What AI should do</h2><p>AI makes the current system easier to perpetuate.</p><p>An employee can generate a polished self-review. Peers can generate feedback. A manager can synthesize both into a persuasive narrative. Another model can summarize it for calibration.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!vVps!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4dfd31b6-b511-477a-a74a-e220bf5ae56b_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!vVps!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4dfd31b6-b511-477a-a74a-e220bf5ae56b_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!vVps!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4dfd31b6-b511-477a-a74a-e220bf5ae56b_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!vVps!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4dfd31b6-b511-477a-a74a-e220bf5ae56b_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!vVps!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4dfd31b6-b511-477a-a74a-e220bf5ae56b_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!vVps!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4dfd31b6-b511-477a-a74a-e220bf5ae56b_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4dfd31b6-b511-477a-a74a-e220bf5ae56b_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;An employee and manager concentrate on AI-generated reviews on their laptops rather than speaking to each other&quot;,&quot;title&quot;:&quot;An employee and manager concentrate on AI-generated reviews on their laptops rather than speaking to each other&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="An employee and manager concentrate on AI-generated reviews on their laptops rather than speaking to each other" title="An employee and manager concentrate on AI-generated reviews on their laptops rather than speaking to each other" srcset="https://substackcdn.com/image/fetch/$s_!vVps!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4dfd31b6-b511-477a-a74a-e220bf5ae56b_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!vVps!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4dfd31b6-b511-477a-a74a-e220bf5ae56b_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!vVps!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4dfd31b6-b511-477a-a74a-e220bf5ae56b_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!vVps!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4dfd31b6-b511-477a-a74a-e220bf5ae56b_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>The prose improves. It is entirely possible that nobody learns anything.</em></p><p>The more useful application is reducing the cost of finding and testing evidence. An approved system can assemble dated examples, link them to their sources, preserve disagreements in peer feedback, and identify claims that have little support. It can help a manager notice that the assessment contains three recent incidents and nothing from the preceding five months.</p><p>It should also be allowed to return an uncomfortable answer: there is not enough evidence here.</p><p>That matters because the record is incomplete by construction. A person who prevents an incident may leave less visible evidence than someone who resolves one. Mentoring can be poorly documented. Quiet coordination may look less impressive than public escalation. Message volume, commits, and meeting attendance are convenient observations with weak claims to being performance measures.</p><p>I would use AI to help interrogate the record. I would not delegate employee ranking or the final compensation, promotion, or termination judgment to it. The manager must inspect the underlying evidence and own the conversation.</p><p>California&#8217;s employment regulations on automated decision systems, effective October 1, 2025, reinforce that accountability. They address discriminatory outcomes from automated systems and require preservation of relevant employment records, including automated-decision data. A human approval step does not make the underlying system irrelevant. <a href="https://calcivilrights.ca.gov/2025/06/30/civil-rights-council-secures-approval-for-regulations-to-protect-against-employment-discrimination-related-to-artificial-intelligence/">California Civil Rights Department</a></p><p>AI could help us remove substantial parts of the review form. If evidence is easier to retrieve and challenge throughout the year, the case for periodically reconstructing it in prose becomes weaker.</p><p>That should be the test for the investment: how much unnecessary work disappears, and whether the remaining decisions improve.</p><h2>Before the next cycle</h2><p>The operating model I would use is fairly small.</p><p>Give feedback while it can still change the work. Address material underperformance when it becomes visible.</p><p>At planning intervals, reset intent and assumptions with the team. Make decision authority explicit. Do not turn each planning discussion into another rating event.</p><p>Once or twice a year, consolidate the important evidence into a short account of leadership, leverage, and results. One page should usually be enough. Ask peers specific questions about work they observed. Let employees correct factual errors.</p><p>Keep compensation allocation explicit. Performance is an input alongside scope, market conditions, internal equity, position in range, and the company&#8217;s budget. Explain the constraint that actually affected the decision. Recognize sustained changes in role when they occur, with consistent promotion standards.</p><p>Use calibration to challenge evidence and reconcile standards. Require a reason for changing a manager&#8217;s assessment. Examine patterns across managers. Do not require every team to resemble the same distribution.</p><p>This can deteriorate too. A one-page memo can become a ten-page packet. Targeted peer input can become selective evidence. Asynchronous promotions can reward access to executives. Continuous feedback can become continuous interruption. The design needs limits and periodic scrutiny, including scrutiny of the managers making the judgments.</p><p>What should a manager say when the system produces an outcome they disagree with?</p><p>Confidentiality is real. You cannot disclose other people&#8217;s compensation or private discussions. But confidentiality does not require pretending a budget decision was a performance conclusion. Explain the facts, your judgment where appropriate, the constraints, and the decision. Do not promise what you cannot deliver. Do not retreat behind &#8220;HR decided&#8221; as if managing the person stopped at the calibration-room door.</p><p>The Valley is a long game. People remember the manager who told them a difficult truth early enough to do something about it. They also remember the manager who praised their work all year and arrived in December with an explanation neither of them believed.</p><p>Trust is what leadership spends when it treats the next review cycle as inevitable.</p><p>Before opening the forms, look at the decisions the company needs to make. Keep the evidence, standards, and conversations that improve those decisions. Remove the work that merely makes them look more objective.</p><p>The employee deserves an honest assessment.</p><p>The manager owes a judgment they can explain.</p><p>No process should relieve leadership of either obligation.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[AI-Native Development Is a Workflow, Not a Coding Agent]]></title><description><![CDATA[What changes when agents become first-class contributors to the software lifecycle]]></description><link>https://www.equationblog.com/p/ai-native-development-is-a-workflow</link><guid isPermaLink="false">https://www.equationblog.com/p/ai-native-development-is-a-workflow</guid><dc:creator><![CDATA[Ruslan Belkin]]></dc:creator><pubDate>Sun, 23 Aug 2026 14:15:39 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!A8Fu!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2bea2cef-5993-4e9c-956e-3bbec9cee65c_2400x1350.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>An AI coding agent is not an engineering system.</p><p>It is an input to one.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>Most teams start by inserting an agent into the development process they already have. The agent gets a ticket, writes a patch, perhaps adds tests, and opens a pull request. Then the old machinery takes over: a human reconstructs intent, fixes the branch, chases CI, explains repository conventions, resolves conflicts, answers review comments, and waits for the merge queue.</p><p>The model may have written the code faster.</p><p>The organization may not have shipped anything faster.</p><p>A faster generator attached to a slow, implicit, human-dependent workflow mostly produces a larger queue.</p><p>The more interesting approach is to redesign the workflow so humans and agents run on the same rails: the same source of truth, operational playbooks, branch and review rules, release gates, evidence, and institutional memory. The agent can take the wheel. A human can take it back at any point. Neither should need a translation layer.</p><p>The operating model I trust is:</p><blockquote><p>the model owns heuristics<br>the workflow owns invariants<br>the human owns judgment<br>the repository owns memory</p></blockquote><p>That is what I mean by agentic development.</p><p>Not autocomplete.</p><p>Not a chat window next to an IDE.</p><p>A software delivery system designed for agents as first-class contributors.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!A8Fu!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2bea2cef-5993-4e9c-956e-3bbec9cee65c_2400x1350.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!A8Fu!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2bea2cef-5993-4e9c-956e-3bbec9cee65c_2400x1350.png 424w, https://substackcdn.com/image/fetch/$s_!A8Fu!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2bea2cef-5993-4e9c-956e-3bbec9cee65c_2400x1350.png 848w, https://substackcdn.com/image/fetch/$s_!A8Fu!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2bea2cef-5993-4e9c-956e-3bbec9cee65c_2400x1350.png 1272w, https://substackcdn.com/image/fetch/$s_!A8Fu!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2bea2cef-5993-4e9c-956e-3bbec9cee65c_2400x1350.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!A8Fu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2bea2cef-5993-4e9c-956e-3bbec9cee65c_2400x1350.png" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2bea2cef-5993-4e9c-956e-3bbec9cee65c_2400x1350.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Diagram contrasting a narrow coding-agent workflow with an end-to-end AI-native development system.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Diagram contrasting a narrow coding-agent workflow with an end-to-end AI-native development system." title="Diagram contrasting a narrow coding-agent workflow with an end-to-end AI-native development system." srcset="https://substackcdn.com/image/fetch/$s_!A8Fu!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2bea2cef-5993-4e9c-956e-3bbec9cee65c_2400x1350.png 424w, https://substackcdn.com/image/fetch/$s_!A8Fu!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2bea2cef-5993-4e9c-956e-3bbec9cee65c_2400x1350.png 848w, https://substackcdn.com/image/fetch/$s_!A8Fu!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2bea2cef-5993-4e9c-956e-3bbec9cee65c_2400x1350.png 1272w, https://substackcdn.com/image/fetch/$s_!A8Fu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2bea2cef-5993-4e9c-956e-3bbec9cee65c_2400x1350.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>Generation is one step. Delivery is the system.</em></p><div><hr></div><h2>The unit of productivity is the landed change</h2><p>Code generation is only one step in a much longer path:</p><blockquote><p>intent &#8594; isolated workspace &#8594; implementation &#8594; testing &#8594; review &#8594; CI &#8594; merge queue &#8594; release proof &#8594; archive &#8594; learning</p></blockquote><p>Each transition has state. Each state has evidence. Each transition can fail for a different reason.</p><p>This matters because the bottleneck moves as generation gets faster. When code is expensive, we optimize code creation. When code becomes cheap, review, integration, testing, release, and human attention become the scarce resources.</p><p>A workflow that needs a human to re-prompt the agent after every external wait is not autonomous. It is interrupt-driven delegation.</p><p>A workflow that lets an agent produce ten pull requests while senior engineers manually reconstruct what happened is not necessarily productive. It may simply be converting model tokens into review inventory.</p><p>The development system has to carry the change, not merely generate it.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!M5QK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a2653ca-5847-448e-a58e-cca5078c68dd_2600x1365.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!M5QK!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a2653ca-5847-448e-a58e-cca5078c68dd_2600x1365.png 424w, https://substackcdn.com/image/fetch/$s_!M5QK!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a2653ca-5847-448e-a58e-cca5078c68dd_2600x1365.png 848w, https://substackcdn.com/image/fetch/$s_!M5QK!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a2653ca-5847-448e-a58e-cca5078c68dd_2600x1365.png 1272w, https://substackcdn.com/image/fetch/$s_!M5QK!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a2653ca-5847-448e-a58e-cca5078c68dd_2600x1365.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!M5QK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a2653ca-5847-448e-a58e-cca5078c68dd_2600x1365.png" width="1456" height="764" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2a2653ca-5847-448e-a58e-cca5078c68dd_2600x1365.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:764,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Nine-stage lifecycle from intent through learning, with evidence at every transition and human judgment at selected gates.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Nine-stage lifecycle from intent through learning, with evidence at every transition and human judgment at selected gates." title="Nine-stage lifecycle from intent through learning, with evidence at every transition and human judgment at selected gates." srcset="https://substackcdn.com/image/fetch/$s_!M5QK!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a2653ca-5847-448e-a58e-cca5078c68dd_2600x1365.png 424w, https://substackcdn.com/image/fetch/$s_!M5QK!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a2653ca-5847-448e-a58e-cca5078c68dd_2600x1365.png 848w, https://substackcdn.com/image/fetch/$s_!M5QK!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a2653ca-5847-448e-a58e-cca5078c68dd_2600x1365.png 1272w, https://substackcdn.com/image/fetch/$s_!M5QK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a2653ca-5847-448e-a58e-cca5078c68dd_2600x1365.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>The relevant unit of output is a change that lands with evidence, release proof, and retained learning.</em></p><div><hr></div><h2>1. Skills, not prompts</h2><p>A prompt is a conversation.</p><p>A skill is an operating asset.</p><p>Prompts disappear into chat history. Skills live with the code. They are versioned, reviewed, discoverable, and available to the next engineer or agent without asking anyone to remember what worked last time.</p><p>A useful skill does one thing well: create an isolated workspace, deploy to a development environment, resolve a merge conflict without erasing someone else&#8217;s work, write a test that will not flake, review a change against known failure modes, or carry a pull request through the merge queue.</p><p>The important part is discovery. The agent should not have to remember that a skill exists. Intent should route to the skill automatically:</p><ul><li><p>&#8220;deploy this change&#8221; loads the deployment procedure</p></li><li><p>&#8220;review this pull request&#8221; loads the review discipline</p></li><li><p>&#8220;add a browser test&#8221; loads the test-design rules</p></li><li><p>&#8220;resolve the conflict&#8221; loads the conflict-resolution playbook</p></li></ul><p>The principle is simple:</p><blockquote><p>do not let the agent improvise an operational sequence that the organization already knows</p></blockquote><p>There is also a useful split inside every skill.</p><p>Put deterministic mechanics in deterministic code: branch validation, workspace creation, naming, parsing, status checks, and machine-readable output.</p><p>Leave only the genuinely non-deterministic parts to the model: interpreting intent, choosing among safe options, explaining a tradeoff, or deciding when to escalate.</p><p>This reduces token use, but the bigger benefit is behavioral consistency. A human and an agent invoke the same operation and get the same rails.</p><p>Skills can also encode architecture, not just procedure. Reaching for a one-off UI element can route the contributor toward the shared component. Adding a new capability can load the feature-flag convention. Touching state management can surface the rules for where that state belongs.</p><p>That is how consistency scales. You do not wait for review to catch every local invention. You make the shared primitive the default at the moment intent is detected.</p><p>One more practical detail matters: author the skill once.</p><p>Different agent tools consume instructions differently, but the institutional knowledge should have one canonical source. Generate the thin adapters each tool needs. Do not maintain three drifting copies of the same operating rule.</p><p>And keep the skill itself lean. A large instruction body is paid for repeatedly as context. The core file should route and disclose. Heavy detail should sit in references that are loaded only when the task actually needs them.</p><p>Agent context is a budget.</p><p>Operational knowledge should be durable, but it should not be noisy.</p><div><hr></div><h2>2. Planning should scale to the work</h2><p>Agentic development needs written intent.</p><p>It does not need a design document for every typo.</p><p>The useful pattern is a ladder:</p><ol><li><p><strong>A one-line cleanup item</strong> for work too small to deserve a ticket.</p></li><li><p><strong>A plan</strong> for one change that carries durable context, sequencing, or a multi-session handoff.</p></li><li><p><strong>A design</strong> for broader implementation strategy across several changes.</p></li><li><p><strong>A specification</strong> for the problem, users, and intended outcome of a substantial capability.</p></li></ol><p>The default at every rung should be the lightest artifact that can safely carry the work&#8212;including no artifact at all.</p><p>This is not just documentation hygiene. The artifacts are the agent&#8217;s durable memory.</p><p>A fresh session should be able to pick up a ticket, find the relevant intent, understand what decisions have already been made, and continue without reconstructing the world from chat history. A different model should be able to do the same.</p><p>That is why planning artifacts have to be the source of truth rather than a ceremonial layer beside the &#8220;real&#8221; work.</p><p>And when the work lands, the artifact should not disappear.</p><p>Archive it with a short retrospective:</p><ul><li><p>what shipped</p></li><li><p>what was cut or deferred</p></li><li><p>where implementation diverged from the plan</p></li><li><p>which pull request landed it</p></li><li><p>what a future maintainer should know</p></li></ul><p>A forward-looking plan without the outcome is an aspiration.</p><p>A plan plus a shipped record is institutional memory.</p><p>The same principle applies to the tracker. Ticket state should follow the real state of the pull request through automation, not depend on a person remembering to drag a card. The repository should tell the truth even when nobody updates a dashboard manually.</p><div><hr></div><h2>3. The workflow is the guardrail</h2><p>Conventions that live only in a document are suggestions.</p><p>Conventions enforced by machinery become the way work happens.</p><p>Every independent task should get its own isolated workspace and branch. That gives parallel agents the same isolation parallel engineers need. One task, one branch, one traceable unit of work.</p><p>The branch should identify the work. The tracker should identify the intent. The pull request should carry the evidence. The merge should close the loop.</p><p>Destructive shortcuts should be unavailable by default. Do not let an agent casually rewrite shared history, force-push over another contributor, or bypass the merge path because it is &#8220;almost done.&#8221; The right workflow should be easier than the dangerous one.</p><p>CI should follow the same pattern.</p><p>Large systems often need more than one build and test system. That is fine. What matters is presenting one clear contract to the merge machinery: a single required gate computed from the underlying evidence.</p><p>The merge queue then has to test against reality.</p><p>A green pull request is green against a snapshot.</p><p>A green merge candidate is green against the mainline plus the changes ahead of it plus itself.</p><p>That distinction catches the failures individual pull requests cannot see: two correct changes that become wrong when combined.</p><p>The queue should also be honest about blame. A red build can come from the change, a known flaky test, or infrastructure. Treating all three as the author&#8217;s fault wastes time and trains people to ignore CI.</p><p>But &#8220;probably not your fault&#8221; is not enough. Retaining a change in the queue should require positive evidence. When the system cannot establish innocence, it should fail toward ownership rather than wave the change through.</p><p>That asymmetry matters.</p><p>Automation should be generous with diagnosis and conservative with authority.</p><div><hr></div><h2>4. Review against failure modes, not vibes</h2><p>A change is not correct because it is internally coherent.</p><p>A mock can agree with the implementation and still model the wrong system. A test can inspect source text and still prove nothing about runtime behavior. A design can match the code and still miss the user&#8217;s intent.</p><p>Self-consistent is not the same as correct.</p><p>Review should therefore start with intent discovery.</p><p>The reviewer walks from the richest source to the poorest:</p><blockquote><p>acceptance criteria &#8594; change description &#8594; title and branch &#8594; commits &#8594; diff alone</p></blockquote><p>It should state what it was able to establish and never invent conformance where the intent was unavailable.</p><p>With intent in hand, review becomes much more useful. It can find not only bugs, but also unmet acceptance criteria and undeclared scope growth.</p><p>The next improvement is to review against concrete failure modes rather than generic taste. Each domain should have a catalog of ways it has actually broken: cleanup that is narrower than the read path, timers that outlive a test, readiness checks that pass before the service is usable, unordered collections asserted in a fixed order, or release code that reads stale state.</p><p>Each checklist entry should come from an observed incident.</p><p>Speculative rules create bureaucracy. Remembered failures create leverage.</p><p>For larger changes, review should converge in passes rather than run once and emit forever:</p><ul><li><p>re-read the original change, not the previous review prose</p></li><li><p>subtract, merge, or demote weak findings before adding new ones</p></li><li><p>replace hedged language with evidence, mark it unverified, or remove it</p></li><li><p>raise the severity bar on later passes</p></li><li><p>stop when new material findings disappear</p></li><li><p>escalate when repeated findings point to a design problem rather than an implementation problem</p></li></ul><p>The result should become sharper and smaller over time.</p><p>Review depth should also scale to risk. A documentation fix does not need a seven-pass investigation. A change touching identity, persistence, release infrastructure, shared types, or test machinery may deserve specialist review in parallel.</p><p>Those specialist reviewers should be read-only. The system that judges the change should not quietly edit the thing it is judging.</p><p>Automated findings also need a disposition path. A serious finding is not cleared because the author typed &#8220;looks good.&#8221; It is cleared because the issue was fixed or because the disagreement contains a reason the system can evaluate.</p><p>And if automated approval is introduced, the policy should be deliberately boring:</p><blockquote><p>veto first &#8594; grant a narrow class &#8594; default deny</p></blockquote><p>High-risk paths go to a human. Eligible changes must pass deterministic corroboration. Missing signals default to false. The policy is read from trusted code outside the change under review, so a pull request cannot rewrite the logic that approves it.</p><p>Run the policy in shadow mode first.</p><p>Trust-sensitive automation should earn authority by producing evidence before it is allowed to exercise authority.</p><div><hr></div><h2>5. Autonomy lives across the waits</h2><p>Writing a patch is the easy part of autonomy.</p><p>The difficult part is carrying the change across everything that does not happen immediately:</p><ul><li><p>waiting for CI</p></li><li><p>waiting for review</p></li><li><p>responding to findings</p></li><li><p>resynchronizing when the mainline moves</p></li><li><p>resolving conflicts</p></li><li><p>entering the merge queue</p></li><li><p>surviving an eviction</p></li><li><p>verifying that the merge actually landed</p></li></ul><p>A useful autonomous loop should not hold the world in model memory. Every time it wakes, it should read the live state again.</p><p>The shape is:</p><blockquote><p>observe &#8594; decide &#8594; act &#8594; wait or escalate &#8594; wake &#8594; observe again</p></blockquote><p>Observation should be read-only. Decision should be side-effect-free. Actions should be explicit and ordered.</p><p>That makes the loop restartable. A crash does not corrupt hidden state. A model switch does not lose the workflow. A stale head commit causes re-derivation rather than an action against yesterday&#8217;s world.</p><p>Waiting is not failure.</p><p>A clean pull request that only needs an approval should park silently and wake when something changes. An agent that keeps asking &#8220;should I continue?&#8221; is transferring coordination cost back to the human.</p><p>Escalation should also be narrow.</p><p>One ambiguous security question should not stop every other safe action on the pull request. Park the uncertain item, label it for a human, and keep working on the rest.</p><p>The load-bearing rule is:</p><blockquote><p>escalate, do not guess</p></blockquote><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!JQ1y!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7935405d-e0ea-413b-9cca-ee2f5e2bf6a9_2400x1553.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!JQ1y!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7935405d-e0ea-413b-9cca-ee2f5e2bf6a9_2400x1553.png 424w, https://substackcdn.com/image/fetch/$s_!JQ1y!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7935405d-e0ea-413b-9cca-ee2f5e2bf6a9_2400x1553.png 848w, https://substackcdn.com/image/fetch/$s_!JQ1y!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7935405d-e0ea-413b-9cca-ee2f5e2bf6a9_2400x1553.png 1272w, https://substackcdn.com/image/fetch/$s_!JQ1y!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7935405d-e0ea-413b-9cca-ee2f5e2bf6a9_2400x1553.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!JQ1y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7935405d-e0ea-413b-9cca-ee2f5e2bf6a9_2400x1553.png" width="1456" height="942" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7935405d-e0ea-413b-9cca-ee2f5e2bf6a9_2400x1553.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:942,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Restartable observe-decide-act loop showing waiting, bounded action, narrow escalation, and terminal outcomes.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Restartable observe-decide-act loop showing waiting, bounded action, narrow escalation, and terminal outcomes." title="Restartable observe-decide-act loop showing waiting, bounded action, narrow escalation, and terminal outcomes." srcset="https://substackcdn.com/image/fetch/$s_!JQ1y!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7935405d-e0ea-413b-9cca-ee2f5e2bf6a9_2400x1553.png 424w, https://substackcdn.com/image/fetch/$s_!JQ1y!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7935405d-e0ea-413b-9cca-ee2f5e2bf6a9_2400x1553.png 848w, https://substackcdn.com/image/fetch/$s_!JQ1y!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7935405d-e0ea-413b-9cca-ee2f5e2bf6a9_2400x1553.png 1272w, https://substackcdn.com/image/fetch/$s_!JQ1y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7935405d-e0ea-413b-9cca-ee2f5e2bf6a9_2400x1553.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>A useful autonomous loop re-reads live state on every wake and escalates only the uncertain item.</em></p><p>Low confidence, destructive operations, security-sensitive changes, ambiguous intent, architectural drift, and actively disputed human feedback belong with a person. Routine synchronization, evidence gathering, bounded retries, and mechanical review cleanup belong with the agent.</p><p>The human should not be in the gears.</p><p>The human should be at the gates.</p><div><hr></div><h2>6. Test where the code runs</h2><p>A test is only meaningful in the runtime it is meant to protect.</p><p>Modern products may span server logic, browser components, containers, deployment templates, end-to-end flows, and agent behavior. Some tests run locally. Some require a live environment. Some exist only in shared CI.</p><p>The test-authoring system should route the contributor to the correct layer before work starts. Knowing that a change requires a slow remote cycle is part of planning, not a surprise after the code is written.</p><p>New tests should also clear a determinism gate.</p><p>The first tier is cheap: screen the test against known flake smells for its layer.</p><p>The second tier is selective: when a smell is present, repeat the test on the cheapest appropriate runner.</p><p>But be honest about what repetition proves.</p><p>Repeated green runs can catch high-frequency nondeterminism inside one test. They do not prove the absence of inter-test failures caused by leaked state, order dependence, shared databases, or sharding. Those appear only in the real suite.</p><p>This is a recurring theme in agentic development: use automation aggressively, but describe its evidence precisely.</p><p>When the product itself contains an agent, conventional assertions are not enough. &#8220;Did the agent behave well?&#8221; is often a rubric question, not a string-equality question.</p><p>Evaluation cases should therefore be first-class test assets:</p><ul><li><p>a prompt or scenario</p></li><li><p>a rubric describing the desired qualities</p></li><li><p>a captured transcript</p></li><li><p>a score and explanation</p></li><li><p>a versioned place in the regression suite</p></li></ul><p>Production traces can become new evaluation cases. A surprising interaction is reviewed, converted into a fixture, and replayed on future versions.</p><p>That is how a production failure stops being an anecdote and becomes a permanent test.</p><p>The caveat matters: rubric-graded evaluations are noisier than deterministic assertions. Some should trend quality rather than block every merge. Judge the distribution and the direction, not one stochastic run.</p><p>Performance tests need the same discipline. Gate on a distribution, not one lucky or unlucky sample.</p><div><hr></div><h2>7. Build once, prove it separately, promote by identity</h2><p>The artifact that passed should be the artifact that ships.</p><p>Do not rebuild it for release. Do not rename it into trust. Freeze it, identify it by its inputs, and promote that exact object through independent quality gates.</p><p>This separates three ideas that are too often conflated:</p><ol><li><p><strong>The change merged.</strong></p></li><li><p><strong>The artifact is releasable.</strong></p></li><li><p><strong>The artifact is deployed.</strong></p></li></ol><p>They are not the same event.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!mtuk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e78f454-f2ef-45bb-a28d-6e8384eec393_2400x1341.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!mtuk!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e78f454-f2ef-45bb-a28d-6e8384eec393_2400x1341.png 424w, https://substackcdn.com/image/fetch/$s_!mtuk!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e78f454-f2ef-45bb-a28d-6e8384eec393_2400x1341.png 848w, https://substackcdn.com/image/fetch/$s_!mtuk!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e78f454-f2ef-45bb-a28d-6e8384eec393_2400x1341.png 1272w, https://substackcdn.com/image/fetch/$s_!mtuk!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e78f454-f2ef-45bb-a28d-6e8384eec393_2400x1341.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!mtuk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e78f454-f2ef-45bb-a28d-6e8384eec393_2400x1341.png" width="1456" height="814" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2e78f454-f2ef-45bb-a28d-6e8384eec393_2400x1341.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:814,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Immutable artifact passes independent quality gates and is either promoted unchanged or held with an audit record.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Immutable artifact passes independent quality gates and is either promoted unchanged or held with an audit record." title="Immutable artifact passes independent quality gates and is either promoted unchanged or held with an audit record." srcset="https://substackcdn.com/image/fetch/$s_!mtuk!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e78f454-f2ef-45bb-a28d-6e8384eec393_2400x1341.png 424w, https://substackcdn.com/image/fetch/$s_!mtuk!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e78f454-f2ef-45bb-a28d-6e8384eec393_2400x1341.png 848w, https://substackcdn.com/image/fetch/$s_!mtuk!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e78f454-f2ef-45bb-a28d-6e8384eec393_2400x1341.png 1272w, https://substackcdn.com/image/fetch/$s_!mtuk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e78f454-f2ef-45bb-a28d-6e8384eec393_2400x1341.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>Build once, prove separately, and promote the same immutable artifact.</em></p><p>A release pipeline should run its own proof: performance, security, coverage, code quality, software approval, and whatever else the product requires. Those gates&#8212;not a vague belief that the mainline looked healthy&#8212;are the release bar.</p><p>Even a blocked release should leave an audit record of what would have shipped. The gate should block the release, not erase the evidence.</p><p>Security follows the same pattern.</p><p>Start with a hardened, consistently pinned base so the scanners have less inherited risk to find. Scan broadly. Unify the findings. Be explicit about what is advisory during development and what becomes blocking at the release boundary.</p><p>Automate safe dependency updates. Escalate major changes, ambiguous fixes, and modifications to the trust machinery itself.</p><p>And fail closed on coverage drift. Adding a new scanner or dependency surface without adding it to the automation&#8217;s coverage map should make the system red rather than silently incomplete.</p><p>The model can classify a finding.</p><p>The release system decides whether the artifact moves.</p><div><hr></div><h2>8. Observe the development system&#8212;and account for it</h2><p>An automated workflow that cannot explain itself will not remain trusted.</p><p>Observe four layers:</p><ul><li><p>the running product</p></li><li><p>each agent turn and tool call</p></li><li><p>CI and merge-queue health</p></li><li><p>the economics of the development loop</p></li></ul><p>The last one is easy to underbuild.</p><p>Agent spend should be attributable to the unit of work: ticket, pull request, person, agent tool, model, and automated review. The pull request is a useful source of truth because it already anchors intent, code, evidence, review, and outcome.</p><p>But token cost alone is not the cost model.</p><p>Track:</p><ul><li><p>cost per accepted change</p></li><li><p>attempts per task</p></li><li><p>human interventions</p></li><li><p>model switches</p></li><li><p>review latency</p></li><li><p>queue retries</p></li><li><p>escaped defects</p></li><li><p>repeated codebase reads</p></li><li><p>duplicated agent work</p></li><li><p>percentage of incidents that improve a skill or evaluation</p></li></ul><p>This is where model routing becomes engineering rather than vibes.</p><p>Routine, bounded work should use the cheapest model that can complete the entire loop safely. Ambiguous architecture, security-sensitive changes, release-critical work, and adversarial review should spend more when the stronger model reduces rework and senior attention.</p><p>The cheapest model call can still produce the most expensive development loop.</p><p>Context size deserves the same scrutiny. A bloated skill that re-enters context on every turn can cost more than its author realizes. Measure it. Move rarely used details behind progressive disclosure. Verify that references remain live.</p><p>Do not merely record agent cost.</p><p>Make it optimizable.</p><div><hr></div><h2>9. The real self-improvement loop</h2><p>The most important rule is about where learning goes.</p><p>When a human or an agent learns something reusable, write it back into the shared system in the same change that exposed it.</p><p>A flaky test becomes a new entry in the flake-smell catalog.</p><p>A merge-queue failure becomes a diagnostic rule.</p><p>A deployment surprise becomes a hardened step in the operating skill.</p><p>A strange production interaction becomes an evaluation case.</p><p>A repeated review finding becomes an architectural convention.</p><p>Do not put the learning in a personal note.</p><p>Do not leave it in a chat session that will disappear.</p><p>Do not create a vague &#8220;lessons learned&#8221; document nobody will load at the moment of action.</p><p>Put it where the next run will discover it automatically.</p><p>The catalog should grow only from observed need. That keeps it credible. A skill registry full of speculative warnings becomes noise; a skill registry built from real incidents becomes compressed organizational experience.</p><p>This is the practical version of a self-improving engineering system.</p><p>The model does not need to remember the incident forever.</p><p>The organization does.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!n1Pr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc8cb97d6-e041-4879-bac6-9f902208147d_2400x1553.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!n1Pr!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc8cb97d6-e041-4879-bac6-9f902208147d_2400x1553.png 424w, https://substackcdn.com/image/fetch/$s_!n1Pr!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc8cb97d6-e041-4879-bac6-9f902208147d_2400x1553.png 848w, https://substackcdn.com/image/fetch/$s_!n1Pr!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc8cb97d6-e041-4879-bac6-9f902208147d_2400x1553.png 1272w, https://substackcdn.com/image/fetch/$s_!n1Pr!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc8cb97d6-e041-4879-bac6-9f902208147d_2400x1553.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!n1Pr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc8cb97d6-e041-4879-bac6-9f902208147d_2400x1553.png" width="1456" height="942" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c8cb97d6-e041-4879-bac6-9f902208147d_2400x1553.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:942,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Flywheel converting incidents into versioned skills, rules, and evaluations that shape the next task.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Flywheel converting incidents into versioned skills, rules, and evaluations that shape the next task." title="Flywheel converting incidents into versioned skills, rules, and evaluations that shape the next task." srcset="https://substackcdn.com/image/fetch/$s_!n1Pr!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc8cb97d6-e041-4879-bac6-9f902208147d_2400x1553.png 424w, https://substackcdn.com/image/fetch/$s_!n1Pr!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc8cb97d6-e041-4879-bac6-9f902208147d_2400x1553.png 848w, https://substackcdn.com/image/fetch/$s_!n1Pr!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc8cb97d6-e041-4879-bac6-9f902208147d_2400x1553.png 1272w, https://substackcdn.com/image/fetch/$s_!n1Pr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc8cb97d6-e041-4879-bac6-9f902208147d_2400x1553.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>The model does not retain institutional memory. The workflow does.</em></p><div><hr></div><h2>The stable equilibrium</h2><p>The models will improve.</p><p>The coding tools will change.</p><p>More of planning, implementation, testing, review, diagnosis, and release preparation will move into the agent loop.</p><p>Good.</p><p>We should want that.</p><p>But the durable boundary remains:</p><blockquote><p>more intelligence inside the model<br>more authority outside the model</p></blockquote><p>The agent can propose.</p><p>The workflow enforces.</p><p>The agent can diagnose.</p><p>The evidence decides what happens next.</p><p>The agent can learn a pattern during one task.</p><p>The repository turns it into institutional memory.</p><p>The advantage is not the percentage of code an agent writes.</p><p>The advantage is a development system that can absorb better models without giving away judgment, release authority, or the history of why the organization works the way it does.</p><p>The model is replaceable.</p><p>The compounding workflow is not.</p><p>Agentic development begins when the agent stops being a novelty bolted onto an old process and becomes a first-class contributor to a workflow designed for both humans and machines.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[Who Owns Engineering Judgment?]]></title><description><![CDATA[Human attention, model budgets, and the control plane for AI-native software organizations]]></description><link>https://www.equationblog.com/p/who-owns-engineering-judgment</link><guid isPermaLink="false">https://www.equationblog.com/p/who-owns-engineering-judgment</guid><dc:creator><![CDATA[Ruslan Belkin]]></dc:creator><pubDate>Sat, 20 Jun 2026 14:15:34 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!5H9i!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ea67afd-a505-4c88-806d-fc121a08dbc2_2287x1525.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Price per token is not a cost model. It is an input to a cost model.</p><p>The AI engineering conversation is still too focused on the wrong budget.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>Tokens, model pricing, and vendor leverage all matter. But the real budget is not tokens.</p><p>The <strong>real budget is senior human attention</strong>: the moments when a model output needs correction, interpretation, architectural judgment, security review, release approval, customer-risk ownership, or institutional context.</p><p>That is where the system breaks.</p><p>There is an even deeper question underneath it: <em>As models get better, who owns engineering judgment?</em></p><p>One answer is: <em>the model vendors</em>.</p><p>That answer is less crazy than people want to admit.</p><p>Coding agents are moving beyond autocomplete into planning, design, implementation, testing, review, deployment, and operations. Full software development lifecycle is increasingly in scope for AI assistance, including planning, design, development, testing, review, and deployment. Agents can draft implementations, modify code across many files, fix build errors, write tests, and produce diff-ready change sets.</p><p>So yes, more workflow logic will move closer to the model.</p><p>That includes planning, review, risk detection, test selection, escalation, human handoff, evidence production, and learned standards of &#8220;good engineering.&#8221;</p><p>The question is not whether this happens. It already has. The question is <strong>what companies should still own?</strong></p><div><hr></div><h2><strong>The uncomfortable fork</strong></h2><p>There are two lazy versions of the future.</p><p>The first says:</p><p>Model vendors will own everything.</p><p>They have the data. They have the compute. They have the distribution. They have the feedback loop. They will absorb all relevant engineering intelligence, including when to involve humans. Your internal workflows will become a thin wrapper around their agents.</p><p>The second says:</p><p>Models are just utilities.</p><p>They are like electricity or cloud compute. Useful, interchangeable, commoditized. The real value stays entirely in business context, process, data, and execution.</p><p>Both are partially true.</p><p>Both are incomplete.</p><p>The more interesting version is this:</p><p>Model vendors will own more generic intelligence than most companies expect. Great companies will still own situated intelligence: context, authority, verification, accountability, and the compounding memory of their own decisions.</p><p>That is the distinction I care about.</p><p>Not &#8220;AI versus human.&#8221;</p><p>Not &#8220;frontier model versus cheap model.&#8221;</p><p>Not &#8220;central platform team versus product teams.&#8221;</p><p>The real boundary is:</p><p>Let the model own heuristics. Keep institutional authority outside the model.</p><div><hr></div><h2><strong>The price card has become strategically relevant again</strong></h2><p>For a while, it was fashionable to say model cost would round to zero.</p><p>Maybe someday. Not yet.</p><p>As of June 15, 2026, public token prices still vary by orders of magnitude across model families, context tiers, output volume, caching, priority mode, batch mode, and whether you are using a seat plan, credit system, or direct API.</p><p>The point is not to memorize today&#8217;s prices. They will change. The point is that model choice is now an operating decision.</p><p>OpenAI&#8217;s public API page currently lists GPT-5.5 at $5.00 per million input tokens, $0.50 per million cached input tokens, and $30.00 per million output tokens under standard processing for shorter context; it lists GPT-5.4 at $2.50 / $0.25 / $15.00 and GPT-5.4 mini at $0.75 / $0.075 / $4.50 for input, cached input, and output respectively.</p><p>Anthropic&#8217;s Claude API pricing page currently lists Claude Fable 5 at $10 input / $1 cache hit / $50 output per million tokens, Claude Opus 4.8 at $5 / $0.50 / $25, Claude Sonnet 4.6 at $3 / $0.30 / $15, and Claude Haiku 4.5 at $1 / $0.10 / $5. The same page says prompt-cache reads cost 0.1x base input price and batch processing applies a 50% discount to both input and output tokens.</p><p>Google&#8217;s Gemini API pricing currently lists Gemini 3.1 Pro Preview at $2 input / $12 output per million tokens for prompts up to 200K tokens, and $4 / $18 above 200K. It also lists Gemini 3.5 Flash at $1.50 / $9, Gemini 3.1 Flash-Lite at $0.25 / $1.50, and Gemini 2.5 Flash-Lite at $0.10 / $0.40, with output pricing including thinking tokens.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!bH1t!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F201ffd27-6562-4588-8c22-d639b5952dcd_2945x1750.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!bH1t!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F201ffd27-6562-4588-8c22-d639b5952dcd_2945x1750.png 424w, https://substackcdn.com/image/fetch/$s_!bH1t!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F201ffd27-6562-4588-8c22-d639b5952dcd_2945x1750.png 848w, https://substackcdn.com/image/fetch/$s_!bH1t!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F201ffd27-6562-4588-8c22-d639b5952dcd_2945x1750.png 1272w, https://substackcdn.com/image/fetch/$s_!bH1t!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F201ffd27-6562-4588-8c22-d639b5952dcd_2945x1750.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!bH1t!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F201ffd27-6562-4588-8c22-d639b5952dcd_2945x1750.png" width="1456" height="865" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/201ffd27-6562-4588-8c22-d639b5952dcd_2945x1750.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:865,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:219741,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.equationblog.com/i/202208096?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F201ffd27-6562-4588-8c22-d639b5952dcd_2945x1750.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!bH1t!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F201ffd27-6562-4588-8c22-d639b5952dcd_2945x1750.png 424w, https://substackcdn.com/image/fetch/$s_!bH1t!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F201ffd27-6562-4588-8c22-d639b5952dcd_2945x1750.png 848w, https://substackcdn.com/image/fetch/$s_!bH1t!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F201ffd27-6562-4588-8c22-d639b5952dcd_2945x1750.png 1272w, https://substackcdn.com/image/fetch/$s_!bH1t!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F201ffd27-6562-4588-8c22-d639b5952dcd_2945x1750.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Public token prices still vary by more than 100x across model tiers; model selection is now an operating discipline.</figcaption></figure></div><p>The first data point executives should internalize: <strong>the same million-token workflow can cost cents, dollars, tens of dollars, or more depending on model, context, output, caching, priority, and tool mode.</strong></p><p>The second data point matters more: <strong>the token bill is often not the expensive part.</strong></p><p><strong>The expensive part is the human recovery loop.</strong></p><div><hr></div><h2><strong>Price per token is the wrong denominator</strong></h2><p>The wrong denominator is: <strong>cost per token</strong></p><p>The better denominator is: <strong>cost per accepted, reviewed, shipped change</strong></p><p>A cheaper model that takes three attempts, creates drift, burns an hour of senior review, and leaves uncertainty around a risky migration may be more expensive than a frontier model that completes the work cleanly in one pass.</p><p>The token bill is visible.</p><p>The senior-engineer recovery cost is usually <strong>hidden</strong>.</p><p>The bad loop looks like this:</p><ol><li><p>Human gives a loose prompt.</p></li><li><p>Agent starts coding.</p></li><li><p>Human watches nervously.</p></li><li><p>Agent drifts.</p></li><li><p>Human interrupts.</p></li><li><p>Agent patches around the correction.</p></li><li><p>Tests fail.</p></li><li><p>Human re-explains the system.</p></li><li><p>Final diff is stitched together from partial attempts.</p></li></ol><p>That is not autonomy.</p><p>That is interrupt-driven delegation.</p><p>The better loop looks like this:</p><ol><li><p>Human approves intent, constraints, risks, and success criteria.</p></li><li><p>Agent produces a plan before implementation.</p></li><li><p>Human gates the plan.</p></li><li><p>Agent implements against the approved plan.</p></li><li><p>A separate review agent attacks the diff.</p></li><li><p>Output is packaged as evidence.</p></li><li><p>Human reviews the evidence, resolves judgment calls, and owns merge/release readiness.</p></li><li><p>Useful discoveries become future context, tests, runbooks, evals, or skills.</p></li></ol><p>The practical target is not &#8220;more AI usage.&#8221;</p><p>The practical target is:</p><ul><li><p>fewer human corrections per accepted diff</p></li><li><p>fewer repeated attempts</p></li><li><p>shorter review time</p></li><li><p>lower escaped-defect rate</p></li><li><p>higher useful concurrency per senior engineer</p></li><li><p>more institutional knowledge captured per completed task</p></li></ul><p>That is the engineering budget.</p><div><hr></div><h2><strong>A simple cost model</strong></h2><p>Here is the model I would use for budgeting:</p><p><em>monthly model cost &#8776; fresh input tokens + cached input tokens + output tokens + tool/runtime costs</em></p><p>More specifically:</p><ul><li><p><strong>Fresh input tokens:</strong> repo context, prompts, docs, issues, logs, traces.</p></li><li><p><strong>Cached input tokens:</strong> repeated context, stable repo instructions, runbooks, API docs, prior plans.</p></li><li><p><strong>Output tokens:</strong> code, tests, plans, reviews, explanations, retries.</p></li><li><p><strong>Tool/runtime costs:</strong> web search, containers, managed agent runtime, code execution, priority/fast mode, storage, observability.</p></li></ul><p>The simple version:</p><p><em>monthly cost &#8776; fresh input MTok &#215; input price + cached input MTok &#215; cache-read price + output MTok &#215; output price + tools</em></p><p>The scary part is output and retries.</p><p>The saving grace is caching.</p><p>Caching turns stable company context into a cost advantage. Anthropic&#8217;s pricing page says prompt-cache reads cost 0.1x base input price and that cache reads pay off quickly after reuse; OpenAI&#8217;s pricing page also shows cached-input rates materially below fresh-input rates for its flagship models.</p><p>So the first-order governance question is not: <strong>How do we cap tokens?</strong></p><p>It is: <strong>How do we make the same context useful repeatedly without reloading the company from scratch every time?</strong></p><div><hr></div><h2><strong>Planning ranges I would use</strong></h2><p>Do not treat the following as a benchmark.</p><p>Treat it as a planning model.</p><p>Also: do not plan any serious AI use case below <strong>$100 per user per month</strong>.</p><p>That does not mean every user will spend $100. It means that below that floor, you are usually not planning for real usage, variance, retries, training, tool overhead, or bursts.</p><p>For engineering, I would not plan below <strong>$200 per engineer per month</strong>.</p><p>Again, not every engineer will spend it every month. But if the budget starts below $200, the organization is implicitly designing for toy usage, not production behavior.</p><p>OpenAI&#8217;s Codex rate card says a typical Codex task using GPT-5.5 may consume 5&#8211;45 credits, and that average Codex cost is roughly $100&#8211;$200 per developer per month, with large variance depending on model, instances, automations, and fast mode. That is a useful public anchor, but I would still set the engineering planning floor at $200 because real production engineering needs room for variance.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!xbAS!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c1f232e-f782-4930-b4b5-5ca7a67b3eac_2947x1571.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xbAS!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c1f232e-f782-4930-b4b5-5ca7a67b3eac_2947x1571.png 424w, https://substackcdn.com/image/fetch/$s_!xbAS!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c1f232e-f782-4930-b4b5-5ca7a67b3eac_2947x1571.png 848w, https://substackcdn.com/image/fetch/$s_!xbAS!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c1f232e-f782-4930-b4b5-5ca7a67b3eac_2947x1571.png 1272w, https://substackcdn.com/image/fetch/$s_!xbAS!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c1f232e-f782-4930-b4b5-5ca7a67b3eac_2947x1571.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!xbAS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c1f232e-f782-4930-b4b5-5ca7a67b3eac_2947x1571.png" width="1456" height="776" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4c1f232e-f782-4930-b4b5-5ca7a67b3eac_2947x1571.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:776,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:212982,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.equationblog.com/i/202208096?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c1f232e-f782-4930-b4b5-5ca7a67b3eac_2947x1571.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!xbAS!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c1f232e-f782-4930-b4b5-5ca7a67b3eac_2947x1571.png 424w, https://substackcdn.com/image/fetch/$s_!xbAS!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c1f232e-f782-4930-b4b5-5ca7a67b3eac_2947x1571.png 848w, https://substackcdn.com/image/fetch/$s_!xbAS!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c1f232e-f782-4930-b4b5-5ca7a67b3eac_2947x1571.png 1272w, https://substackcdn.com/image/fetch/$s_!xbAS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c1f232e-f782-4930-b4b5-5ca7a67b3eac_2947x1571.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Planning envelopes are not entitlements or hard caps; govern through outcomes, not token volume.</figcaption></figure></div><h3><strong>Baseline AI access</strong></h3><p>Expected range: $100&#8211;$250 per user per month</p><p>Use for:</p><ul><li><p>general productivity</p></li><li><p>research</p></li><li><p>writing</p></li><li><p>summarization</p></li><li><p>lightweight analysis</p></li><li><p>occasional technical help</p></li></ul><p>This is the minimum serious floor. Below this, the budget usually becomes administrative theater.</p><h3><strong>Light engineering assistant</strong></h3><p>Expected range: $200&#8211;$750 per engineer per month</p><p>Use for:</p><ul><li><p>code explanations</p></li><li><p>small scripts</p></li><li><p>autocomplete</p></li><li><p>simple unit tests</p></li><li><p>light PR summaries</p></li><li><p>documentation</p></li><li><p>low-risk helper workflows</p></li></ul><p>This is where engineering access should start. A lower floor encourages underuse, rationing, or avoidance.</p><h3><strong>Routine agentic product engineer</strong></h3><p>Expected range: $500&#8211;$1,500 per engineer per month</p><p>Use for:</p><ul><li><p>daily code generation</p></li><li><p>test generation</p></li><li><p>codebase exploration</p></li><li><p>small refactors</p></li><li><p>PR summaries</p></li><li><p>routine debugging</p></li><li><p>first-pass reviews</p></li></ul><p>This is the zone where agents become part of the daily engineering loop, but not yet the primary implementation substrate.</p><h3><strong>AI-heavy product engineer</strong></h3><p>Expected range: $1,500&#8211;$5,000 per engineer per month</p><p>Use for:</p><ul><li><p>long-running implementation loops</p></li><li><p>multi-file refactors</p></li><li><p>design-to-plan-to-code workflows</p></li><li><p>substantial test generation</p></li><li><p>repeated review cycles</p></li><li><p>larger repo context</p></li><li><p>high cached-context reuse</p></li></ul><p>This is plausible when daily work moves from &#8220;assistant&#8221; to &#8220;delegated workflow.&#8221; It should be governed by accepted diffs, review load, and defect outcomes, not by token volume alone.</p><h3><strong>Senior, staff, or tech lead supervising parallel streams</strong></h3><p>Expected range: $3,000&#8211;$10,000 per user per month</p><p>Use for:</p><ul><li><p>architecture exploration</p></li><li><p>cross-service refactors</p></li><li><p>high-risk PR review</p></li><li><p>migration planning</p></li><li><p>release readiness</p></li><li><p>adversarial review</p></li><li><p>parallel agent supervision</p></li><li><p>incident/post-incident investigation</p></li></ul><p>This is the category where hard caps are most dangerous.</p><p>These people amplify many other people and many other agents. Blocking them to save tokens can waste the scarcest resource in the company: senior judgment.</p><h3><strong>AI platform, release, security, or agent-heavy team pool</strong></h3><p>Expected range: $10,000&#8211;$50,000+ per team per month</p><p>Use for:</p><ul><li><p>private evals</p></li><li><p>routing experiments</p></li><li><p>skill registries</p></li><li><p>repo-context services</p></li><li><p>model comparisons</p></li><li><p>automated review infrastructure</p></li><li><p>autonomous regression hunts</p></li><li><p>release and incident automation</p></li><li><p>enterprise observability and governance</p></li></ul><p>This should usually be a pooled budget, not a per-seat cap.</p><p>Platform, incident, release, and evaluation work are all bursty.</p><p>The rule should be: no blank checks, but no dumb throttling during approved high-leverage windows.</p><div><hr></div><h2><strong>The danger of false economy</strong></h2><p>A more capable model can be cheaper even when its tokens are more expensive.</p><p>This feels counterintuitive only if you measure the wrong thing.</p><p>Assume a cheaper model takes three attempts, creates uncertainty, and consumes 75 minutes of senior review.</p><p>Assume a stronger model costs more in tokens but finishes cleanly and consumes 20 minutes of review.</p><p>At a fully loaded senior-engineering cost of even $150&#8211;$300 per hour, the second path can easily be cheaper.</p><p>Not because the tokens are cheaper.</p><p>Because the loop is cheaper.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!5H9i!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ea67afd-a505-4c88-806d-fc121a08dbc2_2287x1525.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!5H9i!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ea67afd-a505-4c88-806d-fc121a08dbc2_2287x1525.png 424w, https://substackcdn.com/image/fetch/$s_!5H9i!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ea67afd-a505-4c88-806d-fc121a08dbc2_2287x1525.png 848w, https://substackcdn.com/image/fetch/$s_!5H9i!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ea67afd-a505-4c88-806d-fc121a08dbc2_2287x1525.png 1272w, https://substackcdn.com/image/fetch/$s_!5H9i!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ea67afd-a505-4c88-806d-fc121a08dbc2_2287x1525.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!5H9i!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ea67afd-a505-4c88-806d-fc121a08dbc2_2287x1525.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1ea67afd-a505-4c88-806d-fc121a08dbc2_2287x1525.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:111899,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.equationblog.com/i/202208096?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ea67afd-a505-4c88-806d-fc121a08dbc2_2287x1525.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!5H9i!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ea67afd-a505-4c88-806d-fc121a08dbc2_2287x1525.png 424w, https://substackcdn.com/image/fetch/$s_!5H9i!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ea67afd-a505-4c88-806d-fc121a08dbc2_2287x1525.png 848w, https://substackcdn.com/image/fetch/$s_!5H9i!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ea67afd-a505-4c88-806d-fc121a08dbc2_2287x1525.png 1272w, https://substackcdn.com/image/fetch/$s_!5H9i!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ea67afd-a505-4c88-806d-fc121a08dbc2_2287x1525.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Human recovery time can dominate model spend; optimize the complete loop, not just token price.</figcaption></figure></div><p>Use cheaper/default models for bounded work:</p><ul><li><p>formatting</p></li><li><p>summarization</p></li><li><p>low-risk docs</p></li><li><p>simple scaffolding</p></li><li><p>rote transformations</p></li><li><p>test naming</p></li></ul><p>Use stronger models when the value is uncertainty reduction:</p><ul><li><p>ambiguous architecture</p></li><li><p>cross-service changes</p></li><li><p>security-sensitive paths</p></li><li><p>data migrations</p></li><li><p>release readiness</p></li><li><p>incident response</p></li><li><p>adversarial review</p></li><li><p>poorly documented internal systems</p></li></ul><p>The mistake is not spending too much on frontier models.</p><p>The mistake is spending frontier tokens where cheap models are enough, then being cheap where a better model would save senior attention, reduce rework, or avoid defects.</p><p>The question is not: <strong>What is the cheapest model that can attempt this?</strong></p><p>The better question is: <strong>What is the cheapest complete loop that can get this safely accepted?</strong></p><div><hr></div><h2><strong>AI amplifies the engineering system you already have</strong></h2><p>DORA&#8217;s 2025 research found near-universal AI adoption among software professionals: 90% of respondents reported using AI at work, and more than 80% believed it increased productivity. The same Google Cloud summary notes that 30% reported little or no trust in AI-generated code, which is exactly the tension engineering leaders need to design around.</p><p>That matches what technical leaders see in practice.</p><p>AI does not fix the engineering system.</p><p>It amplifies it.</p><p>If your specs are crisp, tests are fast, ownership is clean, docs are usable, review norms are strong, and release paths are well instrumented, agents multiply the system.</p><p>If your specs are ambiguous, tests are flaky, docs are stale, ownership is fuzzy, and senior review is overloaded, agents multiply that too.</p><p>More generated code without more validation capacity is not productivity.</p><p>It is inventory.</p><div><hr></div><h2><strong>The model should not become the control plane</strong></h2><p>There is a subtle danger as agents get more competent.</p><p>The workflow starts cleanly:</p><ul><li><p>model proposes a plan</p></li><li><p>model writes code</p></li><li><p>model runs tests</p></li><li><p>model reviews the diff</p></li><li><p>model summarizes risk</p></li><li><p>model recommends human approval</p></li></ul><p>Then the next step is tempting:</p><ul><li><p>model decides whether human approval is needed</p></li></ul><p>Then:</p><ul><li><p>model decides which human</p></li><li><p>model decides whether evidence is sufficient</p></li><li><p>model decides whether a policy applies</p></li><li><p>model decides whether release is safe</p></li></ul><p>At that point, the model is no longer just a cognitive engine.</p><p>It is becoming an authority layer.</p><p>This is where I would be conservative.</p><p>OpenAI&#8217;s Model Spec is a useful public example of how much behavior-level logic is moving into models: instruction following, conflict resolution, intended defaults, safety boundaries, and agentic side-effect control. OpenAI describes the Model Spec as a formal framework for desired model behavior and a target for training and evaluation, while also saying it is not a claim that models behave that way perfectly today.</p><p>That work is necessary.</p><p>It also proves the point: <strong>model behavior is itself a control surface.</strong></p><p>If your engineering workflow depends on hidden model defaults, provider-side routing, or behavior updates you cannot inspect, you do not really own the control plane.</p><p>You are renting it.</p><p>The answer is not to distrust model vendors by default.</p><p>The answer is to separate intelligence from authority.</p><p>The model can advise.</p><p>The system must decide.</p><p>The model can classify.</p><p>The policy layer must enforce.</p><p>The model can produce evidence.</p><p>The evidence standard must be external.</p><p>The model can remember patterns.</p><p>The company must own durable memory.</p><div><hr></div><h2><strong>Heuristics inside the model, invariants outside the model</strong></h2><p>This is the cleanest boundary I see today.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!uh9l!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83b28628-8735-492a-b050-a196cc5b4e2a_2947x1561.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!uh9l!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83b28628-8735-492a-b050-a196cc5b4e2a_2947x1561.png 424w, https://substackcdn.com/image/fetch/$s_!uh9l!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83b28628-8735-492a-b050-a196cc5b4e2a_2947x1561.png 848w, https://substackcdn.com/image/fetch/$s_!uh9l!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83b28628-8735-492a-b050-a196cc5b4e2a_2947x1561.png 1272w, https://substackcdn.com/image/fetch/$s_!uh9l!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83b28628-8735-492a-b050-a196cc5b4e2a_2947x1561.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!uh9l!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83b28628-8735-492a-b050-a196cc5b4e2a_2947x1561.png" width="1456" height="771" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/83b28628-8735-492a-b050-a196cc5b4e2a_2947x1561.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:771,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:195947,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.equationblog.com/i/202208096?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83b28628-8735-492a-b050-a196cc5b4e2a_2947x1561.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!uh9l!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83b28628-8735-492a-b050-a196cc5b4e2a_2947x1561.png 424w, https://substackcdn.com/image/fetch/$s_!uh9l!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83b28628-8735-492a-b050-a196cc5b4e2a_2947x1561.png 848w, https://substackcdn.com/image/fetch/$s_!uh9l!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83b28628-8735-492a-b050-a196cc5b4e2a_2947x1561.png 1272w, https://substackcdn.com/image/fetch/$s_!uh9l!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83b28628-8735-492a-b050-a196cc5b4e2a_2947x1561.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Where the model suggests heuristics, the control plane keeps authority, approvals, and auditability.</figcaption></figure></div><p>A heuristic is: <strong>this looks like a security-sensitive change</strong></p><p>An invariant is: <strong>security-sensitive changes require named approval, passing checks, rollback notes, and an audit trail</strong></p><p>A heuristic is: <strong>these tests are probably relevant</strong></p><p>An invariant is: <strong>no merge without required checks</strong></p><p>A heuristic is: <strong>this migration resembles prior incidents</strong></p><p>An invariant is: <strong>customer-data migrations require staged rollout and post-release monitoring</strong></p><p>A heuristic is: <strong>this evidence package looks complete</strong></p><p>An invariant is: <strong>every agent-authored PR records the model, prompt/context version, tools used, tests run, unresolved risks, and human approver</strong></p><p>A heuristic is: <strong>a human should review this</strong></p><p>An invariant is: <strong>which human, under what SLA, with what responsibility</strong></p><p>This matters because otherwise the model becomes the invisible control plane.</p><p>And that is the failure mode.</p><p>Not &#8220;AI writes bad code.&#8221;</p><p>That is annoying, but manageable.</p><p>The worse failure mode is decision laundering:</p><ol><li><p>The model makes a judgment.</p></li><li><p>The human rubber-stamps it.</p></li><li><p>The audit trail says &#8220;human approved.&#8221;</p></li><li><p>Nobody can reconstruct where the real decision came from.</p></li></ol><p>That is how organizations lose engineering memory while pretending to gain productivity.</p><div><hr></div><h2><strong>Security makes this boundary non-optional</strong></h2><p>Prompt injection is not a side issue. It is a reminder that LLMs are not normal software components.</p><p>The UK National Cyber Security Centre argues that prompt injection is better treated not as normal code injection, but as exploitation of an &#8220;inherently confusable deputy.&#8221; OWASP&#8217;s 2025 Top 10 for LLM and generative AI applications lists prompt injection, sensitive information disclosure, supply-chain risk, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption.</p><p>This has a direct implication for engineering agents: <strong>never make the model its own permission system.</strong></p><p>A model with tool access is powerful.</p><p>It is also confusable.</p><p>The system needs hard constraints outside the model:</p><ul><li><p>permission boundaries</p></li><li><p>sandboxing</p></li><li><p>approval gates</p></li><li><p>allowlists and denylists</p></li><li><p>policy checks</p></li><li><p>secret handling</p></li><li><p>destructive-operation stops</p></li><li><p>release gates</p></li><li><p>audit logs</p></li><li><p>reproducible evidence</p></li></ul><p>If a model can create code, run code, read internal docs, open tickets, comment on PRs, call tools, inspect logs, and recommend releases, then the model is no longer &#8220;just chat.&#8221;</p><p>It is part of the engineering system.</p><p>Treat it that way.</p><div><hr></div><h2><strong>Every non-trivial agent task should start with a plan</strong></h2><p>Coding agents should not jump from prompt to diff for non-trivial work.</p><p>The first artifact should be a plan.</p><p>A useful plan names:</p><ul><li><p>affected files and services</p></li><li><p>assumptions</p></li><li><p>open questions</p></li><li><p>sequencing</p></li><li><p>expected tests</p></li><li><p>risk areas</p></li><li><p>migration concerns</p></li><li><p>rollback concerns</p></li><li><p>expected evidence before review</p></li></ul><p>This is not bureaucracy.</p><p>This is leverage.</p><p>A senior engineer can review a plan much faster than they can reverse-engineer a bad branch.</p><p>Plan review is where humans catch the shape of the work before the model spends tokens and creates state.</p><p>Then implementation happens against the approved plan.</p><p>Then a separate review pass attacks the implementation.</p><p>Then the output is packaged as evidence.</p><p>Then the human reviews the evidence, not just the diff.</p><p>OpenAI&#8217;s AI-native engineering guide makes a similar operational point in its getting-started checklist: start with well-specified tasks, have the agent use a planning tool or write a <code>PLAN.md</code>, check that commands succeed, and iterate on an <code>AGENTS.md</code> file that unlocks loops like running tests and linters.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!7JqJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74d49e25-3fb7-4375-b4cb-363ee06192c6_2942x1077.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!7JqJ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74d49e25-3fb7-4375-b4cb-363ee06192c6_2942x1077.png 424w, https://substackcdn.com/image/fetch/$s_!7JqJ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74d49e25-3fb7-4375-b4cb-363ee06192c6_2942x1077.png 848w, https://substackcdn.com/image/fetch/$s_!7JqJ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74d49e25-3fb7-4375-b4cb-363ee06192c6_2942x1077.png 1272w, https://substackcdn.com/image/fetch/$s_!7JqJ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74d49e25-3fb7-4375-b4cb-363ee06192c6_2942x1077.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!7JqJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74d49e25-3fb7-4375-b4cb-363ee06192c6_2942x1077.png" width="1456" height="533" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/74d49e25-3fb7-4375-b4cb-363ee06192c6_2942x1077.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:533,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:124936,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.equationblog.com/i/202208096?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74d49e25-3fb7-4375-b4cb-363ee06192c6_2942x1077.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!7JqJ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74d49e25-3fb7-4375-b4cb-363ee06192c6_2942x1077.png 424w, https://substackcdn.com/image/fetch/$s_!7JqJ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74d49e25-3fb7-4375-b4cb-363ee06192c6_2942x1077.png 848w, https://substackcdn.com/image/fetch/$s_!7JqJ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74d49e25-3fb7-4375-b4cb-363ee06192c6_2942x1077.png 1272w, https://substackcdn.com/image/fetch/$s_!7JqJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74d49e25-3fb7-4375-b4cb-363ee06192c6_2942x1077.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The goal is fewer interruptions and clearer human gates, not less human accountability.</figcaption></figure></div><p>The minimum evidence package should include:</p><ul><li><p>plan</p></li><li><p>diff summary</p></li><li><p>tests run</p></li><li><p>failing tests</p></li><li><p>unresolved questions</p></li><li><p>risk assessment</p></li><li><p>security notes</p></li><li><p>performance notes</p></li><li><p>rollout notes</p></li><li><p>rollback notes</p></li><li><p>model and context version</p></li><li><p>tool actions taken</p></li></ul><p>This is the practical version of &#8220;human at the gates.&#8221;</p><p>Without the evidence package, the reviewer has to reconstruct the story.</p><p>With it, the reviewer can spend judgment where judgment matters.</p><div><hr></div><h2><strong>&#8220;Human in the loop&#8221; is too vague</strong></h2><p>The phrase &#8220;human in the loop&#8221; can mean anything.</p><p>It can mean thoughtful supervision.</p><p>It can also mean a senior engineer constantly babysitting a model, interrupting it, correcting it, rerunning tests, unwinding drift, and manually stitching together a final diff.</p><p>That is not autonomy.</p><p>That is interrupt-driven delegation.</p><p>The human should not be in the gears.</p><p>The human should be at the gates.</p><h3><strong>Before work starts</strong></h3><p>The human gate is scope:</p><ul><li><p>product intent</p></li><li><p>constraints</p></li><li><p>blast radius</p></li><li><p>test strategy</p></li><li><p>migration risk</p></li><li><p>open questions</p></li></ul><h3><strong>During work</strong></h3><p>The human gate is exception handling:</p><ul><li><p>repeated failure</p></li><li><p>ambiguity</p></li><li><p>high-risk files</p></li><li><p>destructive operations</p></li><li><p>security-sensitive changes</p></li><li><p>missing evidence</p></li></ul><h3><strong>Before merge</strong></h3><p>The human gate is judgment:</p><ul><li><p>architecture</p></li><li><p>maintainability</p></li><li><p>security</p></li><li><p>performance</p></li><li><p>user impact</p></li><li><p>rollout plan</p></li><li><p>rollback path</p></li></ul><h3><strong>After release</strong></h3><p>The human gate is responsibility:</p><ul><li><p>customer impact</p></li><li><p>incident response</p></li><li><p>follow-up prioritization</p></li><li><p>what should become durable organizational memory</p></li></ul><p>The model should do the mechanical reading, mapping, drafting, testing, summarizing, and first-pass review.</p><p>The human should decide what matters.</p><div><hr></div><h2><strong>Routing is necessary, but auto-routing is not authority</strong></h2><p>No single model wins every task.</p><p>That makes routing inevitable.</p><p>LLMRouterBench, a 2026 routing benchmark, evaluates more than 400,000 instances across 21 datasets and 33 models. It confirms strong model complementarity, but also finds that many routing methods perform similarly under unified evaluation and that several recent methods do not reliably beat simple baselines.</p><p>That matches the production reality.</p><p>Auto mode is useful for convenience.</p><p>It is not an authority for high-risk engineering.</p><p>A better routing policy has explicit lanes.</p><h3><strong>Cheap/default lane</strong></h3><p>Use for:</p><ul><li><p>docs</p></li><li><p>formatting</p></li><li><p>summarization</p></li><li><p>boilerplate</p></li><li><p>simple scaffolding</p></li><li><p>low-risk edits</p></li><li><p>test naming</p></li><li><p>mechanical transformations</p></li></ul><h3><strong>Standard frontier lane</strong></h3><p>Use for:</p><ul><li><p>ordinary production implementation</p></li><li><p>debugging</p></li><li><p>feature work</p></li><li><p>test generation</p></li><li><p>codebase exploration</p></li><li><p>moderate refactors</p></li></ul><h3><strong>Top-tier escalation lane</strong></h3><p>Use for:</p><ul><li><p>failed loops</p></li><li><p>repeated test failures</p></li><li><p>ambiguous architecture</p></li><li><p>security-sensitive changes</p></li><li><p>schema/data migrations</p></li><li><p>release-critical code</p></li><li><p>adversarial review</p></li></ul><h3><strong>Human-gated lane</strong></h3><p>Use for:</p><ul><li><p>destructive operations</p></li><li><p>production changes</p></li><li><p>secrets</p></li><li><p>permissions</p></li><li><p>customer-impacting rollout</p></li><li><p>incident response</p></li></ul><p>The labels matter less than the telemetry.</p><p>Track:</p><ul><li><p>wrong-route corrections</p></li><li><p>repeated attempts</p></li><li><p>escalation rate</p></li><li><p>human intervention count</p></li><li><p>model switches</p></li><li><p>review latency</p></li><li><p>escaped defects</p></li><li><p>cost per accepted diff</p></li></ul><p>That is how routing becomes engineering instead of vibes.</p><div><hr></div><h2><strong>Proprietary context is both tax and asset</strong></h2><p>Every company has internal abstractions.</p><p>Some are strategic.</p><p>Many are accidental.</p><p>Models have strong priors for common public stacks. They have weaker priors for your internal framework, your release ritual, your service boundaries, your custom metadata model, your old migration convention, and the bug everyone &#8220;just knows&#8221; to avoid.</p><p>That creates a context tax.</p><p>The model needs more examples, more docs, more traces, more corrective feedback, and more human steering to do the same job.</p><p>The wrong answer is to dump everything into context.</p><p>The right answer is to separate proprietary surface into two buckets.</p><p><strong>First</strong>: proprietary surface that is not differentiated</p><p>Reduce it.</p><p>Standardize it.</p><p>Wrap it.</p><p>Replace it.</p><p>Make it boring to the model.</p><p><strong>Second</strong>: proprietary surface that is differentiated</p><p>Invest in it.</p><p>Make it machine-readable.</p><p>Create canonical examples.</p><p>Build runbooks.</p><p>Version skills.</p><p>Capture golden traces.</p><p>Add private evals.</p><p>Make the hidden system legible.</p><p>Anthropic&#8217;s Agent Skills documentation is a useful public example of this pattern: skills package instructions, metadata, workflows, and optional resources so Claude can load relevant domain-specific expertise on demand instead of repeatedly consuming the same context. The same documentation emphasizes progressive disclosure: metadata is always available, instructions load when triggered, and deeper resources are accessed only as needed.</p><p>Skills should not be treated as prompt snippets.</p><p>They are a mechanism for turning repeated human corrections into reusable workflow capital.</p><p>But skills can also become prompt debt.</p><p>So they need:</p><ul><li><p>owners</p></li><li><p>versioning</p></li><li><p>triggering rules</p></li><li><p>evaluation</p></li><li><p>conflict detection</p></li><li><p>deprecation</p></li><li><p>telemetry</p></li></ul><p>A good bug investigation should not end with a patch.</p><p>It should produce at least one durable artifact:</p><ul><li><p>regression test</p></li><li><p>eval case</p></li><li><p>runbook</p></li><li><p>skill update</p></li><li><p>prompt/context rule</p></li><li><p>better dashboard</p></li><li><p>deleted abstraction</p></li><li><p>clearer API</p></li><li><p>improved release checklist</p></li></ul><p>That is the compounding loop:</p><p><em>workflow &#8594; trace &#8594; evaluation &#8594; correction &#8594; skill &#8594; better workflow</em></p><p>If that loop belongs to the vendor, the company rents intelligence.</p><p>If that loop belongs to the company, the company compounds judgment.</p><div><hr></div><h2><strong>Multi-agent work needs an operating system</strong></h2><p>The future is not one engineer chatting with one assistant.</p><p>It is many agents running in parallel:</p><ul><li><p>one maps the codebase</p></li><li><p>one drafts a plan</p></li><li><p>one implements</p></li><li><p>one writes tests</p></li><li><p>one attacks the diff</p></li><li><p>one checks security</p></li><li><p>one prepares release notes</p></li><li><p>one watches post-release telemetry</p></li><li><p>one turns discoveries into reusable knowledge</p></li></ul><p>Codex can work on tasks in the background, including in parallel, using its own cloud environment; it can also connect to GitHub repositories and create pull requests from its work.</p><p>That sounds powerful.</p><p>It is also how you create an expensive mess if the human becomes the scheduler.</p><p>Past a small number of concurrent streams, the bottleneck is not agent availability.</p><p>The bottleneck is state.</p><ul><li><p>Which branch is blocked?</p></li><li><p>Which tests failed?</p></li><li><p>Which agent repeated work?</p></li><li><p>Which diff conflicts with another diff?</p></li><li><p>Which risk needs a human decision?</p></li><li><p>Which output is ready for serious review?</p></li><li><p>Which run should be stopped?</p></li><li><p>Which context is stale?</p></li><li><p>Which decision has already been made?</p></li></ul><p>Multi-agent engineering needs a command center.</p><p>Not necessarily a fancy product.</p><p>But at least a shared state view:</p><ul><li><p>task</p></li><li><p>branch</p></li><li><p>owner</p></li><li><p>agent role</p></li><li><p>model</p></li><li><p>prompt/context version</p></li><li><p>current plan</p></li><li><p>tests run</p></li><li><p>failures</p></li><li><p>blockers</p></li><li><p>risk level</p></li><li><p>cost so far</p></li><li><p>next required human decision</p></li></ul><p>Without this, the senior engineer stops being an architect and becomes an interrupt router.</p><p>That is a bad trade.</p><div><hr></div><h2><strong>The org chart changes, but not in the lazy way</strong></h2><p>The lazy story is junior replacement.</p><p>The better story is supervision capacity.</p><p>As code generation gets cheaper, verification gets more valuable.</p><p>The shape of engineering work changes.</p><p>Staff and principal engineers spend more time on:</p><ul><li><p>architecture</p></li><li><p>decomposition</p></li><li><p>eval design</p></li><li><p>model steering</p></li><li><p>review standards</p></li><li><p>high-risk release decisions</p></li></ul><p>Senior engineers spend more time on:</p><ul><li><p>running agent pods</p></li><li><p>supervising parallel streams</p></li><li><p>owning service quality</p></li><li><p>converting ambiguous work into reviewable plans</p></li></ul><p>Engineering managers and tech leads spend more time on:</p><ul><li><p>review queues</p></li><li><p>bottleneck removal</p></li><li><p>ownership clarity</p></li><li><p>sequencing</p></li><li><p>cross-team coordination</p></li><li><p>making sure agent output does not overwhelm the human system</p></li></ul><p>Junior and mid-level engineers should not be pushed out of the pipeline.</p><p>Their apprenticeship path changes toward:</p><ul><li><p>verification</p></li><li><p>debugging</p></li><li><p>tests</p></li><li><p>observability</p></li><li><p>code reading</p></li><li><p>small scoped changes</p></li><li><p>agent operation</p></li></ul><p>AI platform engineers become more important:</p><ul><li><p>routing</p></li><li><p>evals</p></li><li><p>skill registries</p></li><li><p>context stores</p></li><li><p>telemetry</p></li><li><p>secure permissions</p></li><li><p>sandboxing</p></li><li><p>cost controls</p></li></ul><p>SRE, QA, release, and security capacity also matter more, not less.</p><p>More generated code without more validation capacity is not productivity.</p><p>It is inventory.</p><div><hr></div><h2><strong>Governance should control the work system, not just total tokens</strong></h2><p>A token cap is easy to understand.</p><p>It is also usually the wrong control surface.</p><p>Better metrics:</p><ul><li><p>cost per accepted diff</p></li><li><p>cost per reviewed and shipped change</p></li><li><p>human interventions per PR</p></li><li><p>repeated attempts per task</p></li><li><p>model switches per task</p></li><li><p>wrong-route corrections</p></li><li><p>review latency</p></li><li><p>escaped defects</p></li><li><p>cache hit rate</p></li><li><p>duplicated agent work</p></li><li><p>percentage of agent tasks with evidence packages</p></li><li><p>percentage of incidents that improve runbooks/evals/skills</p></li><li><p>percentage of high-risk PRs with named human approval</p></li><li><p>percentage of agent-authored PRs with model/context version recorded</p></li></ul><p>Also track the hidden cost:</p><ul><li><p>re-reviewed diffs</p></li><li><p>conflicting branches</p></li><li><p>redundant context loading</p></li><li><p>repeated codebase reads</p></li><li><p>repeated explanations of the same internal abstraction</p></li><li><p>agent runs stopped because ownership was unclear</p></li><li><p>human attention lost to interruptions</p></li></ul><p>That is where the waste hides.</p><p>The governance objective is not to minimize tokens.</p><p>The governance objective is to spend enough tokens to reduce avoidable human touchpoints while preserving human judgment at the places where judgment changes quality.</p><div><hr></div><h2><strong>A narrow operating loop I would actually run</strong></h2><p>Start boring.</p><h4><strong>1. Set floors before caps</strong></h4><p>Do not plan below:</p><ul><li><p><strong>$100/user/month</strong> for any serious AI use case</p></li><li><p><strong>$200/engineer/month</strong> for engineering use cases</p></li></ul><p>The floor does not mean entitlement.</p><p>It means you are taking the workflow seriously enough to support real usage.</p><h4><strong>2. Every non-trivial agent task starts with a plan</strong></h4><p>The plan includes:</p><ul><li><p>affected files/services</p></li><li><p>assumptions</p></li><li><p>open questions</p></li><li><p>risk areas</p></li><li><p>sequencing</p></li><li><p>test strategy</p></li><li><p>rollback concerns</p></li></ul><p>Humans approve plans, not vibes.</p><h4><strong>3. Every agent output includes an evidence package</strong></h4><p>No evidence package, no review.</p><p>Minimum evidence:</p><ul><li><p>plan</p></li><li><p>diff summary</p></li><li><p>tests run</p></li><li><p>failing tests</p></li><li><p>unresolved risks</p></li><li><p>security/performance notes</p></li><li><p>rollout notes</p></li><li><p>rollback notes</p></li><li><p>model and prompt/context version</p></li></ul><h4><strong>4. Use adversarial review from a different model family for high-stakes work</strong></h4><p>Same-model self-review is better than nothing.</p><p>Cross-model review is better for correlated blind spots.</p><h4><strong>5. Put hard stops on repeated failure</strong></h4><p>No infinite retries.</p><p>Stop on:</p><ul><li><p>repeated test failure</p></li><li><p>high-risk file paths</p></li><li><p>destructive operations</p></li><li><p>migration changes</p></li><li><p>missing tests</p></li><li><p>unclear ownership</p></li><li><p>unresolved ambiguity</p></li></ul><h4><strong>6. Build a skill registry</strong></h4><p>Reusable procedures should become skills, not Slack lore.</p><p>But skills need owners, versioning, evals, and deconfliction.</p><h4><strong>7. Build private evals</strong></h4><p>Public benchmarks are useful.</p><p>They are not your production system.</p><p>Measure models on your real task classes:</p><ul><li><p>migrations</p></li><li><p>debugging</p></li><li><p>release readiness</p></li><li><p>internal abstractions</p></li><li><p>security review</p></li><li><p>test repair</p></li><li><p>incident summaries</p></li><li><p>codebase navigation</p></li><li><p>PR evidence quality</p></li></ul><h4><strong>8. Review budgets by outcome</strong></h4><p>Do not ask only:</p><p>who spent the most?</p><p>Ask:</p><ul><li><p>what shipped?</p></li><li><p>how much human review did it consume?</p></li><li><p>how many attempts did it take?</p></li><li><p>what defects escaped?</p></li><li><p>what reusable knowledge was created?</p></li><li><p>what should be routed differently next time?</p></li></ul><p>That is the difference between buying AI tools and building an AI-native engineering system.</p><div><hr></div><h2><strong>The stable equilibrium</strong></h2><p>The model vendors will own a lot.</p><ul><li><p>They will own more generic engineering intelligence than most companies expect.</p></li><li><p>They will make the average code review better.</p></li><li><p>They will make the average migration plan better.</p></li><li><p>They will make the average test-generation loop better.</p></li><li><p>They will make the average junior task cheaper.</p></li><li><p>They will absorb a lot of workflow scaffolding that companies currently think is special.</p></li></ul><p>Good.</p><p>We should want that.</p><p>There is no moat in forcing humans to remember what a model can reliably surface.</p><p>But great companies will still own the harder thing:</p><ul><li><p>context</p></li><li><p>authority</p></li><li><p>verification</p></li><li><p>accountability</p></li><li><p>taste</p></li><li><p>strategy</p></li><li><p>release risk</p></li><li><p>customer commitments</p></li><li><p>institutional memory</p></li><li><p>the learning loop around their own decisions</p></li></ul><p>That is the equilibrium I trust.</p><p>More intelligence inside the model.</p><p>More authority outside the model.</p><p>Use the model as a cognitive engine.</p><p>Do not use it as the court of record.</p><p>Use model vendors aggressively.</p><p>Do not let them become your invisible control plane.</p><p>The future engineering function is not organized around who can type code fastest.</p><p>It is organized around who can define invariants, operate agents, verify outputs, own release risk, and turn failures into durable knowledge.</p><p>The model is not the moat.</p><p>The moat is the learning loop you can inspect, evaluate, govern, and carry across models.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[Data Is the Control Surface for LLMs and Agents]]></title><description><![CDATA[A technical essay on data preparation, curation, mixtures, context engineering, and verifier-backed RL]]></description><link>https://www.equationblog.com/p/data-is-the-control-surface-for-llms</link><guid isPermaLink="false">https://www.equationblog.com/p/data-is-the-control-surface-for-llms</guid><dc:creator><![CDATA[Ruslan Belkin]]></dc:creator><pubDate>Tue, 09 Jun 2026 18:15:25 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!fTkB!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcba638aa-3def-4601-85ce-dde372551499_1800x950.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<blockquote><p><strong>Data quality is not a scalar. Data quality is learnable structure under a compute, context, and objective budget.</strong></p></blockquote><h1>Introduction: data is where intent becomes behavior</h1><p>I have always been biased toward data.</p><p>Not because data work is glamorous. It is not. It is messy, repetitive, expensive, and full of judgment calls.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>But in AI systems, data is where intent becomes behavior.</p><p>A model is not trained on &#8220;knowledge.&#8221; It is trained on sequences, transformations, mixtures, orderings, rubrics, rewards, traces, tool outputs, verifiers, and corrections.</p><p>An agent is not improved by &#8220;prompting&#8221; in the abstract. It is improved by deciding what information belongs in context, what should be retrieved, what should be compressed, what should be isolated, what should be verified, and what should be converted into future training or evaluation data.</p><p>The old pipeline was too small:</p><blockquote><p>collect data &#8594; clean data &#8594; train model</p></blockquote><p>The modern pipeline is closer to:</p><blockquote><p>source &#8594; extract &#8594; clean &#8594; deduplicate &#8594; filter &#8594; transform &#8594; mix &#8594; train &#8594; evaluate &#8594; deploy &#8594; trace &#8594; verify &#8594; compile &#8594; replay</p></blockquote><p>That is the data flywheel behind modern LLMs and agents.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!fTkB!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcba638aa-3def-4601-85ce-dde372551499_1800x950.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!fTkB!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcba638aa-3def-4601-85ce-dde372551499_1800x950.png 424w, https://substackcdn.com/image/fetch/$s_!fTkB!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcba638aa-3def-4601-85ce-dde372551499_1800x950.png 848w, https://substackcdn.com/image/fetch/$s_!fTkB!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcba638aa-3def-4601-85ce-dde372551499_1800x950.png 1272w, https://substackcdn.com/image/fetch/$s_!fTkB!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcba638aa-3def-4601-85ce-dde372551499_1800x950.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!fTkB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcba638aa-3def-4601-85ce-dde372551499_1800x950.png" width="1456" height="768" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/cba638aa-3def-4601-85ce-dde372551499_1800x950.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:768,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:67893,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.equationblog.com/i/201182742?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcba638aa-3def-4601-85ce-dde372551499_1800x950.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!fTkB!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcba638aa-3def-4601-85ce-dde372551499_1800x950.png 424w, https://substackcdn.com/image/fetch/$s_!fTkB!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcba638aa-3def-4601-85ce-dde372551499_1800x950.png 848w, https://substackcdn.com/image/fetch/$s_!fTkB!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcba638aa-3def-4601-85ce-dde372551499_1800x950.png 1272w, https://substackcdn.com/image/fetch/$s_!fTkB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcba638aa-3def-4601-85ce-dde372551499_1800x950.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>Figure 1. Data as a staged control surface across training, evaluation, deployment, verification, and replay.</em></p><p>The important shift is this: data quality is not a scalar. It is learnable structure under a compute, context, and objective budget.</p><h1>1. Epiplexity gives us the right language</h1><p>The best recent framing I have seen is epiplexity.</p><p>Classical information theory gives us a problem. It says deterministic transformations should not create new information. It says information should not depend on data order. It treats likelihood modeling largely as distribution matching.</p><p>Modern LLM practice disagrees.</p><ul><li><p>Synthetic data can help.</p></li><li><p>Curriculum order matters.</p></li><li><p>Reformatting a document into instructions can improve downstream behavior.</p></li><li><p>A verifier can turn a raw rollout into a stronger learning signal.</p></li><li><p>A long context can be useless noise or a high-value playbook.</p></li></ul><p>Epiplexity formalizes this gap by focusing on what a computationally bounded observer can learn from data. The paper introduces epiplexity to capture structured, learnable content for bounded learners, and explicitly addresses why deterministic transformations, ordering, and likelihood modeling can matter in ways classical Shannon or Kolmogorov framings do not capture. <a href="https://arxiv.org/abs/2601.03220">[1]</a></p><p>That is the core thesis of this post: data preparation is not janitorial work. It is the engineering of learnable structure.</p><p>The question is not &#8220;how many tokens do we have?&#8221; The better question is:</p><blockquote><p>What useful structure does this data expose to this model,</p><p>at this stage,</p><p>under this objective,</p><p>with this amount of compute, context, and verification?</p></blockquote><p>That is true for pretraining, post-training, context engineering, reinforcement learning, and production agents.</p><h1>2. Cleaning is not just removing bad text</h1><p>Cleaning is often treated as a low-status preprocessing step. That is a mistake.</p><p>For web-scale training, cleaning is one of the largest capability levers. It determines what the model repeatedly sees, what gets amplified, what gets memorized, and whether the model learns coherent human language or boilerplate, menus, spam, broken markup, affiliate pages, keyword lists, and duplicated templates.</p><p>The best public example is FineWeb. Hugging Face built FineWeb as a 15T-token dataset from 96 Common Crawl snapshots, and the paper is valuable because it documents the cleaning pipeline and ablates the choices instead of presenting them as folklore. The final pipeline extracts from WARC rather than WET, applies base filtering, performs individual per-crawl MinHash deduplication, applies selected C4-style filters, adds custom heuristic filters, and anonymizes email and public IP addresses. <a href="https://arxiv.org/html/2406.17557v1">[2]</a></p><p>The details matter. FineWeb found that WET extraction retained too much boilerplate and menu text. The team instead extracted text from WARC files using trafilatura, and the WARC/trafilatura ablation produced a better model than WET extraction. The base filtering stage then used URL blocklists for adult content, fastText language identification, and MassiveText-style quality and repetition filters, producing roughly 36T tokens after base filtering. <a href="https://arxiv.org/html/2406.17557v1">[2]</a></p><p>Deduplication was more subtle. FineWeb used fuzzy MinHash deduplication with document 5-grams, 112 hash functions, 14 buckets of 8 hashes, and a target of at least 75% similarity. But global deduplication across all 96 snapshots did not work as expected: it reduced the corpus to 4T tokens and gave little improvement. Worse, for an older crawl, the 10% of data retained by global deduplication was visually lower quality than the 90% removed. The better choice was independent per-snapshot MinHash deduplication, which produced 20T tokens and matched RefinedWeb performance in the ablation. <a href="https://arxiv.org/html/2406.17557v1">[2]</a></p><blockquote><p><strong>Deduplication is not automatically good. Deduplication granularity changes the data distribution.</strong></p></blockquote><p>If you deduplicate too aggressively across time, you may accidentally retain the wrong tail. You may remove useful recurring explanations while keeping strange, low-quality pages that only appear once. The goal is not to maximize deletion. The goal is to remove harmful repetition while preserving useful coverage.</p><p>FineWeb also shows how heuristic filtering should be done. The team tested C4-style filters and found that terminal punctuation gave the largest individual HellaSwag boost, but removed about 30% of tokens. They chose all C4 filters except terminal punctuation because the latter deleted too much. Then they built custom filters by collecting more than 50 document statistics, comparing high- and low-quality distributions, selecting candidate thresholds, and validating them with 28B-token ablation runs. The final three custom filters removed about 22% of tokens and improved aggregate benchmark score by about 1% in the ablations. <a href="https://arxiv.org/html/2406.17557v1">[2]</a></p><p>That is how data cleaning should work:</p><p>inspect &#8594; hypothesize &#8594; filter &#8594; train proxy models &#8594; evaluate &#8594; keep only filters that improve behavior</p><p>FineWeb-Edu adds another useful datapoint. It filtered FineWeb into a 1.3T-token educational subset using a classifier trained from LLM annotations. The classifier achieved 82% F1 at the selected threshold; applying it over FineWeb required 6,000 H100 GPU hours. In a 1.82B model trained on 350B tokens, FineWeb-Edu increased MMLU from 33% to 37% and ARC from 46% to 57%, while matching Matrix&#8217;s MMLU performance with almost 10x fewer tokens. <a href="https://arxiv.org/html/2406.17557v1">[2]</a></p><p>That is strong evidence for classifier-based filtering when it is measured. But it also comes with a warning. Apple&#8217;s &#8220;Data-Quality Illusion&#8221; work argues that classifier-based quality filtering can improve downstream task performance while not necessarily improving language modeling on the high-quality target set. The authors challenge the assumption that classifier score captures a universal notion of quality. <a href="https://machinelearning.apple.com/research/data-quality-illusion">[3]</a></p><p>The right conclusion is not &#8220;classifiers are bad.&#8221; The right conclusion is that quality classifiers are instruments, not truth. A quality filter must be attached to a target behavior, benchmark suite, training stage, and ablation record. Otherwise it becomes a magic number.</p><h1>3. The cleaning stack should be empirical</h1><p>A modern LLM data cleaning stack should usually include:</p><ul><li><p>source validation</p></li><li><p>HTML / PDF / code extraction</p></li><li><p>boilerplate removal</p></li><li><p>language identification</p></li><li><p>document normalization</p></li><li><p>PII handling</p></li><li><p>policy filtering</p></li><li><p>exact deduplication</p></li><li><p>fuzzy deduplication</p></li><li><p>near-duplicate cluster analysis</p></li><li><p>repetition filters</p></li><li><p>low-information filters</p></li><li><p>classifier-based quality filters</p></li><li><p>contamination checks</p></li><li><p>source and license metadata</p></li><li><p>benchmark-backed ablations</p></li></ul><p>The important part is not the checklist. The important part is the measurement discipline.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!2Zm_!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97775dd9-2a3c-40df-a375-21b53a37a5b2_1800x950.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!2Zm_!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97775dd9-2a3c-40df-a375-21b53a37a5b2_1800x950.png 424w, https://substackcdn.com/image/fetch/$s_!2Zm_!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97775dd9-2a3c-40df-a375-21b53a37a5b2_1800x950.png 848w, https://substackcdn.com/image/fetch/$s_!2Zm_!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97775dd9-2a3c-40df-a375-21b53a37a5b2_1800x950.png 1272w, https://substackcdn.com/image/fetch/$s_!2Zm_!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97775dd9-2a3c-40df-a375-21b53a37a5b2_1800x950.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!2Zm_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97775dd9-2a3c-40df-a375-21b53a37a5b2_1800x950.png" width="1456" height="768" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/97775dd9-2a3c-40df-a375-21b53a37a5b2_1800x950.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:768,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:90359,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.equationblog.com/i/201182742?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97775dd9-2a3c-40df-a375-21b53a37a5b2_1800x950.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!2Zm_!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97775dd9-2a3c-40df-a375-21b53a37a5b2_1800x950.png 424w, https://substackcdn.com/image/fetch/$s_!2Zm_!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97775dd9-2a3c-40df-a375-21b53a37a5b2_1800x950.png 848w, https://substackcdn.com/image/fetch/$s_!2Zm_!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97775dd9-2a3c-40df-a375-21b53a37a5b2_1800x950.png 1272w, https://substackcdn.com/image/fetch/$s_!2Zm_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97775dd9-2a3c-40df-a375-21b53a37a5b2_1800x950.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>Figure 2. A practical cleaning and filtering stack with an ablation loop.</em></p><p>FineWeb&#8217;s ablation protocol is the right model. The team trained data-ablation models that were identical except for the data, used equal token budgets, ran multiple seeds, and evaluated on benchmarks selected for stable signal at small scale. Their ablations used 1.82B-parameter models, 28B-token filtering runs, 350B-token dedup and cumulative-filtering runs, more than 70 trained models, and roughly 80,000 H100 GPU hours. <a href="https://arxiv.org/html/2406.17557v1">[2]</a></p><p>That is what data-driven data curation means. Cleaning choices should not be made by intuition alone. They should be promoted only if they improve the relevant evaluation suite.</p><h1>4. Data mixtures are learned, not guessed</h1><p>The next mistake is treating data mixture as a fixed human recipe. It is not. Data mixture is a hyperparameter schedule.</p><p>Different training stages need different data distributions:</p><ul><li><p>early pretraining &#8594; broad coverage and diversity</p></li><li><p>late pretraining &#8594; higher-quality and capability-dense data</p></li><li><p>reasoning annealing &#8594; math, code, STEM, synthetic reasoning</p></li><li><p>long-context training &#8594; long coherent documents and long-context QA</p></li><li><p>SFT &#8594; instruction, reasoning, tools, chat, safety</p></li><li><p>preference / DPO &#8594; ranked outputs and rejected alternatives</p></li><li><p>RL &#8594; verifier-backed tasks and environments</p></li><li><p>agent training &#8594; tool trajectories, state transitions, rubrics, rollouts</p></li></ul><p>NVIDIA&#8217;s Nemotron Nano 2 is a good recent example. Its pretraining mixture has thirteen categories, including quality-bucketed crawl data, synthetic high-quality crawl, math, Wikipedia, code, academic data, multilingual data, and synthetic SFT-style data. NVIDIA says it weights higher-quality sources more heavily and uses a three-phase curriculum: Phase 1 emphasizes diversity, Phase 2 shifts toward higher-quality data at the 60% training point, and Phase 3 shifts again at the 90% point. <a href="https://arxiv.org/html/2508.14444v2">[4]</a></p><p>NVIDIA also discloses ablations that show how these choices are made. For multilingual data, they continued a 1B checkpoint for another 100B tokens with 50% multilingual data and 50% default pretraining data, then evaluated on Global-MMLU. Curated Common Crawl averaged 37.0, FineWeb-2 averaged 35.1, DiverseQA-wiki averaged 42.1, and DiverseQA-crawl averaged 47.0. That result led them to upweight DiverseQA-crawl in the multilingual mixture. <a href="https://arxiv.org/html/2508.14444v2">[4]</a></p><p>That is the pattern we want. Not &#8220;we think multilingual web is good.&#8221; Instead: compare multilingual source families under controlled continuation, measure Global-MMLU, and change the mixture.</p><p>The same report shows that adding only 5% fundamental-reasoning SFT-style data into a 100B-token continuation improved Nemotron-H 8B&#8217;s MMLU-Pro from 44.24 to 56.36, with average math increasing by about two points and no decrease in average commonsense or code benchmarks. <a href="https://arxiv.org/html/2508.14444v2">[4]</a></p><p>Qwen3 describes a complementary methodology. It trained on 36T tokens across 119 languages, including coding, STEM, reasoning, books, multilingual data, and synthetic data. It used Qwen2.5-VL to extract text from PDF-like documents, Qwen2.5 to refine that text, and Qwen2.5/Qwen2.5-Math/Qwen2.5-Coder to synthesize trillions of tokens in formats such as textbooks, QA, instructions, and code snippets. Qwen also annotated more than 30T tokens for educational value, domain, safety, and related labels, then optimized the mixture at the instance level through proxy-model ablations rather than only at source or domain level. <a href="https://arxiv.org/html/2505.09388v1">[5]</a></p><p>Qwen3&#8217;s stage schedule is explicit: more than 30T tokens of general pretraining at 4K context, then about 5T higher-quality tokens with increased STEM, coding, reasoning, and synthetic data, then hundreds of billions of long-context tokens, where 75% of texts were 16K&#8211;32K and 25% were 4K&#8211;16K. <a href="https://arxiv.org/html/2505.09388v1">[5]</a></p><p>So the modern methodology looks like this:</p><ul><li><p>label data finely</p></li><li><p>bucket by source and quality</p></li><li><p>run proxy-model ablations</p></li><li><p>evaluate downstream capability deltas</p></li><li><p>schedule mixtures by stage</p></li><li><p>anneal toward capability-dense data</p></li><li><p>validate against regression suites</p></li></ul><p>RegMix formalizes part of this. It treats mixture selection as a regression problem: train many small proxy models on different mixtures, fit a regression model to predict performance, then train the larger model on the predicted best mixture. In its experiments, RegMix trained 512 one-million-parameter models for 1B tokens, used the regression model to choose a mixture, and trained a 1B model for 25B tokens that performed best among 64 candidate mixtures. It also outperformed human selection up to 7B models trained on 100B tokens and matched or exceeded DoReMi with about 10% of the compute. <a href="https://arxiv.org/abs/2407.01492">[6]</a></p><p>The practical point is blunt: if you are not measuring mixture effects, you are guessing.</p><p>And loss is not enough. Nemotron-H reported downstream accuracy jumps after mixture changes and noted cases where validation loss was not a reliable proxy for downstream task performance. FineWeb selected filters using benchmark deltas, not just perplexity. DataComp-LM makes this broader: it provides a 240T-token Common Crawl corpus, training recipes, and 53 downstream evaluations so teams can test deduplication, filtering, and data mixing. Its baseline trained a 7B model on 2.6T tokens to 64% 5-shot MMLU, a 6.6-point MMLU gain over MAP-Neo with 40% less compute. <a href="https://arxiv.org/abs/2406.11794">[7]</a></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!UILY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd23ebb99-fe97-4753-b813-8f6d988c0657_1800x950.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!UILY!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd23ebb99-fe97-4753-b813-8f6d988c0657_1800x950.png 424w, https://substackcdn.com/image/fetch/$s_!UILY!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd23ebb99-fe97-4753-b813-8f6d988c0657_1800x950.png 848w, https://substackcdn.com/image/fetch/$s_!UILY!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd23ebb99-fe97-4753-b813-8f6d988c0657_1800x950.png 1272w, https://substackcdn.com/image/fetch/$s_!UILY!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd23ebb99-fe97-4753-b813-8f6d988c0657_1800x950.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!UILY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd23ebb99-fe97-4753-b813-8f6d988c0657_1800x950.png" width="1456" height="768" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d23ebb99-fe97-4753-b813-8f6d988c0657_1800x950.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:768,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 3&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Figure 3" title="Figure 3" srcset="https://substackcdn.com/image/fetch/$s_!UILY!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd23ebb99-fe97-4753-b813-8f6d988c0657_1800x950.png 424w, https://substackcdn.com/image/fetch/$s_!UILY!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd23ebb99-fe97-4753-b813-8f6d988c0657_1800x950.png 848w, https://substackcdn.com/image/fetch/$s_!UILY!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd23ebb99-fe97-4753-b813-8f6d988c0657_1800x950.png 1272w, https://substackcdn.com/image/fetch/$s_!UILY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd23ebb99-fe97-4753-b813-8f6d988c0657_1800x950.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>Figure 3. Stage-wise mixture design as an experimental loop.</em></p><h1>5. Synthetic data is a compiler, not a shortcut</h1><p>Synthetic data has a bad reputation when it means low-diversity model slop. But the best synthetic data work is not &#8220;ask a model to make more examples.&#8221; It is compilation.</p><p>A raw document is not a dataset. A book is not a dataset. A manual is not a dataset. A GitHub repository is not a dataset. A support transcript is not a dataset. A tool trace is not a dataset. They are source material.</p><p>The data engineering task is to compile source material into useful training, evaluation, retrieval, reward, and context artifacts.</p><p>Instruction Pre-Training augments raw corpora with instruction-response pairs generated by an instruction synthesizer. The paper synthesized 200M instruction-response pairs across 40+ task categories and showed that instruction-augmented pretraining improved base models and made them benefit more from later instruction tuning. <a href="https://arxiv.org/html/2406.14491v1">[17]</a></p><p>MAmmoTH2 recalls relevant web documents, extracts instruction-response pairs, and refines them using open-source LLMs. The project harvested 10M instruction examples from the pretraining web corpus; MAmmoTH2-7B Mistral improved from 11% to 34% on MATH and 36% to 67% on GSM8K without in-domain training data. <a href="https://arxiv.org/html/2405.03548v3">[18]</a></p><p>Web Reconstruction treats each web document as either an instruction or a response and reconstructs the missing side. Its WebR datasets outperformed prior instruction-tuning baselines by up to 16.65% across four instruction-following benchmarks. <a href="https://arxiv.org/html/2504.15573v1">[19]</a></p><p>Qwen3&#8217;s use of synthetic textbooks, QA, instructions, and code snippets is one example. NVIDIA&#8217;s Nemotron Nano 2 uses synthetic high-quality crawl, synthetic SFT-style data, long-document QA, and translated DiverseQA. For long-context extension, NVIDIA used academic documents longer than 32K tokens, split them into 1,024-token chunks, selected 10% of chunks for Qwen2.5-72B-Instruct to generate QA pairs, concatenated those QA pairs, and appended them to the original document. A 20% allocation of this long-context document-QA data improved RULER-128K in ablations. <a href="https://arxiv.org/html/2508.14444v2">[4]</a></p><p>This is the right abstraction:</p><p>source corpus &#8594; compiled training, evaluation, retrieval, reward, and context artifacts</p><p>A book can become:</p><ul><li><p>chapters</p></li><li><p>sections</p></li><li><p>claims</p></li><li><p>definitions</p></li><li><p>examples</p></li><li><p>procedures</p></li><li><p>tables</p></li><li><p>equations</p></li><li><p>edge cases</p></li><li><p>contradictions</p></li><li><p>factual QA</p></li><li><p>procedural QA</p></li><li><p>retrieval tasks</p></li><li><p>citation tasks</p></li><li><p>summaries</p></li><li><p>critiques</p></li><li><p>tool-use tasks</p></li><li><p>rubrics</p></li><li><p>unit tests</p></li><li><p>replay episodes</p></li></ul><p>The invariant is provenance. Every generated artifact should point back to source spans, tool outputs, environment states, or verifier results. Without provenance, synthetic data becomes impossible to debug.</p><p>A generated QA pair is not good because it looks good. It is good if it teaches a capability, survives verification, preserves provenance, improves evals, and does not regress adjacent behavior.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!jJ49!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc852ca9b-0769-4902-a1ec-6839a6da96d7_1800x950.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!jJ49!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc852ca9b-0769-4902-a1ec-6839a6da96d7_1800x950.png 424w, https://substackcdn.com/image/fetch/$s_!jJ49!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc852ca9b-0769-4902-a1ec-6839a6da96d7_1800x950.png 848w, https://substackcdn.com/image/fetch/$s_!jJ49!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc852ca9b-0769-4902-a1ec-6839a6da96d7_1800x950.png 1272w, https://substackcdn.com/image/fetch/$s_!jJ49!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc852ca9b-0769-4902-a1ec-6839a6da96d7_1800x950.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!jJ49!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc852ca9b-0769-4902-a1ec-6839a6da96d7_1800x950.png" width="1456" height="768" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c852ca9b-0769-4902-a1ec-6839a6da96d7_1800x950.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:768,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 4&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Figure 4" title="Figure 4" srcset="https://substackcdn.com/image/fetch/$s_!jJ49!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc852ca9b-0769-4902-a1ec-6839a6da96d7_1800x950.png 424w, https://substackcdn.com/image/fetch/$s_!jJ49!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc852ca9b-0769-4902-a1ec-6839a6da96d7_1800x950.png 848w, https://substackcdn.com/image/fetch/$s_!jJ49!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc852ca9b-0769-4902-a1ec-6839a6da96d7_1800x950.png 1272w, https://substackcdn.com/image/fetch/$s_!jJ49!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc852ca9b-0769-4902-a1ec-6839a6da96d7_1800x950.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>Figure 4. The document-to-dataset compiler.</em></p><h1>6. RL steering is also data engineering</h1><p>Reinforcement learning is usually described as optimization. For LLMs and agents, that is incomplete. RL is a data pipeline.</p><ul><li><p>The prompt distribution is data.</p></li><li><p>The rollout is data.</p></li><li><p>The environment is data.</p></li><li><p>The tool result is data.</p></li><li><p>The verifier is data.</p></li><li><p>The judge rubric is data.</p></li><li><p>The reward vector is data.</p></li><li><p>The failed trajectory is data.</p></li><li><p>The replay set is data.</p></li></ul><p>DeepSeek-R1-Zero is the clean example. It used rule-based rewards for math and code: boxed-answer verification for math, compiler/test feedback for code, and a format reward for &lt;think&gt; tags. DeepSeek explicitly avoided neural outcome or process reward models in that stage because of reward hacking risk and retraining complexity. AIME 2024 pass@1 increased from 15.6% to 71.0%, and majority voting reached 86.7%. <a href="https://arxiv.org/html/2501.12948v1">[8]</a></p><p>This is the key engineering point: if the environment can verify the work, prefer verification over taste.</p><p>For math, code, structured extraction, database queries, tool-use workflows, and many agent tasks, the best reward is often not a learned preference model. It is a test, execution result, schema check, constraint checker, exact match, or environment transition.</p><p>DeepSeek&#8217;s later R1 pipeline added cold-start long-CoT data, reasoning-oriented RL, rejection sampling, roughly 600K reasoning samples, and final training that included broader non-reasoning data. The final DeepSeek-R1 report shows strong benchmark results such as 79.8 AIME 2024, 97.3 MATH-500, 65.9 LiveCodeBench, and 49.2 SWE Verified. <a href="https://arxiv.org/html/2501.12948v1">[8]</a></p><p>Qwen3 shows that RL data does not always need to be huge if the verifier is strong. Its reasoning RL stage used 3,995 query-verifier pairs, selected to be unused in cold start, learnable, challenging, and broad. Qwen3-235B-A22B improved on AIME&#8217;24 from 70.1 to 85.1 over 170 RL steps. <a href="https://arxiv.org/html/2505.09388v1">[5]</a></p><p>Kimi K2 shows the agentic version. Moonshot built a large-scale tool-use synthesis pipeline with three stages: tool spec generation, agent/task generation, and trajectory generation. It fetched 3,000+ real MCP tools from GitHub and generated more than 20,000 synthetic tools; tasks were paired with rubrics specifying success criteria, expected tool-use patterns, and evaluation checkpoints. The simulator maintained state, returned realistic tool feedback, introduced edge cases, and retained only trajectories that passed LLM-judge rubric evaluation. For coding and software engineering, Kimi complemented simulation with real execution sandboxes and test-suite pass rates. <a href="https://arxiv.org/html/2507.20534v2">[9]</a></p><p>NVIDIA Nemotron 3 Super shows the same pattern in an open training pipeline. It reports 25T pretraining tokens, a 7M-sample SFT stage, and a three-stage RL pipeline: multi-environment RLVR across 21 environments and 37 datasets, SWE-RL with OpenHands and Apptainer containers, and RLHF with a principle-following generative reward model. <a href="https://github.com/NVIDIA-NeMo/Nemotron/blob/main/docs/nemotron/super3/README.md">[10]</a></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Run9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60fe8d8a-2735-4821-ad6b-2aa5d9e62338_1800x950.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Run9!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60fe8d8a-2735-4821-ad6b-2aa5d9e62338_1800x950.png 424w, https://substackcdn.com/image/fetch/$s_!Run9!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60fe8d8a-2735-4821-ad6b-2aa5d9e62338_1800x950.png 848w, https://substackcdn.com/image/fetch/$s_!Run9!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60fe8d8a-2735-4821-ad6b-2aa5d9e62338_1800x950.png 1272w, https://substackcdn.com/image/fetch/$s_!Run9!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60fe8d8a-2735-4821-ad6b-2aa5d9e62338_1800x950.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Run9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60fe8d8a-2735-4821-ad6b-2aa5d9e62338_1800x950.png" width="1456" height="768" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/60fe8d8a-2735-4821-ad6b-2aa5d9e62338_1800x950.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:768,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:84424,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.equationblog.com/i/201182742?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60fe8d8a-2735-4821-ad6b-2aa5d9e62338_1800x950.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Run9!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60fe8d8a-2735-4821-ad6b-2aa5d9e62338_1800x950.png 424w, https://substackcdn.com/image/fetch/$s_!Run9!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60fe8d8a-2735-4821-ad6b-2aa5d9e62338_1800x950.png 848w, https://substackcdn.com/image/fetch/$s_!Run9!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60fe8d8a-2735-4821-ad6b-2aa5d9e62338_1800x950.png 1272w, https://substackcdn.com/image/fetch/$s_!Run9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60fe8d8a-2735-4821-ad6b-2aa5d9e62338_1800x950.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>Figure 5. A verifier-backed RL environment stack.</em></p><p>The engineering lesson is simple:</p><p>Use verifiers where possible.</p><p>A scalar reward without provenance is dangerous. A judge without calibration is dangerous. An environment without anti-hacking checks is dangerous. A rollout without replay is lost data.</p><h1>7. Context engineering is runtime data engineering</h1><p>The same data problem now appears at inference time.</p><p>Prompt engineering asks: what should I say to the model?</p><p>Context engineering asks: what information state should the model see before the next action?</p><p>That information state includes instructions, task state, retrieved evidence, examples, memories, tool schemas, tool outputs, prior decisions, constraints, citations, rubrics, and open uncertainties.</p><p>This is not cosmetic. It is runtime curation. An agent context window is a temporary dataset assembled for one forward pass.</p><ul><li><p>If the context is stale, the model reasons from stale evidence.</p></li><li><p>If the context is polluted, the model amplifies pollution.</p></li><li><p>If the context is too large, the model gets distracted.</p></li><li><p>If the context omits a constraint, the model violates it.</p></li><li><p>If the context includes sensitive data unnecessarily, the system creates avoidable risk.</p></li><li><p>If the context is over-compressed, the model loses the structure it needed.</p></li></ul><p>ACE, or Agentic Context Engineering, is one of the most relevant recent papers here. It treats contexts as evolving playbooks rather than static prompts. ACE uses a Generator, Reflector, and Curator to accumulate, refine, and organize strategies through incremental delta updates rather than monolithic rewrites. Across agent and finance benchmarks, ACE reports +10.6% on agents and +8.6% on finance, while adapting without labeled supervision by using execution feedback. <a href="https://arxiv.org/html/2510.04618v1">[11]</a></p><p>The most important ACE result is the failure mode: context collapse. In one AppWorld case study, a context at step 60 contained 18,282 tokens and achieved 66.7 accuracy. One rewrite collapsed it to 122 tokens, and accuracy fell to 57.1, worse than the 63.7 baseline without adaptation. <a href="https://arxiv.org/html/2510.04618v1">[11]</a></p><p>That is epiplexity in practice. Compression removed useful structure. Shorter was not better. Cleaner was not better. A summary destroyed operational knowledge.</p><p>The right rule is not &#8220;use more context.&#8221; The right rule is: preserve useful structure while controlling distraction, staleness, privacy, and cost.</p><p>ACE&#8217;s cost data is also useful. On AppWorld offline adaptation, ACE reduced latency by 82.3% and rollouts by 75.1% versus GEPA. On online FiNER, it reduced latency by 91.5% and token cost by 83.6% versus Dynamic Cheatsheet. That happened because ACE used incremental delta updates and non-LLM merging/deduplication instead of repeatedly rewriting the whole context. <a href="https://arxiv.org/html/2510.04618v1">[11]</a></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!2NLE!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadf88ad8-3c29-4bb5-8a6a-be212ed2b7f2_1800x950.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!2NLE!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadf88ad8-3c29-4bb5-8a6a-be212ed2b7f2_1800x950.png 424w, https://substackcdn.com/image/fetch/$s_!2NLE!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadf88ad8-3c29-4bb5-8a6a-be212ed2b7f2_1800x950.png 848w, https://substackcdn.com/image/fetch/$s_!2NLE!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadf88ad8-3c29-4bb5-8a6a-be212ed2b7f2_1800x950.png 1272w, https://substackcdn.com/image/fetch/$s_!2NLE!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadf88ad8-3c29-4bb5-8a6a-be212ed2b7f2_1800x950.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!2NLE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadf88ad8-3c29-4bb5-8a6a-be212ed2b7f2_1800x950.png" width="1456" height="768" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/adf88ad8-3c29-4bb5-8a6a-be212ed2b7f2_1800x950.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:768,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 6&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Figure 6" title="Figure 6" srcset="https://substackcdn.com/image/fetch/$s_!2NLE!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadf88ad8-3c29-4bb5-8a6a-be212ed2b7f2_1800x950.png 424w, https://substackcdn.com/image/fetch/$s_!2NLE!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadf88ad8-3c29-4bb5-8a6a-be212ed2b7f2_1800x950.png 848w, https://substackcdn.com/image/fetch/$s_!2NLE!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadf88ad8-3c29-4bb5-8a6a-be212ed2b7f2_1800x950.png 1272w, https://substackcdn.com/image/fetch/$s_!2NLE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadf88ad8-3c29-4bb5-8a6a-be212ed2b7f2_1800x950.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>Figure 6. Context engineering as runtime data curation.</em></p><h2>Retrieval should optimize utility, not similarity</h2><p>Most RAG systems still retrieve by semantic similarity. That is not enough for in-context learning.</p><p>ICLERB reframes retrieval for ICL as a recommendation problem: retrieve the documents that maximize downstream LLM utility, not merely the documents most semantically similar to the query. Its benchmark evaluates retrievers by whether retrieved contexts improve LLM accuracy in ICL settings. <a href="https://arxiv.org/html/2411.18947v1">[12]</a></p><p>The results are important. ICLERB found that rankings diverge from MTEB-style retrieval rankings, meaning strong search embeddings are not necessarily strong ICL context selectors. It also found that fine-tuning for semantic similarity can be detrimental for ICL utility in some cases. <a href="https://arxiv.org/html/2411.18947v1">[12]</a></p><p>The RLRAIF method goes further: it fine-tunes a retriever using LLM feedback as a reward signal. With only about 10K DPO values, estimated at 5M tokens, and a small adapter trained on a consumer GPU, the RLRAIF reranker achieved 0.7238 nDCG@10 and 0.7225 nDCG@50, outperforming much larger models such as bge-en-icl and NV-Embed-v2 on ICLERB. <a href="https://arxiv.org/html/2411.18947v1">[12]</a></p><p>The practical implication is direct: do not evaluate retrieval only by relevance. Evaluate whether the retrieved context changes the answer.</p><p>For agents, the retrieval target is not top-k similar chunks. It is the minimum sufficient evidence pack for the next decision.</p><h3>Likely useful in the evidence pack</h3><ul><li><p>task state; hard constraints; source-grounded facts</p></li><li><p>prior decisions; tool schemas; recent tool outputs</p></li><li><p>similar solved cases; known failure modes; rubrics; unit tests</p></li></ul><h3>Usually exclude or isolate</h3><ul><li><p>duplicate chunks; stale state; irrelevant logs</p></li><li><p>raw tool dumps; summaries that erase operational detail</p></li><li><p>sensitive data not needed by the model</p></li></ul><h2>Long-context benchmarks show why this matters</h2><p>Needle-in-a-haystack is too easy.</p><p>RULER was designed because vanilla needle tests measure superficial retrieval. It adds diverse needle types, multiple needles, multi-hop tracing, aggregation, and configurable length/task complexity. In its evaluation of 17 long-context LMs across 13 tasks, many models that were nearly perfect on vanilla needle-in-a-haystack degraded sharply as length and complexity increased; only about half maintained satisfactory performance at 32K despite claiming 32K+ context. <a href="https://openreview.net/forum?id=kIoBbc76Sy">[13]</a></p><p>NoLiMa removes literal overlap between the question and the relevant evidence. That matters because many long-context systems can exploit lexical matching rather than understanding. At 32K context, 11 evaluated models dropped below 50% of their short-context baselines, and GPT-4o dropped from 99.3% to 69.7%. <a href="https://openreview.net/forum?id=0OshX1hiSa">[14]</a></p><p>LongBench v2 tests deeper long-context reasoning with 503 multiple-choice questions, contexts from 8K to 2M words, and six categories including single-document QA, multi-document QA, long in-context learning, long-dialogue history, code-repository understanding, and structured-data understanding. Human experts reached 53.7% under a 15-minute constraint; the best direct-answer model reached 50.1%, while o1-preview reached 57.7% with longer reasoning. <a href="https://arxiv.org/abs/2412.15204">[15]</a></p><p>For coding agents, ContextBench is more directly operational. It contains 1,136 issue-resolution tasks from 66 repositories across eight programming languages, with human-annotated gold contexts. It measures context recall, precision, and efficiency across agent trajectories. <a href="https://arxiv.org/html/2602.05892v2">[16]</a></p><p>That gives us a practical evaluation stack for context engineering:</p><ul><li><p>RULER &#8594; synthetic long-context retrieval, tracing, aggregation</p></li><li><p>NoLiMa &#8594; long-context retrieval without lexical shortcuts</p></li><li><p>LongBench v2 &#8594; realistic deep long-context reasoning</p></li><li><p>ContextBench &#8594; coding-agent context recall, precision, efficiency</p></li><li><p>ICLERB &#8594; utility of retrieved examples/documents for ICL</p></li><li><p>AppWorld/ACE &#8594; adaptive context and agent memory under feedback</p></li></ul><p>The conclusion is direct: context must be evaluated as a data product.</p><p>Every context policy should be ablated:</p><ul><li><p>remove a memory component</p></li><li><p>change the retriever</p></li><li><p>change the reranker</p></li><li><p>change the compression method</p></li><li><p>change the evidence budget</p></li><li><p>change tool-output handling</p></li><li><p>change citation requirements</p></li><li><p>replay the same episodes</p></li><li><p>measure success, grounding, latency, token cost, and regressions</p></li></ul><h1>8. Production traces are the next dataset</h1><p>The highest-value data often appears after deployment.</p><p>Production traces show what users actually ask, what the model retrieves, which tools it calls, where it fails, which evidence it ignores, which judge disagrees, where policies are ambiguous, and where the model sounds confident without being grounded.</p><p>Those traces should not disappear into logs. They should become training, evaluation, context, and reward artifacts.</p><p>The loop is:</p><p>observe &#8594; classify &#8594; verify &#8594; compile &#8594; replay &#8594; promote</p><p>A failure can compile into many things:</p><ul><li><p>bad answer &#8594; corrected SFT example</p></li><li><p>bad retrieval &#8594; new chunking rule or index update</p></li><li><p>bad tool call &#8594; tool-use trajectory or schema constraint</p></li><li><p>bad judgment &#8594; rubric update or judge calibration item</p></li><li><p>unsafe response &#8594; safety eval and reward example</p></li><li><p>missing knowledge &#8594; source update and retrieval test</p></li><li><p>long-context miss &#8594; context benchmark replay</p></li><li><p>agent dead-end &#8594; environment transition test</p></li></ul><p>This is the practical version of the Verifier-Compiler Loop: observe traces and outcomes, evaluate with checks and judges, intervene safely, then compile expert decisions into a versioned behavioral contract. <a href="https://www.equationblog.com/p/the-verifiercompiler-loop-turning">[20]</a></p><p>Do not just patch the prompt. Classify the failure. Compile the correction. Replay the system. Promote only when the change improves the target behavior without regressing adjacent behavior.</p><h1>9. Data quality needs contracts</h1><p>A serious LLM or agent program needs data contracts. Not bureaucracy. Engineering contracts.</p><p>For every dataset:</p><ul><li><p>Dataset name:</p></li><li><p>Training stage:</p></li><li><p>Source classes:</p></li><li><p>License / usage constraints:</p></li><li><p>Freshness window:</p></li><li><p>Extraction method:</p></li><li><p>Filtering rules:</p></li><li><p>Dedup method:</p></li><li><p>PII / policy filters:</p></li><li><p>Contamination checks:</p></li><li><p>Capability tags:</p></li><li><p>Difficulty tags:</p></li><li><p>Mixture weight:</p></li><li><p>Verifier:</p></li><li><p>Judge model / rubric:</p></li><li><p>Known failure modes:</p></li><li><p>Evaluation suite:</p></li><li><p>Ablation evidence:</p></li><li><p>Owner:</p></li><li><p>Promotion criteria:</p></li><li><p>Rollback criteria:</p></li></ul><p>For every context policy:</p><ul><li><p>Context sources:</p></li><li><p>Retrieval policy:</p></li><li><p>Reranking objective:</p></li><li><p>Memory write policy:</p></li><li><p>Compression policy:</p></li><li><p>Isolation boundaries:</p></li><li><p>Citation requirements:</p></li><li><p>Tool-output handling:</p></li><li><p>Sensitive-data policy:</p></li><li><p>Staleness policy:</p></li><li><p>Replay evals:</p></li><li><p>Observed regressions:</p></li></ul><p>For every RL environment:</p><ul><li><p>Task distribution:</p></li><li><p>Environment interface:</p></li><li><p>Allowed tools:</p></li><li><p>State transition rules:</p></li><li><p>Reward components:</p></li><li><p>Verifier implementation:</p></li><li><p>Judge model and version:</p></li><li><p>Rubric:</p></li><li><p>Anti-hacking tests:</p></li><li><p>Timeouts and budgets:</p></li><li><p>Replay set:</p></li><li><p>Human escalation path:</p></li></ul><p>This is where production AI becomes engineering rather than prompt folklore.</p><h1>10. Conclusion</h1><p>The next wave of progress will not come only from larger models.</p><p>It will come from better data control.</p><ul><li><p>Better extraction.</p></li><li><p>Better cleaning.</p></li><li><p>Better deduplication.</p></li><li><p>Better filtering.</p></li><li><p>Better source mixtures.</p></li><li><p>Better ordering.</p></li><li><p>Better synthetic compilers.</p></li><li><p>Better long-context curricula.</p></li><li><p>Better context playbooks.</p></li><li><p>Better verifier-backed environments.</p></li><li><p>Better reward models.</p></li><li><p>Better replay systems.</p></li><li><p>Better governance.</p></li></ul><p>The teams that win will not be the teams with the largest pile of tokens.</p><p>They will be the teams that know which data changes behavior, why it changes behavior, how to measure that change, and how to keep improving it without corrupting the system.</p><p>Data is not just the input to training.</p><p>Data is the control surface.</p><h1>References</h1><p><a href="https://arxiv.org/abs/2601.03220">[1]</a> Epiplexity: Information Theory for Computationally Bounded Learners. <a href="https://arxiv.org/abs/2601.03220">https://arxiv.org/abs/2601.03220</a></p><p><a href="https://arxiv.org/html/2406.17557v1">[2]</a> FineWeb: Decanting the Web for the Finest Text Data at Scale. <a href="https://arxiv.org/html/2406.17557v1">https://arxiv.org/html/2406.17557v1</a></p><p><a href="https://machinelearning.apple.com/research/data-quality-illusion">[3]</a> Apple Machine Learning Research: The Data-Quality Illusion. <a href="https://machinelearning.apple.com/research/data-quality-illusion">https://machinelearning.apple.com/research/data-quality-illusion</a></p><p><a href="https://arxiv.org/html/2508.14444v2">[4]</a> NVIDIA Nemotron Nano 2 Technical Report. <a href="https://arxiv.org/html/2508.14444v2">https://arxiv.org/html/2508.14444v2</a></p><p><a href="https://arxiv.org/html/2505.09388v1">[5]</a> Qwen3 Technical Report. <a href="https://arxiv.org/html/2505.09388v1">https://arxiv.org/html/2505.09388v1</a></p><p><a href="https://arxiv.org/abs/2407.01492">[6]</a> RegMix: Data Mixture as Regression. <a href="https://arxiv.org/abs/2407.01492">https://arxiv.org/abs/2407.01492</a></p><p><a href="https://arxiv.org/abs/2406.11794">[7]</a> DataComp-LM: In Search of the Next Generation of Training Sets for Language Models. <a href="https://arxiv.org/abs/2406.11794">https://arxiv.org/abs/2406.11794</a></p><p><a href="https://arxiv.org/html/2501.12948v1">[8]</a> DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. <a href="https://arxiv.org/html/2501.12948v1">https://arxiv.org/html/2501.12948v1</a></p><p><a href="https://arxiv.org/html/2507.20534v2">[9]</a> Kimi K2 Technical Report. <a href="https://arxiv.org/html/2507.20534v2">https://arxiv.org/html/2507.20534v2</a></p><p><a href="https://github.com/NVIDIA-NeMo/Nemotron/blob/main/docs/nemotron/super3/README.md">[10]</a> NVIDIA Nemotron 3 Super Technical README. <a href="https://github.com/NVIDIA-NeMo/Nemotron/blob/main/docs/nemotron/super3/README.md">https://github.com/NVIDIA-NeMo/Nemotron/blob/main/docs/nemotron/super3/README.md</a></p><p><a href="https://arxiv.org/html/2510.04618v1">[11]</a> ACE: Agentic Context Engineering. <a href="https://arxiv.org/html/2510.04618v1">https://arxiv.org/html/2510.04618v1</a></p><p><a href="https://arxiv.org/html/2411.18947v1">[12]</a> ICLERB and RLRAIF: Learning to Retrieve In-Context Examples by Utility. <a href="https://arxiv.org/html/2411.18947v1">https://arxiv.org/html/2411.18947v1</a></p><p><a href="https://openreview.net/forum?id=kIoBbc76Sy">[13]</a> RULER: What&#8217;s the Real Context Size of Your Long-Context Language Models?. </p><p>https://openreview.net/forum?id=kIoBbc76Sy</p><p><a href="https://openreview.net/forum?id=0OshX1hiSa">[14]</a> NoLiMa: Long-Context Evaluation Beyond Literal Matching. </p><p>https://openreview.net/forum?id=0OshX1hiSa</p><p><a href="https://arxiv.org/abs/2412.15204">[15]</a> LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-Context Multitasks. <a href="https://arxiv.org/abs/2412.15204">https://arxiv.org/abs/2412.15204</a></p><p><a href="https://arxiv.org/html/2602.05892v2">[16]</a> ContextBench: A Benchmark for Measuring Context Engineering in Coding Agents. <a href="https://arxiv.org/html/2602.05892v2">https://arxiv.org/html/2602.05892v2</a></p><p><a href="https://arxiv.org/html/2406.14491v1">[17]</a> Instruction Pre-Training: Language Models are Supervised Multitask Learners. <a href="https://arxiv.org/html/2406.14491v1">https://arxiv.org/html/2406.14491v1</a></p><p><a href="https://arxiv.org/html/2405.03548v3">[18]</a> MAmmoTH2: Scaling Instructions from the Web. <a href="https://arxiv.org/html/2405.03548v3">https://arxiv.org/html/2405.03548v3</a></p><p><a href="https://arxiv.org/html/2504.15573v1">[19]</a> Web Reconstruction: Reconstructing Instruction Data from the Web. <a href="https://arxiv.org/html/2504.15573v1">https://arxiv.org/html/2504.15573v1</a></p><p><a href="https://www.equationblog.com/p/the-verifiercompiler-loop-turning">[20]</a> The Verifier-Compiler Loop. <a href="https://www.equationblog.com/p/the-verifiercompiler-loop-turning">https://www.equationblog.com/p/the-verifiercompiler-loop-turning</a></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[Updated workout routine - Part 3]]></title><description><![CDATA[4-week hybrid strength / power / agility block]]></description><link>https://www.equationblog.com/p/updated-workout-routine-part-3</link><guid isPermaLink="false">https://www.equationblog.com/p/updated-workout-routine-part-3</guid><dc:creator><![CDATA[Ruslan Belkin]]></dc:creator><pubDate>Sat, 09 May 2026 14:15:58 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!-MQk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0187b1d4-c470-4ffa-a463-efc198a6864f_1280x1280.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em>&#8220;Train to stimulate, not annihilate.&#8221; &#8212;Lee Haney</em></p><p>I&#8217;ve recently tweaked my workout schedule again.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>In the previous two posts I described the updated weight-lifting block and the updated functional block. The first one was more muscle / VO2max maintenance focused. The second one was more full-body power / functional movement focused.</p><p>This post is about a different idea: <strong>combining multiple styles of training in the same schedule</strong> &#8212; strength, hypertrophy, power, agility, combat-style conditioning, grip strength, yoga / mobility and VO2max work.</p><p>The goal is not to make the week random. The goal is to make it dense, but still sequenced.</p><p>The main constraint is not just muscular fatigue. It is nervous system fatigue. Muscles can often handle more work than attention, coordination and movement quality can handle. Power, agility, plyometrics, MMA / boxing and interval work are not just &#8220;muscle&#8221; work. They require coordination, speed, braking, balance, bracing and intent. When mental fatigue sets in, form starts getting sloppy before the muscles are truly done.</p><p>Because I work out first thing in the morning, this matters a lot. The workout cannot consume the whole mental budget for the day. I still need to be useful at work at 9am, 1pm and 5pm.</p><p>So this routine is built around a simple idea: enough stimulus to keep progressing, but not so much nervous system fatigue that the rest of the day becomes a tax.</p><p>As before, I work out first thing in the morning, with a 5:30 am wake-up, quick oral/face hygiene, pre-workout supplements, and a 10-15 min meditation.</p><p>Because the schedule is dense, I&#8217;m still skipping pre-workout foam rolling during the work week. I do try to compensate on the weekend with a quick 20 min foam-rolling session once a week, or, when possible, a professional massage every few weeks. If I had unlimited time, I&#8217;d keep the full pre-workout mobility work, but this is the realistic compromise.</p><p>I still use guided workouts, mostly <a href="https://www.beachbodyondemand.com/">Beachbody</a> / BODi historical programs, mainly for timing and pacing, and my week starts on Sundays.</p><p>The general structure: a main workout, followed by an elliptical segment, with optional add-ons for abs, The Challenge, pull-ups or chin-ups depending on the day. The base plan uses 30 min on Monday and Friday and 50 min on Sunday and Saturday; the full-cardio option adds 20 min on Monday and Friday. In slash totals such as 81 / 115, the first number excludes the optional add-on and the second includes it. Beast: Rock-a-Bye Beast (6 min) follows each elliptical session except Thursday in Weeks 1&#8211;3.</p><p>This is a 4-week block. Muscle building remains the primary goal, but athleticism is no longer separated into a completely different block. It is built into the same week.</p><p><strong>Abbreviations below</strong>:</p><p><strong>Body Beast</strong> = Body Beast / Beast Up / Deluxe family. In this block, that includes the Build, Bulk, Tempo, Beast Up, Beast: Abs and Beast: Abs Classic workouts. This is still the main hypertrophy engine of the schedule.</p><p><strong>P90X GN = P90X Generation Next; T30 = Tough Mudder T-MINUS 30; PO4 = The Power of 4; THE WORK = 6 Weeks of THE WORK; Hard Labor = A Week of Hard Labor; Rock-a-Bye Beast = the complete 6-min Beast bedtime stretch routine; S1 and L2 = the short- and long-interval elliptical protocols. All elliptical work is interval, steady, or easy as specified in the tables.</strong></p><p><strong>A note on the elliptical</strong>: I am using it intentionally, not as a compromise because I cannot think of something better. I have an old tendon injury from MMA, so high-volume running is not the best risk/reward trade for me. The elliptical gives me a close enough running-like stimulus for Zone 2 and interval work, but with much lower impact and less tendon irritation. It is not identical to running &#8212; the lack of impact and fixed path are part of the point &#8212; but it gets close enough for my current goals. <strong>The other advantage</strong> is that, with variable incline, the elliptical can <strong>hit the posterior side of the legs</strong> surprisingly well. Higher incline and more deliberate drive turn it into useful work for glutes, hamstrings and calves, not just generic &#8220;cardio.&#8221;</p><h2><strong>Week 1 - accumulation and continuity week</strong></h2><div class="latex-rendered" data-attrs="{&quot;persistentExpression&quot;:&quot;\\begin{array}{|c|l|l|c|c|} \\hline \\text{Day} &amp; \\text{Complete workout(s)} &amp; \\text{Elliptical base/full} &amp; \\text{Base total} &amp; \\text{Full total} \\\\ \\hline \\text{Sun} &amp; \\text{Build: Chest/Tris + Beast: Abs Classic} &amp; \\text{50/50m S1} &amp; \\text{117m} &amp; \\text{117m} \\\\ \\hline \\text{Mon} &amp; \\text{Build: Back/Bis} &amp; \\text{30/50m steady} &amp; \\text{87m} &amp; \\text{107m} \\\\ \\hline \\text{Tue} &amp; \\text{Beast Up: Legs + Challenge (opt.)} &amp; \\text{30/30m easy} &amp; \\text{81/115m} &amp; \\text{81/115m} \\\\ \\hline \\text{Wed} &amp; \\text{GN Cardio Boxing + PO4 Power of Core} &amp; \\text{30/30m steady} &amp; \\text{102m} &amp; \\text{102m} \\\\ \\hline \\text{Thu} &amp; \\text{Build: Shoulders; Bulk: Arms} &amp; \\text{30/30m L2} &amp; \\text{105m} &amp; \\text{105m} \\\\ \\hline \\text{Fri} &amp; \\text{GN Plyometrix + Pull Up (opt.)} &amp; \\text{30/50m steady} &amp; \\text{77/88m} &amp; \\text{97/108m} \\\\ \\hline \\text{Sat} &amp; \\text{PO4 Don't Skip Yoga + Chin Up (opt.)} &amp; \\text{50/50m steady} &amp; \\text{124/136m} &amp; \\text{124/136m} \\\\ \\hline \\end{array}&quot;,&quot;id&quot;:&quot;YYENDXGXER&quot;}" data-component-name="LatexBlockToDOM"></div><p>Week 1 is the classic foundation week. There is a normal Beast chest / back / shoulders structure, but the lower body and athletic work are already present.</p><p>Tuesday is dense even before The Challenge, so The Challenge remains optional and the elliptical stays easy for 30 min. On Thursday, the L2 elliptical sits between Shoulders and Arms, with no stretch.</p><p>Friday is the first athletic / power day. Friday Pull Up and Saturday Chin Up are optional; I choose at most one weekend pull finisher when fully recovered.</p><h2><strong>Week 2 - athletic leg variation week</strong></h2><div class="latex-rendered" data-attrs="{&quot;persistentExpression&quot;:&quot;\\begin{array}{|c|l|l|c|c|} \\hline \\text{Day} &amp; \\text{Complete workout(s)} &amp; \\text{Elliptical base/full} &amp; \\text{Base total} &amp; \\text{Full total} \\\\ \\hline \\text{Sun} &amp; \\text{Beast Up: Chest/Shoulders/Tris + Beast: Abs} &amp; \\text{50/50m S1} &amp; \\text{125m} &amp; \\text{125m} \\\\ \\hline \\text{Mon} &amp; \\text{Tempo: Back/Bis} &amp; \\text{30/50m steady} &amp; \\text{87m} &amp; \\text{107m} \\\\ \\hline \\text{Tue} &amp; \\text{THE WORK: Wk1 Legs + Challenge (opt.)} &amp; \\text{30/30m easy} &amp; \\text{78/112m} &amp; \\text{78/112m} \\\\ \\hline \\text{Wed} &amp; \\text{P90X3 MMX + GN Core Circuit} &amp; \\text{30/30m steady} &amp; \\text{107m} &amp; \\text{107m} \\\\ \\hline \\text{Thu} &amp; \\text{Bulk: Shoulders; Bulk: Arms} &amp; \\text{30/30m L2} &amp; \\text{102m} &amp; \\text{102m} \\\\ \\hline \\text{Fri} &amp; \\text{GN Speed/Agility + Pull Up (opt.)} &amp; \\text{30/50m steady} &amp; \\text{75/86m} &amp; \\text{95/106m} \\\\ \\hline \\text{Sat} &amp; \\text{PO4 Primal Flow + Chin Up (opt.)} &amp; \\text{50/50m steady} &amp; \\text{112/124m} &amp; \\text{112/124m} \\\\ \\hline \\end{array}&quot;,&quot;id&quot;:&quot;EJFHUCUYCP&quot;}" data-component-name="LatexBlockToDOM"></div><p>Week 2 shifts lower-body work to 6 Weeks of THE WORK: Wk1 Legs, an athletic strength session rather than a classic bodybuilding leg day. Saturday uses PO4 Primal Flow as the mobility anchor.</p><p>Friday also becomes a true agility day. This is one of the reasons the block feels different from a standard bodybuilding plan &#8212; the muscle stimulus is still there, but the movement quality matters just as much.</p><p>The upper body still gets enough direct work: Beast Up push, Tempo Back/Bis, shoulders and arms. The Challenge, Pull Up and Chin Up are optional and depend on recovery.</p><h2><strong>Week 3 - peak overload week</strong></h2><div class="latex-rendered" data-attrs="{&quot;persistentExpression&quot;:&quot;\\begin{array}{|c|l|l|c|c|} \\hline \\text{Day} &amp; \\text{Complete workout(s)} &amp; \\text{Elliptical base/full} &amp; \\text{Base total} &amp; \\text{Full total} \\\\ \\hline \\text{Sun} &amp; \\text{Tempo: Chest/Tris} &amp; \\text{50/50m S1} &amp; \\text{109m} &amp; \\text{109m} \\\\ \\hline \\text{Mon} &amp; \\text{Build: Back/Bis} &amp; \\text{30/50m steady} &amp; \\text{87m} &amp; \\text{107m} \\\\ \\hline \\text{Tue} &amp; \\text{Build: Legs + Challenge (opt.)} &amp; \\text{30/30m easy} &amp; \\text{75/109m} &amp; \\text{75/109m} \\\\ \\hline \\text{Wed} &amp; \\text{GN Cardio Boxing + PO4 Power of Core} &amp; \\text{30/30m steady} &amp; \\text{102m} &amp; \\text{102m} \\\\ \\hline \\text{Thu} &amp; \\text{Build: Shoulders; Bulk: Arms} &amp; \\text{30/30m steady} &amp; \\text{105m} &amp; \\text{105m} \\\\ \\hline \\text{Fri} &amp; \\text{GN Full Body Power} &amp; \\text{30/50m steady} &amp; \\text{84m} &amp; \\text{104m} \\\\ \\hline \\text{Sat} &amp; \\text{PO4 Power and Strength Yoga + Chin Up} &amp; \\text{50/50m steady} &amp; \\text{134m} &amp; \\text{134m} \\\\ \\hline \\end{array}&quot;,&quot;id&quot;:&quot;YHBIHBIFZL&quot;}" data-component-name="LatexBlockToDOM"></div><p>Week 3 is the hardest week on paper.</p><p>Tempo Chest/Tris, Build Back/Bis, Build Legs and Full Body Power all land in the same week. Tempo Chest/Tris replaces Bulk Chest plus Abs Classic; its long tempo sets already include core and direct triceps work. This is where form quality matters more than chasing extra load.</p><p>Full Body Power is exactly the kind of workout where the nervous system gets tired faster than the muscles. You may still be able to move the weight, but the movement is no longer crisp. That is usually the signal to reduce weight or stop increasing intensity.</p><p>The goal here is not to survive the workout. The goal is to create a strong enough training stimulus without borrowing too much from the rest of the day. Thursday&#8217;s elliptical stays steady rather than L2 to protect Friday power. Friday has no Pull Up finisher; Saturday&#8217;s Chin Up replaces Ab Attack and stays submaximal before the next chest-and-back day.</p><h2><strong>Week 4 - Hard Labor strength week</strong></h2><div class="latex-rendered" data-attrs="{&quot;persistentExpression&quot;:&quot;\\begin{array}{|c|l|l|c|c|} \\hline \\text{Day} &amp; \\text{Complete workout(s)} &amp; \\text{Elliptical base/full} &amp; \\text{Base total} &amp; \\text{Full total} \\\\ \\hline \\text{Sun} &amp; \\text{Hard Labor: Chest } \\&amp; \\text{ Back + Beast: Abs} &amp; \\text{50/50m steady} &amp; \\text{114m} &amp; \\text{114m} \\\\ \\hline \\text{Mon} &amp; \\text{Hard Labor: Legs} &amp; \\text{30/50m steady} &amp; \\text{81m} &amp; \\text{101m} \\\\ \\hline \\text{Tue} &amp; \\text{Hard Labor: Shoulders } \\&amp; \\text{ Arms} &amp; \\text{30/30m steady} &amp; \\text{80m} &amp; \\text{80m} \\\\ \\hline \\text{Wed} &amp; \\text{P90X3 MMX + GN Core Circuit} &amp; \\text{30/30m steady} &amp; \\text{107m} &amp; \\text{107m} \\\\ \\hline \\text{Thu} &amp; \\text{Hard Labor: Chest } \\&amp; \\text{ Back} &amp; \\text{30/30m steady} &amp; \\text{81m} &amp; \\text{81m} \\\\ \\hline \\text{Fri} &amp; \\text{GN Accel/Decel + Pull Up (opt.)} &amp; \\text{30/50m steady} &amp; \\text{77/88m} &amp; \\text{97/108m} \\\\ \\hline \\text{Sat} &amp; \\text{PO4 Primal Flow + Chin Up (opt.)} &amp; \\text{50/50m steady} &amp; \\text{112/124m} &amp; \\text{112/124m} \\\\ \\hline \\end{array}&quot;,&quot;id&quot;:&quot;PGFFQEIGIL&quot;}" data-component-name="LatexBlockToDOM"></div><p>Week 4 is a Hard Labor strength week rather than a deload.</p><p>A Week of Hard Labor supplies Chest &amp; Back on Sunday and Thursday, Legs on Monday, and Shoulders &amp; Arms on Tuesday. The second Chest &amp; Back exposure is why the Friday Pull Up and Saturday Chin Up finishers remain optional.</p><p>Acceleration / Deceleration on Friday remains a good fit. It is about braking, redirection and movement precision, so it is useful but highly form-dependent. Pull Up is optional after Thursday&#8217;s Chest &amp; Back work.</p><p>All Week 4 elliptical work is steady, and Rock-a-Bye Beast follows every session. Primal Flow anchors Saturday, with Chin Up optional. In Week 5, The Challenge may return as an optional chest/back add-on after Monday leg work; it is not counted in this block.</p><h2><strong>Why this structure</strong></h2><p>In Weeks 1&#8211;3 there are four hypertrophy-led days: push, pull, legs, and shoulders / arms. Week 4 shifts to the Hard Labor split, including two Chest &amp; Back exposures.</p><p>There is one combat / core bridge day per week: Cardio Boxing or MMX, followed by Gen Next Core Circuit or PO4: The Power of Core.</p><p><strong>Note:</strong> That slot is somewhat flexible. Sometimes I follow the prescribed P90X GN Cardio Boxing or P90X3 MMX routine exactly, especially when I feel unimaginative and just want a guided cardio workout. Other times, I use the same slot to practice specific Krav Maga techniques and combinations from prior training: footwork, strikes, kicks, defensive movements, and transitions. The point is not the exact brand-name workout. The point is combat-style conditioning, coordination, rotation, trunk control and movement under fatigue.</p><p>There is one athletic / power day per week: Plyometrix, Speed and Agility, Full Body Power, or Acceleration / Deceleration.</p><p>There is one full yoga anchor per week.</p><p>This may be the opposite for most people, but yoga is one of the hardest routines for me mentally. Not because it is the most metabolically demanding, but because it requires patience, stillness, balance and sustained attention. That is exactly why I put it on Saturday. I do not want to start a dense workday by forcing myself through the routine that costs me the most mentally. Saturday gives me more room to do it properly.</p><p>That is the key difference between &#8220;hard&#8221; and &#8220;expensive.&#8221; Some workouts are hard muscularly. Some workouts are expensive neurologically. I am trying to avoid too many expensive sessions stacked together.</p><p>Tuesday is intentionally controlled in Weeks 1&#8211;3. Leg work comes first, The Challenge is optional, and the elliptical stays easy for 30 min. In Week 4, leg day moves to Monday and Tuesday becomes Hard Labor Shoulders &amp; Arms.</p><p>Wednesday is the combat / core bridge day, followed by a steady 30-min elliptical and the 6-min stretch.</p><p>Friday is the main athletic day. Pull Up is optional in Weeks 1, 2 and 4, and removed in Week 3 to protect Full Body Power and Saturday&#8217;s Chin Up.</p><p>Saturday is the recovery anchor, even though it is not a rest day.</p><h2><strong>P90X3: The Challenge</strong></h2><p>P90X3: The Challenge consists of four blocks. Each block has two sets of pull-ups followed by push-ups, with different variations.</p><p>My current numbers are 15 pull-ups and 40 push-ups per set. That&#8217;s 120 pull-ups and 320 push-ups per workout.</p><p>There&#8217;s also a burnout at the end &#8212; 2 pull-ups and 4 push-ups, no rest, for 6 sets &#8212; so add another 12 pull-ups and 24 push-ups to the total.</p><p>In Weeks 1&#8211;3, The Challenge is an optional Tuesday add-on after leg work. It keeps upper-body endurance and grip strength available without making recovery from Back/Bis mandatory. It is absent in Week 4.</p><p>It is one reason Tuesday&#8217;s elliptical stays easy and limited to 30 min. In Week 5, it may return as an optional Monday add-on after leg work.</p><h2><strong>T-MINUS 30 pull-up and chin-up routines</strong></h2><p>The T30 Pull Up and Chin Up routines are short, roughly 11-12 min.</p><p>They are not meant to replace the main strength work. They are there as grip-strength and pulling-practice finishers.</p><p>My ladders are usually 2-4-8-16. That gives enough volume to matter, but the routines are short enough that I can still treat them as add-ons.</p><p>Friday Pull Up is optional in Weeks 1, 2 and 4, and omitted in Week 3. Saturday Chin Up is optional in Weeks 1, 2 and 4; in Week 3 it is included, submaximal, and replaces Ab Attack.</p><p>When optional, I drop these first if time is short or grip and pulling muscles are not fully recovered. In Weeks 1, 2 and 4 I choose at most one weekend pull finisher.</p><h2><strong>Intervals</strong></h2><p>L2 / long intervals: warm-up, 3 &#215; 5-min hard efforts with 2-min easy segments in between, then cool-down.</p><p>S1 / short intervals: warm-up, 9 &#215; 1-min hard efforts with 1-min easy segments in between, then cool-down.</p><p>All interval and steady/easy cardio in this block is done on the elliptical. For intervals, I adjust resistance and pace to create the hard segments. For steady work, I mostly use incline and resistance to control effort while biasing the posterior chain.</p><p>Sunday uses S1. In Weeks 1&#8211;3, Thursday&#8217;s elliptical sits between Shoulders and Arms: L2 in Weeks 1&#8211;2 and steady in Week 3 to protect Friday power. All Week 4 elliptical work is steady. Other sessions are steady or easy as shown in the tables. Monday and Friday use 30 min in the base plan or 50 min in the full-cardio option.</p><p>Rock-a-Bye Beast, the complete 6-min Beast bedtime stretch routine, follows every elliptical session except Thursday in Weeks 1&#8211;3. It follows every session in Week 4.</p><p>When time is short, separately programmed abs and optional add-ons go first. Then I reduce a 50-min full-cardio session to the 30-min base. If something still has to go, I preserve the main workout rather than rush the primary movement work.</p><h2><strong>Notes on fatigue</strong></h2><p>The base block totals 2,727 min and averages about 97 min per day. The full-cardio option totals 2,887 min and averages about 103 min. Weekly base/full totals are 693/733, 686/726, 696/736 and 652/692 min. Base days range from roughly 75 to 134 min; the longest optional version is about 136 min. All optional strength add-ons would contribute another 171 min across the block.</p><p>That is a lot. The only reason this works is that not every minute has the same recovery cost.</p><p>Heavy lifting, power moves, agility work, boxing / MMA, intervals and high-rep pull-up work all hit differently. The schedule is designed so that the hard work is spread out, and the most coordination-heavy work does not sit on top of maximal lifting every day.</p><p>For me, the nervous system fatigue is the main constraint. If I train too hard in the morning, the problem is not just soreness. The problem is reduced focus later in the day.</p><p>That is a bad trade.</p><p>The point of morning training is to improve the body and improve the day, not to win the workout and lose the workday.</p><h2><strong>Form</strong></h2><p>The form is critical in this block, especially once mental fatigue sets in.</p><p>A separate back-safety exception applies in Build Back/Bis only: I replace its normal deadlift set (15/12/8/8) with weighted floor glute bridges, 4 &#215; 12&#8211;15 with a 2-sec hold and a pain-free load. Tempo Back/Bis has no deadlift and is completed as programmed. Build Legs straight-leg deadlifts and the Hard Labor Legs stiff-leg hinge also remain as programmed. More broadly, form matters even more in power and agility work, where sloppy acceleration, deceleration or plyometrics carry a higher cost.</p><p>Seemingly lower weights can be deceptive. Do not raise the weights unless you can complete the exercise with nearly perfect form. The risk of injury with complex power moves goes up considerably.</p><p>Note: doing exercises with a perfect form is an easy way to enhance effectiveness with lower weights and much reduced risk of injury.</p><p>For tempo workouts, I am not chasing weight. I am chasing tension and control.</p><p>For power workouts, I am not chasing fatigue. I am chasing crisp movement.</p><p>For intervals, I am not trying to turn every segment into a max-effort suffer fest. The goal is repeatable quality.</p><p>As always &#8212; if the last set is too easy, add a few extra reps; next time, increase the weight. The key to performance is sufficient stimulus.</p><h2><strong>When traveling</strong></h2><p>When traveling and the hotel gym with weights is available, the strength portions can mostly continue, depending on space and how crowded the place is.</p><p>If no gym is available, I normally switch to bodyweight conditioning for the duration of travel. Insanity Max 30 Month Two workouts are still a decent fallback. They are not the same stimulus, but they are good enough for maintenance.</p><p>If there is a pull-up bar, I can keep some version of The Challenge or the T30 pull-up / chin-up work. If not, I don&#8217;t overthink it.</p><p>Travel workouts are for maintenance, not perfect replication.</p><p>Again - this is what I do. This is not a recommendation that everyone should train 7 days per week or spend roughly 1.5&#8211;2 hours per day working out.</p><p>Consistency and a purposeful plan are keys and there are many other ways to stay in shape.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[Reliability knobs for agents]]></title><description><![CDATA[Spring 2026 update]]></description><link>https://www.equationblog.com/p/reliability-knobs-for-agents</link><guid isPermaLink="false">https://www.equationblog.com/p/reliability-knobs-for-agents</guid><dc:creator><![CDATA[Ruslan Belkin]]></dc:creator><pubDate>Mon, 30 Mar 2026 14:15:11 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!-MQk!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0187b1d4-c470-4ffa-a463-efc198a6864f_1280x1280.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>The agent conversation is as noisy as it has ever been:</p><ul><li><p>One camp says base models are now good enough, just give them tools.</p></li><li><p>Another (the moi is mostly in. it) says evaluation is the bottleneck, just build better judges.</p></li></ul><p>While I would assign greater weight to the second camp, they are both directionally right and still too coarse.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>The durable gains are showing up somewhere less glamorous and much more useful: new reliability knobs. Not one magical agent architecture, but a stack of control surfaces that help systems lose less intent, preserve more capability, remember more of the right state, and fail in ways teams can actually inspect and correct.</p><p>Several papers of note along these problems:</p><h2><strong>1. PPS: make intent explicit before the model starts guessing</strong></h2><p>Natural-language prompts have a hidden failure mode: intent transmission loss. The user knows what they mean; the model only sees the compressed, underspecified surface form. The PPS paper attacks that directly. Across 60 tasks, 3 domains, 3 models, and 540 generations, natural-language-rendered PPS outperformed both simple prompts and raw JSON on goal alignment. The gains were strongest in ambiguous business tasks, weaker in technical tasks, and actually reversed in low-ambiguity travel planning. That last part matters. It means structured prompting is not magic; it is most valuable when the user&#8217;s objective is fuzzy enough to be misread. The preliminary survey result is also practical: fewer follow-up rounds, from 3.33 to 1.13 on average.</p><p><strong>Why we should care</strong>: a lot of agent failure is downstream of an upstream ambiguity. If the goal, audience, constraints, tone, and success criteria are hazy at turn one, the rest of the trajectory is just confident error propagation. The other useful lesson here is that raw structure is not enough. In the study, rendered PPS beat raw JSON. So the knob is not &#8220;add schema everywhere.&#8221; It is &#8220;make intent explicit in a form the model can actually use.&#8221;</p><p>Paper: <a href="https://arxiv.org/abs/2603.18976">https://arxiv.org/abs/2603.18976</a></p><h2><strong>2. Prompt repetition: a surprisingly cheap robustness hack</strong></h2><p>This one sounds silly until you think about what causal attention is doing.</p><p>The core idea is simple: when reasoning is not enabled, repeat the prompt. The paper shows that prompt repetition improved accuracy across popular models without increasing output length or materially increasing latency in most settings. On their experiments, it won 47 of 70 benchmark-model combinations with 0 losses when reasoning was disabled. The gains were especially strong when the prompt order was hostile to the model, such as options-first multiple choice, and on custom tasks like NameIndex and MiddleMatch.</p><p><strong>Why we should care</strong>: not every model call inside an agent should be a long reasoning trace. Some calls are cheap subroutines: route this, extract that, normalize this, draft tool arguments, re-check the user constraint. For those non-reasoning calls, prompt repetition looks like a genuinely useful default knob. It is not a substitute for reasoning, and it is not free on arbitrarily long prompts, but for short operational subcalls it is exactly the kind of low-cost robustness trick teams tend to underrate.</p><p>Paper: <a href="https://ar5iv.labs.arxiv.org/html/2512.14982v1">https://ar5iv.labs.arxiv.org/html/2512.14982v1</a></p><h2><strong>3. GLM-5: do not buy agentic RL by deleting earlier skills</strong></h2><p>Sequential post-training has a nasty habit: each new stage can quietly sand down the thing the previous stage got good at.</p><p>GLM-5 is interesting partly because it says that out loud. Their pipeline runs sequential RL stages for reasoning, then agentic behavior, then general helpfulness, and then uses on-policy cross-stage distillation as a final refinement to recover skills from earlier stages. Previous stage checkpoints become teachers; the final pass is meant to stop the classic &#8220;more agentic, less sharp&#8221; tradeoff from becoming acceptable collateral damage. On their reported benchmarks, GLM-5 posts about a 20% average gain over GLM-4.7 across agentic, reasoning, and coding tasks, including 77.8 on SWE-bench Verified.</p><p>The more durable lesson is even lower-level than that. The paper is refreshingly explicit that agentic RL stability lives in systems details as much as in objectives. They switched to a deterministic top-k operator because nondeterministic sparse-attention selection caused sharp RL degradation, froze the indexer during RL for stability, and emphasized token-in-token-out handling so the trainer learns on exactly the same token stream produced by the rollout engine. That is the kind of detail that separates &#8220;agent demo&#8221; from &#8220;agent training system.&#8221;</p><p><strong>Why we should care</strong>: a lot of agent improvement work still acts as if new capabilities can simply be stacked. In practice, they interfere. If post-training adds planning but dulls reasoning, or adds autonomy but destabilizes the learning loop, you have not really improved the agent. You just moved the failure somewhere harder to notice.</p><p>Paper: <a href="https://ar5iv.labs.arxiv.org/html/2602.15763v2">https://ar5iv.labs.arxiv.org/html/2602.15763v2</a></p><h2><strong>4. FullStack-Agent: round-trip the artifact, then test the hidden surfaces</strong></h2><p>I like this one because it attacks a very real agent failure mode: the frontend looks right, the demo works, and the backend is still fake.</p><p>FullStack-Agent combines three ideas. First, a multi-agent development workflow with specialized debugging tools for frontend and backend work. Second, FullStack-Bench, which evaluates not just frontend behavior but backend APIs and database state as well. Third, Repository Back-Translation, which converts existing real-world repositories into agent trajectories the model can learn from. The benchmark itself is notable: 101 instructions, 647 frontend tests, 604 backend tests, and 389 database tests. Even better, frontend success is not counted unless the required database interaction is real. That is exactly the kind of hidden-surface check agent evaluation needs more of.</p><p>The results are strong, but the more interesting pattern is the training and evaluation shape. FullStack-Dev with a Qwen backbone reached 64.7 frontend, 77.8 backend, and 77.9 database accuracy, while FullStack-Learn improved a 30B model through self-improvement using repository back-translation and augmentation. The debugging tools also mattered a lot: removing the backend debugging tool increased average backend iterations from 74.9 to 115.5. That is not just a model story. It is a workflow design story.</p><p><strong>Why we should care</strong>: reliable coding agents need falsifiable artifacts. A useful practical extension of this idea is round-tripping: code to spec, spec back to code, compare the two, and inspect the mismatch. That creates a verifier surface instead of treating the codebase as one opaque blob. More broadly, the paper is a reminder that real artifacts and real tests are better teachers than synthetic vibes.</p><p>Paper: <a href="https://arxiv.org/html/2602.03798v1">https://arxiv.org/html/2602.03798v1</a></p><h2><strong>5. MSA: memory should be part of the model, not a retrieval afterthought</strong></h2><p>Agents with long histories do not just need more context. They need memory that stays usable when the history becomes absurd.</p><p>MSA pushes that idea hard. The paper proposes an end-to-end trainable memory framework with sparse attention, document-wise RoPE, KV compression, and a Memory Parallel inference path. The headline is the kind of number people usually ignore until it becomes operationally relevant: less than 9% degradation while scaling from 16K to 100M tokens, with 100M-token inference on 2xA800 GPUs. The other important piece is Memory Interleave, which alternates retrieval, context expansion, and generation so the model can reason across scattered memory segments instead of just pulling one flat chunk and hoping.</p><p><strong>Why we should care</strong>: a lot of current agent memory stacks are really retrieval pipelines wearing a memory costume. That works until the task needs long-range consistency, multi-hop evidence integration, or stable persona/state over time. MSA is interesting because it tries to make memory intrinsic and differentiable rather than bolted on. The real caveat is operational: the current setup still relies on offline pre-encoding of the corpus. So it is not a universal replacement for dynamic knowledge systems yet. But as a direction, it is much closer to agent memory than &#8220;just add bigger RAG.&#8221;</p><p>Paper: <a href="https://arxiv.org/abs/2603.23516">https://arxiv.org/abs/2603.23516</a></p><h2><strong>6. Verifier&#8211;compiler loops: verification is becoming its own stack</strong></h2><p>This is the one I keep coming back to.</p><p>The core production fact is ugly and simple: long workflows multiply small defects. In the verifier&#8211;compiler loop framing, a 1% failure rate across 100 steps leaves only about 36.6% end-to-end success. Even 0.1% per-step failure still leaves only about 90.5%. That is the march-of-nines problem. The implication is that agent reliability is not mainly a prompt problem. It is an error-correction problem. The system needs to observe the episode, judge it against institutional standards, intervene conservatively, replay changes before release, and keep durable evidence of what changed and why. That is also why the distinction between execution knowledge and institutional judgment matters: the agent can know the facts and still fail the organization.</p><p>Recent judge work mostly points in the same direction. JudgeBench shows hard evaluator tasks are genuinely hard, with strong models like GPT-4o only slightly above random on some challenging judge settings. RewardBench 2 makes reward evaluation meaningfully harder than RewardBench 1 and emphasizes correlation with downstream use. DeepSeek&#8217;s GRM/SPCT line is also important because it argues that reward modeling itself can scale with more inference compute through principle generation, critique, and voting, not just with bigger training runs.</p><p>But the field is also getting more honest about calibration. Evaluative Fingerprints found near-zero inter-judge agreement while also showing that judges are individually stable enough to be fingerprinted from their rubric behavior. In other words: they are not random, they are systematically different. Separate work on LLM-as-a-judge reporting shows that evaluator bias and uncertainty should be corrected statistically, not hand-waved. On user simulation, the news is similarly mixed: SimulatorArena suggests profile-conditioned simulators can track human judgments reasonably well on some tasks, but Lost in Simulation shows simulator choice can move measured success rates by up to 9 points and systematically miscalibrate difficulty.</p><p><strong>Why we should care</strong>: one judge score is not a control system. High-reliability agents are going to need judge stacks, not judge monocultures: crisp gates for obvious defects, stronger reasoning judges for nuance, replay before release, disagreement review for hard cases, and humans on the highest-risk boundaries. Simulation will help widen coverage, but only if it is continuously calibrated against real traces.</p><p>Blog: <a href="http://www.equationblog.com/p/the-verifiercompiler-loop-turning">https://www.equationblog.com/p/the-verifiercompiler-loop-turning</a></p><h2><strong>7. IndexCache: systems work is reliability work too</strong></h2><p>This one is more infrastructure than alignment, but it belongs in the same conversation.</p><p>IndexCache starts from a simple observation: in sparse attention, adjacent layers often choose very similar top-k token sets. So instead of recomputing the indexer at every layer, reuse it across layers. On the reported results, that removes up to 75% of indexer computation with negligible quality loss, while reaching 1.82x prefill speedup and 1.48x decode speedup at 200K context. The paper also reports 70&#8211;100% top-k overlap across adjacent layers, which is the structural reason the trick works.</p><p><strong>Why we should care</strong>: efficiency is not separate from reliability. Every unit of inference cost you remove from the serving path can be reinvested into something reliability-shaped: longer context, more retrieval, more search, more verifier passes, more replay budget, or simply lower latency at the same control quality. That is why inference-side engineering keeps mattering more than people think.</p><p>Paper: <a href="https://arxiv.org/html/2603.12201v1">https://arxiv.org/html/2603.12201v1</a></p><h2><strong>The connective tissue</strong></h2><p>If I had to compress the direction into one line, it is this: <em><strong>reliable agents are becoming layered control systems</strong>.</em></p><p>Structured intent reduces loss before the trajectory begins. Prompt repetition stabilizes cheap non-reasoning subcalls. Post-training methods like cross-stage distillation try to make new capabilities additive instead of destructive. Artifact-grounded training and hidden-surface testing make agent outputs more falsifiable. Long-memory work tries to decouple memory capacity from reasoning quality. Judge research is forcing evaluation to become calibrated, replayable, and auditable. Systems work buys the budget to do more of all of it in real time.</p><p>Just a growing stack of knobs that make agent behavior narrower, more inspectable, and a little less mysterious week over week.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[The Verifier–Compiler Loop: Turning Human Preferences into Production Agent Judgment]]></title><description><![CDATA[Why verifier&#8211;compiler loops matter more than another prompt patch]]></description><link>https://www.equationblog.com/p/the-verifiercompiler-loop-turning</link><guid isPermaLink="false">https://www.equationblog.com/p/the-verifiercompiler-loop-turning</guid><dc:creator><![CDATA[Ruslan Belkin]]></dc:creator><pubDate>Mon, 09 Mar 2026 14:14:40 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!31Us!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61ef6284-6a40-4450-b44b-f2e768be2860_1561x820.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Agents do not usually fail in production because a prompt suddenly stopped working. They fail because a workflow that looked 98% fine in isolation turns into 30 turns, six tool calls, two handoffs, a compliance boundary, and a frustrated human on the other side.</p><p>That is the march-of-nines problem (as originally described by Andrey Karpathy). In a long workflow, tiny per-step defects compound into very large end-to-end losses. At 100 steps, a 1% failure rate at each step leaves only about 36.6% end-to-end success; even 0.1% still leaves only about 90.5%. For customer service, regulated operations, and multi-agent workflows, that gap is the difference between a promising demo and a system the business can trust.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>The practical implication is that production reliability should be treated as an error-correction problem. Better base models help, but they are only part of the story. The system also needs a way to observe what happened, judge it against the organization&#8217;s standards, intervene safely when necessary, replay changes before release, and keep durable evidence of what changed and why.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!31Us!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61ef6284-6a40-4450-b44b-f2e768be2860_1561x820.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!31Us!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61ef6284-6a40-4450-b44b-f2e768be2860_1561x820.png 424w, https://substackcdn.com/image/fetch/$s_!31Us!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61ef6284-6a40-4450-b44b-f2e768be2860_1561x820.png 848w, https://substackcdn.com/image/fetch/$s_!31Us!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61ef6284-6a40-4450-b44b-f2e768be2860_1561x820.png 1272w, https://substackcdn.com/image/fetch/$s_!31Us!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61ef6284-6a40-4450-b44b-f2e768be2860_1561x820.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!31Us!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61ef6284-6a40-4450-b44b-f2e768be2860_1561x820.png" width="1456" height="765" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/61ef6284-6a40-4450-b44b-f2e768be2860_1561x820.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:765,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!31Us!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61ef6284-6a40-4450-b44b-f2e768be2860_1561x820.png 424w, https://substackcdn.com/image/fetch/$s_!31Us!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61ef6284-6a40-4450-b44b-f2e768be2860_1561x820.png 848w, https://substackcdn.com/image/fetch/$s_!31Us!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61ef6284-6a40-4450-b44b-f2e768be2860_1561x820.png 1272w, https://substackcdn.com/image/fetch/$s_!31Us!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61ef6284-6a40-4450-b44b-f2e768be2860_1561x820.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p style="text-align: center;"><em>Figure 1. In long workflows, small per-step errors compound into large end-to-end losses.</em></p><p style="text-align: center;">Reliability should come from continuously catching, correcting, compiling, and proving small errors &#8212; not from hoping one prompt patch will cover every edge case.</p><p><strong>Production failures are not only knowledge failures</strong></p><p>A surprisingly large share of production failures are not missing-fact failures. The agent may know the product terms, retrieve the right document, or call the right tool and still fail the institution.</p><p>Four surfaces matter in practice: missing knowledge, institutional judgment, user affect, and evidence. A response can be factually correct but still violate policy, choose the wrong level of certainty, worsen the user&#8217;s emotional trajectory, or leave the team unable to reconstruct what happened well enough to approve a safe fix.</p><ul><li><p><strong>Missing knowledge: </strong>The agent lacks a fact, a tool result, or an updated policy exception.</p></li><li><p><strong>Judgment misalignment: </strong>The facts are present, but the trade-off between speed, certainty, empathy, policy, or escalation is wrong.</p></li><li><p><strong>Affect regression: </strong>The reply is technically valid but increases frustration, distrust, or confusion.</p></li><li><p><strong>Evidence gap: </strong>The team cannot replay the episode, inspect the rewrite, or approve the next release with confidence.</p><p></p></li></ul><p>A banking example makes the distinction concrete. An assistant can know the product, know the account state, and still reply in a way that is too dismissive, too certain, or too slow to escalate.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!3Gry!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03d3aa13-f0e9-4d50-8e18-bafa20b04819_944x496.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!3Gry!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03d3aa13-f0e9-4d50-8e18-bafa20b04819_944x496.png 424w, https://substackcdn.com/image/fetch/$s_!3Gry!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03d3aa13-f0e9-4d50-8e18-bafa20b04819_944x496.png 848w, https://substackcdn.com/image/fetch/$s_!3Gry!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03d3aa13-f0e9-4d50-8e18-bafa20b04819_944x496.png 1272w, https://substackcdn.com/image/fetch/$s_!3Gry!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03d3aa13-f0e9-4d50-8e18-bafa20b04819_944x496.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!3Gry!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03d3aa13-f0e9-4d50-8e18-bafa20b04819_944x496.png" width="944" height="496" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/03d3aa13-f0e9-4d50-8e18-bafa20b04819_944x496.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:496,&quot;width&quot;:944,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:122465,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.equationblog.com/i/190227900?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03d3aa13-f0e9-4d50-8e18-bafa20b04819_944x496.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!3Gry!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03d3aa13-f0e9-4d50-8e18-bafa20b04819_944x496.png 424w, https://substackcdn.com/image/fetch/$s_!3Gry!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03d3aa13-f0e9-4d50-8e18-bafa20b04819_944x496.png 848w, https://substackcdn.com/image/fetch/$s_!3Gry!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03d3aa13-f0e9-4d50-8e18-bafa20b04819_944x496.png 1272w, https://substackcdn.com/image/fetch/$s_!3Gry!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03d3aa13-f0e9-4d50-8e18-bafa20b04819_944x496.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p style="text-align: center;"><em>Figure 2. A customer-facing answer can know the facts and still miss the institution&#8217;s judgment on policy, style, affect, or action.</em></p><p><strong>Judgment should be explicit</strong></p><p>One way to make this tractable is to keep execution knowledge and institutional judgment separate, but versioned together. Skills describe how the work gets done: task-specific success criteria, approved procedures, tool permissions, and required evidence. Judgment describes how the organization wants that work done: risk boundaries, policy rules, quality bars, tone, trade-offs, escalation behavior, and release gates.</p><p>That separation matters because business policy changes faster than workflow logic. If a new escalation rule or compliance boundary requires rewriting every skill from scratch, the system becomes brittle. If judgment is explicit, global changes can be compiled once and enforced across many workflows.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!tGvT!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13cb7838-06b9-42c8-bfaf-46961435e6b1_1561x820.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!tGvT!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13cb7838-06b9-42c8-bfaf-46961435e6b1_1561x820.png 424w, https://substackcdn.com/image/fetch/$s_!tGvT!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13cb7838-06b9-42c8-bfaf-46961435e6b1_1561x820.png 848w, https://substackcdn.com/image/fetch/$s_!tGvT!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13cb7838-06b9-42c8-bfaf-46961435e6b1_1561x820.png 1272w, https://substackcdn.com/image/fetch/$s_!tGvT!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13cb7838-06b9-42c8-bfaf-46961435e6b1_1561x820.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!tGvT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13cb7838-06b9-42c8-bfaf-46961435e6b1_1561x820.png" width="1456" height="765" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/13cb7838-06b9-42c8-bfaf-46961435e6b1_1561x820.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:765,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!tGvT!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13cb7838-06b9-42c8-bfaf-46961435e6b1_1561x820.png 424w, https://substackcdn.com/image/fetch/$s_!tGvT!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13cb7838-06b9-42c8-bfaf-46961435e6b1_1561x820.png 848w, https://substackcdn.com/image/fetch/$s_!tGvT!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13cb7838-06b9-42c8-bfaf-46961435e6b1_1561x820.png 1272w, https://substackcdn.com/image/fetch/$s_!tGvT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13cb7838-06b9-42c8-bfaf-46961435e6b1_1561x820.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p style="text-align: center;"><em>Figure 3. A reliable agent should keep workflow execution and institutional judgment separate, but versioned together as a behavioral contract.</em></p><p><strong>An example of a production flywheel</strong></p><p>A useful production flywheel is not a vague analytics dashboard. It turns one live interaction into an ordered, auditable episode. In one practical pattern, a conversation becomes: user message &#8594; draft response &#8594; affect and intent signals &#8594; initial evaluation &#8594; rewrite decision &#8594; post-rewrite evaluation &#8594; delivered message.</p><p>Once that structure exists, the same episode can be replayed later, reviewed by humans, approved or rejected, and compiled into the next version of the behavioral contract. Runtime interventions can stay conservative &#8212; rewrite, route, pause, slow-halt, or escalate &#8212; while the offline loop decides what should become a durable product change.</p><p>This is the shift from prompt folklore to production engineering. A corrected response is useful at the moment; a replayable, reviewable correction is useful week over week.</p><p><strong>A few operating patterns tend to work</strong></p><ul><li><p>Observe real traces, not only synthetic eval sets.</p></li><li><p>Keep affect, policy, and tool behavior in the same episode view.</p></li><li><p>Replay proposed changes before release rather than patching blindly in production.</p></li><li><p>Treat approvals and evidence as part of the product, not as after-the-fact documentation.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!vYkO!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46e25816-8b6c-412a-b200-ba5ac1d95aae_1561x820.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!vYkO!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46e25816-8b6c-412a-b200-ba5ac1d95aae_1561x820.png 424w, https://substackcdn.com/image/fetch/$s_!vYkO!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46e25816-8b6c-412a-b200-ba5ac1d95aae_1561x820.png 848w, https://substackcdn.com/image/fetch/$s_!vYkO!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46e25816-8b6c-412a-b200-ba5ac1d95aae_1561x820.png 1272w, https://substackcdn.com/image/fetch/$s_!vYkO!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46e25816-8b6c-412a-b200-ba5ac1d95aae_1561x820.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!vYkO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46e25816-8b6c-412a-b200-ba5ac1d95aae_1561x820.png" width="1456" height="765" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/46e25816-8b6c-412a-b200-ba5ac1d95aae_1561x820.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:765,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!vYkO!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46e25816-8b6c-412a-b200-ba5ac1d95aae_1561x820.png 424w, https://substackcdn.com/image/fetch/$s_!vYkO!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46e25816-8b6c-412a-b200-ba5ac1d95aae_1561x820.png 848w, https://substackcdn.com/image/fetch/$s_!vYkO!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46e25816-8b6c-412a-b200-ba5ac1d95aae_1561x820.png 1272w, https://substackcdn.com/image/fetch/$s_!vYkO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46e25816-8b6c-412a-b200-ba5ac1d95aae_1561x820.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p style="text-align: center;"><em>Figure 4. A production flywheel turns one conversation into an ordered, replayable episode that can be inspected, approved, and released with confidence.</em></p><p><strong>Judge quality now depends on calibration</strong></p><p>Recent judge research points in two directions at once. On one hand, judges are improving. Benchmarks such as JudgeBench and RewardBench 2 are making judge quality easier to measure, while DeepSeek&#8217;s GRM / SPCT work suggests that principle generation, critique, and inference-time aggregation can make reward modeling and preference judging much stronger in practice.</p><p>On the other side, the field is getting more honest about calibration. Newer 2025&#8211;2026 work argues that one raw judge score is not enough. Some evaluators show stable but different &#8220;evaluative fingerprints,&#8221; meaning they can be internally consistent while still disagreeing systematically with one another. Other papers show that rubric order, pointwise versus pairwise framing, and judge allocation can shift rankings if those choices are left uncalibrated.</p><p>The production lesson is simple: one judge should not be the whole control system. High-reliability systems tend to use a judge stack &#8212; crisp gates for clear defects, calibrated reasoning judges for nuance, replay for release decisions, disagreement review for hard cases, and humans for the highest-risk boundaries.</p><p style="text-align: center;">The newest question is no longer only &#8220;Which judge scores highest on a benchmark?&#8221; It is also &#8220;Which judge remains stable, interpretable, and properly calibrated inside a release process?&#8221;</p><p><strong>User simulation can help, but only with calibration</strong></p><p>User simulation is becoming necessary because replay without synthetic users does not cover enough edge cases. Recent work such as SimulatorArena suggests that profile-conditioned simulators can track human ratings reasonably well on some multi-turn tasks, especially when the simulator has access to richer user profiles rather than a generic system prompt.</p><p>But simulation should not be treated as ground truth. Lost in Simulation is the warning label: simulator choice can materially change measured success, and simulated populations can drift away from real human behavior. The practical pattern is simulation plus calibration &#8212; use synthetic users to widen test coverage, then measure the gap to hold-out human traces and correct for it.</p><p><strong>Auditability should be part of the product</strong></p><p>If the team cannot reconstruct which draft was blocked, which rewrite was sent, what evidence triggered the intervention, and which version of the behavioral contract approved the change, then it does not really have a production flywheel. It has prompt folklore.</p><p>Auditability is what turns one corrected episode into a durable release decision. In practice, that usually means keeping trace and event correlation, judge verdicts, rewrites, replay results, approval records, and release decisions together. The goal is not paperwork. The goal is to make the next change easier to inspect, safer to ship, and easier to trust.</p><p><strong>How this connects to my ODSC East talk</strong></p><p>At ODSC East 2026, I&#8217;ll go deeper into the mechanics behind this pattern: how judge stacks can be calibrated, how runtime interventions can be chosen without creating new risk, how replay should precede release, and how week-over-week improvements can turn human preferences into durable production agent judgment.</p><p>The talk title is &#8220;<strong>The Verifier&#8211;Compiler Loop: Turning Human Preferences into Production Agent Judgment.</strong>&#8221; This article only sketches the frame. The session will go much deeper into the system design and operating model behind it.</p><p><strong>Selected references</strong></p><ul><li><p><a href="https://arxiv.org/abs/2410.12784">JudgeBench</a></p></li><li><p><a href="https://arxiv.org/abs/2506.01937">RewardBench 2</a></p></li><li><p><a href="https://arxiv.org/abs/2504.02495">Inference-Time Scaling for Generalist Reward Modeling (DeepSeek GRM / SPCT)</a></p></li><li><p><a href="https://arxiv.org/abs/2511.21140">How to Correctly Report LLM-as-a-Judge Evaluations</a></p></li><li><p><a href="https://arxiv.org/abs/2601.05114">Evaluative Fingerprints: Stable and Systematic Differences in LLM Evaluator Behavior</a></p></li><li><p><a href="https://arxiv.org/abs/2602.16610">Who can we trust? LLM-as-a-jury for Comparative Assessment</a></p></li><li><p><a href="https://arxiv.org/abs/2510.05444">SimulatorArena</a></p></li><li><p><a href="https://arxiv.org/abs/2601.17087">Lost in Simulation</a></p></li></ul><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[Agent reliability]]></title><description><![CDATA[A narrow look at what the last ~90 days of research is teaching us]]></description><link>https://www.equationblog.com/p/agent-reliability</link><guid isPermaLink="false">https://www.equationblog.com/p/agent-reliability</guid><dc:creator><![CDATA[Ruslan Belkin]]></dc:creator><pubDate>Mon, 26 Jan 2026 15:15:42 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!-MQk!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0187b1d4-c470-4ffa-a463-efc198a6864f_1280x1280.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>The field is moving extremely fast right now. New agent stacks, new evals, new post-training tricks - the whole ecosystem shifts weekly.</p><p>But if you ship agents, you learn a painful lesson fast:</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p><strong>An agent that succeeds once is not a reliable agent.</strong></p><p>Single-run success rates are demo metrics. Production reliability is a different game:</p><ol><li><p>Consistency across runs (the same task, same setup, multiple attempts)</p></li><li><p>Robustness to &#8220;equivalent&#8221; user inputs (paraphrases, small spec changes, harmless reorderings)</p></li><li><p>Grace under tool/API failures (because they will fail - timeouts, rate limits, partial responses, schema drift)</p></li><li><p>If I had to compress the theme of the last ~90 days into one line, it&#8217;s this:</p></li></ol><p><strong>Reliability is a surface, not a score.</strong></p><p>This post is intentionally narrow: recent work that treats agent reliability as a first-class object - not a vibe.</p><h1>The production reality check: humans are still the reliability layer</h1><p>A paper I keep pointing people to is &#8220;Measuring Agents in Production.&#8221; It&#8217;s one of the rare efforts that asks practitioners what&#8217;s actually working (and what&#8217;s breaking).</p><p>A few findings that stuck with me:</p><ul><li><p>Many production agents are built to be simple and controllable: 68% run at most 10 steps before requiring human intervention.</p></li><li><p>Most teams lean on prompting off-the-shelf models vs weight tuning (70%), and rely primarily on human evaluation (74%).</p></li><li><p>Reliability shows up as the top challenge - especially &#8220;ensuring and evaluating correctness.&#8221;</p></li></ul><p><strong>That&#8217;s the current equilibrium: humans as circuit breakers.</strong></p><p>The real question is how we scale beyond that without lying to ourselves about what &#8220;reliable&#8221; means.</p><h2>ReliabilityBench: measuring reliability as a surface, not a score</h2><p><a href="https://arxiv.org/abs/2601.06112">ReliabilityBench</a> is exactly the kind of benchmark we&#8217;ve needed.</p><p>Instead of asking &#8220;did it succeed,&#8221; it asks:</p><ol><li><p>Does it succeed again (consistency)</p></li><li><p>Does it succeed under equivalent variations of the task (robustness)</p></li><li><p>Does it survive tool/API failures (fault tolerance)</p><p></p></li></ol><p>They formalize this across three dimensions:</p><ul><li><p>pass^k for repeated execution</p></li><li><p>perturbation intensity &#949;</p></li><li><p>fault intensity &#955;</p></li><li><p>...and propose a unified reliability surface: R(k, &#949;, &#955;).</p></li></ul><p>Two ideas here that I think will stick:</p><ol><li><p>Action metamorphic relations: judge correctness by end-state equivalence rather than brittle text matching.</p></li><li><p>Chaos-style fault injection: simulate timeouts, rate limits, partial responses, schema drift.</p></li></ol><p>The reported results are the point:</p><ol><li><p>Perturbations alone reduced success from 96.9% at &#949;=0 to 88.1% at &#949;=0.2.</p></li><li><p>Rate limiting was especially damaging.</p></li></ol><p>This is what &#8220;production-like&#8221; really means: not one clean run, but performance under stress.</p><p><strong>Why we should care:</strong></p><ol><li><p>If you only track single-run pass rates, you end up optimizing for demos.</p></li><li><p>A reliability surface forces the conversation into repeatability, robustness, and failure modes.</p></li></ol><h2>E-valuator: turn &#8220;judge scores&#8221; into runtime decisions (with guarantees)</h2><p>Assume you&#8217;ve built a verifier (LLM judge, PRM, heuristics). You can score trajectories - but can you trust the score enough to make a runtime decision?</p><p><a href="https://arxiv.org/abs/2512.04123">E-valuator</a> reframes this as a sequential hypothesis testing problem: distinguish successful vs unsuccessful trajectories as actions unfold, using a statistically valid test at every step.</p><p>They propose converting any black-box verifier score into a decision rule with controlled false-alarm rates, and show it can both improve monitoring and terminate problematic trajectories early to save tokens.</p><p><strong>Why we should care:</strong></p><ol><li><p>&#8220;Judge reliability&#8221; is now a core dependency for agent reliability.</p></li><li><p>This is one path from heuristics to operational control.</p></li></ol><h2>LLMdoctor: test-time steering as a reliability tool</h2><p>Benchmarks and verifiers tell you &#8220;it broke.&#8221;</p><p>But reliability also requires you &#8220;fix it now.&#8221;</p><p>That&#8217;s why I like test-time alignment approaches that are modular. <a href="https://arxiv.org/abs/2601.10416">LLMdoctor</a> has a clean patient-doctor framing: steer a frozen model with a smaller controller trained on token-level preference signals, via token-level flow-guided preference optimization.</p><p>Even if you ignore the specific algorithm, the pattern matters:</p><ol><li><p>You can steer without retraining the foundation.</p></li><li><p>You can make reliability interventions fast and reversible.</p></li><li><p>You can version and evaluate the controller like a product.</p></li></ol><p><strong>Why we should care:</strong></p><ul><li><p>Most teams treat reliability fixes as either &#8220;change the prompt&#8221; or &#8220;fine-tune and pray.&#8221;</p></li><li><p>Controller-style steering gives a third option: a scoped, testable intervention layer.</p></li></ul><h2>Human-in-the-loop rubrics: reliability is often a &#8220;shared standard&#8221; problem</h2><p>The hardest part of agent reliability isn&#8217;t always the model.</p><p>Sometimes it&#8217;s the absence of a shared, auditable definition of &#8220;correct.&#8221;</p><p>A recent <a href="https://arxiv.org/abs/2511.10865">paper</a> on patch evaluation proposes a simple but scalable framework:</p><ul><li><p>use an LLM to draft a task-specific rubric,</p></li><li><p>have a human review/refine it once,</p></li><li><p>use the rubric-guided LLM judge to evaluate many candidates.</p></li></ul><p>They report improved agreement with human consensus (e.g., Cohen&#8217;s kappa 0.75 on the subset with unanimous human agreement), plus high recall/precision in that setting.</p><p>Even though the domain is program repair, the reliability lesson generalizes:</p><p>When humans disagree, it&#8217;s often because the rubric is implicit. Make it explicit once - then scale it.</p><div><hr></div><h1>A narrow reliability loop I&#8217;d actually run</h1><p>If I had to condense the above into a practical loop (without turning it into a platform pitch), it would look like this:</p><ol><li><p><strong>Define correctness in end-states, not text: </strong>Use metamorphic relations / end-state equivalence where possible.</p></li><li><p><strong>Stress-test, don&#8217;t just benchmark: </strong>Measure a reliability surface across repeated runs (k), perturbations (&#949;), and tool failures (&#955;).</p></li><li><p><strong>Monitor online with calibrated decision rules: </strong>Turn verifier scores into stop / continue / escalate decisions you can defend.</p></li><li><p><strong>Keep humans as reviewers of standards, not full-time graders: </strong>Use human time to approve/refine rubrics and resolve disagreements.</p></li><li><p><strong>Treat steering as a first-class intervention: </strong>Controller models (doctor -&gt; patient) are a pragmatic way to improve behavior without turning every fix into a full retrain.</p></li></ol><div><hr></div><p>Agent reliability is not a single feature and not a single metric. It&#8217;s a contract:</p><ol><li><p>measured under stress</p></li><li><p>monitored online</p></li><li><p>improved with small, controlled interventions</p></li><li><p>audited through shared standards humans can actually read.</p></li></ol><p>The best recent work is finally treating reliability like an object we can engineer - not a hope we can prompt.</p><p><strong>Quick credit</strong>: <strong><a href="https://www.linkedin.com/in/aw-ai/">Andy Wong</a></strong> consistently finds great new papers early, and we end up debating the implications together before they show up in writing.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[Control knobs from recent LLM papers]]></title><description><![CDATA[January 2026 Update]]></description><link>https://www.equationblog.com/p/control-knobs-from-recent-llm-papers</link><guid isPermaLink="false">https://www.equationblog.com/p/control-knobs-from-recent-llm-papers</guid><dc:creator><![CDATA[Ruslan Belkin]]></dc:creator><pubDate>Tue, 20 Jan 2026 15:15:42 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!-MQk!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0187b1d4-c470-4ffa-a463-efc198a6864f_1280x1280.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>The field is moving extremely fast right now. The half-life of a new idea is measured in weeks (sometimes days), and it is getting harder to tell what will actually stick.</p><p>Most weeks, the discourse around language models collapses into one of two modes:</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><ul><li><p>&#8220;Look at this new leaderboard bump.&#8221;</p></li><li><p>&#8220;Agents are coming, everything changes.&#8221;</p></li></ul><p>Both can be true - and still miss the point.</p><p>The stuff that actually moves production outcomes tends to look like new control knobs: better ways to steer models, keep them stable, make them faster, and make failures more legible.</p><p>Credit where it is due: a bunch of these paper finds came from <a href="https://www.linkedin.com/in/aw-ai/">Andy Wong</a>, who consistently surfaces great work. We usually end up debating the implications together before it shows up here.</p><p>So here is my January reading stack: papers that feel unusually primitive-shaped. Each one adds a knob I expect we will keep using.</p><h2>1. <a href="https://www.linkedin.com/posts/rbelkin_251224601-activity-7419125318424145920-Xyaz">Recursive Language Models</a>: &#8220;infinite&#8221; context via self-calls (no retraining)</h2><p>Recursive Language Models (RLMs) are a different answer to long context: do not cram the prompt into the transformer - treat it like part of the environment.</p><p>One concrete instantiation: the prompt becomes a variable inside a Python REPL, and the model writes code to inspect the prompt, decompose it, and recursively call sub-instances of itself over slices of the prompt.</p><p>They report handling inputs two orders of magnitude beyond typical context windows - and even mention strong performance at the 10M+ token scale.</p><p><strong>Why we should care:</strong></p><ul><li><p>This pushes long context from architecture into systems. Not &#8216;train longer,&#8217; but &#8216;reason out-of-core.&#8217;</p></li><li><p>It is an agent-shaped pattern: when the model can write the loop it thinks inside, you get a new class of tool-use + decomposition behaviors.</p></li></ul><p>It reframes the bottleneck: the limit becomes less context length and more how good the model is at building the right indexing + recursion strategy.</p><h2>2. <a href="https://www.linkedin.com/posts/rbelkin_powerpoint-%E6%BC%94%E7%A4%BA%E6%96%87%E7%A8%BF-activity-7418418556901470208-iaSR?utm_source=share&amp;utm_medium=member_desktop&amp;rcm=ACoAAAAPDpcBcLu8_J9WkZjSGCZolugTdGmprHo">LLMdoctor</a>: alignment at test-time, token by token</h2><p>Most alignment work still assumes you either fine-tune the whole model (expensive, slow, often brittle), or you do test-time tricks that are coarse (trajectory-level) and compute-hungry.</p><p>LLMdoctor proposes a clean separation: keep a big &#8216;patient&#8217; model frozen, and steer it with a smaller &#8216;doctor&#8217; model using token-level signals.</p><p>Their claim is that many test-time alignment methods rely on distorted trajectory-level rewards or inefficient sampling that caps performance and harms diversity. The patient-doctor setup extracts token-level preference signals from the patient&#8217;s behavioral variations, then trains the doctor via token-level flow-guided preference optimization (TFPO) to preserve diversity while aligning outputs.</p><p><strong>Why we should care:</strong></p><ul><li><p>Steering becomes modular. Iterate on the doctor without re-baking the patient.</p></li><li><p>Granularity matters. Token-level intervention is the difference between &#8216;mostly aligned&#8217; and &#8216;aligned where it counts.&#8217;</p></li></ul><p>Closer to how agents fail. <strong>Agents do not fail at the end of the trajectory - they fail mid-trajectory</strong>.</p><h2>3. <a href="https://www.linkedin.com/posts/rbelkin_entropy-adaptive-fine-tuning-resolving-confident-activity-7416937560096161792--icu?utm_source=share&amp;utm_medium=member_desktop&amp;rcm=ACoAAAAPDpcBcLu8_J9WkZjSGCZolugTdGmprHo">Entropy-Adaptive Fine-Tuning</a>: a practical take on &#8220;don&#8217;t forget&#8221;</h2><p>Supervised fine-tuning is still the workhorse for specialization - and &#8220;catastrophic&#8221; (or I would rather say &#8220;annoying&#8221;) forgetting is still the bill we pay.</p><p>This paper&#8217;s framing is crisp: it contrasts SFT with on-policy RL and argues the gap comes from distribution mismatch. In RL, the model&#8217;s learning signal is more consistent with its internal beliefs; in SFT, the model is forced to fit external supervision even when that conflicts sharply with what it &#8216;knows.&#8217;</p><p>They focus on confident conflicts: cases where the label token is low-probability under the model, while the model&#8217;s distribution is low entropy (i.e., it is confidently predicting something else). That is where gradients get destructive.</p><p>Their proposal, Entropy-Adaptive Fine-Tuning (EAFT), uses token-level entropy as a gating mechanism: learn aggressively when the model is uncertain; suppress gradients when the model is confident-but-disagreeing.</p><p>From my most recent post: I also think EAFT is a genuinely useful alternative to LoRA in the &#8216;don&#8217;t wreck the base model&#8217; sense - rather than constraining where we update (parameter-efficient adapters), EAFT constrains when updates should matter (skip the destructive ones).</p><p><strong>Why we should care:</strong></p><ul><li><p>This is the kind of idea that turns continuous updates from scary to feasible.</p></li><li><p>It maps to a real production vibe: most of the time we want to learn; sometimes we want to refuse the lesson.</p></li><li><p>It is a different safety knob than LoRA - but it is targeting the same anxiety: regressions.</p></li></ul><h2>4. <a href="https://www.linkedin.com/posts/rbelkin_fig1-activity-7415183051556564993-ws2d?utm_source=share&amp;utm_medium=member_desktop&amp;rcm=ACoAAAAPDpcBcLu8_J9WkZjSGCZolugTdGmprHo">From Entropy to Epiplexity</a>: measuring &#8220;useful information&#8221; for bounded learners</h2><p>Data quality is still the hidden kingmaker. The hard part is: we are not data-rich, we are signal-poor.</p><p>This paper asks a deceptively simple question: can we quantify learnable content in data without tying it to a downstream task?</p><p>They argue classic information measures (Shannon entropy, Kolmogorov complexity) do not capture what matters for computationally bounded learners, and they propose a new measure: epiplexity.</p><p>The vibe: epiplexity is meant to capture structural content while excluding &#8216;time-bounded entropy&#8217; (random/unpredictable content), and the authors claim it helps explain why deterministic transformations and data ordering can still create useful learnable structure in practice.</p><p><strong>Why we should care:</strong></p><ul><li><p>If inputs are becoming the product, we eventually want a metric for the informational value of inputs.</p></li><li><p>Epiplexity feels like a step toward data selection as an engineering discipline, not an art project.</p></li></ul><h2>5. <a href="https://www.linkedin.com/posts/rbelkin_llada20techreportpdf-at-main-inclusionai-activity-7406820279944982528-DmUi?utm_source=share&amp;utm_medium=member_desktop&amp;rcm=ACoAAAAPDpcBcLu8_J9WkZjSGCZolugTdGmprHo">LLaDA2.0</a>: diffusion language models to 100B</h2><p>Autoregressive decoding is powerful, but it is fundamentally serial.</p><p>LLaDA2.0 pushes discrete diffusion language models to 100B parameters via a conversion process: take a pretrained AR model and convert it to a dLLM using a 3-phase block-level training scheme (warm-up with increasing block size, stable full-sequence diffusion, decay back to compact block diffusion).</p><p>They also discuss post-training alignment with SFT and DPO, framing this as a path to frontier-scale efficiency while preserving parallel decoding advantages.</p><p><strong>Why we should care:</strong></p><ul><li><p>Parallel decoding is not just a speed story. It changes how we can spend compute at inference time.</p></li></ul><p>Faster sampling = more room for verification, search, and self-checking within real latency budgets.</p><h2>6. <a href="https://arxiv.org/abs/2509.10534">PoPE</a>: decoupling the &#8220;what&#8221; and &#8220;where&#8221; in positional embeddings</h2><p>I have a soft spot for papers that say: &#8216;this popular thing is entangled in a way that quietly hurts you,&#8217; and then fix it cleanly.</p><p>PoPE (Polar Coordinate Positional Embeddings) argues RoPE entangles content (what) and position (where), which can impair tasks requiring independent matching on the two. They propose PoPE to remove the confound, show better performance on diagnostics and across sequence modeling domains, and highlight strong zero-shot length extrapolation vs RoPE - and even vs YaRN.</p><p><strong>Why we should care:</strong></p><ul><li><p>Long context is now table stakes for serious agent workflows.</p></li><li><p>Works at 8k is not the same as behaves at 80k.</p></li></ul><h1>DeepSeek + stuff that ships</h1><h2><a href="https://www.linkedin.com/posts/rbelkin_engramengrampaperpdf-at-main-deepseek-ai-activity-7416602221993172992-rYqb?utm_source=share&amp;utm_medium=member_desktop&amp;rcm=ACoAAAAPDpcBcLu8_J9WkZjSGCZolugTdGmprHo">Engram</a>: conditional memory as a second sparsity axis</h2><p>The Whale does it again. MoE gave us conditional computation. But knowledge lookup is still mostly simulated via dense compute.</p><p>Engram proposes conditional memory - a complementary axis of sparsity - implemented via an O(1) lookup module modernizing classic N-gram embeddings.</p><p>They describe a &#8216;Sparsity Allocation&#8217; tradeoff between neural computation (MoE) and static memory (Engram), claim a U-shaped scaling law, and report scaling Engram to 27B parameters with gains not just on knowledge tasks but also reasoning, code/math, and long-context retrieval.</p><h2><a href="https://www.linkedin.com/posts/rbelkin_mhc-manifold-constrained-hyper-connections-activity-7412530889903112192-8oM6?utm_source=share&amp;utm_medium=member_desktop&amp;rcm=ACoAAAAPDpcBcLu8_J9WkZjSGCZolugTdGmprHo">mHC</a>: Manifold-Constrained Hyper-Connections</h2><p>This zooms in on a real training pathology: expanding residual streams/connectivity can improve performance, but it can also break the identity mapping property residual connections rely on - leading to instability and scalability issues.</p><p>mHC proposes projecting the residual connection space onto a manifold to restore identity mapping while keeping things efficient.</p><h2><a href="https://www.linkedin.com/posts/rbelkin_ppovsgrpo-activity-7415559502030241792-UhoE?utm_source=share&amp;utm_medium=member_desktop&amp;rcm=ACoAAAAPDpcBcLu8_J9WkZjSGCZolugTdGmprHo">DeepSeek-R1</a>: RL-first reasoning + the GRPO refresher</h2><p>Even if you are excited about the next base model drop, R1&#8217;s training recipe is the more durable lesson.</p><p>The core claim: reasoning behaviors can emerge via pure RL (with a cold-start SFT phase for readability/stability), and they lean on GRPO - which is worth revisiting if your PPO mental model is rusty.</p><p>Quick intuition: GRPO drops the critic and estimates a baseline from grouped samples, which matters a lot for scaling RL in LLM land.</p><h2><a href="https://www.linkedin.com/posts/rbelkin_solving-llm-repetition-problem-in-production-activity-7405340339450023936-p2ve?utm_source=share&amp;utm_medium=member_desktop&amp;rcm=ACoAAAAPDpcBcLu8_J9WkZjSGCZolugTdGmprHo">Solving LLM repetition</a> in production</h2><p>This one earns points for being unapologetically real: repetition loops that stall batch tasks.</p><p>They identify repetition patterns, frame the root cause via Markov analysis + greedy decoding getting stuck in loops, and evaluate mitigations: beam search with early_stopping=True (universal post-hoc), presence_penalty (case-specific), and DPO fine-tuning (model-level universal).</p><h1>The connective tissue</h1><p>If I had to summarize the direction across these papers in one line: we are entering the era of systems that add knobs: inference-time recursion for extreme context, token-level steering, entropy-gated learning, explicit memory, better information measures, disentangled position representations, and faster decoding.</p><p>The fun part: <strong>these knobs compound.</strong></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[Notes on Product Development Process?]]></title><description><![CDATA[&#8220;When a number of people concentrate on a single thought, they can compel the world to accept it.&#8221; &#8212; Paramahansa YoganandaThanks for reading The Equation!]]></description><link>https://www.equationblog.com/p/what-is-your-product-development</link><guid isPermaLink="false">https://www.equationblog.com/p/what-is-your-product-development</guid><dc:creator><![CDATA[Ruslan Belkin]]></dc:creator><pubDate>Fri, 21 Nov 2025 23:15:14 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!-MQk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0187b1d4-c470-4ffa-a463-efc198a6864f_1280x1280.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em>&#8220;When a number of people concentrate on a single thought, they can compel the world to accept it.&#8221; &#8212; Paramahansa Yogananda</em></p><div><hr></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p><strong>&#8220;What is your Product Development Process?&#8221;</strong> -  I&#8217;ve been asked this question several times lately&#8212;including during the <a href="https://elc.community/public/podcasts/specs-as-the-new-source-of-truth-synthetic-data-as-the-next-wave-of-defensibility-product-vision-and-decision-making-frameworks-ruslan-belkini-inflection-ai">ELC podcast</a>.</p><p>I&#8217;ve always been an engineering person. Systems, infrastructure, APIs&#8212;that&#8217;s the way my brain was wired. My interest in product and users started early at LinkedIn. Perhaps I was influenced by my product partner there - <a href="https://www.linkedin.com/in/ablue/">Allen Blue</a>, who was one of the co-founders and amazingly forward looking product visionary. Somewhere in those early cycles it clicked: we do what we do <strong>in service of our users</strong>&#8212;whether those users are developers, end users, or enterprise customers. That realization pulled me from &#8220;just engineering&#8221; into caring about product outcomes. And yet, at the start, I was a complete novice at product (and truthfully probably still am).</p><p>I read the usual suspects (<em><a href="https://www.amazon.com/Lean-Startup-Entrepreneurs-Continuous-Innovation/dp/0307887898">The Lean Startup</a></em>, <em><a href="https://www.amazon.com/Competing-Against-Luck-audiobook/dp/B01IIGPYGC/ref=sr_1_1?crid=2TDML7IMO44ZZ&amp;dib=eyJ2IjoiMSJ9.STwdk378XLZZKzi3XO7n3bZv36x8ZV9v4rW6VEizg5O0ruWENvbsbx8QiiyxXIlYTFATOvJjFYK1kqCrddUrfhG84-yehOvdGzR1lZoTtR5O9_IzmvA6Tfh58QwySM3DpS_KXZOM62jk_ZiAhRrp0S1C6smFUlLBLSuYKpmUKBboEJs1dbZaPld1myGGk8kVfQpEGcHs3DSuNG3HyWVpP1OQUNBfAAQnk0rR7UtjXik.Cz7aTIho21sm5gDrWj4AWkGpUxibmihK2OcR9cmg5m4&amp;dib_tag=se&amp;keywords=Competing+Against+Luck&amp;qid=1761935104&amp;s=books&amp;sprefix=competing+against+luck%2Cstripbooks%2C148&amp;sr=1-1">Competing Against Luck</a></em>, <em><a href="https://www.amazon.com/The-Mom-Test-Rob-Fitzpatrick-audiobook/dp/B07RJZKZ7F/ref=sr_1_1?crid=5SKS7YOVXNPK&amp;dib=eyJ2IjoiMSJ9.-qOPHHPOaY7mEvJQiz1-_SRx_FI3lGO9aA0dhokR7ms.nredWviFhEtciQULac0jny0ttHAiprVQARIcx5uNqEA&amp;dib_tag=se&amp;keywords=The+Mom+Test&amp;qid=1761935127&amp;s=audible&amp;sprefix=the+mom+test%2Caudible%2C142&amp;sr=1-1">The Mom Test</a></em>, and a mountain of blogs) about experiments, rapid iteration, speed, listening to customers, etc. Yet the truth is that a lot of product and company building is about luck&#8212;and if you play the lottery long enough, sometimes you win and the technique was perhaps less important than professed. At some point I came to a realization: there is no such thing as &#8220;finding&#8221; product&#8209;market fit beyond random luck. There is only <strong>compelling the world</strong> to accept your vision of the future.</p><p>So, here&#8217;s how I think about the product product development process today:</p><h2>1. Imagine the future</h2><p>Think of yourself as a sci&#8209;fi <strong>writer</strong>. Imagine the future you want to see. In detail and with maximum nuance. Feel it, smell it, live it, write about it (for yourself mostly). This is not entirely a logical process. Works of art don&#8217;t come from the mind alone; they come from the soul. The more exact, detailed, and precise the imagination&#8212;the better. The approach works not only for big ideas, but also for small problems, design decisions and similar challenges. Try it. Obviously, you need a background in the field so your imagination has grounding.</p><p><strong>Where this can go wrong:</strong></p><ul><li><p><strong>The idea isn&#8217;t yours.</strong> Works of art are not a collective exercise. We often adopt other people&#8217;s ideas unconsciously&#8212;we are imitators by design (see Ren&#233; Girard&#8217;s <em>mimetic</em> theory). Your founding team can absolutely enhance and improve the idea&#8212;just don&#8217;t confuse where it came from. Maybe the product idea isn&#8217;t yours, but you came up with a technological solution that makes it work&#8212;that&#8217;s OK, join the team. The worst failure mode is convincing yourself you like someone else&#8217;s idea on logical merits when you&#8217;re not passionate about it. If you&#8217;re going to be a mercenary, be honest about it&#8212;and love the craft of being a mercenary.</p></li><li><p><strong>The idea isn&#8217;t well envisioned.</strong> It&#8217;s a fleeting, fuzzy thought. The world can&#8217;t materialize a fuzzy image. This failure mode often happens to non-technical people or when we simply don&#8217;t yet have a necessary background to be able to imagine the end product in a precise enough detail.</p></li><li><p><strong>It&#8217;s purely cerebral.</strong> A product born only from logic usually isn&#8217;t art. You need conviction that lives below the neck.</p></li></ul><h2>2. Test it</h2><p>Pitch it to your smartest friends, as well as to friends not in the field. Stress&#8209;test it from different angles. Run experiments to <strong>prove or disprove</strong> your thinking. Is the idea actually new in the first place or you just didn&#8217;t bother to do basic research? Think like an investor; evaluate it as you would an investment pitch. This is the logical part. Is the technology feasible? Is the timeframe right? You have to allow some uncertainty, perhaps a lot of uncertainty. You will hear many &#8220;no&#8221;s and there will never be enough data. But there should be a threshold of probability you can settle on&#8212;and then decide.</p><p><strong>Where this can go wrong:</strong></p><ul><li><p><strong>No serious scrutiny.</strong> Less of an issue with startups, but a very common failure mode in large companies. Apply VC&#8209;type scrutiny: explicit assumptions (why, why now, why this team, how is it different, what category are we creating or capturing, how do we solve the distribution). Even for small projects - do this, it can be done very quickly with modern tools.</p></li><li><p><strong>Never&#8209;ending search for 100% conviction.</strong> It doesn&#8217;t exist. Set the risk level you can carry and move.</p></li></ul><h2>3. Focus on it</h2><p>This is about <strong>compelling the world</strong> to accept your vision. It requires assembling as many smart, values&#8209;aligned people as you can and maintaining persistent focus on execution. This is also the place where much traditional advice can work&#8212;to an extent.</p><p><strong>Where this can go wrong:</strong></p><ul><li><p><strong>Insufficient critical mass of smart people</strong>, or settling for the wrong, mediocre team (or a team of mercenaries) that isn&#8217;t excited or hasn&#8217;t adopted the idea almost as a new religion. I have yet to hear from anyone that they have a mediocre team (&#8220;we have a very high bar clich&#233;&#8221;). Be honest, is it really true? Most teams in fact are mediocre or worse.</p></li><li><p><strong>Insufficient funding. </strong>It is perhaps a sad truth - but funding is a competitive differentiation for startups. Inadequate capital could impair both the ambition and execution when the speed matters most.</p></li><li><p><strong>Loss of focus</strong> during execution. Could happen for variety of reasons - loss of conviction on a part of the team, poor growth execution / team dilution (by hiring too many or wrong people too fast), sometimes a function of too much funding.</p></li><li><p><strong>Bailing too early</strong>&#8212;not giving the idea enough time to work. This one is hard as you never know for sure.</p></li></ul><div><hr></div><p>I&#8217;m writing this for myself&#8212;as a reminder&#8212;as much as for you. It&#8217;s incredibly easy to fall into these traps.</p><p><strong>Imagine clearly. Test honestly. Focus completely.<br>Compel the world.</strong></p><p>Love to hear your thoughts.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p>]]></content:encoded></item><item><title><![CDATA[Agents That Feel the Room—and Fix Themselves]]></title><description><![CDATA[Where agentic AI is headed, and why intelligent data flywheels matter]]></description><link>https://www.equationblog.com/p/agents-that-feel-the-roomand-fix</link><guid isPermaLink="false">https://www.equationblog.com/p/agents-that-feel-the-roomand-fix</guid><dc:creator><![CDATA[Ruslan Belkin]]></dc:creator><pubDate>Sat, 25 Oct 2025 14:15:25 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!-MQk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0187b1d4-c470-4ffa-a463-efc198a6864f_1280x1280.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>If you watch today&#8217;s agents for more than a few minutes, you see the mood swings. In one chat they&#8217;re warm and crisp; in the next they over&#8209;explain, miss an obvious cue, or bungle a tool call with the wrong parameter. And even when there&#8217;s no &#8220;human feel&#8221; at all&#8212;just a heads down task&#8212;they drift off spec: skip required steps, optimize the wrong objective, or simply neglect or misinterpret human inputs. </p><p>I don&#8217;t think this is a &#8220;just scale it&#8221; problem. It&#8217;s a feedback problem. We&#8217;re still treating behavior&#8212;tone, choices, task adherence, even API hygiene&#8212;like vibes in a prompt instead of something we can observe, grade, and steadily improve. That&#8217;s why I keep coming back to intelligent data flywheels: small, concrete loops that turn context &#8596; behavior trace into a living artifact for training, evaluation, and live steering.</p><h3>What I mean by &#8220;intelligent data flywheels&#8221;</h3><p>The picture in my head is simple. Keep a <strong>map</strong> of the situations your agent encounters (support vs. sales, calm vs. upset, low&#8209;stakes vs. high&#8209;stakes) and the <strong>behaviors</strong> you want in each (clarity, empathy, brevity, caution, brand voice and values). Use that map to (a) <strong>generate</strong> realistic multi&#8209;turn data&#8212;by having a &#8220;Human LLM&#8221; act out the human side of conversations&#8212;and (b) <strong>judge</strong> the agent against the behavior playbook with a grader that you also train. Then run the same primitives at three speeds: <em><strong>observability</strong></em> (find where it breaks or excel), <em><strong>inference&#8209;time nudging</strong></em> (steer it right now), and <em><strong>training</strong></em> (make the fix stick). When human raters and the judge disagree beyond a reasonable margin, you update the judge; when they align, you let the judge carry more load and feed the next enhancement round. <strong>That&#8217;s the loop</strong>.</p><p>I still care a lot about <em>emotional intelligence</em>&#8212;human-facing agents that communicate across different media and modalities&#8212;but the same flywheel helps with boring, high&#8209;impact stuff that isn&#8217;t &#8220;EQ&#8221; at all: tool use, retrieval grounding, latency/cost tradeoffs, and safety drift.</p><h3>Why this suddenly feels practical</h3><p>A few research threads clicked into place:</p><ul><li><p><strong>Principle&#8209;following reward models</strong> showed you can align behavior to a collection of human-written rubrics instead of massive preference sets</p></li><li><p><strong>Inference&#8209;time scaling for judges</strong> matured (e.g. DeepSeek&#8217;s GRM). Spend more tests&#8209;time compute&#8212;parallel samples plus a meta&#8209;judge&#8212;and you get more reliable reward signals for both training and live guardrails.</p></li><li><p><strong>Judges that actually reason</strong> generate a case-specific rubric before scoring, which humans can verify, align, and generalize..</p></li><li><p><strong>Open, strong preference data</strong> (e.g. helpsteer3) and <strong>closed, experiential data from your deployed agent</strong> provides a solid base you can specialize with your own behavior map.</p></li></ul><p><strong>Caveat</strong>: LLM&#8209;as&#8209;a&#8209;judge isn&#8217;t magic. Benchmarks like JudgeBench show judges can be brittle or biased if you don&#8217;t treat them like first&#8209;class products&#8212;versioned, monitored, retrained, and contexted.. That&#8217;s another reason to put them inside the flywheel.</p><h3>The unglamorous stuff agents fail at</h3><ul><li><p><strong>Function calling &amp; schema correctness.</strong> Even top models still fumble basic format rules (quote this string; ISO date there) and multi&#8209;step tool chains. Recent work&#8212;BFCL, JSONSchemaBench, IFEval&#8209;FC&#8212;quantifies how often calls are syntactically valid yet semantically wrong. In my head, the &#8220;judge&#8221; can be a schema/trace checker with scenario&#8209;aware penalties, and the generator can synthesize tricky, long&#8209;horizon tool graphs to close the gap.</p></li><li><p><strong>Grounding &amp; hallucinations in RAG.</strong> Datasets like RAGTruth and newer lenses like HalluLens keep reminding us that extrinsic hallucinations haven&#8217;t vanished; high&#8209;certainty hallucinations are especially sneaky. A flywheel can grade answers on entailment against retrieved context and choose the next hard cases to label or synthesize.</p></li><li><p><strong>Open&#8209;world task reliability.</strong> Real agent work looks like OS&#8209;level workflows and the messy web. OSWorld, WebArena, and AgentBench have moved the bar here and highlight recurring failure modes&#8212;state tracking, planning depth, visual grounding. Using their task taxonomies as &#8220;contexts&#8221; and step&#8209;level success as &#8220;behaviors&#8221; gives you a clean contract for the flywheel to optimize.</p></li><li><p><strong>Safety and the &#8220;persona dial.&#8221;</strong> OpenAI&#8217;s emergent&#8209;misalignment results point to interpretable <em>persona features</em>&#8212;directions in activation space that modulate toxic or deceptive modes. That turns safety from a black box into a dial you can monitor and counter&#8209;steer inside the flywheel.</p></li></ul><p>And yes, <strong>EQ still matters</strong> because humans are in the loop. Benchmarks like EmoBench, EmotionQueen, and multimodal EmoBench&#8209;M show a persistent gap to humans on &#8220;understand + respond appropriately.&#8221; That&#8217;s the sweet spot for a behavior&#8209;by&#8209;context map coupled to a judge that also reasons about emotion.</p><h3>How this looks in practice&#8212;my mental movie</h3><p>I picture an analytics view that doesn&#8217;t just say &#8220;CSAT dropped,&#8221; but <em>where and why</em>: &#8220;In escalations from upset users, brevity overrode clarity; schema errors spiked in step&#8209;3 tool calls; the judge drifted on empathy.&#8221; From there, the loop suggests more of what it needs: a batch of synthetic escalations with complicated tool chains; a judge tune on Emotion&#8209;Application items; a tweak to the behavior weights for this scenario. We close the loop in three places: surface the issue (observability), compensate now (inference), and make it permanent (training). Publish the change, re&#8209;benchmark, repeat. </p><p>Over time you get a system that not only <em>thinks</em> better, but <em>behaves</em> predictably under stress&#8212;because behavior stopped being vibes and started being data.</p><p>And one more thing: over time, those results compound into a differentiated, market-adapting playbook &#8212; a living operational memory co-built by your team and the system, shaped by every success, failure, and fix.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p>]]></content:encoded></item><item><title><![CDATA[Is code still the source of truth?]]></title><description><![CDATA[Summary of my presentation at 2025 ELC Annual]]></description><link>https://www.equationblog.com/p/is-code-still-the-source-of-truth</link><guid isPermaLink="false">https://www.equationblog.com/p/is-code-still-the-source-of-truth</guid><dc:creator><![CDATA[Ruslan Belkin]]></dc:creator><pubDate>Wed, 15 Oct 2025 14:15:48 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!37ed!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbe693e6-8ace-42a5-954f-7eeb50a5a876_1617x1131.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>I argued at <a href="https://youtu.be/y6M-SCOhnfE?si=Q5_LBSFsRbrGqoqj">ELC Annual</a> that the center of gravity is drifting&#8212;from code to <em>inputs about the code</em>. Prompts, datasets, models, and evals are becoming the primary artifacts. That&#8217;s not a prediction; it&#8217;s merely connecting the dots <em>backward</em> and noticing what&#8217;s already changed.</p><div class="native-video-embed" data-component-name="VideoPlaceholder" data-attrs="{&quot;mediaUploadId&quot;:&quot;3b5bef46-7b41-4c50-a547-16223e09c54f&quot;,&quot;duration&quot;:null}"></div><div><hr></div><h3>The constraints we actually fight</h3><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>Most of engineering is just pressure-testing three constraints: <strong>human cognition</strong> (complexity, coordination, bugs), <strong>provisioning at scale</strong> (deploy, redundancy, cost), and <strong>change risk</strong> (feedback loops). The tools that moved the needle before LLMs&#8212;managed runtimes, DVCS+CI/CD, containerization/cloud, observability&#8212;were all bets against those constraints.</p><div><hr></div><h3>What shifted with LLMs</h3><p>Treating the model like a &#8220;compiler&#8221; works&#8212;until it doesn&#8217;t. Traditional compilers are deterministic; auto&#8209;regressive models aren&#8217;t. Tiny prompt edits (or swaps between model families) yield materially different outcomes. That puts <em>inputs</em>&#8212;prompts, retrieval/context rules, tool schemas&#8212;and <em>evaluation</em> at the center. The inputs are the product.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!37ed!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbe693e6-8ace-42a5-954f-7eeb50a5a876_1617x1131.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!37ed!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbe693e6-8ace-42a5-954f-7eeb50a5a876_1617x1131.png 424w, https://substackcdn.com/image/fetch/$s_!37ed!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbe693e6-8ace-42a5-954f-7eeb50a5a876_1617x1131.png 848w, https://substackcdn.com/image/fetch/$s_!37ed!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbe693e6-8ace-42a5-954f-7eeb50a5a876_1617x1131.png 1272w, https://substackcdn.com/image/fetch/$s_!37ed!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbe693e6-8ace-42a5-954f-7eeb50a5a876_1617x1131.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!37ed!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbe693e6-8ace-42a5-954f-7eeb50a5a876_1617x1131.png" width="1456" height="1018" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/dbe693e6-8ace-42a5-954f-7eeb50a5a876_1617x1131.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1018,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:177727,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.equationblog.com/i/175477719?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbe693e6-8ace-42a5-954f-7eeb50a5a876_1617x1131.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!37ed!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbe693e6-8ace-42a5-954f-7eeb50a5a876_1617x1131.png 424w, https://substackcdn.com/image/fetch/$s_!37ed!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbe693e6-8ace-42a5-954f-7eeb50a5a876_1617x1131.png 848w, https://substackcdn.com/image/fetch/$s_!37ed!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbe693e6-8ace-42a5-954f-7eeb50a5a876_1617x1131.png 1272w, https://substackcdn.com/image/fetch/$s_!37ed!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbe693e6-8ace-42a5-954f-7eeb50a5a876_1617x1131.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p></p><p>Two practical implications:</p><ol><li><p><strong>Inputs as first&#8209;class artifacts.</strong><br>Prompts (and flows), input datasets (docs, tickets, logs), models/fine&#8209;tunes, and eval suites/user simulators all need real versioning, lineage, and regression checks&#8212;because models change under your feet.</p></li><li><p><strong>Evaluation is your safety rail.</strong><br>You won&#8217;t guarantee determinism; you <em>can</em> bound behavior and catch drift. Invest in evals and forward simulation before you put agents anywhere near money or production.</p></li></ol><div><hr></div><h3>Legacy, repos&#8212;and the new &#8220;rewrite&#8221;</h3><p>&#8220;Full rewrite&#8221; used to be a dirty word. With LLM&#8209;accelerated throughput, wholesale rewrites are increasingly viable when entropy makes understanding costlier than regeneration + hardening with evals. </p><div><hr></div><h3>What engineering is becoming</h3><p>Zoom out and the job collapses into <strong>tooling</strong> and <strong>data</strong>.</p><p>Humans become what the models aren&#8217;t:  <strong>(a) evaluators</strong> and <strong>(b) carriers of undocumented institutional knowledge</strong>. That&#8217;s where leverage lives.</p><blockquote><p><strong>Tech debt is dirty data.</strong><br>Clean it, or it compounds.</p></blockquote><h3>A minimal operating checklist (until better tooling is available):</h3><ul><li><p><strong>Check in prompts with code.</strong> Include tool schemas, context rules, guardrails, and tests.</p></li><li><p><strong>Pin and record models.</strong> Track families/versions and fine&#8209;tuning metadata like compiler flags. Expect drift.</p></li><li><p><strong>Build evals before features.</strong> Scenarios, simulators, acceptance criteria&#8212;gate releases on them.</p></li><li><p><strong>Prefer rewrite when entropy wins.</strong> If understanding cost &gt; regeneration + eval hardening, start over.</p></li><li><p><strong>Instrument everything.</strong> You can&#8217;t lead probabilistic systems blind.</p></li></ul><p>None of this requires prophecy (Bohr and Feynman would approve);  tempered by the reminder that auto&#8209;regressive generation diverges without control. </p><p>The link to the full <a href="https://youtu.be/y6M-SCOhnfE?si=kpKoXJiL-BxIDwpE">video</a></p><p>&#8212; <strong>Ruslan</strong></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[Updated workout routine - Part 2/2]]></title><description><![CDATA[6-weeks functional block]]></description><link>https://www.equationblog.com/p/updated-workout-routine-part-22</link><guid isPermaLink="false">https://www.equationblog.com/p/updated-workout-routine-part-22</guid><dc:creator><![CDATA[Ruslan Belkin]]></dc:creator><pubDate>Sat, 11 Oct 2025 14:15:37 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!-MQk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0187b1d4-c470-4ffa-a463-efc198a6864f_1280x1280.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em>&#8220;I am good at pullups&#8221;</em>  &#8212;Tony Horton</p><div><hr></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>As mentioned in the previous post - I&#8217;ve recently tweaked my workout schedule to harmonize the functional block (focused on full body power movements) and weight lifting block (focused on muscle building) to make them both be 6-weeks in length each. I alternate those blocks.</p><p>In my <a href="https://www.equationblog.com/p/my-current-workout-routine">previous</a> post I described the prior routine: two blocks &#8212; a 6-week full-body (mostly power-focused) functional block, followed by a 4-week weight-lifting block.</p><p>In this post, I&#8217;m outlining a modified functional block. This is the hardest block due to high demand on neuro-musculature system (even though the weights are considerably lower) and the follow on weight lifting block always feels like a break.</p><p>As before, I work out first thing in the morning (work scheduling), with a 5:30 am wake-up, quick oral/face hygiene, pre-workout <a href="https://www.equationblog.com/p/the-supplements">supplements</a>, and a 10-15 min meditation. </p><p>Because the blocks are now a bit longer, I&#8217;m skipping pre-workout foam rolling during the work week. If I had unlimited time, I&#8217;d keep it.</p><p>I still use guided workouts (<a href="https://www.bodi.com/us/en/s/subscription?code=SEMB_BODI_GOOGLE&amp;g_acctid=821-296-1770&amp;g_adgroupid=166824054903&amp;g_adid=733099013197&amp;g_adtype=search&amp;g_campaign=SEM_Brand_All&amp;g_campaignid=21755424360&amp;g_keyword=beachbody&amp;g_keywordid=kwd-44858520&amp;g_network=g&amp;gad_campaignid=21755424360&amp;gad_source=1&amp;gclid=CjwKCAjwkvbEBhApEiwAKUz6-2h6-q4czV8nmyhecXe2sn_CuI1tjw6RDjhU3QcsCBlzqMjexYOB5BoCG38QAvD_BwE&amp;gclsrc=aw.ds&amp;nb_adtype=&amp;nb_ap=&amp;nb_fii=&amp;nb_kwd=beachbody&amp;nb_li_ms=&amp;nb_lp_ms=&amp;nb_mi=&amp;nb_mt=e&amp;nb_pc=&amp;nb_pi=&amp;nb_placement=&amp;nb_ppi=&amp;nb_si=%7Bsourceid%7D&amp;nb_ti=kwd-44858520&amp;nbt=nb%3Aadwords%3Ag%3A21755424360%3A166824054903%3A733099013197&amp;utm_campaign=SEM_Brand_All&amp;utm_content=21755424360_166824054903&amp;utm_medium=SEM&amp;utm_source=GOOGLE&amp;utm_term=beachbody_">Beachbody</a> &#8212; historical) mainly for timing and pacing.</p><p>The core workout is built around <a href="https://www.beachbodyondemand.com/programs/6-weeks-of-the-work/start-here?locale=en_US">The 6-weeks of the work</a> program by Amolia Ceasar and it consists of 3 blocks, 2 weeks each. The workouts get progressively more complex (although not necessarily harder) in each subsequent block. Since the program has a formal rest day and a really easy 20-25m stretch routine day, I simply skip the rest day and tag on the stretching routine to the tail end of the last workout of the week. So the program&#8217;s 2 weeks end up being 10 days for me. To make it still be 6 calendar weeks I simply repeat week 1 or week 5 in the end (week 1 is actually one of the hardest weeks if you push the weights up). For reference - I am sharing my workout sheets <a href="https://docs.google.com/document/d/15WEBWvQ095jwGwg1DS_jHmzMD2OrZr9DgTgV7P7VjFA/edit?usp=sharing">here</a>.</p><p>Below are two blocks each matching alternating week in the program.</p><p>Weeks 1/3/5:</p><div class="latex-rendered" data-attrs="{&quot;persistentExpression&quot;:&quot;\\begin{array}{c c c c}\n\\textbf{Core workout} &amp; \\textbf{Supplemental 1} &amp; \\textbf{Supplemental 2} &amp; \\textbf{Supplemental 3} \\\\\n\\hline\nPush &amp; 10m\\ Ab\\ Hummer  &amp; Short\\ Intervals  &amp; 30m \\ Zone\\ 2 \\\\\nLegs &amp; P90X3\\ Challenge  &amp; 30m \\ Zone\\ 2 \\\\\nEndurance\\ \\&amp;\\ Agility &amp; P90X\\ Ab\\ Ripper  &amp; 50m \\ Zone\\ 2 \\\\\nPull &amp; IM30\\ 10m\\ Abs &amp; Long\\ Intervals &amp; 30m \\ Zone\\ 2 \\\\\nFull\\ Body\\ Tempo &amp; Range\\ \\&amp;\\ Repair &amp; 50m \\ Zone\\ 2 \\\\\n\n\\end{array}&quot;,&quot;id&quot;:&quot;GGMEAQIFIO&quot;}" data-component-name="LatexBlockToDOM"></div><p>Weeks 2/4/6:</p><div class="latex-rendered" data-attrs="{&quot;persistentExpression&quot;:&quot;\\begin{array}{c c c c}\n\\textbf{Core workout} &amp; \\textbf{Supplemental 1} &amp; \\textbf{Supplemental 2} &amp; \\textbf{Supplemental 3} \\\\\n\\hline\nTotal\\ Body\\ Push\\ Pull &amp; 10m\\ Ab\\ Hummer  &amp; Short\\ Intervals  &amp; 30m \\ Zone\\ 2 \\\\\nStrength\\ \\&amp;\\ Power  &amp; 50m \\ Zone\\ 2 \\\\\nCardio\\ \\&amp;\\ Core &amp; P90X3\\ Challenge  &amp; 30m \\ Zone\\ 2 \\\\\nIsometrics &amp; IM30\\ 10m\\ Abs &amp; Long\\ Intervals &amp; 30m \\ Zone\\ 2 \\\\\nCrucible &amp; Range\\ \\&amp;\\ Repair &amp; 50m \\ Zone\\ 2 \\\\\n\n\\end{array}&quot;,&quot;id&quot;:&quot;LJIUQCJXIC&quot;}" data-component-name="LatexBlockToDOM"></div><p></p><ul><li><p><a href="https://www.beachbodyondemand.com/programs/p90x3/start-here?locale=en_US">P90X3</a> Challenge consists of four blocks. Each block has two sets of pull-ups followed by push-ups (different variations). My current numbers are: 15 pull-ups, 40 push-ups per set. That&#8217;s 120 pull-ups and 320 push-ups per workout. There&#8217;s also a burnout at the end &#8212; 2 pull-ups and 4 push-ups, no rest, for 6 sets &#8212; so add 12 pull-ups and 24 push-ups to the total.</p></li><li><p>10min Ab Hummer is from <a href="https://www.beachbodyondemand.com/programs/the-masters-hammer-and-chisel/start-here?locale=en_US">Hammer and Chisel</a> program, P90X Ab Ripper is from <a href="https://www.bodi.com/us/en/s/fitness/p90x?code=Shopping_BOD_SMART_GOOGLE&amp;utm_campaign=BB_US_PLA_GGL_PerformanceMax_ProgramsBR&amp;utm_content=21698947640_&amp;utm_term=&amp;g_acctid=821-296-1770&amp;g_adgroupid=&amp;g_adid=&amp;g_adtype=none&amp;g_campaign=BB_US_PLA_GGL_PerformanceMax_ProgramsBR&amp;g_campaignid=21698947640&amp;g_keyword=&amp;g_keywordid=&amp;g_network=x&amp;gclsrc=aw.ds&amp;nb_adtype=&amp;nb_ap=&amp;nb_fii=&amp;nb_kwd=&amp;nb_li_ms=&amp;nb_lp_ms=&amp;nb_mi=&amp;nb_mt=&amp;nb_pc=&amp;nb_pi=&amp;nb_placement=&amp;nb_ppi=&amp;nb_si={sourceid}&amp;nb_ti=&amp;nbt=nb%3Aadwords%3Ax%3A21698947640%3A%3A&amp;gad_source=1&amp;gad_campaignid=21692483691&amp;gclid=CjwKCAjwuePGBhBZEiwAIGCVS_Gq1bOqaavHimcR4HvsA7h1UhD3I5tu3ZGynnrRMKpyVGTYCBIhpRoCZmoQAvD_BwE">P90X</a> and 10min Abs is from <a href="https://www.beachbodyondemand.com/programs/insanity-max30/start-here?locale=en_US">Insanity Max 30.</a></p></li></ul><p><strong>Intervals</strong></p><ul><li><p><strong>Long intervals:</strong> 3-min warm-up jog; 3 &#215; 5-min runs with 2-min walks in between; cool-down jog/walk.</p></li><li><p><strong>Short intervals:</strong> 3-min warm-up jog; 9 &#215; 1-min max-effort sprints with 1-min walks; cool-down jog/walk.</p></li></ul><ul><li><p><strong>When time is short</strong>, abs get dropped first. If Zone 2 is 50 min, I cut it to 30; if Zone 2 follows intervals, I drop it.</p></li></ul><p>The form is critical for these workouts, especially mental fatigue sets in. Seemingly lower weights can be deceptive - do not raise the weights unless you can complete the exercise with nearly a perfect form, as the risk of injury with complex power moves goes up considerably.</p><p><strong>Note</strong>: doing exercises with a perfect form is an easy way to enhance effectiveness  with lower weights and much reduced risk of injury.</p><p><strong>When traveling</strong></p><p>When traveling and the hotel gym with weights is available, both circuits can continue, depending on the space and how crowded the place is. When no gym is available I normally switch to Insanity Max 30 Month Two workouts for the duration. Those are body weight only and while not sufficient by themselves, they are decent enough for maintenance while traveling.</p><p>Again - this is what I do. Consistency and a purposeful plan are keys and there are many other ways to stay in shape.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[Updated workout routine - Part 1/2]]></title><description><![CDATA[&#8220;Key to performance: fit the benchmark.&#8221; &#8212;unspoken rule at major AI labs]]></description><link>https://www.equationblog.com/p/updated-workout-routine-part-12</link><guid isPermaLink="false">https://www.equationblog.com/p/updated-workout-routine-part-12</guid><dc:creator><![CDATA[Ruslan Belkin]]></dc:creator><pubDate>Sat, 23 Aug 2025 14:15:23 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!-MQk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0187b1d4-c470-4ffa-a463-efc198a6864f_1280x1280.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em>&#8220;Key to performance: fit the benchmark.&#8221;</em>  &#8212;unspoken rule at major AI labs</p><div><hr></div><p><a href="https://my.clevelandclinic.org/health/diagnostics/10683-dexa-dxa-scan-bone-density-test">DEXA</a> Update (August, 2025):</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!r5dI!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa225d11-33fc-42c9-90da-f82391a9e3d1_1912x224.png 424w, https://substackcdn.com/image/fetch/$s_!r5dI!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa225d11-33fc-42c9-90da-f82391a9e3d1_1912x224.png 848w, https://substackcdn.com/image/fetch/$s_!r5dI!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa225d11-33fc-42c9-90da-f82391a9e3d1_1912x224.png 1272w, https://substackcdn.com/image/fetch/$s_!r5dI!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa225d11-33fc-42c9-90da-f82391a9e3d1_1912x224.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!r5dI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa225d11-33fc-42c9-90da-f82391a9e3d1_1912x224.png" width="1456" height="171" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/aa225d11-33fc-42c9-90da-f82391a9e3d1_1912x224.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:171,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:61973,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://www.equationblog.com/i/170830684?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa225d11-33fc-42c9-90da-f82391a9e3d1_1912x224.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!r5dI!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa225d11-33fc-42c9-90da-f82391a9e3d1_1912x224.png 424w, https://substackcdn.com/image/fetch/$s_!r5dI!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa225d11-33fc-42c9-90da-f82391a9e3d1_1912x224.png 848w, https://substackcdn.com/image/fetch/$s_!r5dI!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa225d11-33fc-42c9-90da-f82391a9e3d1_1912x224.png 1272w, https://substackcdn.com/image/fetch/$s_!r5dI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa225d11-33fc-42c9-90da-f82391a9e3d1_1912x224.png 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><p>As you can see - total body fat went up by 3.3 pounds, while total lean mass went up by 4.4 pounds since the last scan. As you can see - it is actually very difficult to increase lean mass without gaining some fat mass.</p><div><hr></div><p>I&#8217;ve recently tweaked my workout schedule, shifting more toward muscle and <a href="https://en.wikipedia.org/wiki/VO2_max">VO2max</a> maintenance.</p><p>In my <a href="https://www.equationblog.com/p/my-current-workout-routine">previous</a> post I described the prior routine: two blocks &#8212; a 6-week full-body (mostly power-focused) functional block, followed by a 4-week weight-lifting block.</p><p>In this post, I&#8217;m outlining a modified weight-lifting block that&#8217;s now 6 weeks to match the functional power block.</p><p>As before, I work out first thing in the morning (work scheduling), with a 5:30 am wake-up, quick oral/face hygiene, pre-workout <a href="https://www.equationblog.com/p/the-supplements">supplements</a>, and a 10-15 min meditation. </p><p>Because the blocks are a bit longer, I&#8217;m skipping pre-workout foam rolling during the work week. If I had unlimited time, I&#8217;d keep it.</p><p>I still use guided workouts (<a href="https://www.bodi.com/us/en/s/subscription?code=SEMB_BODI_GOOGLE&amp;g_acctid=821-296-1770&amp;g_adgroupid=166824054903&amp;g_adid=733099013197&amp;g_adtype=search&amp;g_campaign=SEM_Brand_All&amp;g_campaignid=21755424360&amp;g_keyword=beachbody&amp;g_keywordid=kwd-44858520&amp;g_network=g&amp;gad_campaignid=21755424360&amp;gad_source=1&amp;gclid=CjwKCAjwkvbEBhApEiwAKUz6-2h6-q4czV8nmyhecXe2sn_CuI1tjw6RDjhU3QcsCBlzqMjexYOB5BoCG38QAvD_BwE&amp;gclsrc=aw.ds&amp;nb_adtype=&amp;nb_ap=&amp;nb_fii=&amp;nb_kwd=beachbody&amp;nb_li_ms=&amp;nb_lp_ms=&amp;nb_mi=&amp;nb_mt=e&amp;nb_pc=&amp;nb_pi=&amp;nb_placement=&amp;nb_ppi=&amp;nb_si=%7Bsourceid%7D&amp;nb_ti=kwd-44858520&amp;nbt=nb%3Aadwords%3Ag%3A21755424360%3A166824054903%3A733099013197&amp;utm_campaign=SEM_Brand_All&amp;utm_content=21755424360_166824054903&amp;utm_medium=SEM&amp;utm_source=GOOGLE&amp;utm_term=beachbody_">Beachbody</a> &#8212; historical) mainly for timing and pacing, and my week starts on <strong>Sundays</strong>.</p><p>The general structure: a <strong>weight-lifting segment followed by a cardio segment</strong> (where I listen to podcasts or other educational audio).</p><p></p><div class="latex-rendered" data-attrs="{&quot;persistentExpression&quot;:&quot;\\begin{array}{c c c c}\n\\textbf{Core workout} &amp; \\textbf{Supplemental 1} &amp; \\textbf{Supplemental 2} &amp; \\textbf{Supplemental 3} \\\\\n\\hline\nBuild\\ Chest\\ \\&amp;\\ Tries &amp; Beast\\ Abs\\ Classic  &amp; Short\\ Intervals  &amp; 30m \\ Zone\\ 2 \\\\\nBuild\\ or\\ Bulk\\ Legs &amp; P90X3\\ Challenge  &amp; 30m \\ Zone\\ 2 \\\\\nBuild\\ Shoulders &amp; Beast\\ Abs  &amp; 50m \\ Zone\\ 2 \\\\\nBuild\\ Back\\ \\&amp;\\ Biceps &amp; 50m \\ Zone\\ 2 \\\\\nBulk\\ Chest &amp; Beast\\ Abs &amp; Long\\ Intervals &amp; 30m \\ Zone\\ 2 \\\\\nBulk\\ Arms &amp; T30\\ Pull &amp; 50m \\ Zone\\ 2 \\\\\nBulk\\ Shoulders &amp; T30\\ Chin\\ ups &amp; IM30\\ 10m\\ Abs&amp; 50m \\ Zone\\ 2 \\\\\n\n\\end{array}&quot;,&quot;id&quot;:&quot;LFBENJWANL&quot;}" data-component-name="LatexBlockToDOM"></div><p></p><ul><li><p>The core workouts are built around <a href="https://www.beachbodyondemand.com/programs/body-beast/start-here?locale=en_US">Body Beast</a> workout with additional components added on.</p></li><li><p><a href="https://www.beachbodyondemand.com/programs/p90x3/start-here?locale=en_US">P90X3</a> Challenge consists of four blocks. Each block has two sets of pull-ups followed by push-ups (different variations). My current numbers are: 15 pull-ups, 40 push-ups per set. That&#8217;s 120 pull-ups and 320 push-ups per workout. There&#8217;s also a burnout at the end &#8212; 2 pull-ups and 4 push-ups, no rest, for 6 sets &#8212; so add 12 pull-ups and 24 push-ups to the total.</p></li><li><p><a href="https://www.beachbodyondemand.com/programs/tough-mudder-t-minus-30/start-here?locale=en_US">T-30 </a>Chin-up and Pull-up routines are short (~12m) 2X chin-up/pull-up ladders followed by hanging leg raises, monkey bar simulation and a burn-out. My ladder is usually 2-4-8-16 (30 total in each ladder).</p></li></ul><p><strong>When time is short</strong>, abs get dropped first. If Zone 2 is 50 min, I cut it to 30; if Zone 2 follows intervals, I drop it.</p><p><strong>Intervals</strong></p><ul><li><p><strong>Long intervals:</strong> 3-min warm-up jog; 3 &#215; 5-min runs with 2-min walks between; cool-down jog/walk.</p></li><li><p><strong>Short intervals:</strong> 3-min warm-up jog; 9 &#215; 1-min max-effort sprints with 1-min walks; cool-down jog/walk.</p></li></ul><p><strong>Notes on the weight-lifting pattern</strong></p><p>Standard sets follow a <strong>light&#8211;medium&#8211;heavy&#8211;drop</strong>-set pattern.</p><ul><li><p><strong>Light set (15 reps):</strong> warm-up for the muscle group and a chance to fine-tune the form.</p></li><li><p><strong>Medium set (&#8776;12 reps):</strong> calibration (heavy but not maximal). Sometimes I skip medium and go straight to a heavy 8-rep set, then a second heavy set after 45&#8211;60 s rest.</p></li><li><p><strong>Heavy set (6&#8211;8 reps):</strong> stop 1&#8211;2 reps shy of failure, immediately followed by a drop set at medium weight for another 6&#8211;8 reps.</p></li></ul><p>There are no true power moves in this block, but I often simulate them by moving fast on the concentric and slowing down on the eccentric (e.g., a rapid pull-up followed by a slow controlled descent).</p><p><strong>Note 1</strong>: doing exercises with a perfect form is an easy way to enhance effectiveness  with lower weights and much reduced risk of injury.</p><p><strong>Note 2</strong>: shoulder upright row is bio mechanically not a great exercise as it puts shoulders into an impingement position (EZ-bar or not), so I replace it with a <strong>dumbbell high pull</strong> as the closest alternative.</p><p>As always&#8212;if the last set is too easy, add a few extra reps; next time, increase the weight. The key to performance is sufficient stimulus.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[Why Inverting AI Workflows Is Key to Enterprise Accuracy]]></title><description><![CDATA[How leveraging human judgment and reversing traditional AI workflows can overcome dirty data challenges.]]></description><link>https://www.equationblog.com/p/why-inverting-ai-workflows-is-key</link><guid isPermaLink="false">https://www.equationblog.com/p/why-inverting-ai-workflows-is-key</guid><dc:creator><![CDATA[Ruslan Belkin]]></dc:creator><pubDate>Sun, 08 Dec 2024 15:15:47 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!SSKm!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9f8f1cd-667b-4d51-9dc3-4f35c98e1acd_1024x1024.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>In this post, I want to highlight a persistent challenge that often stalls efforts to improve AI accuracy in enterprise environments. While many acknowledge that insufficient data can be a limiting factor&#8212;especially when proprietary information is needed to enhance a general-purpose large language model (LLM)&#8212;the real issue goes beyond mere scarcity. To achieve top-tier domain performance, enterprises may need to fine-tune LLMs so that they internalize specialized knowledge and context. Unlike Retrieval-Augmented Generation (RAG) or prompt-based methods, which can inject additional information at inference time without altering the model&#8217;s parameters, fine-tuning permanently integrates the provided training data into the model&#8217;s internal representations.</p><p>This direct integration makes data quality crucial. Fine-tuning can deliver significant improvements in accuracy and domain alignment, but only if the training data is both reliable and relevant. Unfortunately, many enterprise datasets&#8212;sourced from CRMs, ERPs, and other operational systems&#8212;are incomplete, inconsistent, and lack clear indicators of quality. These repositories often contain &#8220;dirty&#8221; data that, even after cleaning, remains difficult to differentiate in terms of importance or strategic value. Merely connecting an LLM to these data sources, hoping they will serve as a source of refined knowledge, is insufficient. Without a methodology to identify which subsets truly matter, organizations risk overwhelming the model with undifferentiated content, ultimately diluting its domain expertise rather than enhancing it.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>Only with robust frameworks for scoring, filtering, and weighting data can organizations ensure that their fine-tuned LLMs prioritize the most accurate, timely, and contextually relevant information. In the absence of these data differentiation strategies, the promise of fine-tuning remains out of reach, and valuable proprietary knowledge fails to translate into tangible performance gains for the AI model.</p><h2><strong>An Even Bigger Problem: Human Knowledge Gaps</strong></h2><p>Even if enterprise data could be perfectly cleaned, quantified and integrated, it still wouldn&#8217;t guarantee optimal AI-driven decisions. A significant portion of critical knowledge remains locked in human minds&#8212;varied in expertise, judgment, and subjectivity. This disparity is why organizations hold meetings, rely on structured decision-making processes, and depend on human interpretation to bridge data with hopefully sound judgment.</p><p>To overcome this challenge, enterprises must rethink their AI workflows. Rather than AI serving as a co-pilot to human decision-makers, it should take the role of the pilot. In this model, humans shift from being the central drivers of decision-making to acting as data providers or experts who contribute insights only when the AI specifically requests them. The ideal scenario minimizes human intervention, enabling the AI to steer autonomously.</p><p>Extending the aviation analogy, this transition must be continuous. Instead of a static handover, the AI system should operate within a near real-time decision loop, constantly refining its approach based on new information and feedback. In doing so, the enterprise moves toward a state where human judgment is available on demand, but rarely required at the forefront.</p><div><hr></div><h2><strong>A Real-World Example: Customer Service Chatbots</strong></h2><p>Let&#8217;s consider a simple but real-world scenario: a customer service chatbot. Imagine this bot has access to all relevant information&#8212;customer databases, Jira tickets, FAQs, and documented workflows. For simplicity, assume it achieves perfect accuracy in intent detection.</p><p>What failure modes remain?</p><ol><li><p>The data is incomplete (e.g., it doesn&#8217;t know the answer).</p></li><li><p>The data is wrong (e.g., outdated FAQs or incorrect workflows).</p></li></ol><p>In these scenarios, the fallback is often to engage a human. This is illustrative because it highlights what the AI system should ideally do: engage humans strategically when gaps arise.</p><div><hr></div><h3><strong>Addressing Failure Scenarios</strong></h3><p>Two critical questions emerge:</p><ol><li><p><strong>What should happen next after a failure is detected and a human is engaged?</strong></p></li><li><p><strong>How could the failure have been avoided in the first place?</strong></p></li></ol><p>For the first question, the optimal response is to escalate the issue to a knowledgeable human (or multiple humans for high-stakes decisions), supported by the AI system in data lookup and verification. These escalated humans are likely not the same as the first-line responder&#8212;they must have the expertise to guide the system to the best resolution. Think of this as a "pager duty" for data management.</p><p> For the second question, prevention lies in robust simulations. For example, problem queries could be simulated using past customer interactions, exploring various branches of inquiry to proactively identify gaps in data or workflows.</p><p>This approach scales beyond customer service to more complex systems, such as financial modeling, supply chain management, and strategic decision-making.</p><div><hr></div><h2><strong>Scaling the Solution</strong></h2><p>Inverting traditional workflows&#8212;placing AI in the pilot&#8217;s seat and using humans as expert data sources&#8212;can fundamentally reshape enterprise operations. When properly implemented, this approach not only taps into the best of human expertise on demand, but also ensures that AI systems maintain efficiency, consistency, and domain relevance over time. Yet, to realize these benefits, organizations must couple this human-in-the-loop paradigm with robust data-centric strategies, particularly when it comes to fine-tuning AI models. A carefully curated and differentiated dataset ensures that the AI&#8217;s internal parameters remain aligned with business goals and processes and reflective of the most reliable, high-quality information.</p><p>For this approach to succeed, enterprises should:</p><ol><li><p><strong>Automatically Query Human Input:</strong><br>Treat humans as dynamic, on-call domain experts who provide clarifications, judgments, and domain-specific insights exactly when the situation demands. This continuous, as-needed engagement ensures that fine-tuned models incorporate high-value human knowledge without depending too heavily on subjective inputs.</p></li><li><p><strong>Simulate Task Inputs:</strong><br>Integrate continuous simulation runs. By proactively refining data quality and selection, the AI system stays prepared to handle new inputs effectively.</p></li><li><p><strong>Continuously Update Data and Models</strong><br>Implement iterative pipelines that not only refine data selection, weighting, and filtering, but also continuously adjust the model&#8217;s fine-tuning parameters as it encounters new scenarios.</p></li></ol><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!SSKm!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9f8f1cd-667b-4d51-9dc3-4f35c98e1acd_1024x1024.webp" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!SSKm!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9f8f1cd-667b-4d51-9dc3-4f35c98e1acd_1024x1024.webp 424w, https://substackcdn.com/image/fetch/$s_!SSKm!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9f8f1cd-667b-4d51-9dc3-4f35c98e1acd_1024x1024.webp 848w, https://substackcdn.com/image/fetch/$s_!SSKm!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9f8f1cd-667b-4d51-9dc3-4f35c98e1acd_1024x1024.webp 1272w, https://substackcdn.com/image/fetch/$s_!SSKm!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9f8f1cd-667b-4d51-9dc3-4f35c98e1acd_1024x1024.webp 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!SSKm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9f8f1cd-667b-4d51-9dc3-4f35c98e1acd_1024x1024.webp" width="1024" height="1024" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f9f8f1cd-667b-4d51-9dc3-4f35c98e1acd_1024x1024.webp&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1024,&quot;width&quot;:1024,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:496458,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/webp&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!SSKm!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9f8f1cd-667b-4d51-9dc3-4f35c98e1acd_1024x1024.webp 424w, https://substackcdn.com/image/fetch/$s_!SSKm!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9f8f1cd-667b-4d51-9dc3-4f35c98e1acd_1024x1024.webp 848w, https://substackcdn.com/image/fetch/$s_!SSKm!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9f8f1cd-667b-4d51-9dc3-4f35c98e1acd_1024x1024.webp 1272w, https://substackcdn.com/image/fetch/$s_!SSKm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9f8f1cd-667b-4d51-9dc3-4f35c98e1acd_1024x1024.webp 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p></p><p>Enterprise AI systems can&#8217;t succeed if they&#8217;re treated as mere co-pilots. The future of AI in the enterprise requires rethinking workflows, prioritizing AI as the primary decision-maker, and leveraging humans as on-demand knowledge sources&#8212;brought into the loop only when their judgment or domain expertise is truly needed. This shift in perspective not only demands robust strategies for filtering, scoring, and fine-tuning proprietary data, but also calls for continuous refinement of how organizations manage, interpret, and integrate information into AI models.</p><p>By giving the AI system the reins, enterprises can better address the persistent issues of &#8220;dirty&#8221; data, the hidden complexities of human judgment, and the inherent challenges of operating at scale. The goal is not to diminish human input, but to transform it into a strategic resource that the AI can query as needed. This inversion&#8212;where humans assist AI, rather than the other way around&#8212;represents the logical next step in unlocking the full potential of AI within the future enterprise landscape.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[Eating out hacks]]></title><description><![CDATA[Disclaimer: The following is based on my personal experience and is not intended as medical advice.]]></description><link>https://www.equationblog.com/p/eating-out-hacks</link><guid isPermaLink="false">https://www.equationblog.com/p/eating-out-hacks</guid><dc:creator><![CDATA[Ruslan Belkin]]></dc:creator><pubDate>Sat, 14 Sep 2024 14:15:58 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0187b1d4-c470-4ffa-a463-efc198a6864f_1280x1280.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><strong>Disclaimer:</strong> The following is based on my personal experience and is not intended as medical advice. Everyone's body and health situation are different, and what works for one person may not work for another. Always consult with a healthcare professional or physician before starting or changing any medication, supplement, or diet plan.</p><p>Let&#8217;s face it&#8212;we can&#8217;t always control what we eat, especially if we want to participate in business and social life.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>You eat at work, attend business or family functions, year-end parties, and let&#8217;s not even mention holidays and vacations. The energy balance equation can quickly get out of whack.</p><p>Let&#8217;s first address the topic everyone is talking about: GLP-1 receptor agonists. While I don&#8217;t have personal experience with them (don&#8217;t need it), I can see how they could be beneficial for many people. From what I understand, I don&#8217;t share many of the concerns about them. In biology, there&#8217;s no such thing as a "free lunch"&#8212;there&#8217;s always a risk/reward balance, and in the general case, being overweight and metabolically unhealthy presents a much higher risk.</p><p>GLP-1 agonists, such as <strong><a href="https://www.drugs.com/semaglutide.html">semaglutide</a></strong> (sold under brand names like Ozempic and Wegovy), <strong><a href="https://www.drugs.com/mtm/liraglutide.html">liraglutide</a></strong> (Saxenda), and <strong><a href="https://www.drugs.com/tirzepatide.html">tirzepatide</a></strong> (Mounjaro), have been used for years in the treatment of diabetes and off label in the body building community. These medications work by enhancing insulin secretion, slowing gastric emptying, and increasing satiety, leading to reduced food intake. The mechanism of action is well understood. However, the downsides are also known. One of the primary concerns is the loss of muscle mass along with fat, which can be a serious issue for sedentary individuals. So yes&#8212;if you&#8217;re considering using these medications, lift weights and eat plenty of protein.</p><p>There are some lesser-known issues, such as reduced heart rate variability (HRV), so I wouldn&#8217;t suggest casually using these drugs just to lose 10 pounds. But honestly, I see a lot of people who could have benefited from starting on them yesterday.</p><p>As for nutraceuticals, there are plenty of marketing claims about manipulating GLP-1 levels with probiotics, herbs, and other supplements. However, these claims are not supported by solid science. Mechanistically, to achieve the same effect as pharmaceutical GLP-1 agonists, you would need a 1,000-fold increase in the efficacy of these substances. So don&#8217;t bother.</p><p>The second approach is purely behavioral and has worked for me and many people I know&#8212;aligning your meal schedule with your circadian rhythm. This means eating three meals (breakfast, lunch, and dinner) at as consistent a time as possible each day, while absolutely avoiding any snacking between meals. Doing this helps prevent the constant need to eat by keeping your body in a more natural rhythm.</p><p>Regarding hunger between meals, there&#8217;s a remedy that can help (though, if you avoid snacking long enough, you will not need it). This involves using <strong>bitters</strong>, an ancient remedy believed to stimulate digestion and reduce appetite. Bitters work by activating bitter taste receptors in the mouth and gastrointestinal tract, which in turn stimulates the release of digestive enzymes and bile, potentially promoting a feeling of fullness or reducing cravings.</p><p>One specific supplement is <strong>Amarasate Extract</strong>, derived from New Zealand&#8217;s <strong>Gentian root</strong>. It contains bitter compounds that target receptors in the gut, signaling the brain to trigger satiety, which can help suppress appetite between meals. While these supplements aren&#8217;t highly effective for everyone, they can be a useful temporary hack until you get into the no-snacking habit.</p><p>Now, assuming you&#8217;ve got the basics under control but find yourself at a function with an overwhelming amount of tasty food (think authentic pasta dishes), self-control might not be enough&#8212;except for leaving early (which is often a good idea for other reasons).</p><p>First, <strong>front-loading with protein-rich foods</strong> when available (instead of carbs or alcohol) can help. This will increase satiety upfront, making you less likely to overeat.</p><p>Secondly, there are two pharmaceutical options you can consider:</p><ol><li><p><strong><a href="https://www.drugs.com/pro/orlistat-capsules.html">Orlistat</a></strong> blocks <strong>pancreatic lipase</strong>, the enzyme necessary to digest fats. The over-the-counter dose is 60 mg, which is mildly effective, while the prescription dose of 120 mg is more effective, reducing fat calorie absorption by about 30%. However, a warning: overuse will reliably lead to gastrointestinal side effects, especially diarrhea.</p></li><li><p><a href="https://www.drugs.com/mtm/acarbose.html">Acarbose</a> is a reversible alpha-glucosidase inhibitor that slows the breakdown of complex carbohydrates (such as pasta, rice, and bread) into simple sugars, delaying glucose absorption. Acarbose can lower postprandial blood glucose by about 20-40 mg/dL (closer to 20 mg/dL for me) per 25 mg dose, up to around 75 mg. Most of the calories from these carbs will eventually be absorbed, but some may pass into the large intestine, where they are fermented by gut bacteria. Beyond metabolic benefits, avoiding blood sugar spikes helps control hunger. Fun fact: Acarbose is only one of 3 drugs found to prolong lifespan in <a href="https://www.nia.nih.gov/research/dab/interventions-testing-program-itp">ITP</a> (Interventions Testing Program) studies (the other two being <a href="https://www.drugs.com/mtm/sirolimus.html">rapamycin</a> and <a href="https://www.drugs.com/drug-class/sglt-2-inhibitors.html">SGLT2 inhibitors</a>).</p></li></ol><p>Both of these medications must be taken at the right time&#8212;specifically with meals. The official recommendation is to take them with the first bite of food, but I&#8217;ve personally found they work a little better if you take them a few bites into the meal.</p><p>When it comes to simple sugars, that&#8217;s the toughest challenge. Unfortunately, there&#8217;s no highly effective remedy for handling them (aside from SGLT2 inhibitors, which wouldn&#8217;t qualify as a party hack). One supplement with some evidence, though not strong, is <strong>Gymnema Sylvestre extract</strong>. This herb contains compounds known as gymnemic acids, which are thought to work by blocking the sugar receptors on your taste buds, reducing the sweetness sensation and potentially curbing sugar cravings. It may also slow the absorption of sugar in the intestines by interacting with glucose transporters. However, while Gymnema Sylvestre might help a little if taken about 30 minutes before a meal, especially with sugary foods, it&#8217;s far from a magic bullet for controlling simple sugars.</p><p>In conclusion, these strategies can help you shave off some extra calories and reduce hunger, allowing you to get through meals with minimal damage.</p><p><em>Hope this helps and love to hear your feedback,</em></p><p><em>Ruslan</em></p><p><strong>Note</strong>: when I say something does or does not work for me, I rely on objective measurements, such as CGM in real-time, as well as calorie tracking (using <a href="https://cronometer.com/">Cronometer</a> app), calorie expenditure (using Apple Watch),  body composition (a scale as well as periodic DEXA scans).</p><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[Differentiation in SaaS AI]]></title><description><![CDATA[A lot has happened in the last few months as the field has advanced (new models, new choices, lower prices), several companies have "inflected," and some things (CUDA dominance) have stayed the same.]]></description><link>https://www.equationblog.com/p/differentiation-in-saas-ai</link><guid isPermaLink="false">https://www.equationblog.com/p/differentiation-in-saas-ai</guid><dc:creator><![CDATA[Ruslan Belkin]]></dc:creator><pubDate>Thu, 12 Sep 2024 17:30:04 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0187b1d4-c470-4ffa-a463-efc198a6864f_1280x1280.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>A lot has happened in the last few months as the field has advanced (new models, new choices, lower prices), several companies have "inflected," and some things (CUDA dominance) have stayed the same.</p><p>What has remained elusive, however, is where differentiation will ultimately lie, specifically among startups.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>In the interest of minimizing noise, I will omit opining on the most common themes that other people have been discussing at length (inference vs. training hardware, open-source vs. closed-source models, large vs. small models, various picks and shovels, and fine-tuning&#8212;though the latter remains an unsolved problem for application developers). All these areas have one thing in common: they require a lot of money and are thus mostly suitable for larger players.</p><p>The question is&#8212;where is, or is there, a differentiated opportunity? (And by differentiated, I mean outside of dumping a ton of money into a seed-stage company against a deck and a team&#8212;which can be differentiating on its own&#8212;or likely not.)</p><p>Let&#8217;s look at the basic structure of an LLM-first SaaS app. What does it need?</p><ol><li><p>Access to a set of competent LLMs of different sizes at a reasonable cost and speed (many choices, nothing super differentiated).</p></li><li><p>A RAG system (i.e., pgvector)&#8212;not differentiated, especially with larger context windows, where the emphasis on retrieval accuracy is somewhat diminished.</p></li><li><p>An orchestration infrastructure: there&#8217;s basically a classic flow, such as intent detection, query rewriting, summarization, output checking, and correction. These frameworks will evolve, and there will be many options. In fact, with more and more LLM usage to write code itself, one could argue that frameworks like LangChain add more layers of complexity and problems than they solve&#8212;though I could be proven wrong. There could be new ways to think about frameworks in a world where most of the code would be generated.</p></li><li><p>A testing  and observability infrastructure&#8212;definitely a major pain, but the problem area is obvious and many people are working on it. The question is whether it will spur s slew of new SaaS companies, or - considering immaturity of frameworks it will be more tightly coupled with service providers offering models and data services?</p></li><li><p>Coding tools (Copilot, Cursor, etc.)&#8212;again, quite obvious, and various tools will exist wrapping around ever-better models. The IDE becomes more of a very complex prompting interface and a feedback collection tool. There are several possibilities here:</p><ul><li><p>A revolutionary new interface that we have not yet seen or thought of</p></li><li><p>A plethora of Cursor-like forks and VSCode plug-ins, though it&#8217;s difficult to see how they win, unless Microsoft simply yields. Cursor being superior to GitHub Copilot is mostly a function of Cursor RAG-ing your codebase, and GitHub being presumably more &#8220;privacy&#8221; conscious. A simple business decision by <a href="https://www.linkedin.com/in/jkevinscott/">Kevin Scott</a> can alter that equation overnight.</p></li></ul></li><li><p>Differential fine-tuning and associated dataset management&#8212;no great solutions, but probably best addressed by model providers and companies like <a href="https://scale.ai">Scale.AI</a>.</p><p></p></li></ol><p>Alright. Some winners could emerge (especially around testing, and observability). It will likely converge into a few larger winners, most closely tied to model and data providers.</p><p>What are we overlooking? Can building an LLM wrapper be intrinsically differentiated? By intrinsically, I mean: is there an accumulation of leverage that creates a competitive moat over time? There&#8217;s always an opportunity to build a better product&#8212;better UX, clever influencer marketing, superior GTM&#8212;that leads to the acquisition of market share, and that can be differentiated, albeit hardly a sure thing.</p><p>There is likely one area that can be a source of leverage. We know proprietary data is that source in AI. But that&#8217;s too unstructured; to be differentiated, it needs to lead to a vastly superior outcome that is not obtainable by just using commonly available models.</p><p>So, what form is that, and how might it work?</p><p>When we look at a classic SaaS workflow (for example, responding to a customer service query), what happens is quite specific to each customer in two ways:</p><ol><li><p>Prompts are likely different for each domain and further different for each customer.</p></li><li><p>Prompts involving function calling (i.e., opening a Jira ticket) are definitely different, especially if a custom integration is involved.</p></li><li><p>Workflow routing logic may be different (though this is technically outside LLM data).</p></li><li><p>Error checking and validation at the end of the action can likewise be very specific to a customer domain and environment.</p></li></ol><p>Moreover, in other cases where user feedback is part of the loop, prompts may be dynamically updated (via LLM).</p><p>So, these prompts, in fact, constitute a customer-specific and user-specific dataset that grows over time, requires deep knowledge of customer workflows (and thus is not easily replicable), and grows over time.</p><p>Is this enough differentiation? Unclear&#8212;I think it depends on whether business workflows can converge and be serviced by a more intelligent model. However, to do so will require understanding these workflows first. While I think ultimately it will converge, there&#8217;s probably room on the way there.</p><p>Love to hear your thoughts,  </p><p>Ruslan</p><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[The supplements]]></title><description><![CDATA[Update (Nov, 2026)]]></description><link>https://www.equationblog.com/p/the-supplements</link><guid isPermaLink="false">https://www.equationblog.com/p/the-supplements</guid><dc:creator><![CDATA[Ruslan Belkin]]></dc:creator><pubDate>Sun, 26 May 2024 14:15:57 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0187b1d4-c470-4ffa-a463-efc198a6864f_1280x1280.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div><hr></div><p><strong>Update (Nov, 2026)</strong></p><p>The rule I am using for this pass is simple: a mechanism has to be sufficiently distinct, supported by human evidence, and likely to add something not already covered more directly elsewhere in the stack. Mechanistic novelty alone is not enough.</p><p>Removed:</p><ul><li><p><strong><a href="https://www.amazon.com/gp/product/B07BH4JG35/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">PQQ</a>: 1 cap, 20mg (AM)</strong></p><ul><li><p>PQQ and Urolithin A are not mechanistic duplicates: PQQ is proposed to stimulate mitochondrial biogenesis, while Urolithin A targets mitophagy. But a distinct mechanism is not enough on its own. Exercise already provides the stronger biogenesis stimulus, the small 20mg human trial increased a mitochondrial signaling marker without improving aerobic performance, and another personal trial produced no reproducible subjective or objective benefit. With no demonstrated incremental value, PQQ did not earn a permanent place in the stack.</p></li></ul></li><li><p><strong><a href="https://nootropicsdepot.com/smart-ps-phosphatide-complex-softgel-capsules-100mg-phosphatidylserine/">Smart PS&#8482; Phosphatidylserine</a>: 1 serving, 1 cap</strong></p><ul><li><p>Phosphatidylserine itself is not a duplicate of Alpha-GPC, EPA/DHA, or Bacopa, but this formulation was not strong enough to justify its own slot. One softgel provides 100mg phosphatidylserine&#8212;below the roughly 300&#8211;400mg used in most positive cognition and stress trials&#8212;plus only 25mg phosphatidylcholine and 2.5mg phosphatidylethanolamine, amounts too small to matter in an already choline- and phospholipid-rich stack. Without a defined memory or stress target or a reproducible personal result, this was an underdosed, untargeted layer of cognitive redundancy.</p></li></ul></li><li><p><strong><a href="https://www.amazon.com/gp/product/B000POSA2Q/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">Aged Garlic Extract</a>: 1/2 serving, 1 cap, 300mg 2x/day</strong></p><ul><li><p>The original rationale was to hedge against possible TMAO production from ALCAR and choline precursors. ALCAR is now gone, and direct human evidence for aged garlic lowering TMAO is weak. At 600mg/day, the dose was also below the 1.2g/day used in a positive blood-pressure trial. The remaining plausible effects&#8212;modest changes in blood pressure, endothelial function, lipids, oxidative stress, and platelet activity&#8212;mostly duplicate endpoints already addressed more directly by telmisartan, tadalafil, rosuvastatin/ezetimibe, and EPA/DHA. That left pharmacodynamic clutter without a clear new target.</p></li></ul></li></ul><p><strong>Update (Mar, 2026)</strong></p><p><strong>Added:</strong></p><ul><li><p><strong><a href="https://www.amazon.com/dp/B01A68H2BA?ref=ppx_yo2ov_dt_b_fed_asin_title&amp;th=1">TUDCA</a>: 1 service, 250mg AM/PM</strong></p><ul><li><p>TUDCA is here primarily as a liver and bile-flow support compound. Unlike many &#8220;liver support&#8221; botanicals, it is a bile acid derivative, so the rationale is more targeted: it helps make the bile acid pool more hydrophilic, may reduce stress on hepatocytes, and has a cleaner mechanistic case in the setting of a crowded stack plus mild liver-enzyme drift. I&#8217;m treating it as pragmatic insurance rather than a performance supplement &#8212; the goal is preserving liver resilience and bile handling, not expecting any noticeable day-to-day effect.</p></li></ul></li></ul><p><strong>Removed:</strong></p><ul><li><p><strong><a href="https://www.amazon.com/gp/product/B00T5A1SD4/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">Acetyl L-Carnitine HCI</a></strong>: 1 serving, 1g</p><ul><li><p>It does help transport fatty acids into mitochondria, acetyl-CoA group donor providing brain health benefits and good nootropic effects, and I did feel moderate benefits for the workouts. The downside - it feeds bacteria in the gut that produce TMAO. While TMAO impact is not completely settled, I am removing ALCAR on risk/reward basis, especially considering absorption problems with oral forms in general.</p></li></ul></li><li><p><strong><a href="https://www.amazon.com/gp/product/B07WVSYH5P/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">Cordyceps</a></strong>: 1 serving, 2g</p><ul><li><p>Preliminary research suggested that Cordyceps may have positive effects on telomere length. Subjectively I did not feel any effect and upon further examination telomere length rationale is quite weak. Removing to simplify the stack.</p></li></ul></li></ul><p></p><p><strong>Removed</strong>:</p><ul><li><p><strong><a href="https://www.amazon.com/dp/B001DNV5CA?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1&amp;th=1">EGCG</a></strong> 1/2 serving, 1 cap, 200mg 2x/day</p><ul><li><p>Originally the thesis here was colorectal risk mitigation (there is mechanistic plausibility + some human data in the adenoma space), but on the latest labs I saw a mild upward drift in liver enzymes. Green tea extracts are one of the more common &#8220;usually fine, but occasionally not&#8221; botanicals from a liver standpoint &#8212; and when you&#8217;re monitoring LFTs, anything with even a small hepatotoxicity signal is a poor fit. Net: risk/reward no longer pencils out for me right now, so it&#8217;s out until LFTs normalize and a future re-challenge (if any) can be done cleanly.</p></li></ul></li><li><p><strong><a href="https://www.amazon.com/dp/B00LPJJHRC?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1">AC-11</a></strong>  1cap 350mg into the morning shake</p><ul><li><p>This was a pure experiment around the &#8220;DNA repair / resilience&#8221; claims. After a reasonable run, the subjective benefit was barely detectable and I couldn&#8217;t find anything objective to justify keeping it in. Given how crowded the stack already is, anything that isn&#8217;t clearly pulling its weight gets cut.</p></li></ul></li><li><p><strong><a href="https://www.amazon.com/dp/B07H8NK8Y1?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1&amp;th=1">Gotu Kola extract</a></strong>: 1/2 serving, 1 cap, 120mg</p><ul><li><p>I like Gotu Kola for microcirculation / skin-collagen support, but similarly to EGCG it&#8217;s a botanical with rare (but real) hepatotoxicity case reports. With LFTs already trending the wrong way, this becomes an obvious &#8220;first things to pull&#8221; candidate. Removed for now.</p></li></ul></li></ul><p><strong>Added</strong>:</p><ul><li><p><strong><a href="https://www.amazon.com/dp/B0DNMWS7WX?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1">Urolithin A</a></strong>, 1/2 service, 1 cap, 1000mg, PM</p><ul><li><p>Urolithin A is the only item in this stack aimed primarily at mitochondrial quality control through mitophagy&#8212;the selective clearance of damaged mitochondria. That is different from creatine&#8217;s ATP buffering, CoQ10&#8217;s role in electron transport, and exercise-driven mitochondrial biogenesis. Supplementing also bypasses the large person-to-person variation in producing urolithins from food. Human trials show bioavailability and mitochondrial target engagement, with some signals in muscle endurance and strength, but no consistent improvement in walking distance, ATP production, or vascular function. That earns it a monitored experiment, not high conviction.</p></li></ul></li><li><p><strong><a href="https://www.amazon.com/gp/product/B07BH4JG35/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">PQQ</a></strong>: 1 cap, 20mg (AM)</p><ul><li><p>PQQ is a redox-active compound with a plausible role in mitochondrial biogenesis / cellular energy metabolism, and there are a few small human studies with signals on inflammation markers and cognitive endpoints &#8212; nothing definitive. I previously ran it and couldn&#8217;t find a clean personal signal, so conviction remains low. I&#8217;m re-running it only because it pairs conceptually with Urolithin A (different mechanism, same &#8220;mitochondrial cleanup + support&#8221; direction) and it&#8217;s easy to discontinue if it stays silent.</p></li></ul></li><li><p><strong><a href="https://www.amazon.com/dp/B0D3SNSJ47?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1">Colostrum</a></strong> 1 scoop, 1g, morning shake</p><ul><li><p>Colostrum is a &#8220;gut insurance&#8221; lever for me: bovine colostrum concentrates immunoglobulins (IgG), lactoferrin, and other bioactives that support barrier integrity and innate immune function. The best human data I&#8217;ve seen is in athletes / high training stress contexts, where colostrum has been associated with improved gut permeability markers and fewer URTI-type issues. I&#8217;m using a conservative dose (1g/day) and treating it as supportive &#8212; not magic. Avoid if dairy allergy is a concern.</p></li></ul></li></ul><p><strong>Updated</strong>:</p><ul><li><p><strong><a href="https://nootropicsdepot.com/synapsa-bacopa-monnieri-extract-whole-plant-powder/">Bacopa Monirelli</a></strong> (Synapsa): 1 serving, 320mg</p><ul><li><p>Formulation changed to <a href="https://www.amazon.com/dp/B0BQWY5BB9?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1&amp;th=1">Cognance</a> 1 serving, 100mg with rationale being a better overall profile with no potential sedation effects.</p></li></ul><p></p></li></ul><p><strong>Update (Oct, 2025)</strong></p><ul><li><p><strong>Replaced <a href="https://www.amazon.com/dp/B07QNMCXHG?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1">Reishi mushroom powder</a> </strong>with<strong> <a href="https://www.amazon.com/dp/B0F5F24B1K?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1&amp;th=1">Turkey Tail </a>mushroom powder</strong>: 1 serving, 1g (2 days on, 1 day off - non overlapping with Lion&#8217;s Mane).<strong> </strong>Reasons:</p><ul><li><p><strong>Safety/monitoring</strong>&#8212;Reishi has rare but documented hepatotoxicity case reports and inclusion in LiverTox, which matters if you&#8217;re watching LFTs; Turkey Tail has a long clinical track record as an adjuvant (PSK) with generally favorable tolerability.</p></li><li><p><strong>Evidence fit</strong>&#8212;Turkey Tail&#8217;s PSK/PSP have been studied extensively as immune modulators (including RCTs), with signals on NK-cell activity and broader immune function, making it a reasonable like-for-like immunologic substitute</p></li><li><p><strong>Potential 5-AR inhibition</strong>: There&#8217;s some weak evidence that Reishi may have an anti-androgenic potential, which we obviously don&#8217;t want</p></li></ul></li><li><p><strong>Added another</strong> <strong><a href="https://www.amazon.com/gp/product/B00E9M4XFI/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">Creatine</a></strong>: 1 serving, 5g to the morning shake for 10g total (between pre-workout and the morning shake). New research strongly suggest cognitive benefits (especially when short on sleep) from this dosing and subjectively I feel a noticeable boost and no afternoon slump at all.</p></li></ul><p><strong>Update (Aug, 2025)</strong></p><p><strong>Added</strong>:</p><ul><li><p>Tongat Ali was replaced <a href="https://nootropicsdepot.com/eurycomax-optimized-tongkat-ali/?searchid=14818292&amp;search_query=eury">Eurycomax</a> and correspondingly 2.5mg of Pregnenolone was removed. In total between Cistamax, Eurycomax and direct supplementation DHEA and Pregnenolone dosages stayed at 5mg each.  So far appears to be a much smoother stack.</p></li><li><p><strong><a href="https://nootropicsdepot.com/maca-extract-powder/?searchid=14818332&amp;search_query=maca+root">Maca</a></strong> extract powder was added to the morning shake at 200mg. See notes.</p></li><li><p><strong><a href="https://nootropicsdepot.com/search.php?search_query=gotu%20kola">Gotu Kola</a></strong> was added into the morning shake along with the evening dose.</p></li><li><p><strong><a href="https://www.amazon.com/dp/B001DNV5CA?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1&amp;th=1">EGCG</a></strong> was added into the morning shake along with the evening dose. See notes.</p></li><li><p><strong><a href="https://www.amazon.com/dp/B0014UECK4?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1">Stinging Nettle Root Root</a></strong> extract was added into the morning shake along with the evening dose.</p></li><li><p><strong>NMN/NR</strong> stack was restructured and replaced with <a href="https://www.amazon.com/dp/B07TK4VYWS?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1&amp;th=1">NR</a> (Tru Niagen) plus Pterostilbene+Quercetin. See detailed discussion.</p></li><li><p><strong><a href="https://www.amazon.com/dp/B00LPJJHRC?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1">AC-11</a></strong> was added experimentally, 1cap 350mg into the morning shake. So far seem to generate a slight mental kick. Data-wise not yet convinced.</p></li><li><p><strong>Creatine</strong> was moved from morning shake to pre-workout. Seemed to make a significant difference, probably due to transient DHT elevation.</p></li><li><p><strong><a href="https://www.drugs.com/rosuvastatin.html">Rosuvastatin</a></strong> (Crestor) along with <strong><a href="https://www.amazon.com/dp/B0CKC3TXCC?psc=1&amp;ref=ppx_yo2ov_dt_b_product_details">Geranylgeraniol</a></strong> were moved from <strong>AM</strong> to <strong>PM</strong> to avoid (theoretical) transporter conflicts with other supplements.</p></li><li><p><strong><a href="https://www.amazon.com/dp/B07ZPH99CQ?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1&amp;th=1">H2 Molecular Hydrogen</a> </strong>was added along with the morning supplements. See notes.</p></li></ul><p><strong>Removed</strong>:</p><ul><li><p><strong><a href="https://www.amazon.com/gp/product/B0036THML2/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1https://www.amazon.com/gp/product/B0036THML2/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;th=1">Gaia Herbs Adrenal Health Daily</a> -</strong> I took this for years; it made my <a href="https://www.equationblog.com/p/my-personal-daily-routine">original list</a> because it seemed helpful during a heavy-stress period. That said, I&#8217;ve always been skeptical of &#8220;adrenal support&#8221; blends&#8212;sensible ingredients, but usually well below clinically studied doses. After multiple on/off cycles, I&#8217;m convinced it isn&#8217;t doing anything meaningful (and at worst may slightly lower morning cortisol from the small amount of ashwagandha).</p></li></ul><p><strong>Update (Sep, 2024</strong>):</p><p><strong>Added</strong>:</p><ul><li><p><strong>Spermidine </strong>(see AM morning shake section)</p></li><li><p><strong>Magnesium Glycinate</strong>, 200mg in the evening replacing Magnesium complex</p></li></ul><p><strong>Removed</strong>:</p><ol><li><p><strong><a href="https://www.amazon.com/dp/B08NCFV5RJ?psc=1&amp;ref=ppx_yo2ov_dt_b_product_details">C60</a></strong> (in olive oil): 1 tsp</p><p>The premise: an interesting compound due to it&#8217;s direct and unique anti-oxidant action that doesn&#8217;t blunt effects of exercise.</p><p>It was an experiment and thus far I have not found neither subjective nor objective evidence of it doing anything (perhaps it was a problem with a specific brand).</p></li><li><p><strong><a href="https://www.amazon.com/gp/product/B07BH4JG35/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">PQQ</a></strong>: 1 cap, 20mg</p><p>The premise: research around mitochondrial support effects is quite solid. Quantitatively or subjectively I couldn&#8217;t see or feel anything. </p></li><li><p><strong>Probiotics</strong></p><p>After experimenting with several over the years, I found them to be useful for a specific purpose, but not as part of a maintenance routine.</p><p></p></li></ol><div><hr></div><p>Everyone is asking what supplements I take. I am going to share full list upfront and then we are going have a discussion.</p><p>Several disclaimers:</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><ul><li><p>This list is highly personalized and is periodically revised based on  quantitative testing, research updates, and subjective evaluation. What you see here is a snapshot as of today.</p></li><li><p>None of this is a medical advice. Please, do not apply this list to yourself without proper medical guidance.</p></li><li><p>It's crucial to remember that, from a scientific perspective, supplements (and even diet) are not as significant as one might think compared to exercise. Nutrition and supplement studies are typically of low quality and should be viewed with skepticism. They are most often based on epidemiology, which is unreliable due to confounding factors, biases, and difficulty in establishing causality. Although Mendelian randomization trials are more robust in inferring causality, they are limited in number due to costs and incentives. When they are done - these trials are often underpowered, too short, and tend to focus on populations that may not be relevant to our purposes.</p></li><li><p>Therefore, the totality of data, including observational data from various traditions, along with personal biomarkers, functional fitness metrics, and subjective evaluations, must all be considered.</p></li><li><p>Given the difficulty of evaluating efficacy, why do I pursue this? Partially, it's for the fun of biohacking.</p></li><li><p>The main goals for me are health span and performance (not longevity by itself - who wants to live longer without being able to do things anyway).</p></li></ul><p>Here&#8217;s how to read the list:</p><ul><li><p><strong>Supplement</strong>: (including the specific brand I use and dosages; note that I have no affiliation with any of the companies mentioned).</p></li><li><p><strong>Commentary</strong>: I&#8217;ll provide brief commentary without references (for the sake of my time). Accuracy is not guaranteed. Feel free to discuss it further with ChatGPT.</p></li><li><p><strong>Risk</strong>: low/medium/high (I don&#8217;t use anything I consider high risk). No supplement is zero risk as unknown effects may exist, and there could be discrepancies versus the label due to manufacturing or quality control issues.</p></li><li><p><strong>Personal outcome</strong>: none/minor/notable/significant - a subjective evaluation of the benefit based on an aggregate of biomarker, functional testing, and subjective feelings of well-being.</p></li><li><p><strong>Conviction</strong>: low/medium/high - my current level of conviction in the supplement's efficacy based on the aggregate of data, research and personal outcome.</p></li></ul><p>Let&#8217;s get right to it.</p><h1>AM - Pre-workout</h1><p>As you know from my <a href="https://open.substack.com/pub/ruslan/p/my-personal-daily-routine?r=jthe7&amp;utm_campaign=post&amp;utm_medium=web">earlier post</a> - I <a href="https://open.substack.com/pub/ruslan/p/my-current-workout-routine?r=jthe7&amp;utm_campaign=post&amp;utm_medium=web">workout</a> first thing in the morning and that drives some of the logic here (if I was to work out in the afternoon or the evening - the supplementation would be different)</p><p>The following supplements are mixed in as powders into a pre-workout drink:</p><p><strong><a href="https://www.amazon.com/gp/product/B09SP1Q3T9/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">Essential Amino Acids</a></strong> (EAAs): 1 serving, 11g</p><p>As I work out in the morning and we know that in a hypo-caloric state the body can break down muscle for energy, having circulating amino acids (in addition to the stimulus from exercise) helps preserve muscle mass. Essential Amino Acids (EAAs) are beneficial because they are already in their simplest form and are readily available for muscle protein synthesis. This makes them easier for the body to use immediately compared to whole proteins, which need to be broken down first. Additionally, EAAs are easier on the digestive system.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>: significant</p><p><strong>Conviction</strong>: high</p><p><strong><a href="https://www.amazon.com/gp/product/B00QYZ6MLG/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">Legion Pulse</a></strong> Pre-Workout drink: 1/2 serving, 11.75g</p><p>This product serves as both a nootropic supplement and a pre-workout aid. It&#8217;s a well-designed product with no filler or questionable ingredients. The main components are: a moderate dose of caffeine for increased energy and focus, L-Citrulline for nitric oxide enhancement and improved blood flow, Betaine/TMG which helps control homocysteine levels (this ingredient combines nicely with other supplements, such as NAD precursors) and has several performance benefits, L-Theanine for its calming and focus-enhancing effects, Alpha-GPC as an effective acetylcholine precursor for cognitive enhancement, and electrolytes (sodium and potassium) to support hydration and muscle function.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>: significant</p><p><strong>Conviction</strong>: high</p><p><strong><a href="https://www.amazon.com/gp/product/B00VAOKFO6/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">Taurine</a></strong>: 1 serving, 2g</p><p>The research on taurine is impressive on many levels, demonstrating benefits for cardiovascular health and muscle function. However, I am not sure it&#8217;s doing anything noticeable for me personally. Taurine supplementation might be more important for people on a vegetarian or vegan diet, as they may have lower taurine levels due to the absence of taurine-rich animal products in their diet.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>: none</p><p><strong>Conviction</strong>: low</p><p><strong><a href="https://www.amazon.com/gp/product/B081PBMGYJ/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">Collagen</a></strong>: 1 serving, 10.29g</p><p>Collagen helps complement the amino acid profile provided by EAAs, offering specific amino acids like glycine, proline, and hydroxyproline. Research suggests that taking collagen prior to a workout, assuming adequate exercise stimulus, can support connective tissue and joint health, with some evidence also indicating benefits for bone health. This particular product uses special formulation of collagen and has additional ingredients such as Hyaluronic Acid, ch-OSA, and Buffered Vitamin C, which are known for their synergistic benefits for joint health.</p><p><strong>A side note</strong>: excess collagen, due to conversion of hydroxyproline into oxalates. Between this dose and the morning shake it comes down to 20g / day for me,  which may be close to an upper safe dose.</p><p><strong>Risk</strong>: moderate</p><p><strong>Personal outcome</strong>: notable</p><p><strong>Conviction</strong>: high</p><p><a href="https://www.amazon.com/gp/product/B0791NCWPL/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">N-Acetyl L-Tyrosine</a> (occasionally): 1 serving, 0.4g</p><p>N-Acetyl L-Tyrosine (NALT) is a precursor to neurotransmitters such as dopamine, norepinephrine, and epinephrine, as well as thyroid hormones like thyroxine (T4) and triiodothyronine (T3),, which can enhance mental focus and cognitive performance. I take it occasionally for an extra kick during workouts, as it helps improve alertness and mental clarity, contributing to a more energized workout experience.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>: notable</p><p><strong>Conviction</strong>: moderate</p><p>The following supplements are taken separately as capsules along with the pre-workout drink at the same time:</p><p><strong><a href="https://nootropicsdepot.com/cistamax-capsules/">CistaMax</a></strong> (5 days on, 2 days off, 2 weeks washout every 10 weeks): 1 serving, 1 cap</p><p>This is an amazing combo and you have to admire the thinking that went into designing it. It would take a couple of pages to dissect it - so for the sake of time do your own research here.</p><p><strong>Risk</strong>: moderate</p><p><strong>Personal outcome</strong>: significant</p><p><strong>Conviction</strong>: high</p><p><strong><a href="https://nootropicsdepot.com/eurycomax-optimized-tongkat-ali/?searchid=14818292&amp;search_query=eury">Eurycomax</a></strong> (5 days on, 2 days off, 2 weeks washout every 10 weeks): 1 serving, 2 caps</p><p>Eurycomax is the new great combo based around Tongat Ali with additional ingredients, and it seems to avoid some of the concerns with pure Tongat by better balancing estrogen suppression. A reminder - anytime you are tinkering with hormones even a little, quantitative testing is a must to ensure safety and efficacy.</p><p><strong>Risk</strong>: moderate</p><p><strong>Personal outcome</strong>: significant</p><p><strong>Conviction</strong>: high</p><p><strong><a href="https://nootropicsdepot.com/dhea-quick-dissolve-tablets-micronized/?searchid=14818447&amp;search_query=dhea">DHEA</a></strong>: 1/2 serving, 1/2 tab, 2.5mg (5 days on, 2 days off)</p><p>At very low doses - such in in my case for a total of  5mg of DHEA and 5mg or Pregnenolone between DHEA tablet, Eurycomax and Cystamax this helps nudge, especially as we age the process of steroidogenesis in the right direction, towards testosterone synthesis without triggering negative effects and feedback loops.</p><p><strong>Risk</strong>: moderate</p><p><strong>Personal outcome</strong>: significant</p><p><strong>Conviction</strong>: high</p><p><strong><a href="https://www.amazon.com/gp/product/B00E9M4XFI/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">Creatine</a></strong>: 1 serving, 5g</p><p>The benefits of creatine are well known - both for muscle endurance and brain (and even systemically). </p><p>There&#8217;s a plausible risk of DHT elevation (and resulting hair loss). The one frequently quoted  study was not replicated and there&#8217;re other issues with it (however, anecdotally there are a lot of bold body builders). Either way keeping an eye on DHT levels is recommended if hair loss is a concern.</p><p>Secondarily, creatine supplementation will elevate blood creatinine levels that will show up on a standard metabolic panel test (such as CMP). This is not a concern - if the cause is Creatine supplementation. However tell your doctor, so a more accurate biomarker can be used to assess kidney health (such as Cystatin C).</p><p>Finally - note that creatinine as well as liver enzymes (ALT most notably) can also be elevated due to normal muscle breakdown as a result of a strength training session, so likewise tell your doctor or use a wash out period (72 hours - I am not doing it).</p><p><strong>Note</strong>: when you severely underslept for whatever reason, additional 5mg of Creatine later in the day (I&#8217;d just add that extra dose into the morning shake) could be quite helpful as a temporary fix.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>: significant</p><p><strong>Conviction</strong>: high</p><h1>AM - Breakfast (Post workout)</h1><p>These are taken in conjunction with my morning shake and whenever possible I use powders (instead of capsules, to avoid filler ingredients and flow agents) and capsules (whenever a powder form is not available) are broken up and put into the shake. Using powders is also more cost effective.</p><h2>Core Morning Shake recipe</h2><p><strong><a href="https://www.amazon.com/gp/product/B06XWXTX49/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">Whey Protein</a></strong>: 25g</p><p><strong><a href="https://www.amazon.com/gp/product/B08L6Y12Q4/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">Collagen Protein</a></strong>: 9g</p><p><strong><a href="https://www.amazon.com/dp/B0F22L62DC?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1">Fiber Powder</a></strong>  (most days, but sometimes I skip adding it when I want to give my gut a break from extra fiber)</p><p><strong><a href="https://www.amazon.com/gp/product/B08KYLFGH7/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">Wild blueberry</a></strong> or <strong><a href="https://www.amazon.com/gp/product/B07QTCDMBW/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">elderberry powder</a></strong>: 5g (alternating) </p><p><strong><a href="https://www.amazon.com/gp/product/B001TNW23U/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">Pomegranate powder</a></strong>: 1 tbs</p><p><strong><a href="https://www.amazon.com/gp/product/B07G3KLNM7/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">Tart cherry powder</a></strong>: 4.8g (among many other benefits this helps to keep uric acid at low levels).</p><p><strong>Olive oil</strong>, 2 tsps (to make sure there&#8217;s a little bit of fat to help with absorption of fat soluble vitamins).</p><p>Some sort of fruit from whatever is around, could fresh or be frozen: banana, a cup of strawberries, raspberries, an apple, 2 peaches, 3 apricots, etc.</p><h2>Supplements that are mixed into the shake</h2><p><strong><a href="https://www.amazon.com/gp/product/B01BGZVXC6/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">O.N.E. multivitamin</a></strong>: 1 serving, 1 cap</p><p>Think of it as an insurance policy, although given my genetics - methylated B vitamins are great. There&#8217;s no perfect multi-vitamin and it&#8217;s a compromise for convenience.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>: minor</p><p><strong>Conviction</strong>: high</p><p><strong><a href="https://www.amazon.com/gp/product/B00BN4OMW4/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">Magnesium</a></strong> complex: 1/2 serving, 1 cap, 120mg</p><p>The benefits are well documented, especially considering that it is nearly impossible to get enough magnesium through the regular modern diet (with all the soil depletion).</p><p><strong>Risk: low</strong></p><p><strong>Personal outcome: </strong>minor</p><p><strong>Conviction: </strong>high</p><p><strong><a href="https://www.amazon.com/gp/product/B00NG1N8OU/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">Glucosamin HCI</a></strong>: 1 serving, 1g</p><p>Decent research for joint (formation and repair of cartilage, synovial fluid production) and even brain health.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>: minor</p><p><strong>Conviction</strong>: moderate</p><p><strong><a href="https://www.amazon.com/dp/B0D3SNSJ47?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1">Colostrum</a></strong> 1 scoop, 1g, morning shake</p><p>Colostrum is a &#8220;gut insurance&#8221; lever for me: bovine colostrum concentrates immunoglobulins (IgG), lactoferrin, and other bioactives that may support barrier integrity and innate immune function. The best human data I&#8217;ve seen is in athletes / high training stress contexts, where colostrum has been associated with improved gut permeability markers and fewer URTI-type issues. I&#8217;m using a conservative dose (1g/day) and treating it as supportive &#8212; not magic. </p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>: minor</p><p><strong>Conviction</strong>: moderate</p><p><strong><a href="https://www.amazon.com/dp/B0014UECK4?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1">Stinging Nettle Root Extract</a></strong>, 1/2 serving, 1 cap, 250mg</p><p>Mechanism of action is anti-androgen activity specific to prostate, thus not affecting androgens systemically, which is quite nice. The outcomes are supported by solid data, including human trials.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>: significant</p><p><strong>Conviction</strong>: high</p><p></p><p><strong><a href="https://www.amazon.com/dp/B0BQWY5BB9?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1&amp;th=1">Bacopa Monirelli</a></strong><a href="https://www.amazon.com/dp/B0BQWY5BB9?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1&amp;th=1"> </a>(Cognance): 1 serving, 100mg</p><p>Bacopa is a long-running cognitive &#8220;insurance&#8221; herb with human trial data suggesting benefits for memory/processing after weeks of consistent use (with GI side effects being the most common downside). The trade-off with traditional bacopa extracts is that some people feel slightly sedated or &#8220;flat.&#8221; Cognance is a newer bacopa extract with a different standardization profile, and the bet here is keeping the brain-support upside while avoiding the sedation tax.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>: minor</p><p><strong>Conviction</strong>: moderate</p><p><strong><a href="https://nootropicsdepot.com/maca-extract-powder/?searchid=14900538&amp;search_query=maca+extract">Maca Extract Powder</a></strong> (5% macamides): 1 serving, 125mg</p><p>An interesting compound that inhibits <strong>FAAH</strong> (the enzyme that breaks down anandamide), which can raise endocannabinoid tone. The result is a noticeable mood and well being elevation. The safety record is excellent and it appears not to affect hormones or major neurotransmitters.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome:</strong> notable</p><p><strong>Conviction</strong>: moderate</p><p><strong><a href="https://www.amazon.com/dp/B07W8TL9DZ?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1&amp;th=1">Lion&#8217;s Mane</a></strong>: 1 serving, 1g,  1 day on, 2 days off</p><p>For brain benefits as it is known to increase both BDNF (Brain-Derived Neurotrophic Factor)) and especially NGF (Nerve Growth Factor).</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>: minor</p><p><strong>Conviction</strong>: moderate</p><p><strong><a href="https://www.amazon.com/dp/B0F5F24B1K?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1&amp;th=1">Turkey Tail</a> mushroom powder</strong>: 1 serving, 1g (2 days on, 1 day off - non overlapping with Lion&#8217;s Mane)</p><p>The idea is to boost NK-cell activity as a preventative. Turkey Tail &#8212;notably its PSK/PSP fractions and mycelium&#8212;has human data showing increased NK-cell tumoricidal activity and higher circulating NK cells vs. control: a phase-I study in breast-cancer survivors reported improved NK function and lymphocyte counts; a randomized double-blind trial linked rises in the NK-activation marker CD69 to functional gains; and oncology RCTs using PSK showed peripheral NK-cell increases compared with control.</p><p><strong>Note</strong>: if you have any kind of autoimmunity, I would avoid.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>: minor</p><p><strong>Conviction</strong>: moderate</p><p><strong><a href="https://www.amazon.com/gp/product/B08QC4V25L/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">Inositol</a></strong>: 1 serving, 1g</p><p>Supports healthy thyroid function and helps keeps TSH levels in the optimal range.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>:: notable</p><p><strong>Conviction</strong>: high</p><p><strong><a href="https://nootropicsdepot.com/milk-thistle-4-1-extract-silybum-marianum/">Milk Thisle</a></strong>, 1 serving, 100mg</p><p>Milk thistle is well-researched for its liver support properties, primarily due to its active compound, silymarin. Silymarin has antioxidant, anti-inflammatory, and antifibrotic effects, which help protect liver cells from damage, support liver regeneration, and improve liver function. Probably a good insurance with all the other supplements I am taking &#128578;. </p><p>While research supports these benefits, it hasn&#8217;t been enough time to ascertain the results for me personally.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>: none</p><p><strong>Conviction</strong>: low</p><p><strong><a href="https://www.amazon.com/dp/B09WDJZYSQ?ref=ppx_yo2ov_dt_b_fed_asin_title">Spermidine</a></strong>, 1 service, 2 caps, 13mg equivalent</p><p>Spermidine induces autophagy by inhibiting the activity of key enzymes involved in acetylation, such as histone acetyltransferases. This inhibition leads to the deacetylation of autophagy-related genes, which promotes the formation of autophagosomes&#8212;vesicles that engulf and degrade damaged proteins, organelles, and cellular debris.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>: none</p><p><strong>Conviction</strong>: moderate</p><p><strong><a href="https://www.amazon.com/dp/B07TK4VYWS?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1&amp;th=1">TruNiagen NR</a></strong>: 1 serving, 1 cap, 300mg, combined with <strong><a href="https://www.amazon.com/dp/B07QB7DP4G?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_6">Quercetin + Pterostilbene</a></strong>, 1 serving, 2 caps, 500mg Quercetin, 50m of Pterosilbene)</p><p>This deserves a bit of discussion. The whole idea of using NAD+ precursors came out of research by David Sinclair. We know NAD+ decreases as we age and it is important with 1000s different things, most interestingly with Sirtuin activation (Resveratrol or similar compounds, such as Pterostilbene).</p><p>However there&#8217;re many issues with it:</p><ol><li><p>No positive effects have been observed in humans thus far (except for one study on cognitive health with NR)</p></li><li><p>Both NR and Resveratrol failed <a href="https://www.nia.nih.gov/research/dab/interventions-testing-program-itp">ITP</a> studies (which thus far is a gold standard for longevity research).</p></li><li><p>The original Resveratrol study has been thoroughly discredited due to study design (in fact it turned out to be so bad, people wonder how it (and it&#8217;s authors) got so much attention in the first place),</p></li><li><p>There is a rodent study showing cancer growth acceleration effects from NR (for pre-existing cancers - mechanistically this makes sense as NAD+ is quickly depleted by hungry cancer cells, and replenishing NAD+ will only accelerate that process)</p></li><li><p>There is a concern of up-regulating CD38, although in moderate doses this should not be a concern (there&#8217;s yet to be determined U-shape curve here that may depend on many variables). This could potentially be mitigated by Quercetin.</p></li><li><p>Bio-availability of Resveratrol is low (I am using Pterostilbene instead for that reason, although it&#8217;s not clear that it is doing anything either).</p></li><li><p>In addition any of these precursors need to be supplemented with TMG (in my case there&#8217;s plenty in Legion pre-workout drink)  to prevent elevation of homocysteine due to back conversion to nicotinamide.</p></li></ol><p>That said - mechanistic theory around how all this could work for longevity is quite compelling and it is possible that it would take considerable time for it to work.</p><p>Subjective effects for me in terms of feeling more energetic, however the question is whether  supplementing with plain old nicotinamide would accomplish the same outcome (including NAD+ support) for a fraction of the cost.</p><p>Finally - there&#8217;s still a serious, unresolved conundrum here. NAD+ enhances cellular energy by participating in mitochondrial redox reactions, boosting ATP synthesis (hence feeling energetic). It is also essential for sirtuin activation, which signals autophagy and repair&#8212;processes that act as evolutionary sensors for low nutrient levels.</p><p>However, consuming a lot of protein, necessary for maintaining muscle mass, stimulates the mTOR pathway, contradicting these aims. The mTOR pathway promotes cell growth and inhibits autophagy, counteracting the benefits. An alternative approach involving cycling and agents like <a href="https://www.drugs.com/mtm/sirolimus.html">Rapamycin</a> might be better, but the exact protocols and pros and cons are unclear. Most importantly, I would have no idea how to reconcile it with my everyday workout routine.</p><p><strong>Risk</strong>: moderate</p><p><strong>Personal outcome</strong>:: notable</p><p><strong>Conviction</strong>: low</p><h2>Supplements taken along with the shake / breakfast</h2><p>These are supplements that are not (or cannot be) available in a powder form or breakable capsules.</p><p><strong><a href="https://www.amazon.com/gp/product/B07VF9HB74/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">EPA/DHA</a></strong>: 1/2 serving, 1 cap</p><p>Not much to opine on, the benefits of EPA/DHA are well researched. I keep my omega index &gt;8 (8.94% as of last test) on <a href="https://omegaquant.com/">OmegaQuant</a> test (your dose may vary, so test it).</p><p><strong>Risk: low</strong></p><p><strong>Personal outcome: </strong>minor</p><p><strong>Conviction: </strong>high</p><p>Allergy Research Group Tocomin SupraBio Tocotrienols: 1 serving, 1 softgel, 200mg</p><p>Tocotrienols occupy a narrower experimental slot. They are vitamin E isoforms with different membrane distribution and biological activity from alpha-tocopherol, although this palm-derived product also supplies 45.6mg of alpha-tocopherol per softgel. I use its mixed alpha, beta, gamma, and delta tocotrienol fraction with a self-emulsifying delivery system because absorption is one of the practical limitations of oral tocotrienols. Small human trials reported favorable signals in arterial stiffness and carotid-plaque progression, but a larger trial at roughly this daily dose found no improvement in vascular function or cardiovascular risk markers. I treat this as exploration of a distinct layer of arterial biology rather than established additional cardiovascular protection.</p><p>Risk: low to moderate (review with the prescriber if using anticoagulant or antiplatelet medication)</p><p>Personal outcome: none (preventative)</p><p>Conviction: moderate</p><p><strong><a href="https://www.lifeextension.com/vitamins-supplements/item02334/super-k">Vitamin K1/MK-4/MK-9</a></strong>: 1 serving, 1 cap</p><p>A must considering supplementation with Vitamin D (O.N.E. multivitamin has 2000 IU), plus other vascular and bone health benefits.</p><p><strong>Risk: low</strong></p><p><strong>Personal outcome: </strong>none (as far as subjective feeling, calcium score continues to stay at 0 and bone density on DEXA went up - so that&#8217;s be pretty good)</p><p><strong>Conviction: </strong>high</p><p><strong><a href="https://www.amazon.com/gp/product/B073VL4WKN/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">COQ10</a></strong>: 1 serving, 1 cap, 100mg</p><p>The idea here is to counteract effects of a statin that whacks HMG-CoA enzyme and decreases COQ10 levels as a result. While studies are conflicted, the subjective feeling is noticeable for me.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>:: notable</p><p><strong>Conviction</strong>: high</p><p><strong><a href="https://www.amazon.com/dp/B01A68H2BA?ref=ppx_yo2ov_dt_b_fed_asin_title&amp;th=1">TUDCA</a>: 1 service, 250mg</strong></p><p>TUDCA is here primarily as a liver and bile-flow support compound. Unlike many &#8220;liver support&#8221; botanicals, it is a bile acid derivative, so the rationale is more targeted: it helps make the bile acid pool more hydrophilic, may reduce stress on hepatocytes, and has a cleaner mechanistic case in the setting of a crowded stack plus mild liver-enzyme drift. I&#8217;m treating it as pragmatic insurance rather than a performance supplement &#8212; the goal is preserving liver resilience and bile handling, not expecting any noticeable day-to-day effect.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>:: none</p><p><strong>Conviction</strong>: high</p><p></p><p></p><p><strong><a href="https://www.amazon.com/dp/B07ZPH99CQ?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1&amp;th=1">H2 Molecular Hydrogen</a></strong>: 1 serving, 1 tablet (80mg magnesium, 8ppm), dissolved in water when taking other supplements.</p><p>There is a significant body of research suggesting that H2 is a selective antioxidant neutralizing mostly the harsh stuff (&#8226;OH, ONOO&#8315;) while sparing signaling ROS (e.g., H&#8322;O&#8322;) that drive exercise adaptations.</p><p>Risk: low</p><p>Personal outcome: none</p><p>Conviction: low</p><h2>Prescription medications</h2><p>Goes without saying that these always must be discussed with and prescribed by your doctor.</p><p><strong><a href="https://www.drugs.com/rosuvastatin.html">Rosuvastatin</a></strong> (Crestor): 5mg <strong>(PM)</strong></p><p>While there&#8217;s a lot of controversy around the use of statins - I am quite convinced based on available research. ASCVD progression is a stochastic process and keeping APO-B concentrations low is, in my opinion very much worth it.</p><p>That said - statins are not perfect drugs with a lot of off target effects. First, they work by suppressing HMG-CoA cascade, that causes all kinds of problems. Secondly it seems there&#8217;s causal evidence of increase in insulin resistance overtime. On top of that, there&#8217;re negative effects on liver (observed by modest elevation of liver enzymes and despite unclear clinical significance - this is still not cool).</p><p>That said, for me Rosuvastatin is still the best (of imperfect) choices. First - it is the only effective hydrophilic statin that (unlike lipophilic statins, such as <a href="https://www.drugs.com/atorvastatin.html">Atorvastatin</a> (Lipitor) doesn&#8217;t massively diffuse into tissues or crosses the blood-brain barrier (thus reducing potential side effects, such muscle soreness).</p><p><strong>Risk</strong>: moderate</p><p><strong>Personal outcome</strong>: signigicant (as measured by calcium score staying at zero)</p><p><strong>Conviction</strong>: high</p><p><strong><a href="https://www.drugs.com/search.php?searchterm=ezetimibe&amp;a=1">Ezetimibe</a></strong> (Zetia): 10mg (<strong>PM</strong>)</p><p>Ezetimibe works through a completely different mechanism than statins. Instead of reducing cholesterol synthesis in the liver, it blocks cholesterol absorption in the small intestine. This makes it a particularly attractive companion to low-dose statin therapy because it can provide meaningful additional LDL-C and APO-B reduction without substantially increasing statin-related side effects.</p><p>What convinced me was the growing body of evidence suggesting that lower cumulative lifetime exposure to APO-B-containing particles is strongly associated with lower cardiovascular risk. If lowering APO-B is the objective, adding Ezetimibe often appears to be one of the most efficient and lowest-risk ways to push levels down further before considering more aggressive interventions.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>: notable</p><p><strong>Conviction</strong>: high</p><p><strong><a href="https://www.drugs.com/tadalafil.html">Tadalafil</a></strong> (Cialis): 5mg <strong>(AM)</strong></p><p>While developed as an ED drug, I take it for it&#8217;s off target systemic effects on improving the blood flow. As we know - most ills stem from poor blood flow as we age. Improved circulation also enhances workout performance and supports overall cardiovascular health, contributing to smoother functioning of many bodily systems. Tadalafil also slightly lowers blood pressure, helping to maintain it within the optimal range.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>:: significant</p><p><strong>Conviction</strong>: high</p><p><strong><a href="https://www.drugs.com/cons/minoxidil-oral.html">Minoxidil</a></strong> (Loniten): 2.5 mg (<strong>PM</strong>)</p><p>I am using it as a preventative for hair loss at low dose (it is an old, repurposed blood pressure medication). It doesn&#8217;t seem to affect the blood pressure at all for me at this micro dose. Otherwise no noticeable side effects.</p><p><strong><a href="https://www.drugs.com/mtm/telmisartan.html">Telmisartan</a> </strong>(Micardis): 20mg (<strong>PM</strong>)</p><p>Telmisartan is an ARB, so the primary reason I take it is simple: keep blood pressure consistently in the optimal range, with minimal spikes and day-to-day variance. For vascular and brain health, I care not only about average blood pressure, but also about the repeated mechanical stress that pressure excursions place on small arteries over time.</p><p>What makes Telmisartan more interesting than a generic &#8220;blood pressure drug&#8221; is its broader vascular profile. Angiotensin II signaling contributes to vasoconstriction, oxidative stress, inflammation, collagen deposition, and adverse arterial remodeling. By blocking the AT1 receptor, Telmisartan  helps reduce some of that remodeling pressure, not just lower the number on the cuff.</p><p>It is also one of the longer-acting ARBs, which matters because smoother 24-hour coverage. Telmisartan also has partial PPAR-&#947; activity, which may explain some of its reported effects on insulin sensitivity, endothelial function, and vascular inflammation.</p><p>So the short version is: I take Telmisartan mainly for tight, stable blood pressure control, but I like that it may also improve endothelial function, arterial stiffness, and the biology of vascular remodeling.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>: significant</p><p><strong>Conviction</strong>: high</p><h3>Coffee</h3><p>The discussion would not be complete without coffee. Coffee is one of the most widely used drugs in the world. 67% of Americans drink coffee (that&#8217;s why I suppose it just got removed from the CPI by the BLS as the price of coffee has been going up).</p><p>I take my coffee black (Americano) with no milk or sugar of course. I do add 1 tsp of <a href="https://www.amazon.com/gp/product/B07XTZRXCV/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">C8</a> MCT oil into my morning coffee.</p><p>On most days that be just one mug, but sometimes I also get the second cup around lunchtime. More than that starts adversely affecting my sleep quality - so if I must, I switch to decaf.</p><h1>PM (with dinner)</h1><p><strong><a href="https://www.amazon.com/gp/product/B07VF9HB74/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">EPA/DHA</a></strong>: 1/2 serving, 1 cap</p><p>The second dose as in AM.</p><p>Allergy Research Group Tocomin SupraBio Tocotrienols: 1 serving, 1 softgel, 200mg</p><p>The second dose as in AM.</p><p><a href="https://www.amazon.com/Magnesium-Glycinate-Capsules-Supplement-Supports/dp/B078SH5C4S/ref=sr_1_1_sspa?crid=39XNN6LQ7W7N3&amp;dib=eyJ2IjoiMSJ9.UoMfJKxej4_me1ExwJ82ID5RGK4DCBn6omrTt8b-GrY7E-hbviVg6W6eK5FQwWw24g_4PFd048HxT39tRHJ5C0MBu1wKgRRcCcoCqfpEbOPx1OyDFGSNyJsJmE4MKqpFoMIfvXr623pH3y6pqwsXXVDOcPSa5SsHWJruC3ISEGqbXTbeTyFiYRetLEdVknegxSOcXj4FZA_c2l1abmZR0hwQ1qbKluOQqJO0pXz9nceIzZ8KSqfU1-J6qk_9kQ7z3Fj6vq18g9uJSCk2ZnoQnLWtIde8NJSfHO_OYkOYiXI.pS_wK9zIxObxSpYq4A1pSsjrkrEluFX7mDAnDL8rOlM&amp;dib_tag=se&amp;keywords=magnesium%2Bglycinate&amp;qid=1730938542&amp;rdc=1&amp;sprefix=magnesium%2Bgl%2Caps%2C231&amp;sr=8-1-spons&amp;sp_csd=d2lkZ2V0TmFtZT1zcF9hdGY&amp;th=1">Magnesium</a> glycinate: 1 serving, 1 cap, 200mg</p><p>Seem to be more effective than than standard magnesium complex and has a notable calming effect for sleep.</p><p><strong>Risk: low</strong></p><p><strong>Personal outcome: </strong>significant</p><p><strong>Conviction: </strong>high</p><p><strong><a href="https://www.amazon.com/gp/product/B07BH65296/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">Magnesium L-Threonate</a></strong>: 2/3 serving, 2 caps, 96mg</p><p>Brain health benefits and a minor calming effect for sleep.</p><p><strong>Risk: low</strong></p><p><strong>Personal outcome: </strong>minor</p><p><strong>Conviction: </strong>low</p><p><strong><a href="https://www.amazon.com/gp/product/B073VL4WKN/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">COQ10</a></strong>: 1 serving, 1 cap, 100mg</p><p>The second dose as in AM.</p><p><strong><a href="https://www.amazon.com/dp/B01A68H2BA?ref=ppx_yo2ov_dt_b_fed_asin_title&amp;th=1">TUDCA</a>: 1 service, 250mg</strong></p><p>The second dose as in AM.</p><p><strong><a href="https://www.amazon.com/gp/product/B01D15LMCK/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&amp;psc=1">BroccoMax</a></strong>: 1 serving, 2 caps</p><p>The research is decent, especially around colon cancer prevention effects. However I can&#8217;t quantitatively put a handle on it for me personally thus far.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>:: none</p><p><strong>Conviction</strong>: low</p><p><strong><a href="https://www.amazon.com/dp/B0DNMWS7WX?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1">Urolithin A</a>,</strong> 1/2 service, 1 cap, 1000mg, PM</p><p>Urolithin A is the only item in this stack aimed primarily at mitochondrial quality control through mitophagy&#8212;the selective clearance of damaged mitochondria. That is different from creatine&#8217;s ATP buffering, CoQ10&#8217;s role in electron transport, and exercise-driven mitochondrial biogenesis. Supplementing also bypasses the large person-to-person variation in producing urolithins from food. Human trials show bioavailability and mitochondrial target engagement, with some signals in muscle endurance and strength, but no consistent improvement in walking distance, ATP production, or vascular function. That earns it a monitored experiment, not high conviction.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>: none</p><p><strong>Conviction</strong>: moderate</p><p><strong><a href="https://www.amazon.com/dp/B0014UECK4?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1">Stinging Nettle Root Extract</a></strong>, 1/2 serving, 1 cap, 250mg</p><p>The second dose as in AM.</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>: significant</p><p><strong>Conviction</strong>: high</p><p></p><p><strong><a href="https://www.amazon.com/dp/B0CKC3TXCC?psc=1&amp;ref=ppx_yo2ov_dt_b_product_details">Geranylgeraniol</a></strong>: 1 serving, 1 cap</p><p>A fairly new compound that also counteracts negative effects of statins, by a different mechanism. </p><p>Geranylgeraniol (GGOH) is a critical intermediate in the mevalonate pathway, downstream of HMG-CoA reductase and it is necessary for the prenylation of small GTP-binding proteins. Prenylation is a post-translational modification that allows these proteins to anchor to cell membranes and function properly.</p><p>By supplementing with GGOH, the normal function of these prenylated proteins can be restored. Subjectively for me it appears to be even more efficacious than COQ10 (although I would keep both as mechanisms and effects are different).</p><p><strong>Risk</strong>: low</p><p><strong>Personal outcome</strong>: notable</p><p><strong>Conviction</strong>: high</p><h1>Discussion</h1><p>So, that's a lot of supplements. The question is: can you achieve the same results with just a good diet? Probably not.</p><p>Even if you source high-quality ingredients and are prepared to spend an inordinate amount of time cooking everything perfectly, it's unlikely you'll achieve all the effects and benefits tailored to your genetics, personal health circumstances, and performance goals without some scientific enhancement.</p><p>The second question: can you avoid supplements altogether? Of course, you can. The human body is very resilient and can tolerate suboptimal conditions for a while. However, if performance optimization is your goal, then supplementation should be considered.</p><p>Another point: ingredients in food are presumably in more natural ratios than in a synthetically constructed supplement regimen. True. However, our body is highly adaptable and can handle a lot of variation. Besides, common processed foods like french fries and soda are far from natural or balanced.</p><p>Another concern is supplement quality, fillers, and labeling. This is valid. Fillers are problematic, which is why I opt for powders, choose the cleanest possible options, and stick with reputable brands. Risks, such as heavy metal contamination, do exist, but they are manageable if you monitor important biomarkers regularly. On the flip side - common food products are hardly immune from this problem, from Glyphosate, micro-plastics, heavy metals and other types of contaminants.</p><p>Does it take a lot of work? Some, but not a lot, as I've fine-tuned my routine over the years. I also batch my shakes, preparing and mixing them on the weekend for the week, so the actual breakfast prep takes very little time.</p><p>Hope this helps - in whatever way it does &#128578; </p><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[Cold Sales Outreach Emails Inefficacy - a sign of upcoming disruption?]]></title><description><![CDATA[Everyone does it, everyone hates it, large companies are built on it (Outreach, Apollo) and it underlines the foundation of CRM systems (like Salesforce and HubSpot). What prompted me to opine here was an accidental feature we developed at Jelled.AI that sorts communications that you don&#8217;t need to pay attention to out. Looking at the results of it - it got me thinking. I will explain.]]></description><link>https://www.equationblog.com/p/cold-sales-outreach-emails-inefficacy</link><guid isPermaLink="false">https://www.equationblog.com/p/cold-sales-outreach-emails-inefficacy</guid><dc:creator><![CDATA[Ruslan Belkin]]></dc:creator><pubDate>Sun, 12 May 2024 14:15:58 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0187b1d4-c470-4ffa-a463-efc198a6864f_1280x1280.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Everyone does it, everyone hates it, large companies are built on it (Outreach, Apollo) and it underlines the foundation of CRM systems (like Salesforce and HubSpot).</p><p>What prompted me to opine here was an accidental feature we developed at <a href="https://jelled.ai">Jelled.AI</a> that sorts communications that you don&#8217;t need to pay attention to out. Looking at the results of it - it got me thinking. I will explain.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>But first - there is no surprise the efficacy of cold outreach emails has been declining (despite all the hacks - that largely do not work) and probably reached the point (with the introduction of AI tools into sales workflows) where the cost to send them is near zero, but the efficacy is an asymptote of zero. In fact I would argue there&#8217;s a high probability of a net negative return as receiving cold emails could drain whatever good will the customer could have had before opening them.</p><p>My personal routine handling inbound sales outreach was as follows:</p><ol><li><p>I never respond to cold sales outreach, except if it&#8217;s from founders</p></li><li><p>I will mark repeated sales outreach email as spam (although that doesn&#8217;t seem to help due to various email distribution tricks people are using) and will block the sender.</p></li></ol><p>Nonetheless - that&#8217;s still annoying.</p><p>So - with the new feature, <a href="https://jelled.ai">Jelled.ai</a> automatically detects inbound sales emails (so far with 100% accuracy) and sorts them out into a separate folder (the AI engine itself keeps the content if you ever need to do a research on it in the future).  As a result - my inbox is super clean and I no longer get annoyed by the all the inbound.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!stWi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b41aca-5771-4f9f-aadd-5e8eac7b1a7e_3146x1812.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!stWi!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b41aca-5771-4f9f-aadd-5e8eac7b1a7e_3146x1812.png 424w, https://substackcdn.com/image/fetch/$s_!stWi!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b41aca-5771-4f9f-aadd-5e8eac7b1a7e_3146x1812.png 848w, https://substackcdn.com/image/fetch/$s_!stWi!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b41aca-5771-4f9f-aadd-5e8eac7b1a7e_3146x1812.png 1272w, https://substackcdn.com/image/fetch/$s_!stWi!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b41aca-5771-4f9f-aadd-5e8eac7b1a7e_3146x1812.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!stWi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b41aca-5771-4f9f-aadd-5e8eac7b1a7e_3146x1812.png" width="1456" height="839" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c0b41aca-5771-4f9f-aadd-5e8eac7b1a7e_3146x1812.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:839,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:368798,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!stWi!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b41aca-5771-4f9f-aadd-5e8eac7b1a7e_3146x1812.png 424w, https://substackcdn.com/image/fetch/$s_!stWi!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b41aca-5771-4f9f-aadd-5e8eac7b1a7e_3146x1812.png 848w, https://substackcdn.com/image/fetch/$s_!stWi!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b41aca-5771-4f9f-aadd-5e8eac7b1a7e_3146x1812.png 1272w, https://substackcdn.com/image/fetch/$s_!stWi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b41aca-5771-4f9f-aadd-5e8eac7b1a7e_3146x1812.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p></p><p>However - this brings up several interesting observations. While using LLMs to generate emails is practically free (comparing to other methods involving humans) - getting rid of these emails is also free. Therefor the usefulness of the cold outbound channel is NIL and the only people making money here are from OpenAI.</p><p>The voice channel was already useless (I never ever pick up the phone in the first place if you are not in my address book). Text will largely meet similar fate (including mentions).</p><p>So, if cold outbound is of no value - that means that the value of SDRs is just an expense (the introduction research can be done by AI agents, while the introductions will still need to be done through the influencer network).</p><p>What does this do? Does It invert the SaaS sales model as it empowers the buyer to run a competitive bidding process through AI - which is already aware of company plans, budgets, timelines, etc.?</p><p>This, being on top of other AI advances, brings into question the need for CRM systems&#8212;which really have nothing to do with customer relationship management but are designed to manage the salesforce. Humans are forgetful, unorganized, need complex compensation plans, and are generally flawed. AI-first designed systems will obviously not have any of those problems. Additionally, as we have fewer humans, we will need fewer seats&#8212;there goes your pricing model as well.</p><p>Are we going to see an upheaval in the world of Salesforce? Does it apply to other professionals (in marketing for example)?</p><p>Something possibly to keep an eye on in the near future.</p><p>Thoughts welcome!</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.equationblog.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Equation! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item></channel></rss>