loading
loading
Blanket usage caps are akin to a hiring freeze in the AI era. Many enterprises have turned to uniform caps to control AI spend, but they are blunt instruments that limit productivity. They hold back the engineers who are the most efficient users of AI. And engineers who use AI less effectively don’t get feedback on what to change. The better approach is to allocate spend based on engineering efficiency and give less efficient engineers precise suggestions on how to improve. This has historically been hard to manually scale. So, today we’re introducing three enterprise features to help: - Efficiency Score grades how efficiently each engineer uses Devin. For each session, including investigations and planning that never produce a PR, Devin estimates the associated productive engineering hours, meaning how long a human engineer would have needed for the same work. The grade reflects the productive hours an engineer gets per unit of spend across recent sessions, with a smaller credit for volume. - Devin Coach gives engineers feedback on a prompt before it runs. It flags common mistakes, such as a vague request, an expensive mode for a simple task, or unrelated tasks in one session, and suggests a fix. - Efficiency-based approvals raise limits automatically for efficient engineers. This enables more precise cost controls without ballooning admins’ review burden. When an engineer with a Healthy or Satisfactory grade nears their limit, Devin raises it automatically, up to a ceiling the admin sets. Other engineers’ requests are sent to an admin for review. Giving each engineer an Efficiency Score To set granular limits, you need to know how much each engineer gets out of Devin relative to what they spend. The Efficiency Score is a 0–100 rating of each engineer’s recent Devin usage. It is calculated from recent Devin sessions in three parts: Devin identifies the sessions that produced useful work, estimates the engineering hours that work represents, and converts those hours into efficiency and volume ratings that combine into the score. Recognizing work that doesn’t produce a merged PR Many organizations measure AI output with merged PRs. The limitation of that approach is that a lot of useful work never produces a merged PR, such as planning, triage, data analysis, and investigations. The Efficiency Score thus builds on “productive engineering hours,” the metric we introduced earlier this year to estimate the value of Devin sessions. First, Devin checks whether each session produced useful work. For a session with a PR, that means the PR was merged. For a session without one, a classifier judges whether Devin made real progress on the task, like triaging the bug or answering the analytics question. That way, a bug investigation gets credit even though it never produces a PR. Second, for every session that passes, it estimates how many hours the same work would have taken a human engineer. We validated these estimates against a dataset of engineers’ own assessments of the equivalent engineering hours from their Devin sessions. Combining efficiency and volume into a score At Cognition, we calculate spend via Agent Compute Units (ACUs), which combine LLM and cloud infrastructure cost into one unit of measure. The score looks at an engineer’s productivity in two ways: - Efficiency (E) is productive hours per ACU. It shows whether spend turns into productive work. An engineer who writes clear prompts, picks well-scoped tasks, and uses the right model gets more productive hours from each ACU. - Volume (V) is productive hours per day. It shows how much productive work the engineer delivers overall, regardless of cost. We weigh efficiency more heavily, since the goal is to give more capacity to engineers who get the most out of each ACU. Volume earns points because we also want to credit how much productive work an engineer delivers. Each measure is first converted to a rating between 0 and 1 that levels off at high values. The two ratings then combine into the score: An engineer’s efficiency rating sets their starting score: a rating of 0.50 starts at 50. Volume can then add up to a fifth of the points left between that starting score and 100. An engineer starting at 50 can gain up to 10 points from volume, and one starting at 80 can gain up to 4. For example, here are four engineers based on their most recent 200 ACUs. The power user and the typical engineer are equally efficient, but the power user’s higher volume earns them 5 more points. The heavy but wasteful engineer matches the power user’s volume, but low efficiency puts them nearly 20 points lower. Giving engineers a path to improve The score reflects recent usage rather than a lifetime average, so early sessions don’t count against engineers indefinitely. Better task selection, planning, and model choices improve future scores. Engineers can see their score in My Analytics, under Session efficiency, as one of four grades: Healthy, Satisfactory, Needs Improvement, or Unknown when there isn’t enough data. Supporting metrics help explain the score: merged PRs per day, ACUs per merged PR, sessions with delivered outcomes, and consumption by session size. Guiding engineers before they spend with Devin Coach We’ve found that new users tend to run into the same few issues: prompts too general to accomplish a concrete task, defaulting to the most expensive model, or combining several unrelated tasks in one session. Because these patterns are so consistent, we saw an opportunity to productize training. Devin Coach gives engineers feedback on their prompts before they run, so they can build better habits over time without requiring a series of training sessions. Example: no plan before implementation Suppose the user writes, “Rewrite our billing system to support usage-based pricing.” The main problem with such a prompt is that it leaves major design decisions open. Devin Coach suggests making a plan first. This allows the engineer to correct wrong assumptions before spend goes into an implementation. Example: using an overpowered model for the task Another example: for simple tasks such as changing a button label, Devin Coach may suggest “Switch mode” to replace Ultra, the highest-powered and most expensive choice, with the default mode. As smaller models have improved, they are a more efficient choice for many tasks. To avoid noisy suggestions, Coach uses a small LLM classifier that checks a fixed set of behaviors and returns no suggestion when a prompt is fine. We tuned it on several hundred labeled examples, weighted toward cases where Coach should leave a good prompt alone. Devin Coach analytics shows admins which suggestions users see, accept, and reject, with acceptance rates, daily trends, and breakdowns by type. This helps teams identify recurring problems, and can be used by the company’s DevEx teams or by Cognition’s forward-deployed engineering teams for tailored enablement sessions. Devin Coach will not silently rewrite prompts. Admins choose how much intervention they want: inline suggestions that engineers can ignore (the default), required confirmation before a flagged new-session prompt is sent, or no checks at all. Raising limits automatically with efficiency-based approvals Enterprises using static limits typically have two options: figure out manually who their power users are and give them a higher allowance, or set a uniform cap that limits them. We built efficiency-based approvals in Devin as a third option. Devin uses each engineer’s Efficiency Score to raise their limit automatically, which lets enterprises scale personalized cost controls to hundreds, if not thousands, of engineers without a growing review queue. Admins turn this on by selecting “Approve based on efficiency” in Usage policies and setting the maximum amount Devin can approve automatically. When an engineer with a Healthy or Satisfactory grade approaches their limit, Devin increases it without a request, so they can keep working without waiting for an admin. Increases stop at a ceiling the admin sets, and organization-wide limits still apply. If an engineer requests more than the ceiling allows, Devin grants up to the ceiling and sends the remainder to an admin. It never denies a request automatically. Engineers with a Needs Improvement or Unknown grade can still request more usage, but their requests go to an admin for review. When reviewing a request, admins see the engineer’s reason alongside their efficiency, model mix, and consumption, the metrics enterprise admins told us matter most when allocating spend. The chart below follows two real engineers at different companies. Both were among the most productive users on their accounts, and both hit their monthly cap. At the first company, which required manual approvals (left), the engineer was blocked for four days until an admin raised the limit, and their merged PRs fell to a third of their usual rate that week before recovering. At the second company, which had auto-approvals enabled (right), the engineer hit their cap 22 times, and Devin lifted it automatically each time. They kept shipping, going from 30 to nearly 80 merged PRs a week over the period. Allocating spend where it has the most impact As AI investments grow, organizations need greater precision and automation to manage spend effectively across thousands of developers. Efficiency Score, Devin Coach, and efficiency-based approvals are intended to help organizations avoid blanket limits. Over the coming months, we will introduce flexible budgeting controls that allow enterprise administrators to allocate budgets to department leaders, who can then distribute usage limits to sub-teams and handle request approvals. Alongside these controls, we will continue to optimize our efficiency scoring and coaching capabilities to help enterprises maximize the engineering output delivered for every dollar spent. Auto-approvals are available today for all customers on a Devin Enterprise plan. Get in touch to try it out.
document fetched 2026-10-24 11:26