loading
loading
Artificial Intelligence Tokenomics at scale: How Jamf built real-time spend enforcement for Amazon Bedrock Generative AI spend behaves unlike any cost line before it. Traditional compute scales with provisioned capacity. AI spend scales with behavior: a single engineer running an agentic coding loop against a premium model can burn more tokens in a few hours than a team does in a week. This is the tokenomics problem: usage is invisible until the bill arrives, making both cost control and return on investment (ROI) hard to prove. Before expanding AI access, leadership wants three answers: What do we spend per person? Can we cap it without slowing the engineers down? And do the productivity gains justify the cost? Jamf, trusted by more than 76,000 organizations to manage and secure Apple devices at scale, faced this challenge directly. To accelerate AI-assisted development, Jamf gave its engineering organization broad access to Amazon Bedrock. Productivity climbed, but so did the need for AI FinOps: per-user visibility and cost accountability. Jamf built a production system that solves this at the individual level. In this post, you learn how to build this architecture using AWS Identity and Access Management (IAM) Customer Managed Policies (CMPs), an Amazon Athena-based cost view, and a serverless AWS Lambda enforcement loop. By the end, you will have a working, production-tested pattern that enforces tiered spend limits in near-real-time without disrupting active sessions. This is an AI cost governance capability. Solution overview Jamf’s solution tracks each engineer’s daily Amazon Bedrock spending, then applies tiered model restrictions as they approach their budget. For example, it denies Anthropic Claude Opus access at 80% of the daily budget and denies Anthropic Claude Sonnet access at 100%. It does not block Anthropic Claude Haiku, so engineers retain a low-cost model to keep working. Restrictions take effect within minutes, require no re-authentication, and revert automatically at the next daily reset. For engineers who legitimately need more, a documented exception process grants time-boxed higher limits. Architecture diagram The architecture addresses three concerns: measuring spending, deciding who to restrict and notifying them, and enforcing the restriction. Each is handled by a purpose-built, serverless component. The end-to-end flow works as follows: Measure - Invocation and logging — When an engineer calls Amazon Bedrock (bedrock:InvokeModel) through their AWS IAM Identity Center single sign-on (SSO) session, Amazon Bedrock logs invocations (including model ID, input and output token counts, and user identity) to the Amazon Simple Storage Service (Amazon S3) bucket you configure. Decide and notify - Cost measurement — an Amazon Athena view (bedrock_cost_today ) reads those logs and computes per-user daily spend by multiplying token counts against published model rates. Athena queries the raw logs in place, avoiding the need for a separate data pipeline. - Enforcement decision — An AWS Lambda enforcement handler runs every 15 minutes, triggered by an Amazon EventBridge schedule. It reads the current day’s spend from the Athena view and cross-references an Amazon DynamoDB exceptions table that holds any custom, time-boxed limits granted to individual engineers. - Notification — The same enforcement handler reads each user’s previous state from the Amazon DynamoDB state table, and when spend crosses a new threshold, sends the engineer a one-time Slack direct message for that tier, so restrictions are not a surprise. Enforce - Enforcement action — For each engineer over a threshold, the Lambda publishes a new version of the appropriate CMP by using iam:CreatePolicyVersion. The policy targets specific users through a saml:sub condition key. - Live evaluation — The CMPs are attached to the IAM permission set. On the engineer’s next Amazon Bedrock call, IAM evaluates the updated policy and allows or denies the request accordingly, with no re-authentication required. Prerequisites To deploy this solution, you need: - AWS account with permissions to create IAM roles, Customer Managed Policies, AWS Lambda functions, Amazon Athena workgroups, Amazon S3 buckets, and Amazon DynamoDB tables. - AWS IAM Identity Center with a permission set that you configure and assign to your users. - Configure Amazon Bedrock model invocation logging to deliver logs to your Amazon S3 bucket. - AWS Command Line Interface (AWS CLI) configured with appropriate credentials. - Slack workspace with an app configured for slash commands, interactivity, and bot messages. Deployment steps You can access the code from this post at https://github.com/aws-samples/sample-bedrock-spend-enforcement. At a high-level, the deployment steps can be summarized as follows: Step 1: Create the Amazon Athena cost view Amazon Bedrock invocation logs land on Amazon S3 as JSON. Start by creating an Athena table over the log location, then a view that translates token counts into dollars. The view multiplies input and output tokens by the published per-token rate for each model, grouped by user identity and the current date. Replace the rate constants with the current Amazon Bedrock pricing for your Region. Each model family needs an explicit pricing branch. An unmapped model is priced at the highest tier as a fail-safe (not $0), so an unrecognized model cannot bypass enforcement. Watch the accompanying alert and add the model’s real rate promptly. Step 2: Create the Customer Managed Policies Create the enforcement policies that deny a model family for a specific set of users, identified by their saml:sub value. Start with an empty user list. The Lambda populates it at runtime by publishing new policy versions. Attach the policies to the IAM permission set. When you update these customer-managed policies by using iam:CreatePolicyVersion, your changes take effect immediately without re-provisioning. Step 3: Deploy the enforcement Lambda and schedule Deploy the enforcement handler and schedule it to run every 15 minutes with Amazon EventBridge. On each run, the handler queries the Athena view, reads the DynamoDB exceptions table, computes the restricted-user list for each tier, and publishes updated CMP versions. This design is idempotent by construction: each run recomputes the full restricted-user list from that day’s cumulative spend, rather than applying an incremental change. Running the handler twice in a row, or missing a run entirely, produces the same result once it catches up. There is nothing to double-apply and nothing to roll back. The daily reset is implicit too. The Athena view scopes spend to a rolling daily window (00:00 in your chosen reference time zone). Once that window rolls over, the next run’s recomputed list omits users who are no longer over threshold, and their CMP restriction lifts automatically on the following iam:CreatePolicyVersion call. There is no separate unblock code path to maintain or get out of sync. Step 4: Add the exception workflow Some engineers legitimately need higher limits for a large migration, a customer escalation, or a model evaluation. Rather than manually editing policies, expose a Slack slash command (/bedrock-limit) that admins can use to grant a time-boxed custom limit. The command writes an entry to the DynamoDB exceptions table with the engineer’s identity, the elevated limit, and an expiry timestamp. It also records an audit trail of who granted it, when, and optionally which ticket authorized it. Set a DynamoDB Time to Live (TTL) attribute on the expiry timestamp, so exceptions clean themselves up automatically. On its next run, the enforcement Lambda reads the active exceptions and adjusts each engineer’s threshold accordingly. Learnings Running this system in production surfaced lessons that are worth sharing. - Cost: For this use case, AWS Lambda, DynamoDB, and S3 generated costs well under $10/month for hundreds of engineers. Amazon Athena is the one line item to size carefully: the bill scales with scan scope and run frequency, so keep the invocation-log schema lean (token counts and identity/model metadata only) and query the pre-aggregated cost view once per run rather than scanning raw logs repeatedly. - JSON logs mean every query scans every byte, regardless of columns selected. Athena must deserialize each row before applying any filter, so four differently filtered queries against the same view each scanned the same ~11 GB. Column pruning and predicate pushdown do not help against row-oriented JSON. Fix it by combining derived queries into one SELECT ... GROUP BY and splitting results in application code, or go further and convert logs to a columnar format like Parquet. - Governance accelerates adoption rather than restricting it. The strategic insight is counterintuitive: putting a hard per-user cap in place made leadership comfortable expanding access, not contracting it. Because spending is now observable, Jamf could confidently grow the number of AI-enabled engineers. - Always available model. Keeping a low-cost model always available was a deliberate choice. An engineer at 100% of the budget can still get work done, so enforcement does not fully stop productivity. - New models need an explicit pricing branch. Treat the pricing map as a first-class operational artifact and add new models at the moment they are enabled. - IAM policy version limit. A managed policy retains a maximum of five versions. The enforcement Lambda must delete the oldest non-default version before creating a new one, or iam:CreatePolicyVersion will fail. - Account for the asynchronous query model of Amazon Athena. Athena queries are submitted and polled, not returned synchronously. The Lambda handler must start the query, wait for completion, and then read results. Set the function timeout accordingly. Cleanup To avoid ongoing charges and remove the enforcement controls, delete the resources you created. Conclusion In this post, you learned how Jamf built real-time, per-user spend enforcement for Amazon Bedrock using Customer Managed Policies, an Amazon Athena cost view, and a serverless AWS Lambda loop. As generative AI adoption scales, the organizations that win will be the ones that can answer the tokenomics question with confidence: we know what each person spends, we can cap it without slowing anyone down, and we can prove the value. Cost governance is what makes it safe to say yes to more AI, not less. Now, it’s your turn to start exploring. We are here to help, and if you need further assistance, reach out to AWS Support and your AWS account team.
document fetched 2026-09-02 06:30 · canonical https://aws.amazon.com/blogs/machine-learning/tokenomics-at-scale-how-jamf-built-real-time-spend-enforcement-for-amazon-bedrock