Exowatt beats frontier models on accuracy with SageMaker serverless model customization

extract

Artificial Intelligence Exowatt beats frontier models on accuracy with SageMaker serverless model customization Exowatt builds modular, dispatchable solar systems that deliver reliable power for energy-intensive applications such as AI data centers. Its flagship product, the Exowatt P3, captures solar energy through optical lenses, stores it as heat in a long-duration thermal battery, and dispatches electricity on demand through a heat engine. Unlike grid-tied photovoltaic solar systems, P3 powers data centers directly behind the meter, helping data centers come online without waiting in multi-year grid interconnection queues. Exowatt builds a fleet of AI agents that monitor and triage live alarms and support operational decisions for solar systems, often at the edge, with limited or no connectivity. As its fleet scales from test benches toward manufacturing and thousands of deployed field units, the intelligence layer that operates these systems must scale with it. In this post, we show how Exowatt used Amazon SageMaker AI serverless model customization to build a repeatable AI pipeline. The pipeline produces task-specific fine-tuned models for two deployment targets, edge and cloud, achieving higher accuracy than frontier models at up to 49× lower cost. The two models described here are the first outputs of that pipeline rather than the end goal. The more durable result of the engagement was the pipeline itself: a repeatable path from raw operational data to a certified training set, a fine-tuned model, and an evaluated deployment artifact. Exowatt can now run this path in days rather than months. Alarm triage and energy-market analysis are two of many narrow, high-value tasks where the same approach applies. The challenge: AI on physical energy infrastructure Exowatt’s energy management system (EMS) runs across a growing fleet of operational sites. Each site continuously produces alarms: a hardware component operating out of spec, a dropped connection, an out-of-range voltage reading, and similar events. As the fleet scales toward thousands of field units, a human operator can’t monitor every site in real time. Exowatt needed an AI agent that could read the alarm state on each site and take or recommend the correct next action autonomously and reliably. A frontier model couldn’t solve this problem for three reasons: - The task is narrow and proprietary. Alarm triage on Exowatt’s EMS depends on their specific hardware, data, and operating logic. A general-purpose frontier model isn’t specialized for it. - It must run at the edge, with no GPU and often no internet. Many sites have limited or intermittent connectivity, so an agent that depends on a cloud call to a frontier model leaves the operator without coverage when the connection drops. - Cost doesn’t scale at fleet level. Routing every operational decision to a frontier model across thousands of sites is not economical for a decision that repeats continuously. This is where Exowatt turned to model customization. Exowatt’s initial fine-tuning workflow relied on selecting an open-weight model, building a container, and creating a custom training recipe before any experiment could run. This was a major blocker for model experimentation. Additionally, they had to decide GPU hardware requirements of model fine-tuning. As the team increased the pace of experimentation, each run first required choosing a specific GPU instance type and then waiting for that self-selected capacity to become available. When a chosen instance type was unavailable, this could delay the start of a run or block it entirely. They needed a model small enough to run on a no-GPU edge controller, yet accurate enough compared to the frontier model on the specific task to act autonomously. Fine-tuning closed that gap: a compact, task-specialized model that is both deployable in the cloud and at the edge. Solution overview Exowatt addressed these challenges by adopting a cloud-native, serverless model customization approach on Amazon SageMaker AI. The same training pipeline produces fine-tuned models for two deployment targets: the edge and the cloud. Exowatt implemented this approach on the Qwen3-1.7B model for the first use case, edge alarm triage, then applied the same approach to the Qwen3-14B model for the second use case, cloud energy-market analysis. The company plans to reuse this solution to build smaller, task-specific models across the fleet. The following seven AWS services power the AI pipeline, with no instances, clusters, or endpoints to manage: The workflow is organized into three stages: Model customization (cloud): Exowatt fine-tunes open-weight base models using serverless model customization, Low-Rank Adaptation (LoRA) and supervised fine-tuning (SFT) on SageMaker AI. Because serverless training allocates capacity on demand, the team runs multiple experiments per day rather than waiting for self-selected GPU. After evaluation, the best model is registered in the SageMaker AI Model Registry. Edge deployment: For latency-sensitive, connectivity-constrained work such as alarm triage, the fine-tuned Qwen3-1.7B model is frozen, quantized to the GGUF format, and run with llama.cpp inside Kubernetes pods at the edge using on-site controllers. This path requires no GPU and operates without internet connectivity. Cloud deployment (elastic): For internet-connected, fleet-level analysis, the fine-tuned Qwen3-14B energy-market model is served through Amazon Bedrock Custom Model Import, which runs with no idle cost and bills only while the model is active. Why Amazon SageMaker AI serverless model customization Amazon SageMaker AI serverless model customization service provides a wide selection of open-weight models with ready-made fine-tuning recipes from the SageMaker Public Hub, so the team avoided building and optimizing a custom training container, work that can otherwise take weeks. For Exowatt, this turned fine-tuning from a scarce, scheduled event into a routine, on-demand one. This makes a rapid experimentation-and-test loop practical while keeping experimentation cost low. Fine-tuning on provisioned GPUs requires choosing a specific instance type, requesting that capacity, waiting for it to become available, running the job, and paying for the instance for as long as it is held. The held time typically includes the idle time between experiments when a dedicated instance is kept running. For a small team running many experiments, this slows down innovation and increases cost. Amazon SageMaker AI serverless model customization automatically manages and orchestrates the infrastructure required to run the fine-tuning job, removing the pain of capacity management, and charges only for the time the fine-tuning job runs. Use case 1: Edge alarm triage with Qwen3-1.7B Exowatt sites raise alarms around the clock. A compact on-site model reads each alarm together with the site’s own status and telemetry and returns one of a small fixed set of classifications with cited evidence (such as an offline system, derated component, or sensor mismatch) or an explicit abstention when the evidence is insufficient. Model selection and training data The team first tried Qwen3-8B with prompting and guardrails, which plateaued below the target (0.735 macro-F1). A fine-tuned Qwen3-1.7B changed that: it outperformed the Qwen3-8B plus prompting setup on this narrow, single-turn task at roughly one-fifth the parameters, while staying small enough for GPU-free edge inference. Thirteen fine-tuned Qwen3-8B candidates across 12 dataset versions never beat the prompted 8B baseline. The smaller model with more varied data did. The training set was small and curated: 2,522 examples built from 684 verified decisions on generated site scenarios plus 1,838 certified paraphrases that kept each decision fixed while rewriting the prose. The paraphrases were generated by Qwen3-235B on Amazon Bedrock for about $3.72. Each paraphrase was kept only if deterministic checks confirmed that the decision, labels, evidence citations, and every number were unchanged (1,838 of 2,218 candidates passed). One fault family was held out of training entirely, and 320 cases were reserved for testing. Training and deployment Training completed in about 32 minutes on SageMaker serverless fine-tuning for roughly $20–$25. Quantized to 8 bits (about 1.71 GiB), the model runs offline in a 6 GiB llama.cpp pod on the site’s existing CPU server. It also runs on a t3.2xlarge instance type. Use case 2: Cloud energy-market analysis with Qwen3-14B Dispatchable storage turns timing into an operational decision: each site must continually decide how much stored energy to hold for its load and when to release it, and for grid-connected sites market conditions feed into that choice. The fine-tuned model reads a six-hour packet of live market data (about 90 rows of real-time prices, day-ahead forecasts, and grid notices) and returns one of five market events (such as a price spike or scarcity adder) in about four seconds. Every claim is tied to a specific data row to inform the dispatch strategy. Why Qwen3-14B Qwen3-14B was chosen because it is supported by Amazon Bedrock Custom Model Import, it ships as a SageMaker Public Hub model with a ready LoRA recipe (no custom container or instance type to manage), and at about 28 GiB it is small enough to serve cost-effectively. Training data and process Exowatt fine-tuned the model on public Electric Reliability Council of Texas (ERCOT) market data drawn from five years of history (about 163 million rows across 12 tables). The team rule-labeled this data into 25,000 examples (22,000 for training, 1,000 for validation, and 2,000 held out for testing). They then expanded the training set to 86,281 rows by having Anthropic’s Claude Sonnet 4.6 on Amazon Bedrock rewrite the prose of each example while its labels and numeric fields stayed fixed. Training was a single LoRA epoch (rank 16, alpha 32), completed in about 8.5 hours of managed serverless fine-tuning with up to four jobs running at once. The team ran 23 jobs and about 155 hours of managed training in the six days from initial plan to a champion model serving in Cloud EMS. The decisive improvement came from paraphrase diversity: adding three rewrites per answer lifted macro-F1 from 0.950 to 0.975. The base model scored only 0.415 macro-F1. It produced valid JSON but did not yet know the market. Fine-tuning fixed that. Deployment The merged checkpoint was imported to Amazon Bedrock Custom Model Import in about 14 minutes. The model runs with no idle cost and bills at $6.86 per hour only while active. Benchmark results On the edge alarm-triage task the fine-tuned Qwen3-1.7B went from a model that couldn’t produce a valid answer to 0.911 macro-F1 with 320 test cases, ahead of every frontier baseline. Applying the identical fine-tune-and-evaluate loop to the energy-market task with 2,000 test cases, the Qwen3-14B reached 0.975 macro-F1, again ahead of the frontier baseline and at a fraction of the cost. In both cases a small, fine-tuned model beat far larger general-purpose models on the specific task. Both models return strict JSON in which every claim cites a specific input data row. Any unsupported claim is dropped before it reaches an operator, which is what lets the team trust small models in autonomous, physical settings. Table 1: Edge alarm triage results, fine-tuned Qwen3-1.7B compared to frontier models $0 per 1,000 decisions means no metered API or GPU inference cost. The quantized model runs on the site’s existing CPU. Table 2: Cloud energy-market analysis results, fine-tuned Qwen3-14B compared to frontier models Costs are Amazon Bedrock list prices times measured tokens per answer. Fine-tuning lifted an open model from last place to first: +0.35 macro-F1 over Claude Opus 4.7 (95 percent confidence interval 0.324 to 0.371) at about 49× lower cost per 1,000 answers ($0.60 compared to $29.26). The model held 0.968 macro-F1 on a second, later test set. Business benefits Faster iteration and faster time to market: Exowatt moved its customized model from experimentation to production in only nine days. This process could have taken months if the team had to build and optimize custom containers and training recipes, provision GPUs, and manage infrastructure. With SageMaker AI serverless model customization, the team trained Qwen3-1.7B in about 32 minutes and Qwen3-14B in about 8.5 hours, with no capacity requests and up to four training jobs running concurrently. For the energy-market use case, Exowatt ran 23 jobs totaling about 155 hours of managed training and moved from the base model to a fine-tuned model serving in Cloud EMS in six days. The merged checkpoint was imported into Amazon Bedrock in about 14 minutes. The end-to-end pipeline used seven AWS serverless services, with no instances, clusters, or endpoints for Exowatt to manage. Higher accuracy compared to frontier models: Both fine-tuned models beat the larger frontier models on their own task: Qwen3-1.7B at 0.911 macro-F1 (compared to 0.698 for Claude Sonnet 4.6) and Qwen3-14B at 0.975 (compared to 0.625 for Claude Opus 4.7). Lower cost across both deployments: In the cloud, the Qwen3-14B serves at $0.60 per 1,000 answers versus $29.26 for Claude Opus 4.7 (about 49× lower), with no idle cost and $6.86 per hour only while active on Amazon Bedrock Custom Model Import. At the edge, the quantized Qwen3-1.7B runs on the site’s existing CPU with no per-inference cost and no dependency on connectivity. Grounded, low-risk outputs: Across both models, the evidence fence removed every uncited claim (any evidence citation whose source, field, or value does not literally match the input), and the edge model abstains rather than guesses when unsure. This is what let the team run it autonomously in shadow mode on a cloud-hosted replica of a site. Lessons learned For specific tasks, smaller fine-tuned large language models can outperform large frontier models: Fine-tuned Qwen3-1.7B and Qwen3-14B models each outperformed frontier models on the target task, delivering higher accuracy at a fraction of the cost with significantly smaller models. Data quality and variety outweigh raw volume and extra epochs: For tightly scoped tasks, data quality mattered more than model size or data volume. On the alarm-triage model, more verified training variety moved macro-F1 from 0.584 (trained on the 684 verified decisions alone) to 0.911 (with the paraphrase-expanded 2,522-row set). Simply running more epochs over the same rows did worse: a control run that trained on the 684 original rows for 11 epochs (the same number of steps) reached only 0.879. Curated data mattered more than dataset size throughout. Removing the provisioning step changes how a team works: When a training run no longer requires waiting for GPU capacity, fine-tuning shifts from a rare event to a routine, iterative one. One training pipeline serves two deployment targets: Separating model training from deployment lets Exowatt use the same pipeline for both offline edge deployment and connected fleet-level analysis, simplifying operations and accelerating model deployment in the cloud. Conclusion Exowatt now has one reusable AI pipeline that took a task from model specific data preparation to model deployment. Rather than tuning one model for different tasks, Exowatt plans to apply the same fine-tuning principles to different tasks, each with its own data, specific models, and evaluations, across the fleet as new device types and market tasks come online. The pattern generalizes beyond energy industry segments. AI engineers, data scientists, and customers across industry segments can adopt the same principles. Use Amazon SageMaker AI serverless model customization as a low-overhead AI pipeline to fine-tune small, task-specific models, then deploy them where the work happens: at the edge when connectivity or latency requires it, and through a managed cloud endpoint such as Amazon SageMaker AI for elastic workloads. To learn more, see Amazon SageMaker AI serverless model customization and Amazon Bedrock.