loading
loading
Run Jev inside an Open Agent Harness Agents burn a lot of tokens on decisions that are really just picking one option from a short list, like matching a client request to the right GitHub issue. So we built a weekly client follow-up agent that does exactly that. We tested it on 40 requests, and it was as accurate as letting Opus make every match, at 150x lower decision cost. Here’s the tech stack we used: - TrueForge as the agent harness - Jev for the matching decisions - Daytona as the sandbox for code execution Jev is a decision model from TypeSafe AI. It takes the options from your app and returns a structured choice with a calibrated confidence score, without generating any text. Low-confidence matches go back to Opus for review, and in our test, that review caught 5 wrong matches that would otherwise have gone through. TrueForge is TrueFoundry’s harness, and it keeps context small. It loads tool definitions only when needed, writes large tool outputs to files, and lets the agent process data in scripts instead of the conversation. On DevRev’s Enterprise Bench, it solved about as many tasks as Claude Managed Agents on roughly a third of the tokens. It also keeps credentials outside the sandbox and holds every write for approval. If you want to try this out, TrueForge is fully open source (MIT-licensed) and runs on your own machine with a single command. TrueForge GitHub: https://github.com/truefoundry/trueforge (don’t forget to star 🌟) Lastly, to help you reproduce my work and understand all the integration details, we wrote a detailed article. How to turn an LLM into a decision model? Say we have a support ticket that says, “I was charged twice for invoice 4411. Please refund.” Our application needs to make two decisions. Which team should handle the ticket, and whether to issue a refund. A language model would then generate text token by token, while a decision model would score the available choices directly. Here’s how to turn an LLM into one: Define the questions and choices: Give the model the ticket alongside the decisions to make. For the team, the choices are billing, technical, and sales. For the refund, they are yes and no. Keep the backbone and freeze its weights: The backbone processes the input and builds internal representations of it. Add LoRA, small trainable components that adapt those representations while the original weights stay frozen. Replace the language model head: The language model head predicts the next token. Replace it with a decision head that gathers evidence from the ticket for each option, lets the questions and options interact, and produces one score per choice. Convert scores into probabilities: Apply softmax, which converts scores into probabilities, separately for each question. You get one probability distribution over the teams and another over the refund choices. Train on the correct answers: For this training example, the labels are billing and yes. Cross-entropy loss measures how well the predictions match those labels, and training updates only the decision head and LoRA components. Calibrate on held-out examples. After training, use examples excluded from training to adjust the model’s confidence. Its predicted probabilities should reflect how often those predictions are correct. At inference time, one forward pass scores all the choices. Your application gets probabilities it can use to route the ticket and make the refund decision. We trained a JEV-style decision model locally using Qwen3.5 0.8B. Accuracy went from 37% to 65% in just 60 steps, with 10 minutes of training on 4GB of VRAM. You can watch our full video walkthrough below, from setup and training to evaluation and inference: Good day!
document fetched 2026-10-24 10:04