Fix evaluation bias to unlock better models on Amazon Bedrock

extract

Artificial Intelligence Fix evaluation bias to unlock better models on Amazon Bedrock This post was co-authored with Khaled Thabit, Meriem Boughrara, and Mohamed Desouky from Property Finder. The engineering team at Property Finder, a digital real estate application, discovered that 96.8 percent of their evaluation labels had been inherited directly from a single model’s predictions rather than verified by human experts. Their ground truth was systematically skewed toward the incumbent model, making every alternative appear to underperform. The labels also had errors, but the bias was what blocked objective model comparison. They uncovered this when migrating their vision AI workload to a more cost-effective option on Amazon Bedrock. Property Finder processes tens of thousands of property images every day across the Middle East and North Africa (MENA). Their system evaluates 11 quality attributes per image. These cover watermark detection, room classification, furnishing status, photo quality, and construction indicators. That’s thousands of new property listings with an average of 15 images each. All need quality assessment before going live. Property Finder needed a more cost-effective option. But measured against those labels, any alternative was at a disadvantage. The hardest part was not finding a better model. It was discovering that their evaluation methodology itself was the blocker. In this post, we share how Property Finder identified evaluation bias, built unbiased ground truth, and applied evolutionary prompt optimization to select the right model on Amazon Bedrock, achieving higher accuracy at 86 percent lower cost. We walk through the methodology and iteration examples, with lessons you can apply to your own workloads. The following figure illustrates the prompt optimization loop that Property Finder and the AWS Generative AI Innovation Center team used to iteratively improve classification accuracy on Amazon Bedrock. The diagram shows the iterative prompt optimization cycle. A property image is processed by Amazon Bedrock, which outputs the predicted attributes in JSON format. During optimization, these predictions are scored with Micro F1 against the 256-image expert-annotated development set. Error analysis then inspects failed images and identifies visual patterns, which inform the next prompt version. The loop ran from v0 to v27. AI assistants helped refine the prompts, and later versions used automated optimization. Your evaluation might be lying Many teams encounter this scenario. You want to compare vision models, so you build an evaluation dataset. The most common way to get labels is to run your current model and treat its outputs as ground truth. The model has been running in production and nobody complained, so the labels must be good enough. But this creates a circular evaluation where any alternative gets judged by how closely it matches the incumbent’s behavior, not by actual correctness. Property Finder’s evaluation dataset had exactly this problem. Of their labeled samples, 96.8 percent came directly from the predictions of the existing vision language model (VLM) rather than independent human verification. Any alternative model was structurally disadvantaged before a single image was even processed. Hidden errors confirmed the severity. Watermark text extraction had character-level mistakes marked as correct, including hallucinated brand names and missing characters. Photo classification forced single labels onto images containing multiple room types. In the remaining 3.2 percent of human-labeled samples, annotators had introduced categories outside the original specification, like Office appearing where no such label existed in the prompt. A model comparison loses validity when the answer key was written by the model you are trying to replace. Building ground truth you can trust To break the cycle, you need human experts with real domain knowledge doing fresh annotations independently. No reference to previous model outputs. No access to what the incumbent predicted. Dataset scope Property Finder’s team hand-annotated 356 UAE images and a separate set of Egypt images. The team selected these from unlabeled production listings across representative industry segments. This post reports results on the UAE images. The Egypt images served as a cross-regional check on 230 images and are not reported here. The first 256 UAE images to be annotated formed the development set for tuning. The evaluation set for the final model comparison uses all 356, adding 100 images annotated after the development runs. A set of 356 images is roughly sufficient to detect moderate differences between models. Rare classes within individual attributes have fewer examples, so smaller differences on those attributes might not be detectable. Expert annotation A strategy and analytics expert annotated all images. They brought specialized knowledge of real estate listing quality standards. The AWS Generative AI Innovation Center team conducted systematic quality reviews through dedicated clarification sessions. Those reviews settled ambiguous cases and kept the labeling consistent across the dataset. Multi-label transition Real estate images frequently show multiple rooms through doorways or display overlapping quality characteristics. The switch from single-label to multi-label classification alleviated a major source of artificial ambiguity. A hallway photo showing a bedroom through a doorway is genuinely both. After you allow multiple labels, you measure what the model gets right rather than penalizing it for reasonable secondary predictions. A hybrid approach could complement this methodology: models from different families surface candidate labels and disagreements, and human experts adjudicate every case, so no model output is used directly as a label. This would reduce the annotation burden for future dataset expansion. Metrics selection Micro F1 served as the primary metric because it weights each prediction equally, which matches the business goal of overall label correctness. Macro F1 was reported alongside it because Micro F1 is dominated by frequent classes and can hide weak performance on rare attributes. Per-label precision and recall helped identify specific failure modes. Per-sample intersection over union (IoU) measured how closely each image’s predicted labels overlapped the expert’s labels. The nine single-label attributes, such as rotation or watermark detection, have one right answer per image. The AWS team also scored them with exact-match accuracy, the share of images where the answer matches the expert label. Prompt optimization as reinforcement learning With unbiased evaluation in place, you can start improving. You cannot improve what you cannot measure correctly. The optimization follows a loop analogous to reinforcement learning. Think of each prompt version as an agent, the Micro F1 score as the reward signal, and error analysis as the gradient for improvement. The idea resembles GEPA (Genetic-Pareto reflective prompt evolution), where you mutate prompt components, evaluate fitness, select survivors, and iterate. In this engagement, the AWS team refined most versions from error analysis, with AI assistants reviewing failed images and proposing changes. MIPROv2, an automated prompt optimization framework from the DSPy library, complemented this manual process for the photo classification task in v27. Where it started The first prompt (v0) used generic classification instructions, simplified here for illustration: “Classify this image into one of the following categories: Living Room, Kitchen, Bedroom, Bathroom…” Proprietary model A, Property Finder’s production baseline, scored a Micro F1 of 0.55 on the 256-image development set. It used v22, a prompt tuned for its model family. Scores before v21 used the earlier single-label setup and aren’t directly comparable. Iteration by iteration The AWS team tested early versions on several candidate models and added structured output formatting and attribute-specific definitions. The most instructive fix came from image-level error analysis. In the early single-label versions, the model consistently classified rooms visible through doorways as the secondary room type. The AWS Generative AI Innovation Center team added explicit instructions to classify based on the room occupying the majority of the frame. The later switch to multi-label output kept the main room as a label and allowed rooms visible through doorways as additional labels. Regional customization produced the next gain. In some regions, listing photos carry decorative Arabic text overlays that are not watermarks. Rules for that text removed a whole category of false positives. The v21 prompt switched photo classification and room category to multi-label output and became the shared starting prompt for every model. With it, Llama 4 Maverick 17B Instruct from Meta scored a Micro F1 of 0.75 on the development set. From there, the AWS team tuned each model’s prompt separately. For this model, v24 targeted room size overestimation, multi-label handling, and output format. Then v25, an automatically optimized version, moved the critical tasks first. In v27, MIPROv2 restructured the photo classification task, and the model’s Micro F1 on that task reached 0.76. Where it ended The final prompt, v27_PF, is v27 plus rules that Property Finder added for renders, images with borders or frames, and corridors. Like v27, it keeps the Arabic text rules and output format instructions. A normalization step after each model call handles remaining format differences. With v27_PF, Llama 4 Maverick 17B Instruct scored a Micro F1 of 0.80 on the 356-image evaluation set, where proprietary model A scored 0.56 with v22. The evaluation set includes the 256 development images, and both models’ prompts were tuned on those images. These scores are therefore optimistic. The switch from proprietary model A to Llama 4 Maverick 17B Instruct, each on its own tuned prompt, lifted Micro F1 from 0.55 to 0.81 on the development set. That is a 47 percent relative gain. Prompt tuning alone, on Llama 4 Maverick 17B Instruct, raised it from 0.75 with the shared v21 prompt to 0.81 with v27. Lessons from failed approaches Not everything succeeded. The 2-bucket approach put the watermark attributes and the other attributes in two separate API calls. It brought no meaningful accuracy gain. Boxed-image detection also dropped sharply. Without the watermark attributes in the same prompt, the models mistook watermark overlays for image borders. Generic prompts also failed regardless of which model you ran them on. Real estate imagery has too much domain specificity for a one-size-fits-all instruction. Results: Cost, accuracy, and speed The AWS Generative AI Innovation Center team evaluated multiple models on Amazon Bedrock with the unbiased evaluation framework. All models started from the shared v21 prompt and were then tuned for their model family on the development set. Proprietary model A got one tuning round, v22. Llama 4 Maverick 17B Instruct got several, v24 to v27, plus Property Finder’s edge-case rules, so it had more tuning effort behind it. The final comparison against proprietary model A used the 356-image evaluation set. Property Finder selected Llama 4 Maverick 17B Instruct for their production workload based on accuracy across all 11 attributes and cost efficiency. For model availability by Region, refer to Supported models by AWS Region in Amazon Bedrock. The following table shows the exact-match accuracy that Llama 4 Maverick 17B Instruct (v27_PF) reached on each single-label attribute, scored on the 356-image evaluation set. It scored 90 percent or higher on six of the nine, and matched or exceeded proprietary model A on all 11 attributes. On photo classification, which can have several labels per image, the two scored the same. Cost per 1,000 images Property Finder scores tens of thousands of images a day, so per-image cost matters. Llama 4 Maverick 17B Instruct is a cost-effective open-weight model, and the pipeline gets all attributes for an image in a single call. These figures are estimates as of December 2025 in the US East (Ohio) Region. The Llama 4 Maverick 17B Instruct figure assumes 9,000 input tokens and 200 output tokens per image, from the original two-call cost analysis. A single call with the production prompt carries about 7,500–8,000 input tokens, so this is an upper estimate. The 86 percent compares it with the cost analysis estimate for Property Finder’s previous setup, which used the v21 prompt. The accuracy comparison used v22, which was tuned from v21, so its cost might differ. For current pricing, refer to Amazon Bedrock pricing. Inference speed Property Finder runs this optimized pipeline in production, processing tens of thousands of images daily with Llama 4 Maverick 17B Instruct on Amazon Bedrock. In their own monitoring, Property Finder observed a 40–60 percent reduction in inference latency compared with their previous setup. Apply this to your workloads Start by auditing your evaluation labels. If they came from model predictions rather than human experts, you have the same bias problem Property Finder found. Build domain-expert ground truth for a few hundred representative samples, and keep a separate test split that you never use for tuning. Adopt multi-label metrics where your classification task involves overlapping attributes. Then iterate on prompts using image error analysis, treating each iteration as a generation in an evolutionary optimization loop. When you deploy a pipeline like this, add responsible AI controls. For example, you can use Amazon Bedrock Guardrails to enforce content policies and filter inappropriate model outputs. The attribute predictions in this pipeline don’t come with a calibrated confidence score, so use a practical signal to choose images for human review. Model disagreement is one such signal. When two models give different answers for an attribute, the image is often ambiguous. A second model on every image adds cost, so you can start with a sample. Check how often the signal catches real errors on your development set before you rely on it. Conclusion In this post, we walked through how Property Finder identified evaluation bias, built unbiased ground truth, and applied evolutionary prompt optimization to achieve better accuracy at lower cost through model selection on Amazon Bedrock. Three elements needed to come together for this result: unbiased evaluation to measure honestly, the flexibility to evaluate and switch between model families without re-platforming, and carefully evolved prompts tailored to the domain. Amazon Bedrock provided the foundation. It gave Property Finder access to multiple model families through a single API, inherited AWS governance and security controls, and the ability to adopt new models as they become available without changing their infrastructure. To get started with model evaluation and selection on Amazon Bedrock, visit the Amazon Bedrock documentation. If you process vision workloads at scale and want help building an unbiased evaluation framework, reach out to the AWS Generative AI Innovation Center. Share your experiences with model evaluation and prompt optimization in the comments. Acknowledgments Additional contributors from AWS include Yousuf Albannai, Amine Sehili, Thomas Ounnas, Aris Tsakpinis, Rafik Soliman, Nikhil Ulhas Pandit, and Sneh Sharma.