FUNDING · MODELS · #2670
Property Finder fixes evaluation bias and cuts vision-AI costs on Amazon Bedrock via prompt optimization
Property Finder found that 96.8% of its image-quality evaluation labels were inherited from a single incumbent vision model, biasing any alternative models. The team re-annotated a development and evaluation set with domain experts (256 and 356 UAE images), then used iterative/evolutionary prompt optimization on Amazon Bedrock (prompt loop reached v27), achieving higher accuracy while reducing cost by 86% for their vision workload.
KEY POINTS
- Property Finder found that 96.8% of its image-quality evaluation labels were inherited from a single incumbent vision model, biasing any alternative models.
- The team re-annotated a development and evaluation set with domain experts (256 and 356 UAE images), then used iterative/evolutionary prompt optimization on Amazon Bedrock (prompt loop reached v27), achieving higher accuracy while reducing cost by 86% for their vision workload.
- This matters because evaluation datasets derived from incumbent model outputs can falsely favor the incumbent; independent expert annotation plus iterative prompt tuning on Bedrock enabled an objective comparison and large cost/accuracy gains.
WHY IT MATTERS
This matters because evaluation datasets derived from incumbent model outputs can falsely favor the incumbent; independent expert annotation plus iterative prompt tuning on Bedrock enabled an objective comparison and large cost/accuracy gains.