RESEARCH · RESEARCH · #2003
FDE-Bench: benchmark for end-to-end delivery from underspecified business requests (arXiv:2610.14539v1)
FDE-Bench is a new benchmark built from 49 real customer projects that evaluates agents on end-to-end delivery tasks requiring interactive requirement clarification and iterative optimization (cases span combinatorial optimization and ML). Tested across 18 agent configurations, the top mean delivery score is 0.5510 (FDE reference = 1.0) and the highest delivery success rate is 6.12%; paper arXiv:2610.14539v1 and materials are at https://baidubce.github.io/FM-Agent/FDE-Bench/.
KEY POINTS
- FDE-Bench is a new benchmark built from 49 real customer projects that evaluates agents on end-to-end delivery tasks requiring interactive requirement clarification and iterative optimization (cases span combinatorial optimization and ML).
- Tested across 18 agent configurations, the top mean delivery score is 0.5510 (FDE reference = 1.0) and the highest delivery success rate is 6.12%; paper arXiv:2610.14539v1 and materials are at https://baidubce.github.io/FM-Agent/FDE-Bench/.
- Because it gives a realistic, end-to-end evaluation of agents on underspecified real-world business tasks and shows current agents fall far short of human FDE references, setting a concrete target for research and deployment work.
WHY IT MATTERS
Because it gives a realistic, end-to-end evaluation of agents on underspecified real-world business tasks and shows current agents fall far short of human FDE references, setting a concrete target for research and deployment work.