RESEARCH · RESEARCH · #2309
TxnQA: benchmark for analytical reasoning and evidence grounding over transaction event sequences (arXiv:2610.27256v1)
TxnQA is a new QA benchmark (arXiv:2610.27256v1) for analytical reasoning and evidence grounding over synthetic, privacy-mitigated transaction event streams. It includes 27 task families (81 templates) plus a 5-scenario fraud-risk reasoning suite; evaluates format-following rate (FFR), conditional content accuracy (CA), and end-to-end correctness (E2E); and finds that LoRA-adapted open models can enforce output schemas but still struggle with arithmetic and evidence attribution — e.g., Gemma-4-31B-IT and Qwen3-8B reach E2E scores of 0.867 and 0.816, MoE checkpoints 0.632–0.713, and on the 500-instance fraud suite verdict accuracy often >80% while joint (verdict+exact evidence) accuracy falls to ≈23%.
KEY POINTS
- TxnQA is a new QA benchmark (arXiv:2610.27256v1) for analytical reasoning and evidence grounding over synthetic, privacy-mitigated transaction event streams.
- It includes 27 task families (81 templates) plus a 5-scenario fraud-risk reasoning suite; evaluates format-following rate (FFR), conditional content accuracy (CA), and end-to-end correctness (E2E); and finds that LoRA-adapted open models can enforce output schemas but still struggle with arithmetic and evidence attribution — e.g., Gemma-4-31B-IT and Qwen3-8B reach E2E scores of 0.867 and 0.816, MoE checkpoints 0.632–0.713, and on the 500-instance fraud suite verdict accuracy often >80% while joint (verdict+exact evidence) accuracy falls to ≈23%.
- The benchmark highlights distinct failure modes — schema compliance, semantic calculation, and evidence attribution — critical for deploying QA models on transaction histories and financial audit tasks.
WHY IT MATTERS
The benchmark highlights distinct failure modes — schema compliance, semantic calculation, and evidence attribution — critical for deploying QA models on transaction histories and financial audit tasks.