TxnQA: benchmark for analytical reasoning and evidence grounding over transaction event sequences (arXiv:2610.27256v1)
TxnQA is a new QA benchmark (arXiv:2610.27256v1) for analytical reasoning and evidence grounding over synthetic, privacy-mitigated transaction event streams. It includes 27 task families (81 templates) plus a 5-scenario fraud-risk reasoning suite; evaluates format-following rate (FFR), conditional content accuracy (CA), and end-to-end correctness (E2E); and finds that LoRA-adapted open models can enforce output schemas but still struggle with arithmetic and evidence attribution — e.g., Gemma-4-31B-IT and Qwen3-8B reach E2E scores of 0.867 and 0.816, MoE checkpoints 0.632–0.713, and on the 500-instance fraud suite verdict accuracy often >80% while joint (verdict+exact evidence) accuracy falls to ≈23%.