BoardSafety benchmark evaluates LLMs on board-specific safety using manufacturer documentation
The BoardSafety benchmark (arXiv:2610.33448v1) converts vendor datasheets and hardware guides into board-specific safety rule bases and labeled operating scenarios (safe, unsafe, insufficient-info) to test whether LLMs can judge if proposed operations are safe for specific development boards. The authors evaluate models with varying access to extracted rules and structured-rationale prompting, finding that rule access generally improves accuracy but current models still cannot reliably determine safety.