large language models

2026-11-02

RESEARCH · 1 SOURCE · arXiv cs.AI

Paper proposes a portable specification for the "agent harness" used with LLMs

The arXiv preprint (arXiv:2610.40917v1) decomposes the "agent harness"—the infrastructure surrounding large language models—into eight functional components and proposes a portable, versioned specification framework that records which model ran in which harness. The authors argue such a record is needed because harness configurations can cause multi‑fold differences in reported scores, comprise much of an agent's attack surface, and are not captured by existing model or agent cards or current AI documentation laws.

6.0

2026-11-02

RESEARCH · 1 SOURCE · arXiv cs.AI

Nedimo: multilingual, evidence-grounded pipeline for low-arousal AI-generated sleep stories (arXiv v1)

The paper presents Nedimo, a multilingual pipeline that separates deterministic planning from LLM generation to produce low-arousal narrated sleep stories. It includes a four-phase attention taper, structural and lexical validation with bounded error-aware retries and quarantining of unresolved outputs, provider-agnostic TTS with paragraph-aware joining and two-pass loudness normalization, a publication gate enforcing loudness/true-peak limits, an evidence-to-requirement mapping, technical evaluation criteria, and a prospective protocol for testing sleep-onset effects; no clinical efficacy is claimed.

5.0

2026-10-31

RESEARCH · 1 SOURCE · arXiv cs.AI

OpenFrontiers: dataset and benchmark for LLM scientific rediscovery (arXiv:2610.37739v1)

OpenFrontiers is a cross-disciplinary resource combining a structured corpus of 12,151 open research problems with a retrospective benchmark of 200 mathematics problems (110 resolved in 2026, 90 still open), each formalized as Lean 4 statements with automated checks and independent reviews; a higher-quality HQ-50 subset is provided. The authors publish an evaluation protocol (closed-book, frozen-retrieval, live-web) and report results evaluating seven LLM agents on HQ-50 under a standardized budget, finding closed-book rediscovery rare (at most 7 of 40 resolved problems) while live-web successes largely reflect retrieval of published solutions; the dataset is hosted at the linked Hugging Face collection.

7.0

2026-10-31

RESEARCH · 1 SOURCE · arXiv cs.AI

Paper (arXiv:2610.38100v1) argues CLadder-generated tests embed superficial patterns that inflate LLM causal reasoning claims

A new arXiv paper (2610.38100v1) examines the CLadder procedural generator used to benchmark formal causal reasoning in LLMs and finds that CLadder embeds highly predictive superficial patterns. The study shows simple models (e.g., random forests) can perform well after training on generated samples, that commercial LLM performance on CLadder tests is brittle, and that removing text needed for formal causal reasoning often does not reduce model accuracy, suggesting LLMs may exploit generator-specific artifacts rather than true causal reasoning.

7.0

2026-10-27

RESEARCH · 1 SOURCE · arXiv cs.AI

Smart Coder Framework: structuring human–LLM collaborative qualitative coding (arXiv:2610.33205v1)

The paper proposes the Smart Coder Framework, a three‑phase approach (Instruction, Prompt Strategy, Reliability Check) to structure human–LLM collaborative qualitative coding. The authors evaluated the framework on 72 middle‑school math teacher interview transcripts (12 for prompt development, 60 for formal evaluation) against a human‑consensus benchmark using a factorial design comparing instructed vs uninstructed prompts, few‑shot vs few‑shot+chain‑of‑thought, and single‑pass vs self‑consistency; instruction consistently improved agreement, chain‑of‑thought helped further under adequate instruction, and self‑consistency gave smaller theme‑dependent gains while flagging unstable decisions for human review.

6.0

2026-10-23

RESEARCH · 1 SOURCE · arXiv cs.AI

SpecEvo: multi-LLM speculative-evolution framework for cost-efficient scientific discovery

The paper introduces SpecEvo, a multi-LLM framework that coordinates inexpensive 'Speculators' and an 'Advisor' with occasional interventions from a frontier-model 'Navigator' to consolidate search evidence and direct exploration. Across 176 tasks in four scientific and algorithmic discovery domains, SpecEvo matches or outperforms frontier-model baselines while cutting API cost by up to 80%, improving equation-discovery evaluation efficiency by up to 1.5x, and beating the strongest CO-Bench baseline by 3.8%.

6.0

2026-10-21

RESEARCH · 1 SOURCE · arXiv cs.AI

Navalia framework uses verifiable task synthesis and long-horizon post-training to improve Cowork Agents

Navalia is a framework that builds verifiable analytical tasks from corporate financial reports and applies long-horizon post-training (SFT and RL) to improve Cowork Agent performance. Initialized from Qwen3.6-35B-A3B, the approach uses semantic evidence selection, executable computation chains, and independent verification to generate training trajectories and yields consistent gains across five benchmarks (OfficeQA Pro, FinSearchComp, Finance Agent Benchmark, APEX-Agents-AA, Workspace-Bench-Lite).

7.0

2026-10-19

RESEARCH · 1 SOURCE · arXiv cs.AI

DRAGO: deterministic diagnosis framework to repair LLM-generated formal specifications

The paper introduces DRAGO, a repair framework that replaces ungrounded LLM critiques with deterministic, formalism-specific diagnosis and only accepts rewrites when a reference-free acceptance predicate shows reduced diagnosed error weight. Evaluated on SMT-LIB2, MiniZinc and PDDL benchmarks, DRAGO cuts specification regressions from 11.8% to 1.3% and raises answer accuracy by 12.9 percentage points over zero-shot formalization and 8.7 percentage points over LLM-diagnostic repair; the authors also identify and correct rendering defects that made half of the Planetarium split inconsistent with references.

7.0

2026-10-15

RESEARCH · 1 SOURCE · arXiv cs.AI

Paper finds widespread methodological failures in agentic data science systems

This arXiv paper evaluates "agentic" data science systems beyond accuracy using (1) a large-scale analysis across 220 data science tasks, (2) a reusable test suite targeting six specific failure modes, and (3) qualitative comparisons to expert Kaggle solutions. It finds most agentic pipelines contain at least one methodological failure (e.g., data leakage, improper validation), that data leakage often correlates with higher reported performance, and that agents lack creativity and domain knowledge compared to top human practitioners.

7.0

2026-10-12

RESEARCH · 1 SOURCE · Microsoft Research

StudentSim: personalized AI student simulators and StudentSimEval benchmark for testing AI tutors

StudentSim is a research system that trains personalized AI student simulators from publicly available, de-identified learner records to model how individual students answer questions and respond to tutor guidance. The work also introduces StudentSimEval, a benchmark spanning chess, second-language English writing, and mathematics; the research team reports that StudentSim outperforms leading baselines and is intended for research use only, not classroom deployment or teacher replacement.

6.0

2026-10-09

COMPANIES · 1 SOURCE · AWS Machine Learning

Postman runs Agent Mode for 40 million developers on Amazon Bedrock

Postman describes how its Agent Mode — an AI-native interface for API testing, documentation, discovery, and implementation — is run in production for ~40 million developers using Amazon Bedrock. The post details architectural patterns (controlling tool sprawl, schema-based reads, treating context as the bottleneck), operational choices (geographically scoped cross-Region inference, model-dependent zero-data-retention, multi-tier prompt caching), human approval before state changes, and use of Amazon Bedrock Guardrails to redact PII; it also explains narrowing >170 tools to ~15 via a vector database of tool embeddings to limit the model's tool surface.

7.0

2026-10-05

RESEARCH · 1 SOURCE · arXiv cs.AI

Tropical Reinforcement Learning (TROPIC) swaps sum for max to enable compositional solutions

The arXiv preprint proposes Tropical Reinforcement Learning, which replaces the usual sum-over-trajectories objective with a max (the tropical semiring) so state values reflect the log-probability of the most likely verified solution and yield explicit replayable paths. The authors present TROPIC, a training algorithm for deterministic, resettable environments with verifiable outcomes, and report up to 16 percentage points improvement over strong on-policy baselines on four tasks (Sokoban, Countdown, FrozenLake, WebShop).

7.0

2026-10-02

RESEARCH · 1 SOURCE · arXiv cs.AI

Benchmarking automatic prompt optimization with a 1,118‑puzzle chess benchmark (arXiv:2610.00416v1)

The paper introduces a chess-based benchmark built from 1,118 Lichess puzzles to evaluate automatic prompt optimization (APO) for frozen LLMs, offering exact-match scoring, engine-based move evaluation, renewability, and low cost (the study ran for about $800). The authors evaluate six APO algorithms on eight target models (reporting that Gemini 3.5 Flash solves ~55% of puzzles), and release puzzles, code, and dataset-renewal scripts on GitHub.

6.0

2026-09-25

RESEARCH · 1 SOURCE · arXiv cs.AI

AdvRole: adversarial closed-loop curriculum for RL role-playing agents

An arXiv paper (arXiv:2609.28609v1) introduces AdvRole, an adversarial context‑rewriting framework that alternates between an Actor (learning to role‑play) and a Rewriter (editing character profiles and dialogue contexts) to create a closed‑loop curriculum; the Rewriter is trained with a performance‑gap reward so scenarios evolve to target the Actor's weaknesses. The authors report consistent improvements over baselines on three English/Chinese benchmarks and release a new multilingual benchmark.

7.0

2026-09-25

RESEARCH · 1 SOURCE · arXiv cs.AI

Epydemix Agent Framework enables LLM agents to run stochastic epidemic models (arXiv:2609.28692v1)

The paper introduces the Epydemix Agent Framework, an additive layer over the open-source Epydemix Python library for stochastic compartmental epidemic modeling that lets AI agents based on large language models discover available models and parameters, validate declarative scenario specifications, execute models through tested library code, and inspect results. The authors demonstrate an end-to-end workflow comparing vaccination strategies for a novel respiratory virus and evaluate the framework across 50 agent sessions and five modeling tasks, finding it reduced interaction turns, output tokens, and cost on most tasks while noting tradeoffs when allocating resources for per-point reproducibility.

6.0

2026-09-25

RESEARCH · 1 SOURCE · arXiv cs.AI

Human–AI hypothesis testing: SCALE policy for cost-aware selective AI scoring and human escalation

This arXiv preprint (arXiv:2609.28859v1) studies using AI judgments together with selective human verification to perform valid hypothesis tests with controlled type‑I/II errors at minimum cost. The authors derive an information‑theoretic lower bound on cost, characterize a report‑dependent information frontier, and introduce SCALE, a sequential cost‑aware policy that adaptively combines selective AI scoring with human escalation; SCALE is finite‑sample valid, matches the bound to first order as target error probabilities vanish, and can be extended using paired AI–human pilot data.

6.0

2026-09-23

RESEARCH · 1 SOURCE · arXiv cs.AI

ArXiv preprint shows LLMs benefit from deliberation across four tasks

The arXiv preprint arXiv:2609.22497v1 adapts a three-stage human deliberation paradigm to multiple large language model families and tests it across four domains (visual numerical estimation, ML paper peer review, detecting hidden malicious AI behavior, and sports forecasting). Deliberating LLM groups produced lower collective error than passive aggregation, individual model judgments improved after deliberation, and the advantage depended on model diversity (clones of a single model did not benefit).

7.0

2026-09-23

RESEARCH · 1 SOURCE · arXiv cs.AI

MSBD: Multi-Split Boundary Decision for LLM-based page stream segmentation

The paper introduces Multi-Split Boundary Decision (MSBD), a zero-shot approach that predicts multiple document boundaries within a page window in a single LLM call to reduce inference requests. Evaluation across multiple language models, collections, input modalities, and window sizes shows MSBD can substantially improve inference efficiency while preserving segmentation accuracy within a model- and corpus-dependent operating range, but accuracy falls off at larger windows with distinct over- and under-segmentation behaviors.

6.0

2026-09-23

RESEARCH · 1 SOURCE · arXiv cs.AI

arXiv paper proposes failure taxonomy and TADL lifecycle for trustworthy agentic AI

The new arXiv paper reviews security and operational risks specific to agentic AI built on large language models, organizes failure modes (e.g., indirect prompt injection, backdoor triggers, goal misgeneralization, memory contamination, cross-session leakage) into a unified taxonomy, and surveys mitigation approaches. It introduces the six-phase Trustworthy Agent Development Lifecycle (TADL) mapping trust activities and evidence across specification, design, training, evaluation, deployment, and monitoring, but notes TADL has not yet been empirically validated and highlights gaps in benchmarks and future research priorities.

7.0

2026-09-16

RESEARCH · 1 SOURCE · Meta AI

Brain2Qwerty v2 decodes real-time sentences from non‑invasive MEG; code released

Researchers released Brain2Qwerty v2, an end‑to‑end AI pipeline that decodes real‑time sentences from non‑invasive magnetoencephalography (MEG) recordings, reporting 61% word accuracy overall (78% for the best participant) after training on ~22,000 sentences from nine volunteers; the team is also releasing full training code for v1 and v2 and a partner (BCBL) is releasing the v1 dataset. The approach uses end‑to‑end deep learning and fine‑tuned large language models alongside tools like Tribev2, NeuralSet, and NeuralBench, and the authors report decoding accuracy improves log‑linearly with more data, narrowing the gap with invasive methods.

8.5

2026-08-26

RESEARCH · 1 SOURCE · Apple Machine Learning Research

IDEA Prune: An integrated enlarge-and-prune pipeline for generative language model pretraining

The paper advocates incorporating enlarged-model pretraining into structured pruning pipelines and treats the enlarge-and-prune process as a single integrated system. It studies whether pretraining a larger model is worthwhile even if the larger model is never deployed and how to optimize the pipeline for token efficiency.

6.0

2024-03-11

RESEARCH · 1 SOURCE · Google Research

Chain-of-Table: Iteratively evolving tables as a reasoning chain for improved table understanding

Researchers (Zilong Wang and Chen-Yu Lee of the Cloud AI Team) propose Chain-of-Table, a framework that trains LLMs via in-context learning to iteratively generate table operations and update intermediate tables as an explicit reasoning chain; this transforms complex tables into simpler, question-aligned views and reportedly achieves new state-of-the-art results on WikiTQ, TabFact, and FeTaQA benchmarks.

7.0