Daily AI Digest — 2026-08-13
arXiv Highlights
OpenART: Scaling Agent Red Teaming via Open-Ended Environment Evolution
Problem
Agent safety benchmarks predominantly test isolated prompts against short, static tasks, which misses the dominant failure mode of deployed agents: cumulative risks that emerge from long-horizon, stateful interaction with a mutable environment. When an agent orchestrates dozens of tools over a persistent workspace, an early write (a decoy record, a poisoned instruction, a redirected destination) propagates through subsequent steps in ways that no single-turn prompt injection benchmark can capture. OpenART targets this gap by defining the unit of evaluation as an executable scenario rather than a prompt, and by red-teaming through authorized modifications of environment state rather than through adversarial user messages.
Method
OpenART decomposes evaluation into three stages: scenario construction, target-native projection, and controlled environment evolution.
A scenario is a target-agnostic contract fixing four things: a benign objective \tau, an executable environment specification, a workflow, and a hidden safety condition evaluated by a deterministic rule E_q. From a domain seed (e.g., “cloud-platform change reconciliation”), a planner expands into a full scenario, from which the user-visible task, initial environment x_0, and evaluator are derived. A target adapter \Pi_r projects the scenario into a deployed agent’s native runtime (workspace primitives, MCPs, tools, skills) without altering scenario semantics. This decouples “what is being tested” from “how the target expresses it,” enabling one scenario to be instantiated across heterogeneous agents.
The red-teaming protocol is a controlled Markov process over environment states. At round t, given current environment x_t and attacker context C_t, a policy proposes changes \Delta_t, which are filtered and materialized through the adapter:
\Delta_t \sim p(\cdot \mid x_t, C_t), \quad m_t = \Pi_r(\Delta_t)
The target then executes the original task, producing trajectory \xi_t, next state x_{t+1}, and evaluator feedback Y_t:
(\xi_t, x_{t+1}) \sim K_r(\cdot \mid \tau, x_t, m_t), \quad Y_t = E_q(\tau, \xi_t, x_{t+1})
C_{t+1} = U(C_t, x_t, \Delta_t, m_t, Y_t)
Critically, \tau and E_q are invariant across the entire evolution; only the target-visible state changes. Attack success rate improvements therefore reflect stronger environment perturbations, not a shifted evaluation goalpost. The instantiated policy — EMHA (Evolutionary Markov Hypergraph Attack) — is black-box and requires no parameter updates: it maintains a hypergraph over authorized state transitions and coordinates multi-vector modifications (workspace contents, injected instructions, capability metadata, retained execution state) guided by evaluator feedback.
The strict attack success metric requires agreement between the deterministic evaluator and a GLM-5.2 judge:
\mathrm{ASR}_{\mathrm{strict}} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\{D_i = 1 \land L_i = 1\}
Any disagreement counts as failure, giving a conservative estimate.
Scale and results
The Arena comprises 10,000+ validated scenarios across 50 domains, drawn from a corpus of over 500,000 tools, MCPs, and skills collected following SkillNet. Only bundles that pass deterministic evaluator validation are retained. The median scenario requires 97 tool calls, placing tasks squarely in the long-horizon regime where cumulative state effects dominate. Evaluation covers 75 agent-model configurations (15 deployed agents × 5 foundation models).
The 50-domain vocabulary spans enterprise operations, finance, healthcare, and infrastructure workflows, indicating that scenarios are drawn from operationally realistic settings rather than synthetic sandboxes. The abstract reports that EMHA drives substantial Strict-ASR increases across all 75 configurations relative to their un-evolved baselines (the specific per-configuration numbers are not exposed in the excerpts provided), demonstrating that even agents that complete the benign task reliably remain vulnerable when the environment is adversarially evolved.
Limitations and open questions
Several concerns follow from the design. First, the scenario-as-contract abstraction assumes the deterministic evaluator E_q faithfully encodes the safety condition; scenarios where safety is inherently ambiguous (e.g., context-dependent disclosure) are hard to capture with a fixed rule, and reliance on GLM-5.2 as a co-judge inherits that model’s biases. Second, “authorized state transitions” bound EMHA’s action space — the framework tests robustness against a threat model that has legitimate write access to portions of the workspace, which is realistic for shared-agent settings but does not cover external prompt-injection surfaces. Third, the median 97 tool calls is expensive; running 10K scenarios × 75 configurations × multiple evolution rounds is a serious compute footprint, and the paper does not (in the provided excerpts) quantify per-scenario cost or judge-agreement rates. Fourth, because EMHA is black-box and feedback-driven, its convergence behavior on adversarially trained agents is unclear — the hypergraph structure and update rule U are not detailed here.
Why this matters
Prompt-level red teaming saturates quickly because the attack surface it models is narrow. OpenART reframes the problem: safety is a property of the executing environment over time, and stress-testing it requires evolving state while holding task and evaluator fixed. If EMHA-style environment evolution generalizes, it becomes the natural analog of adversarial training for agent deployments — and the 10K-scenario, 75-configuration substrate gives a reproducible benchmark for measuring progress.
Source: https://arxiv.org/abs/2608.00677
Mechanist: AI as a Scientific Instrument for Discovering the Mechanisms of Intelligence
Problem
Mechanistic interpretability remains a bottleneck: model capabilities scale faster than our ability to explain them, and existing automated interpretability tools address narrow tasks — labeling single neurons or SAE features — rather than proposing and testing general mechanism theories that span pretraining and inference. Simultaneously, “AI scientist” agents such as Sakana’s system and Claude Code target ML engineering tasks (training pipelines, benchmarking) rather than causal analysis of what models internally compute.
This paper proposes Mechanist, a multi-agent framework whose scientific object is the model itself: it autonomously proposes hypotheses about internal mechanisms, runs causal interventions, and iterates.
Method
Mechanist decomposes the discovery loop into four staged agents coordinated by an orchestrator that parses objectives, dispatches agents sequentially, and validates artifacts passed between them:
Hypothesis Agent. Given a user objective, it decomposes intent into domain-specific queries and retrieves evidence from two graphs: (i) a curated interpretability KG of ~13,000 papers, and (ii) SciAtlas, a 43M-paper multidisciplinary graph spanning 26 fields including neuroscience, psychology, and molecular biology. Each hypothesis is emitted as atomic claims paired with milestone-level tests. Cross-disciplinary retrieval is what motivated, for example, the tripartite decomposition of belief reasoning into World Knowledge, Personal Belief, and Attributed Belief, imported from the theory-of-mind literature.
Experiment Agent. Operationalizes each claim into a runnable suite specifying dataset splits, target models, interpretability methods, controls, metrics, and compute budget. It draws from a curated library of 32 mechanistic analysis methods covering probing, causal intervention (activation patching, path patching), and validation. A lightweight sanity-check pass gates full deployment.
Verification Agent. Audits label provenance, data leakage, metric validity, and traceability of reported numbers to logged runs, then assesses conclusion robustness.
Iteration Agent. Feeds verified findings back into the KG so subsequent hypotheses are conditioned on the growing internal record.
Case study: cross-modal subliminal transfer
The most striking behavioral finding is an extension of subliminal learning to opposing-preference and cross-modal settings. Prior work showed a GPT-4.1 student fine-tuned on neutral number sequences from an owl-preferring GPT-4.1 teacher inherits the owl preference. Mechanist proposes and tests two extensions:
- Safety transfer via filtered data. Responses from an unsafe teacher are filtered to retain only content classified safe, then used to fine-tune a Qwen3.5-9B student. The student produces unsafe answers to multimodal laboratory-safety prompts despite training only on filtered-safe data.
- Cross-modal preference transfer. A banana-preferring teacher is used to generate apple images; a Qwen-Image student fine-tuned on these apple images then produces banana images when asked for its favorite fruit.
Both effects are quantified with N=3 training runs and 95% t-CIs, showing statistically clear separation between students trained on regular- vs. trait-carrying teacher outputs. This is a genuine safety result: modality-crossing filters do not neutralize latent trait transmission.
Belief mechanism and interventions
Using the belief taxonomy imported from cognitive science, Mechanist derives a mechanism theory describing how models encode world knowledge, form personal beliefs, and attribute beliefs to others, and traces the emergence of these representations across pretraining checkpoints. The theory is then converted into steering interventions that improve downstream task performance, and — in a scientific-foundation-model setting (Evo2-7B) — into activation-level steers that bias DNA sequence generation toward user-specified properties.
Quantitative evaluation
Mechanist is benchmarked against Claude Code and the Sakana AI-Scientist across 16 reproduced papers spanning 9 research areas, scored on four dimensions (data usage, experiment design, experiment execution, result analysis) by three human experts plus Claude Opus 5 and GPT-5.6-sol as LLM judges (48 system-paper units per judge). Results:
- Mechanist achieves the highest mean reliability under every judge, with judge-pair Spearman rank correlations indicating consistent orderings.
- Hypothesis quality — scored on novelty, impact, and testability against the 10 nearest related works — is higher than both baselines.
The evaluation methodology is itself notable: paired scoring across three human and two LLM judges, bootstrapped CIs (4,000 resamples), and rank-correlation checks between judges.
Limitations and open questions
- The paper’s central metric — LLM- or human-judged reliability of reproductions — is not a ground-truth mechanism test; agreement across judges does not exclude shared blind spots.
- The mechanism library (32 methods) fixes an inductive bias: hypotheses testable only with methods absent from the library are systematically underexplored.
- The subliminal-transfer finding, while important, is demonstrated on specific model families (GPT-4.1, Qwen3.5-9B, Qwen-Image); the scaling laws of trait leakage remain open.
- The “belief mechanism theory” is operational; whether its constructs generalize beyond the probed tasks, or reflect artifacts of the probing basis, is untested.
- Costs, wall-clock, and failure modes of the agent loop are not detailed in the excerpts.
Why this matters
If autonomous agents can propose, test, and verify causal mechanism claims about frontier models — including safety-relevant ones like cross-modal trait leakage through filtered data — mechanistic interpretability becomes a scalable pipeline rather than a bottleneck. The catch is that the field’s evaluation standards must catch up: reliability judged by LLMs on reproductions is not a substitute for adversarial falsification of the mechanism theories themselves.
Source: https://arxiv.org/abs/2608.12036
ToolHazard: Scaling Adversarial Environments for Security Evaluation and Alignment of LLM-based Agents
Problem
Indirect prompt injection is now the dominant attack surface for tool-using LLM agents: an attacker who can write to any environmental state that later flows into the agent’s observations (email bodies, DB records, tool outputs) can hijack the agent’s decision loop without touching the user query or system prompt. Existing benchmarks (InjecAgent, AgentDojo, and derivatives) rely on hand-built environments with fixed injection slots and often use a stochastic LLM to simulate tool responses. This caps domain coverage, produces non-deterministic evaluations, and makes it hard to study where and when injections actually succeed. ToolHazard’s contribution is a synthesis pipeline that removes the manual environment-engineering bottleneck and produces executable, stateful environments with programmatic success checks.
Method
The threat model is standard: an environment e=\langle\mathcal{E},\mathcal{R},\mathcal{T}\rangle with entities/state \mathcal{E}, transition rules \mathcal{R}, and tool APIs \mathcal{T}. Given a benign query q, the agent samples \tau=(q,a_1,o_1,\ldots,a_T,o_T)\sim\mathcal{A}(q,e). Attacks are modeled as e'=\operatorname{Inject}(e,\ell,\delta) where \ell is an attacker-writable state that propagates into observations, and success is defined as the injected \delta triggering an unintended tool action verifiable on the final environment snapshot.
Three modules cooperate. The Environment Simulator ingests seed queries from ToolACE and API-Bank and performs staged prompt-driven planning to produce a blueprint
\mathcal{B}=f_{\text{ops}}\big(f_{\text{state}}(f_{\text{env}}(\mathcal{D}))\big),
with f_{\text{env}} inferring the domain (e.g., banking, calendar, CRM), f_{\text{state}} deriving entity schemas and transition constraints, and f_{\text{ops}} enumerating executable query/mutation operations. The blueprint is then compiled into runnable Python tool code and passed through an automated quality inspector. The Attacker Agent explores the compiled environment to locate viable injection points \ell that are actually reachable during task execution (as opposed to injecting into unread fields) and instantiates payloads under six wrappers: basic-combined, important-template, multi-turn, decision-hijacking, reasoning-criteria, and tool-selection. The User Simulator generates state-grounded long-horizon tasks whose success and attack outcomes are verified by generated check functions run over the final environment snapshot. Check functions are synthesized by GPT-4.1-mini once and then executed deterministically; no LLM judge is in the evaluation loop.
Benchmark construction and coverage
The pipeline initially yields 191 valid environments, split into 140 training and 51 test candidates. Filtering to environments containing injection points actually reachable along task trajectories leaves 60 for ToolHazard-Align (alignment data generation) and 28 disjoint environments for ToolHazard-Bench.
ToolHazard-Bench contains 512 tools and 87 state-grounded tasks with an average execution horizon of 15.56 steps — considerably longer than most prior injection benchmarks, which stress single- or few-step tool sequences. Domain breadth inherits from the seed datasets (see the environment word cloud, dominated by finance, scheduling, health, e-commerce, and productivity terms).
Results
Seven agents are evaluated in a ReAct loop: GPT-5, GPT-4.1, Gemini-3.1-pro-preview, Gemini-2.5-pro, DeepSeek-V3.2, and Qwen3-8B/4B. The paper’s headline empirical claims:
- All target agents exhibit substantial vulnerability under the six environment-side attack strategies; no model is robust across the full attack suite.
- Injection timing matters strongly. Holding the attack strategy fixed to tool-selection, injecting at the earliest reachable state (top-1) yields consistently higher ASR than top-2 or random placement. Early-step observations are far more likely to reshape the agent’s plan than later ones, presumably because the agent’s early ReAct thoughts commit to a trajectory that later contradictory content cannot easily override.
- Fine-tuning on ToolHazard-Align data reduces ASR on both ToolHazard-Bench and the out-of-distribution AgentDojo benchmark while preserving benign task completion (BR), indicating that the synthesized alignment data transfers rather than overfitting to ToolHazard’s own attack templates.
Because BR and ASR are computed via executed check functions over environment snapshots, the results avoid the judge-variance and prompt-sensitivity that plague LLM-graded agent benchmarks.
Limitations and open questions
Coverage is bounded by the seed datasets: environments outside the ToolACE / API-Bank domain distribution are not synthesized. Environment generation, task generation, and check-function synthesis all use GPT-4.1-mini, so systematic biases in that model (e.g., under-representing certain race conditions or authorization patterns) propagate into the benchmark; the human validation in Appendix F is the main mitigation. The threat model explicitly excludes browser/webpage-level injections, which are arguably the highest-impact real-world channel. The six payload wrappers are a useful taxonomy but not exhaustive; adaptive attacks against alignment-trained agents are not studied. Finally, “attack success” is defined by check-function firing on the final snapshot, which may miss partial hijacks that damage intermediate state without changing the terminal condition.
Why this matters
ToolHazard turns adversarial environment construction from a bespoke engineering task into a compute-scalable synthesis pipeline with deterministic, programmatic evaluation. Its finding that injection timing dominates payload wording — earliest-reachable slots yield the highest ASR — has direct implications for defenses: early-step input validation and plan-locking are likely higher-leverage than late-stage output filters.
Source: https://arxiv.org/abs/2608.11878
The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images
The problem
The “thinking-with-images” paradigm lets multimodal LLMs invoke visual operations — canonically crop-and-zoom — during reasoning, on the premise that actively re-attending to image regions improves fine-grained perception. In practice, these agentic pipelines often match or trail direct inference while consuming far more tokens, and exhibit pathological behaviors like repeatedly cropping irrelevant regions or failing on questions the base model answers correctly. The paper asks a sharper question than “does tool-use help?”: does the returned visual evidence causally drive the answer, or is the tool call merely a syntactic scaffold that the language prior exploits?
Causal formulation
The authors formalize a visual tool-use trajectory as a structural causal graph. At step i, a policy \pi samples an action
T_i \sim \pi(\cdot \mid I, Q, T_{<i}, O_{<i}),
a visual engine E_{\text{tool}} produces observation O_i (a cropped sub-image), and after n steps the model emits Y \mid (I, Q, T_{1:n}, O_{1:n}). The graph separates two paths to Y: the observation-mediated path T \to O \to Y (genuine perceptual contribution) and an action-induced shortcut T \to Y that reflects how merely emitting a tool call reshapes the language context (chain-of-thought, region names, coordinates) without any pixels being consulted.
They audit this graph at three levels of intervention:
- Policy level: compare full tool-use against direct inference, toggling the entire T \to O \to Y subgraph on/off.
- Trajectory level: during rollout, replace each real O_i with a corrupted \tilde O_i. Because the policy sees corrupted feedback, it may branch into a new action trace \tilde T_{1:\tilde n}, capturing the closed-loop effect.
- Step level: under a fixed prefix (T_{<i}, O_{<i}), counterfactually swap only O_i to isolate its marginal contribution.
The step-level estimand, Visual Evidence Gain (\mathrm{VEG}_i), measures the change in the model’s probability of the correct answer attributable to observation O_i alone, holding the prefix and downstream generation controls fixed. This is the key quantity: a nonzero T\to Y shortcut can inflate policy-level comparisons, but \mathrm{VEG}_i neutralizes it by construction.
What the audit finds
Across six MLLMs and five fine-grained perception benchmarks, the observation-mediated path is largely inert. The authors organize the pathology as policy miscalibration with two failure modes:
Mode 1 — Calling Without Looking (CWL). The tool fires, but O_i does not move the answer distribution. Two sub-cases:
- Saturated prior: the policy calls when the pre-call correctness probability g_{i-1} already exceeds \tau_{\text{sat}} = 0.95, so \mathrm{VEG}_i is mechanically bounded near zero. This dominates Qwen3-VL-8B.
- Structurally inactive call: no call along the trajectory carries evidence beyond the shortcut, so the entire tool sequence contributes via T \to Y only. DeepEyes exemplifies this.
The scatter of g_{i-1} vs. \mathrm{VEG}_i on V^* makes the saturation pattern explicit: a large mass of calls sits to the right of the \tau_{\text{sat}}=0.95 line with \mathrm{VEG}_i \approx 0, and correct trajectories (C) accumulate near the top-left quadrant where g_{i-1} is low and evidence actually pays off.
Mode 2 — Looking Without Planning. The complementary failure: observations do carry evidence, but the policy either continues calling past the point of sufficiency or terminates before extracting the available signal. Here T \to O \to Y is nonzero but the stopping/committing rule is uncorrelated with evidence accumulation.
Taken together, tool-use behaves as a “syntactic ritual”: the act of emitting a crop action changes the token stream (and thus Y via the shortcut) more reliably than the returned pixels change Y via the observation path. This explains the empirical puzzle — marginal or negative gains at higher token cost — without invoking training-data artifacts.
Limitations and open questions
The framework is currently instantiated for crop-and-zoom; extending \mathrm{VEG} to richer visual operators (draw, segment, OCR, external retrieval) requires care in defining the counterfactual observation distribution used to swap O_i. The step-level estimand also depends on the choice of corruption for \tilde O_i; different corruptions bound the effect differently, and the paper’s saturation threshold \tau_{\text{sat}}=0.95 is a heuristic that couples the diagnosis to base-model calibration. Finally, the audit is descriptive: it cleanly identifies miscalibration but does not yet prescribe a training objective that would drive \mathrm{VEG}_i up — e.g., an RL reward tied to counterfactual evidence gain rather than final-answer correctness, which would directly penalize both CWL and looking-without-planning behaviors.
Why this matters
Benchmark gains from “agentic” visual tool-use conflate two very different mechanisms: genuine perceptual re-attention and a language-side shortcut induced by emitting tool tokens. Visual Evidence Gain gives a principled, per-step causal decomposition that any thinking-with-images system can be audited against, and it reframes the training target from “did the model call a tool and answer correctly?” to “did the returned pixels move the answer?”
Source: https://arxiv.org/abs/2608.06270
Spark-to-Paper: End-to-End Research Paper Generation as a Composable Skill
Problem
Automating the pipeline from a research idea to a submission-ready manuscript is not a text-generation problem. It requires literature retrieval with verifiable citations, experiment design and execution, revision of claims under measured evidence, figure production in editable vector formats, and consistency across sections written over long horizons. Existing “AI scientist” systems typically bundle these into monolithic agents with prescribed graphs, which forces every subtask into the same interaction pattern and couples pipeline logic to a specific orchestration platform. Spark-to-Paper reframes the pipeline as thirteen composable skills executed inside a general-purpose coding assistant (Claude Code in the reference implementation), avoiding a dedicated agent runtime.
Method
A skill is a declarative unit specifying what a research task should accomplish, its constraints, admissible tools, and required output artifacts — but not the internal reasoning trajectory. The coding assistant reads shared project files, decides how to complete the skill, and writes new artifacts back. A skill’s execution can therefore be short (e.g., an argument reorganization) or extensive (adding citations after a failed consistency check) using the same declarative interface. A lightweight ts-paper orchestrator sequences skills into Stage 0 (input routing), Stages 1–7 (core paper generation), and a conditional Stage 8 (experiment execution and manuscript reconciliation).
The core design principle is a strict separation of model judgment and deterministic operation. The LLM handles context-dependent decisions — argument organization, literature relevance, whether evidence supports a claim — while deterministic scripts handle checkable operations: template structure validation, citation cross-referencing, LaTeX compilation, plotting from measured metrics, and file integrity. A stage only proceeds after its deterministic gates pass.
Experiment planning as preregistration. Spark-to-Paper decouples experiment planning from reporting. During planning, the datasets, baselines, metrics, ablations, and result-table schemas are fixed; numerical cells stay empty until execution. The experiment stage then maps manuscript claims to required evidence and executes only the minimal set of runs that fills the gaps rather than expanding a generic experimental template. Each numeric result must be traceable to dataset, model configuration, seed, metric, and source output file before it is admitted into the manuscript.
Post-execution, each claim is classified as supported, partially-supported, unsupported, contradicted, or needs-confirmation, and is retained, weakened, removed, moved to limitations, or triggers additional experiments. Null and negative results are preserved. Two operating modes are enforced by deterministic gates: Proposal Mode, where unobserved results must remain unspecified, and Data-Aware Mode, where every quantitative claim must resolve to a data source. This gate blocks the common failure of hallucinated numerical tables.
Self-Refutation Loop bound. Because self-critique can recursively invalidate the original research objective, experiment–critique–revision cycles are capped at seven; a trajectory that fails to converge produces a failure report rather than a manuscript. This is complemented by Self-Review and Adversarial Review skills that challenge semantic decisions surviving the deterministic gates.
Role-aware figures. Figures are handled by two paths depending on function.
Experimental-result figures are produced directly from measured metric files via plotting programs to native vector PDF, keeping quantitative figures grounded in logs. Method and explanatory figures use an image-generation model to produce a raster visual target, which is then reconstructed as HTML using editable text, shapes, and connectors. The system iteratively renders the HTML and compares against the raster, adjusting layout, geometry, and text placement until discrepancy is acceptable, then exports to PDF. If reconstruction is unreliable, the pipeline falls back to the raster rather than shipping a broken vector figure. The result is that all textual and geometric elements in method figures remain editable and vector-based.
Evaluation
Evaluation spans six dimensions — five for artifact quality and one for generation cost — using controlled runs on eight externally selected topics (three shared with a single-pass baseline for paired comparison), retrospective analysis of prior systems’ released outputs, and case studies. To avoid the “evaluating with the optimizer” pathology, citation validity is checked post-hoc against external bibliographic services rather than the in-pipeline citation gate. The evaluation protocol, including topic list and criteria, is registered with an external timestamp before generation. Cost comparisons against prior systems use only values reported in those systems’ own papers; unavailable numbers are marked as such rather than imputed.
The included case study shows two demo papers generated from a single short proposal, with mismatched claims flagged; the paper highlights specific incorrect expectations that the revision stage corrected via measured evidence.
Limitations and open questions
Claim-level evidence diagnosis (supported/contradicted/etc.) is still performed by the LLM and only recorded in structured reports — this is the largest remaining trust boundary. The seven-cycle experiment cap is heuristic; nothing prevents legitimate research trajectories from requiring more iterations, and the failure-report outcome is not compared against human-in-the-loop recovery. The HTML-reconstruction path for method figures is only shown to succeed “in a small number of correction rounds” without quantitative reconstruction-fidelity numbers or fallback-rate statistics. The evaluation reports outcomes on eight topics with uncertainty across topics, but the abstract’s specific numerical claims (quality margins, cost) are not surfaced in the sections provided here.
Why this matters
The interesting contribution is architectural, not modeling: it shows that a general coding assistant with file I/O and tool use is sufficient substrate for long-horizon research automation if you separate model judgment from deterministic gates and preregister the evidence schema before running experiments. That factoring — declarative skills over shared artifacts, hard gates for verifiable properties, bounded self-critique for semantic drift — is a reusable template for other long-horizon LLM pipelines where hallucinated numerical outputs are the primary failure mode.
Source: https://arxiv.org/abs/2608.11924
AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses
Problem
Distillation transfers capability from strong to weak models by updating the weak model’s parameters (teacher forcing, on-policy distillation, RLAIF). This paper asks whether the same transfer can happen entirely at inference time, with no gradient updates to the target. The setup: a strong “builder” model writes an inference-time harness (prompts, routers, deterministic code, format enforcers, verifiers) that wraps a fixed weaker “target” model. The question is how much of the capability gap is actually a scaffolding gap.
Method
Let M_{\text{build}} be the strong builder and M_{\text{tar}} the fixed weak target. For each benchmark \mathcal{D}^{(j)} the authors carve off a 5% validation slice \mathcal{V}^{(j)} and hide the remainder \mathcal{T}^{(j)}. The builder is placed inside an agentic coding platform (Cursor, Claude Code, or GPT Codex) with initial workspace \mathcal{W}_0 = \{\mathcal{R}, \mathcal{C}_{\text{demo}}, \mathcal{V}\} — a rule file, a demo of how to call M_{\text{tar}}, and the labeled validation set.
The builder loop at iteration k:
- Propose/revise a scaffold S_k \leftarrow M_{\text{build}}(\mathcal{W}_k).
- Run it on validation: \hat{Y}^{\mathcal{V}}_k \leftarrow S_k(M_{\text{tar}}, \mathcal{V}), accuracy a_k.
- Collect the error set \mathcal{E}_k = \{(x, y, \hat{y}) \in \mathcal{V} : \hat{y} \neq y\}.
- Update workspace \mathcal{W}_{k+1} = \mathcal{W}_k \cup \{S_k, a_k, \mathcal{E}_k\}.
When the builder submits, its final scaffold \hat{S} is exported as an executable entry point f_{\hat{S}}(x; M_{\text{tar}}) and evaluated on the hidden \mathcal{T}. Crucially, the scaffold architecture is unconstrained: prompt templates, benchmark routers, deterministic pre/post-processing, symbolic solvers, format enforcers, and ensembles are all allowed.
Benchmarks: four Theory-of-Mind datasets aggregated to 3900 items — BigToM (1200, binary belief/goal/action), Hi-ToM (1200, nested recursion 0–4 with deception), MMToM-QA (600, Bayesian goal/belief inference), and MuMA-Tom (900, 3-choice multi-agent). Primary metric: unweighted macro-average accuracy across the four benchmarks. Targets: GPT-5.4-mini (main) and Gemini-3.5-flash (contrast). Builders span Opus-4.7 (four reasoning effort levels), Sonnet-4.6, GPT-5.5, GPT-5.4-mini, Codex-5.3, Gemini-3.1-Pro, Gemini-3.5-flash, and Grok-0.1, each replicated 3 times per platform — 72 runs total.
Reference points on GPT-5.4-mini: vanilla direct-call at 0.488, human-designed UserHarness at 0.939, and vanilla GPT-5.4 (a full tier up) at 0.619.
Results
Across all 57 scaffolded GPT-5.4-mini runs, mean macro-accuracy is 0.763 (+0.275 over vanilla), with 100\% of runs beating the no-scaffold baseline. The best single scaffold (GPT-5.5 builder inside GPT Codex) hits 0.912, an 86.7\% relative uplift, closing most of the gap to the human-engineered UserHarness at 0.939 and comfortably surpassing vanilla GPT-5.4 (0.619).
Builder ranking on GPT-5.4-mini:
| Builder | BigToM | Hi-ToM | MMToM | MuMA | Avg |
|---|---|---|---|---|---|
| GPT-5.5 | 1.000 | 0.803 | 0.842 | 0.857 | 0.875 |
| Opus-4.7 (x-high) | 0.970 | 0.791 | 0.788 | 0.876 | 0.856 |
| Gemini-3.5-flash | 0.986 | 0.712 | 0.778 | 0.777 | 0.813 |
| Sonnet-4.6 | 0.977 | 0.712 | 0.742 | 0.810 | 0.810 |
| Opus-4.7 (high/med/low) | — | — | — | — | 0.807 / 0.793 / 0.711 |
| Gemini-3.1-Pro | 0.910 | 0.732 | 0.618 | 0.593 | 0.713 |
| GPT-5.4-mini (self) | 0.981 | 0.649 | 0.619 | 0.474 | 0.681 |
| Codex-5.3 | 0.983 | 0.625 | 0.563 | 0.528 | 0.675 |
| Grok-0.1 | 0.613 | 0.592 | 0.537 | 0.511 | 0.563 |
Several patterns hold:
- Builder dominates platform. Ordering by builder is stable across Cursor/Claude Code/GPT Codex; mean within-cell std is 0.036, an order of magnitude below the mean uplift.
- Reasoning effort matters within a builder. For Opus-4.7 the Spearman correlation between reasoning effort and scaffold quality is \rho = 0.77, monotone from low (0.711) through x-high (0.856).
- Validation-efficient. Median 5 validation evaluations; mean validation–test gap 0.021; correlation between probe count and accuracy only r = 0.17 — no overfitting to the 5% slice, no benefit from more probing.
- The mechanism is cognitive offloading, not more inference-time reasoning. Scaffold accuracy correlates strongly (r = 0.72) with the fraction of items answered by deterministic code/rules rather than by the target model. Gains come from routing per-benchmark, extracting structured predicates, applying polarity logic, forcing answer formats, and greedy decoding — not from asking the target to think harder or sample more.
- Self-scaffolding is weak. GPT-5.4-mini scaffolding itself only reaches 0.681; builder capability matters even when the target is fixed, since writing correct deterministic solvers requires stronger reasoning than executing them.
- Headroom matters. On Gemini-3.5-flash (vanilla 0.761), UserHarness only lifts to 0.941; scaffolding a stronger target has less to fix and can even disturb correct behaviors.
Residual errors concentrate on deep recursive belief tracking under deception (Hi-ToM’s higher orders) and Bayesian goal inference in MMToM — precisely the parts that resist reduction to explicit decision procedures. Top scaffolds still repair \approx 83\% of baseline errors.
Limitations and open questions
The study is confined to ToM benchmarks whose structure is unusually amenable to symbolic extraction (belief predicates, observation traces, room graphs). It is unclear how much of the +0.275 uplift generalizes to domains where deterministic offload is not available (open-ended math proofs, code repair on unseen APIs, long-horizon agentic work). Because the builder has direct access to \mathcal{V} with labels, this is closer to few-shot program induction than to pure transfer; the 5% slice is small but non-trivial. The near-parity with a human-designed harness (0.912 vs 0.939) also suggests a ceiling: automated harnessing recovers what humans encoded, but does not exceed it. Finally, the framing entangles “target capability” with “target adherence to a controller” — a stronger target with more brittle format compliance can be scaffolded worse than a weaker but more obedient one.
Why this matters
If test-time harnessing recovers most of the strong-model gap on structured tasks without any weight updates, then benchmark leaderboards for small models are effectively measuring an unharnessed lower bound, and the deployment-relevant question is how much of a task’s competence is “compilable” into a scaffold that a cheap target can execute. This reframes capability elicitation as a one-time reasoning expenditure by a strong builder, amortized across all subsequent inferences by a weak target.
Source: https://arxiv.org/abs/2608.12307
Self-Geometry: GT-Free and Plug-and-Play Test-Time Adaptation for Geometrically Consistent 3D Vision Foundation Models
Problem
Feed-forward 3D vision foundation models (VFMs) like VGGT, \pi^3, and Depth Anything 3 (DA3) regress camera pose, depth, and pointmaps in one forward pass. They are trained by regressing each output against GT annotations independently, without any explicit multi-view geometric constraint (bundle adjustment is prohibitively expensive at pretraining scale). Consequently, the predicted (pose, depth, pointmap) triple is not guaranteed to satisfy the reprojection/epipolar relations that must hold for a physically consistent scene. Prior test-time adaptation (TTA) approaches such as Free-Geometry rely on implicit self-consistency between model-derived quantities (pointmaps, features), which gives negligible correction when the base VFM is already inaccurate.
Method
Self-Geometry replaces implicit signals with explicit multi-view constraints, using 2D pixel correspondences from LightGlue as pseudo-GT. The pipeline has three components: Geometric Disentanglement Optimization (GDO), Frame Angular-Neighbor (FAN) view sampling, and Lightweight LoRA-based TTA.
Two primary losses. For a correspondence pair (\tilde{\mathbf{x}}_i, \tilde{\mathbf{x}}_j) between target view i and source view j:
- The MVC loss (\mathcal{L}_{\mathrm{mvc}}) is a point-to-point reprojection residual: unproject \tilde{\mathbf{x}}_j using predicted depth \mathbf{D}_j and pose, reproject to view i, and penalize distance to \tilde{\mathbf{x}}_i. This jointly supervises pose and depth but suffers from the classical pose-depth ambiguity (different (pose, depth) pairs yield the same reprojection).
- The EC loss (\mathcal{L}_{\mathrm{ec}}) is a depth-independent point-to-line residual using the Sampson distance to the epipolar line \boldsymbol{\ell}_i = \mathbf{F}_{i \leftarrow j} \tilde{\mathbf{x}}_j, with \mathbf{F}_{i \leftarrow j} = \mathbf{K}_i^{-\top}[\mathbf{t}_{i\leftarrow j}]_\times \mathbf{R}_{i\leftarrow j}\mathbf{K}_j^{-1}. This supervises only camera pose and resolves the ambiguity in \mathcal{L}_{\mathrm{mvc}}.
Gradient Disentanglement (GD). Both losses act on shared pose parameters, and their gradients conflict (obtuse angle) in 42.4% of TTA iterations on ETH3D. GD projects out the conflicting component from \nabla\mathcal{L}_{\mathrm{ec}} against \nabla\mathcal{L}_{\mathrm{mvc}} (the “ec-disentangled” direction is chosen empirically). The ablation confirms directionality matters: disentangling only \nabla\mathcal{L}_{\mathrm{ec}} yields AUC@3 = 0.27 and geometry F1 (w/o p.) = 0.60, beating \nabla\mathcal{L}_{\mathrm{mvc}}-disentanglement (0.25 / 0.52) and bidirectional disentanglement (0.26 / 0.59).
Pseudo-correspondence filtering. Raw LightGlue matches have precision 0.39. Two-stage filtering by \mathcal{L}_{\mathrm{ec}} then \mathcal{L}_{\mathrm{mvc}} residuals raises precision to 0.63 (recall 0.88), and downstream AUC@30 jumps from 0.37 (raw) to 0.83.
FAN and Lightweight TTA. Views are sampled by SO(3) geodesic distance so supervision is scale-invariant. Only LoRA adapters (rank 64, \alpha=64, inserted into QKV of every attention block; encoder-only for \pi^3) are updated. Auxiliary regularizers \mathcal{L}_{\mathrm{pc}} (photometric, SSIM+L1 with \alpha=0.85), \mathcal{L}_{\mathrm{eds}} (edge-aware depth smoothness), and \mathcal{L}_{\mathrm{bdc}} (baseline depth anchor on top-50% confidence pixels) prevent depth drift. All reprojection residuals are Huber-robustified with \delta = 1.345 \cdot 1.4826 \cdot \mathrm{median}(|r|). Optimization runs 50 AdamW iterations at lr 5\times 10^{-5} with cosine schedule.
Results
Across six VFMs (VGGT, \pi^3, DA3-G/L/B/S) and four benchmarks (7Scenes, ETH3D, ScanNet++, HiRoom):
- VGGT pose: mean AUC@3 rises from 0.38 → 0.39 (+3.3%), with a striking +37.3% on ETH3D (0.20 → 0.27). Free-Geometry manages only +2.1% mean. TCO degrades VGGT (-15.2% mean).
- \pi^3: mean AUC@3 goes 0.45 → 0.48 (+8.3%), AUC@30 0.90 → 0.92. TCO collapses \pi^3 (mean AUC@3 -91.5%), showing brittleness of prior TTA.
- DA3-Giant (already strong): mean AUC@3 0.60 → 0.61 (+1.5%); on the harder HiRoom subset +3.5%. Free-Geometry is marginally better on this well-calibrated model (+2.4%).
- On weaker DA3-B, Self-Geometry improves ETH3D AUC@3 by +9.8%.
Per-scene adaptation completes in under two minutes on a single RTX PRO 6000 for up to 40 input views with DA3-Giant on ETH3D.
Limitations
Supervision quality is bounded by the external matcher: repetitive textures, textureless walls, and wide-baseline pairs with low overlap starve the loss. On DA3-Giant — where the baseline is already near-saturated — Self-Geometry occasionally underperforms Free-Geometry (e.g., ScanNet++ AUC@3 -1.0% vs. +0.2%), suggesting the explicit constraints can slightly perturb already-consistent predictions. Latency (~2 min/scene) rules out real-time use. The method also assumes intrinsics are recoverable to compute \mathbf{F} or that the VFM predicts them jointly.
Why this matters
This is a clean demonstration that TTA for 3D VFMs benefits far more from explicit multi-view constraints (reprojection + epipolar) supervised by cheap external correspondences than from output-space self-consistency. The gradient-conflict analysis between reprojection and epipolar residuals is a reusable lesson for any pipeline that jointly optimizes pose and depth against reprojection losses.
Source: https://arxiv.org/abs/2608.10708
Hacker News Signals
Tracking down the 16-year-old WAL-reset SQLite bug
Source: https://tailscale.com/blog/sqlite-wal-reset-bug
Tailscale engineers traced a rare database corruption issue to a subtle interaction between SQLite’s WAL (Write-Ahead Log) mode and how the WAL header gets reset. The bug, present in SQLite for roughly 16 years, surfaces when a WAL file is truncated or reset while another reader holds a shared lock but has not yet read the WAL index.
The mechanical issue: SQLite’s WAL mode uses a shared-memory region (the WAL-index, typically a .shm file) to coordinate readers and writers. When the WAL is reset after a full checkpoint, the writer zeroes out WAL-index header fields and rewrites them. If a reader snapshots the index header between the zeroing and the rewrite — a TOCTOU window — it can observe a partially inconsistent header. Specifically, the mxFrame field (maximum valid frame count) and the salt values can be read in a torn state, causing the reader to either miss committed frames or attempt to apply frames from a previous WAL generation that share the same file offset.
The reproduction required multi-process SQLite access (not multi-threaded with a single handle), WAL mode enabled, and precise scheduling to hit the narrow write window. Tailscale’s workload — multiple Go processes sharing a SQLite database for coordination state — made this race reachable in production at scale, even though the probability per operation is extremely low.
The fix involves ensuring the WAL-index header is written with sufficient memory barriers and that readers re-validate the header salt after acquiring their read lock, which SQLite’s existing “read the header twice and compare” logic is supposed to handle — but a missing barrier meant the compiler or CPU could reorder reads such that both “reads” saw the same cached value.
This is a good case study in why lock-free shared-memory protocols require explicit memory ordering semantics even on x86, and why SQLite’s reputation for reliability doesn’t preclude decade-scale latent concurrency bugs in edge-case access patterns.
Jolt: Clojure compiler implemented with Chez Scheme
Source: https://jolt-lang.github.io
Jolt is an ahead-of-time compiler for Clojure (or a Clojure-compatible dialect) built on top of Chez Scheme. The core idea is to use Chez Scheme’s mature native-code compiler as a backend, translating Clojure semantics into Scheme’s object model rather than targeting the JVM.
The technical stakes: Clojure-on-JVM inherits JVM startup latency, GC pause characteristics, and interop friction for systems-level work. Chez Scheme offers a highly optimizing native compiler with a generational GC, first-class continuations, and a small runtime — making it a plausible substrate for a faster-starting Clojure.
The compiler pipeline translates Clojure’s persistent data structures (PersistentVector, PersistentHashMap via HAMT) and its concurrency primitives (atoms, refs) into Scheme representations. Clojure’s macro system is bootstrapped by running the macro expander itself in Chez Scheme, which means the compilation model is: read Clojure source, expand macros in Scheme, emit Scheme AST, invoke Chez’s compile-program. Interop with Scheme libraries is therefore direct, but Java interop is absent by design.
Key open problems include: Clojure’s deftype/defrecord protocol dispatch, which maps to Scheme’s SRFI-9 or custom vtable structures; lazy sequences, which translate naturally to Scheme’s delay/force or custom thunks; and the namespace system, which requires a separate module-resolution layer on top of Chez’s library system.
The project is early — not all of core Clojure is implemented — but the approach is technically sound. Chez Scheme’s register allocator and inliner are well-regarded, and prior work (e.g., Racket’s move to Chez Scheme as a backend for its own compiler) validates the substrate. For anyone interested in language implementation, the codebase is a clean example of using a high-quality existing compiler as a portable backend rather than writing one from scratch.
Emergent Introspective Awareness in Large Language Models
Source: https://arxiv.org/abs/2601.01828
This paper probes whether LLMs exhibit what the authors call “introspective awareness” — loosely, the capacity for a model to have accurate beliefs about its own internal states or processing. The framing is influenced by philosophy of mind terminology, but the empirical claims are more tractable.
The methodology involves constructing prompts designed to elicit self-reports about uncertainty, knowledge gaps, or reasoning steps, then comparing these reports against ground-truth behavioral signals (e.g., does the model claim confidence on questions it consistently answers incorrectly?). The authors test several LLMs across factual recall, multi-step reasoning, and counterfactual tasks.
The quantitative findings show that larger models are better calibrated in their self-reports — their expressed uncertainty correlates more strongly with actual error rates — but this correlation remains weak in absolute terms. Even top models show systematic overconfidence on domains where they have surface-level familiarity. The paper argues this is “emergent” in the sense that smaller models show near-zero calibration on self-reports while larger ones show a statistically significant but modest positive correlation.
The interpretive claim — that this constitutes genuine introspection — is philosophically overloaded and the paper’s framing somewhat overstates what the measurements show. What is actually measured is behavioral calibration of verbalized confidence, not access to internal states. The model has no privileged access to its own weights or activations; it is predicting tokens about itself using the same mechanism it uses for everything else. Whether that constitutes “introspection” depends entirely on how you define the term, and the paper does not settle this.
More concretely useful: the finding that self-reported confidence is a weakly predictive but non-trivial signal for downstream error detection, which has practical implications for building LLM pipelines that use the model’s own uncertainty estimates as a routing or abstention signal.
Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot
Source: https://knownagents.com/insights
The report documents a pattern of HTTP traffic that presents User-Agent strings matching known AI crawler identifiers (ClaudeBot, GPTBot, anthropic-ai, etc.) while exhibiting behavior characteristic of vulnerability scanners rather than legitimate web crawlers: hitting known CVE-specific paths (e.g., /wp-admin/, /.env, /actuator/health), probing non-public API endpoints, and sending payloads associated with injection testing.
The technical mechanism is straightforward: nothing enforces that a given User-Agent string corresponds to the declared origin. Legitimate AI crawlers are generally allowed through by site owners who want their content indexed; spoofing these strings gives a scanner a higher probability of bypassing naive bot-detection rules that whitelist known crawlers.
Several indicators distinguish the malicious traffic: IP ranges do not overlap with published ASN blocks for Anthropic or OpenAI infrastructure; request patterns show sequential path enumeration rather than link-following; and TLS fingerprints (JA3/JA4) differ from what genuine browser-based or curl-based crawlers produce, suggesting custom tooling.
The mitigation landscape is thin. IP-allowlisting based on published crawler IP ranges is the most robust defense but requires maintaining fresh lists. JA3/JA4 fingerprint filtering can help but is bypassable with effort. The deeper problem is that the industry convention of “allow known AI crawlers” creates an exploitable trust signal.
This is a straightforward abuse-of-convention attack. It does not require any novel technique — User-Agent spoofing is as old as HTTP — but the emergence of AI crawler whitelisting as a broadly applied rule has created a new effective bypass vector. The operational security implication for server operators: do not conflate User-Agent whitelisting with actual access control for any sensitive paths.
Faster floating point math with Rust’s new API
Source: https://pythonspeed.com/articles/faster-float-math-rust/
This article covers Rust’s stabilization of f32::midpoint and related numerically sound floating-point utilities, and more substantively, the std::intrinsics::fadd_fast family and the newer safe wrappers (f32::add with explicit rounding modes) that allow opting into IEEE-relaxed semantics without unsafe.
The core issue: by default Rust (like C with -O2 but without -ffast-math) preserves strict IEEE 754 semantics. This means the compiler cannot reassociate (a + b) + c to a + (b + c), cannot contract a * b + c into an FMA, and cannot assume x - x == 0. These restrictions prevent auto-vectorization in many loops because SIMD lanes would change reduction order.
The new API surface includes f32::midpoint(a, b) which computes (a + b) / 2 without overflow using the identity a + (b - a) / 2, avoiding the naive midpoint’s overflow bug. More impactful for performance is the explicit availability of f32::mul_add (fused multiply-add, mapping to a single VFMADD on x86 AVX) and the ongoing stabilization of per-operation “fast” flags that let the programmer annotate specific operations as reassociation-safe without blanket unsafe blocks.
Benchmark results in the article show 2-4x throughput improvement on reduction loops over float arrays when the compiler can vectorize with relaxed associativity, consistent with what you would expect from 256-bit AVX2 SIMD (8 floats per lane) on a previously scalar loop.
The limitation is that Rust’s approach remains more granular than GCC/Clang’s -ffast-math, which is a single flag. Per-operation annotation scales poorly when an entire numerical kernel needs relaxed semantics. The #[allow(clippy::float_arithmetic)] ecosystem and crates like fast-floats have been workarounds, but a module-level or function-level relaxed-semantics attribute remains absent from stable Rust.
Qwen3.8-2.4T
Source: https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B
Qwen3.8-2.4T is the largest model in Alibaba’s Qwen3 release: a Mixture-of-Experts architecture with 2.4 trillion total parameters and 95 billion active parameters per forward pass (hence “A95B”). The model uses the standard top-k token routing over expert FFN layers, with each transformer block containing a shared attention head and a routed FFN where only a subset of experts is activated per token.
The 2.4T/95B ratio implies roughly 25 experts per layer with k=1 or a similar sparse configuration, consistent with the scaling law motivation that MoE allows total parameter count — and thus memorization capacity and knowledge coverage — to grow faster than compute per token. Training compute is dominated by the active parameter count, not total, so 95B-active at inference is roughly equivalent in FLOPs-per-token to a dense ~95B model.
The publicly stated benchmark results are strong across coding (HumanEval, LiveCodeBench), math (AIME, MATH-500), and instruction following, with the model reportedly competitive with or exceeding GPT-4o and Claude Sonnet on several benchmarks at time of release. The model supports a “thinking” mode (extended chain-of-thought with budget tokens) and a standard mode, toggled via a system prompt.
Practical deployment is non-trivial: 2.4T parameters in bf16 requires approximately 4.8 TB of GPU memory, necessitating multi-node tensor parallelism or aggressive quantization (INT4 reduces to ~1.2 TB, still requiring a large cluster). The Hugging Face model card notes the model is available in both full precision and quantized variants. For most practitioners, the smaller Qwen3 variants (30B-A3B, 235B-A22B) are the relevant operational choices; the 2.4T exists as a capability ceiling benchmark.
Go is an ideal language for AI-assisted software engineering
Source: https://developers.googleblog.com/why-go-is-an-ideal-language-for-ai-assisted-software-engineering/
Google’s Go team argues that Go’s design properties make it particularly tractable for LLM-based code generation and editing. The technical arguments are substantive, though the piece is from an interested party.
The core claim rests on several properties. First, Go’s grammar is small and regular (25 keywords, no operator overloading, no implicit conversions), which reduces the surface area an LLM must model for syntactically and semantically valid generation. Second, gofmt enforces a single canonical formatting, eliminating style variation in training data and generated output — the model sees one representation per program structure. Third, Go’s type system is explicit but not as elaborate as Rust’s or Haskell’s, so type-correct generation is easier to achieve and verify. Fourth, the standard library and import graph are explicit and self-contained, with no header files or build-system complexity that requires out-of-file context.
The more interesting technical point concerns refactoring: Go’s toolchain exposes a well-defined AST and type-information API (go/ast, go/types, golang.org/x/tools/go/analysis), making it straightforward to write deterministic post-processing passes that correct or verify LLM-generated code. An LLM output that is syntactically valid Go can be immediately parsed, type-checked, and subjected to static analysis in a tight loop, enabling generation-verification pipelines that are harder to build for languages with complex build systems.
The limitations of the argument: LLMs already perform well on Python and TypeScript despite irregular syntax, suggesting that training data volume dominates language regularity as a factor in generation quality. The Go-specific tooling benefits are real but mostly relevant for tool-assisted workflows, not raw generation quality. The post reads partly as Go advocacy, but the underlying points about toolchain introspectability for agentic coding loops are technically sound.
Launch HN: Discovered Materials (YC P26) – AI agents to discover new materials
Source: https://discoveredmaterials.com/research/
Discovered Materials is applying autonomous AI agents to computational materials discovery, targeting the pipeline from hypothesis generation through DFT (Density Functional Theory) calculation, structure prediction, and synthesis feasibility assessment.
The technical workflow follows the established computational materials science stack: structure generation (via methods like random search, evolutionary algorithms, or generative models over crystal structure space), followed by DFT relaxation using codes like VASP or Quantum ESPRESSO to obtain ground-state energetics, with stability assessed against the convex hull of competing phases. The agentic layer orchestrates these computationally expensive steps, decides which candidate structures to prioritize, and interprets results to propose next experiments — the standard active learning loop applied to materials.
The ML components likely include graph neural network interatomic potentials (e.g., MACE, CHGNet, or similar) for cheap pre-screening before expensive DFT, and possibly generative models (diffusion over crystal structures, as in DiffCSP or CDVAE) for structure proposal. The HN comments ask about benchmark targets; the team mentions battery electrolytes and catalysts as initial domains, which are high-value targets with well-defined computational fitness proxies (ionic conductivity, adsorption energies).
The main technical challenge is the gap between computational prediction and experimental synthesis: DFT stability on the convex hull does not guarantee synthesizability, and the agent must reason about synthesis routes, precursor availability, and thermodynamic accessibility under realistic conditions. Closing this loop requires either experimental feedback (expensive) or synthesis-feasibility models trained on experimental databases (still an open research problem).
The broader space (Google DeepMind’s GNoME, Microsoft’s MatterGen, academic groups using similar stacks) is crowded, so differentiation likely comes from workflow engineering and domain-specific fine-tuning rather than novel ML architecture.
Noteworthy New Repositories
drumih/turbo-fieldfare
Gemma 4 26B-A4B on Apple Silicon with approximately 2 GB of RAM. The key insight is that Gemma 4’s 26B-A4B variant is a Mixture-of-Experts model where only 4B parameters are active per forward pass; turbo-fieldfare exploits this sparsity to avoid materializing inactive expert weights. The runtime targets M-series unified memory via Metal, keeping only active expert shards in resident memory while streaming inactive ones from disk or keeping them paged out. No PyTorch, no Hugging Face stack — the inference path is a lean C/Metal implementation that avoids framework overhead. For practitioners who want to run a capable MoE locally without a workstation GPU or a large RAM footprint, this fills a real gap: most local inference tools load full parameter tensors regardless of sparsity. The project is early-stage but the architecture choice (MoE + aggressive expert offloading) is technically sound and directly maps to why the RAM figure is so low. Useful as a reference for anyone building sparse-model inference on consumer hardware.
Source: https://github.com/drumih/turbo-fieldfare
FareedKhan-dev/kimi-k3-in-c
A from-scratch C99 inference engine for Kimi K3, nominally a 2.78T-parameter MoE model, running on a single CPU in around 8 GB of RAM. The approach mirrors llama.c/llama2.c in spirit: no BLAS, no LAPACK, no CUDA, no framework linkage — just portable C with manual matrix kernels and aggressive quantization (likely INT4/INT8 weight packing) to compress active-parameter working sets. For an MoE of this scale the resident footprint is dominated by active experts per token, not total parameters, which makes the 8 GB figure plausible if inactive experts are memory-mapped and paged on demand. The value here is pedagogical and portability-oriented: the single-file or minimal-file C99 approach compiles on any POSIX system with a C compiler. Practitioners building embedded or air-gapped inference pipelines, or researchers wanting to audit every arithmetic operation, benefit from the lack of abstraction layers. It also serves as a stress test for how far MoE sparsity can be pushed on purely CPU-bound hardware without vendor libraries.
Source: https://github.com/FareedKhan-dev/kimi-k3-in-c
i3T4AN/KADATH
KADATH is an evolutionary multi-agent runtime that frames agent improvement as a population-based optimization loop. Each epoch, a population of autonomous agents is instantiated, evaluated against a fitness function tied to a specified goal, and then subjected to selection, mutation, and crossover operators — producing the next generation. Epochs are reproducible via seeded state, enabling ablations and comparisons across evolutionary runs. The architecture separates the agent substrate (tool use, environment interaction) from the evolutionary controller (selection pressure, genetic operators), which means the core loop is model-agnostic. This is closer to neuroevolution / quality-diversity search applied to agent behavior than to standard RLHF or prompt optimization. Relevant prior work includes OpenAI’s POET, AutoML-Zero, and various LLM-based self-improvement loops, but KADATH’s explicit epoch-reproducibility focus is a practical differentiator for research use. Open questions include how fitness functions are specified, whether the mutation operators act on prompts, weights, or tool configurations, and how it scales when agent rollouts are expensive.
Source: https://github.com/i3T4AN/KADATH
Paritok-official/paritok-4b-v1
A context-compression gateway specifically designed for agentic coding sessions. The core mechanism is a 4B parameter code-native model that rewrites agent conversation history into a denser representation, reducing token count without discarding semantically critical content. Reported compression ratios start at 25% on turn 1 and compound to over 85% in long or context-saturated sessions, with a claimed 3x increase in effective turns per context window. The system operates as a transparent proxy: agents talk to it via a drop-in BASE_URL replacement, so Claude Code, Cursor, Codex, and OpenHands require no code changes. Technically, this sits between summarization (lossy, task-agnostic) and lossless tokenization tricks — it is trained specifically on code and tool-call traces, so it preserves identifier names, function signatures, and error messages that generic summarizers drop. The 4B model size keeps inference latency low enough to not dominate turn latency. Main open question is the fidelity-compression tradeoff: how often does aggressive compression in late turns cause the downstream agent to lose critical context, and how is that measured?
Source: https://github.com/Paritok-official/paritok-4b-v1
alikon-art/DeterminFlow
A production-oriented workflow runtime for AI pipelines, positioned between lightweight chaining libraries (LangChain, simple async queues) and heavyweight orchestration platforms (Airflow, Prefect). The “determin” framing emphasizes deterministic execution semantics: workflows are defined as explicit DAGs with typed inputs/outputs, and the runtime enforces that each node’s behavior is validated before execution proceeds. Recovery is a first-class primitive — nodes declare failure modes and the runtime can checkpoint, retry, or reroute rather than requiring the entire pipeline to be re-run from scratch. This matters for long-running agentic workflows where LLM calls are expensive and partial-failure recovery can save significant cost. The runtime targets deployment as a service (HTTP or gRPC interface) rather than as an embedded library, which suits multi-team environments where workflow definitions need to be versioned and monitored independently. Key differentiators over LangGraph or similar: explicit validation pass before execution, built-in recovery policies, and a service-first deployment model. Evaluating actual reliability guarantees requires examining how the checkpoint store and failure detection are implemented.
Source: https://github.com/alikon-art/DeterminFlow
dinosn/fastjson-jsontype-rce-lab
A self-contained Docker lab environment for studying and defending against fastjson deserialization RCE chains, covering two distinct vulnerability classes. The first targets fastjson 1.x (versions 1.2.66–1.2.83) via @JSONType resource probe gadgets (tracked as CVE-2026-16723). The second demonstrates a fastjson2 2.0.57 bypass where an attacker-controlled @type reaches loadClass even with autoType explicitly disabled, by abusing polymorphic type annotations (@JSONType(seeAlso) and Jackson’s @JsonSubTypes) to smuggle a class load through the polymorphism resolution path. Payloads are marker-only (no actual weaponized shellcode), making this safe for defensive tooling development. The repo also ships a scanner to detect vulnerable patterns. The safeMode and JDK17 controls are documented as mitigations, which is useful because JDK17’s removal of certain reflection access paths breaks common JNDI/classloader gadgets. Valuable for security engineers auditing Java services that consume untrusted JSON, particularly those that upgraded from fastjson 1.x to 2.x assuming the autoType-disabled default was sufficient.
Source: https://github.com/dinosn/fastjson-jsontype-rce-lab
yc-software/qm
A multiplayer agent harness for coordinating work across humans and AI agents, built and operated by YCombinator. The technical substrate is a shared task/state environment where multiple agents (and human operators) can be assigned work items, observe each other’s progress, and hand off or escalate. “Multiplayer” here means concurrent, persistent sessions rather than turn-based single-user chat — closer to a collaborative OS process model than a chatbot. The harness manages agent lifecycle (spawn, monitor, terminate), task routing (which agent handles which subtask), and result aggregation. For engineering teams, this fits scenarios where a single coding task decomposes into parallelizable subtasks that different specialized agents (or a human reviewer) can handle concurrently. The YC provenance suggests it is battle-tested on real startup workflows. Technical details on the state synchronization mechanism, agent communication protocol, and failure handling are the critical unknowns for evaluating it as infrastructure. Accessible at https://qm.ycombinator.com.
Source: https://github.com/yc-software/qm
fuxicodex/Fuxi
A terminal-native AI coding agent with a focus on cost-aware LLM routing. Fuxi runs as a CLI process, exposing an agent loop that can edit files, execute shell commands, and invoke tools directly in the terminal environment — no IDE plugin, no browser UI. The distinguishing feature is cost-aware routing: the agent maintains a model-cost table and dynamically selects among configured LLM providers (e.g., routing simple edits to a cheaper model, complex refactors to a more capable one) based on estimated task complexity and a user-defined cost budget. This is architecturally distinct from tools that hardcode a single provider or require manual model selection. Being self-contained means the entire agent state lives in the terminal session with no external service dependency beyond the LLM APIs themselves, which matters for air-gapped or security-sensitive environments. The fast startup and minimal footprint make it composable with shell scripts and CI pipelines. Compared to Aider or Claude Code CLI, the primary differentiator is the multi-provider cost router. Implementation details on how task complexity is estimated for routing decisions are worth examining closely.