Daily AI Digest — 2026-07-02
arXiv Highlights
PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception
Problem
Standard multimodal captioning and perception benchmarks have saturated: leaderboard scores compress across models that behave very differently under real use. The authors attribute this to two failure modes of existing metrics. First, holistic semantic matching (CIDEr, BLEU, LLM-as-judge on the whole caption) rewards fluent global summaries and tolerates missing or hallucinated fine details. Second, linear averaging over sub-criteria lets a model compensate for critical factual errors by getting many easy items right. As a concrete illustration, on DOCCI several frontier models receive near-identical scores despite clearly different human-preference Elo ratings.
Method
PerceptionRubrics reframes evaluation as atomic auditing against instance-specific rubrics rather than reference-based similarity. The pipeline has three components.
Golden captions via Circular Peer-Review. For each of the 1,038 information-dense images (crowded scenes, dense documents, charts, UI screens, and other categories, see Figure 3), multiple MLLMs generate long descriptions and then critique/verify each other’s outputs in a circular consensus loop. The surviving statements form a golden caption whose length is deliberately large — mean 770.42 words, median 569 words, with a right-skewed tail exceeding 3,400 words. Long captions are necessary to anchor rubrics for images that contain hundreds of individually verifiable visual facts.
Dual-stream rubric distillation. Each golden caption is decomposed into atomic propositions and split into two disjoint pools:
- Must-Right: essential facts a competent viewer would not miss (identity of the main subject, headline text of a document, counts of salient objects). Failure here indicates fundamental perceptual breakdown.
- Easy-Wrong: fine-grained, error-prone details (occluded object attributes, small text, spatial relations among many entities) that separate strong from mediocre models.
The benchmark contains 10,718 atomic rubrics total: 4,053 Must-Right and 6,665 Easy-Wrong, averaging 10.33 rubrics per image.
Gated Scoring. Rather than a linear mean, the score is conjunctive over Must-Right and additive over Easy-Wrong. Let M_i \in \{0,1\} denote satisfaction of the i-th Must-Right rubric and E_j \in \{0,1\} for Easy-Wrong. The per-image score is
S \;=\; \mathbb{1}\!\left[\textstyle\sum_i M_i \geq \tau_M\right] \cdot \left(\alpha \cdot \frac{\sum_i M_i}{|M|} + (1-\alpha) \cdot \frac{\sum_j E_j}{|E|}\right),
where the indicator implements a sharp binary penalty: models that miss a threshold \tau_M of essential facts get zero credit regardless of how many fine details they enumerate. This directly targets the failure mode where models list many peripheral attributes while getting the main subject wrong.
Rubric verification itself is performed by an LLM judge given only the model’s candidate caption and the atomic proposition; because each rubric is a yes/no atomic claim, judge variance is far lower than for holistic caption scoring.
Results
Three findings are called out.
Reliability gap. Aggregated per-rubric accuracy (linear average) is high across frontier models, but Gated scores drop sharply — models pass many individual checks yet routinely fail the conjunctive Must-Right constraint on any given image. This quantifies brittleness that saturated benchmarks hide.
Open–closed stratification. Despite recent claims that open-source models have closed the gap on reasoning benchmarks, PerceptionRubrics exposes a persistent ~8% deficit on dense-perception tasks between the best open-source models and proprietary frontiers. Perception, not reasoning, is where the gap has stayed open.
Human alignment. Against Vision Arena Elo scores on GPT-5.4, Qwen3-VL-235B, GPT-4o, Kimi-K2.6, and MiMo-V2.5, PerceptionRubrics achieves Pearson r = 0.916 and Spearman \rho = 1.000 (perfect rank agreement on this five-model slice). DOCCI and DetailCaps produce near-degenerate rankings and substantially lower correlation. The gated mechanism, not merely the rubric decomposition, appears to drive the alignment: replacing gating with a linear average measurably degrades rank correlation in their ablation.
Limitations and open questions
The benchmark is small in image count (1,038) and evaluation relies on an LLM judge over atomic propositions, which shifts the reliability question from caption-level judging to atomic verification — cheaper but not free of bias, especially for OCR-dependent Easy-Wrong items. The Circular Peer-Review pipeline uses MLLMs to construct ground truth for MLLMs, so systematic blind spots shared across generators (e.g., certain scripts, chart types) may propagate into rubrics unchecked. The 5-model human-alignment slice inflates the Spearman number to 1.000; rank stability under a larger model set is unreported. Finally, the threshold \tau_M and weight \alpha in gated scoring are design choices whose sensitivity is not fully characterized in the excerpts.
Why this matters
Saturation on multimodal captioning benchmarks has largely been a scoring artifact: linear averages let models trade essential facts for peripheral verbosity. Gated conjunctive scoring over atomic rubrics is a simple, mechanically clean fix that recovers discrimination among frontier models and aligns with human preference (Spearman 1.000 on the tested slice), while revealing an 8% open-vs-closed perception gap that reasoning benchmarks miss.
Source: https://arxiv.org/abs/2606.28322
Multimodal Continuous Reasoning via Asymmetric Mutual Variational Learning
Problem
Multimodal LLMs typically externalize reasoning as discrete tokens (chain-of-thought in language space). For perception-heavy tasks this discretization is lossy: fine-grained spatial or attribute information gets projected onto a text vocabulary before it can be used. Continuous latent reasoning — inserting learned latent vectors \mathbf{Z} = [\mathbf{z}_1,\ldots,\mathbf{z}_k] \in \mathbb{R}^{k \times d} into the autoregressive stream — sidesteps this bottleneck, but there is no supervision on what those latents should be. Variational inference is the natural tool: treat \mathbf{Z} as unobserved and maximize
\log p_\theta(\mathbf{y}\mid\mathbf{x}) = \log \int p_\theta(\mathbf{y}\mid \mathbf{x},\mathbf{Z})\, p_\theta(\mathbf{Z}\mid\mathbf{x})\, d\mathbf{Z}.
The catch is a train–inference asymmetry. The posterior q_\phi(\mathbf{Z}\mid\mathbf{x},\mathbf{y}) used during training sees the target \mathbf{y}; the prior p_\theta(\mathbf{Z}\mid\mathbf{x}) used at inference does not. Under the standard ELBO,
\log p_\theta(\mathbf{y}\mid\mathbf{x}) \geq \mathbb{E}_{q_\phi}\!\left[\log p_\theta(\mathbf{y}\mid\mathbf{x},\mathbf{Z})\right] - D_{\mathrm{KL}}\!\left(q_\phi(\mathbf{Z}\mid\mathbf{x},\mathbf{y})\,\|\,p_\theta(\mathbf{Z}\mid\mathbf{x})\right),
the forward KL pulls the prior toward a posterior that has taken an answer-conditioned shortcut. The authors call this answer leakage: q_\phi can encode \mathbf{y} almost verbatim into \mathbf{Z}, and the prior is then trained to hallucinate that content from \mathbf{x} alone.
Method
AMVL keeps the ELBO structure but adds a reverse KL that regularizes the posterior back toward the prior, producing a bidirectional calibration objective. Concretely, the framework parameterizes both distributions from the LLM itself: k latent slots are inserted into the autoregressive sequence, and the hidden states at those positions parameterize a Gaussian prior (conditioning only on \mathbf{x}) and a Gaussian posterior (conditioning on \mathbf{x} and \mathbf{y}).
Training combines three terms: the reconstruction likelihood \mathbb{E}_{q_\phi}[\log p_\theta(\mathbf{y}\mid\mathbf{x},\mathbf{Z})], the standard forward KL D_{\mathrm{KL}}(q_\phi \| p_\theta), and a reverse KL D_{\mathrm{KL}}(p_\theta \| q_\phi). The two KLs play complementary geometric roles. The forward direction is mode-seeking — it sharpens the prior onto a mode of the posterior — while the reverse direction is mass-covering, pushing the posterior to keep support wherever the prior places mass. On a bimodal example, forward KL alone collapses the approximating distribution to one mode, while the reverse term restores support over regions the inference-time prior can actually reach.
The mechanism against leakage is straightforward: if q_\phi collapses onto an answer-conditioned region that p_\theta(\mathbf{Z}\mid\mathbf{x}) cannot access, then D_{\mathrm{KL}}(p_\theta \| q_\phi) blows up because the prior places mass where the posterior does not. Minimizing the reverse KL therefore penalizes exactly the “inference-incompatible” posterior configurations the authors formalize as prior contamination.
Implementation is on Qwen2.5-VL-7B-Instruct with k=8 latent tokens of dimension d=512. Vision encoder is frozen; the LM backbone and the variational heads are jointly trained on a mixture of Visual-CoT, ReFocus, CogCoM, and Zebra-CoT. At inference only the prior is used, so the compute overhead is k extra latent positions per query.
Results
On high-resolution perception (V*, HRBench4K, HRBench8K), AMVL-7B reaches 84.29 / 72.12 / 68.50 overall (avg. 74.97), compared to Qwen2.5-VL-7B at 76.44 / 68.00 / 63.75 (avg. 69.40) — an absolute gain of +5.57 over the base model. It beats the strongest continuous-latent baseline Monet (74.08 avg.) and discrete-reasoning baselines DeepEyes (73.21) and PixelReasoner (73.09). Notably on V* the gain is +7.85 over base and +1.04 over Monet; on HRBench8K FSP (fine-grained single-instance perception) the gain is +9.25.
Vision-centric BLINK shows a larger effect. AMVL-7B averages 66.91 vs. 56.08 for the base model (+10.83), and beats the next best (PixelReasoner at 62.70, Mull-Tokens at 62.30) by roughly 4 points. Task-level gains are heavily concentrated on the abstract/perceptual categories: IQ Test +12.67, Jigsaw +32.00, Multi-view Reasoning +14.29, Visual Similarity +5.93. Spatial Relation is unchanged (88.81), suggesting the latent reasoning helps most where language-space CoT is a poor substrate.
Token-level relevance maps indicate that the learned latents attend to task-relevant image regions rather than functioning as generic register tokens.
Limitations and open questions
The evaluation is confined to a single 7B backbone; whether the reverse-KL correction is still necessary — or sufficient — at larger scales where the posterior might rely less on shortcuts is untested. The latent budget is small and fixed (k=8); scaling behavior in k, and whether adaptive latent counts would help on longer-horizon reasoning, are open. The claim that reverse KL specifically neutralizes answer leakage rests on a formalization of “prior contamination” that would benefit from a direct causal probe (e.g., measuring mutual information I(\mathbf{Z}; \mathbf{y} \mid \mathbf{x}) under the posterior with and without the reverse term). The KL scheduling details, which are known to be brittle in latent-variable LMs, are relegated to the appendix. Finally, no comparison is given against simple non-variational baselines like BC-Z-style teacher forcing on synthetic reasoning traces.
Why this matters
Continuous latent reasoning is a plausible route around the language-token bottleneck for perception-heavy multimodal tasks, but it has been dogged by unstable training. Framing the failure mode as posterior shortcut usage — and fixing it with a mass-covering reverse KL — is a clean, theoretically grounded intervention that produces double-digit gains on abstract-visual benchmarks where discrete CoT plateaus.
Source: https://arxiv.org/abs/2607.00461
ASPIRE: Agentic /Skills Discovery for Robotics
ASPIRE (Agentic Skill Programming through Iterative Robot Exploration) is a continual-learning system that autonomously writes, debugs, and refines robot control programs under the code-as-policy paradigm, while distilling validated fixes into a growing skill library. The premise is that most current LLM-driven robot programming pipelines are one-shot: an agent generates code, it fails, and either a human intervenes or the agent retries against opaque scene-level feedback. ASPIRE turns each failure into a structured debugging problem with per-primitive multimodal evidence, and treats successful repairs as first-class, transferable knowledge.
System architecture
ASPIRE is organized as a coordinator–actor loop (Fig. 1). A central coordinator owns the shared skill library and dispatches actor coding agents (Claude Opus 4.6 with a 1M-token window in sim; Codex GPT-5.5 on the real robot) to individual tasks. Actors do not share raw trajectories or chat histories; only distilled skills are promoted to the library. Each actor’s context stays narrow: task spec, current program, and structured execution traces near the failing primitive.
Three components close the loop:
Robot execution engine. For every primitive call (perception, geometry, motion planning, control) the engine records the API, inputs, outputs, return status, and relevant multimodal artifacts — RGB keyframes just before/after the call, overlays, grasp candidates, object poses, and planner outputs. The agent never receives full video; it gets an evidence-scoped trace tied to the call chain implicated in the failure. This is a deliberate design point: too little evidence hides the failing primitive, too much raw video distracts from the causal chain.
Skill library. Validated repairs — not human-written primitives — are compiled into reusable in-context skills. Categories that emerge include localization disambiguation, motion-primitive construction, navigation recovery, object-level grasping, scene understanding, and debugging workflows.
Evolutionary search. Rather than trust single-trajectory refinement, the coordinator samples diverse candidate programs \pi_0, \ldots, \pi_k across tasks, enabling exploration of alternative strategies that a purely local repair loop would miss.
Trace-guided debugging, concretely
Figure 2 illustrates the mechanism on a BEHAVIOR-1K navigate-and-pick-up-radio task. Ego-view keyframes show the robot localizing the radio but repeatedly failing to approach it. The primitive trace attributes the failure to recurring PLANNING_ERROR returns — candidate navigation goals fall inside the table’s collision-avoidance buffer. The actor synthesizes a patch: a multi-angle approach routine that re-perceives the radio from a reachable side. The repair validates in closed loop and is promoted to the library as a “Multi-Angle Approach” skill.
Crucially, that skill is now available in-context for future navigation-with-clutter tasks and — as shown later — even for a different embodiment on real hardware.
Experimental setup and results
The coding agent, environment, and API surface are held fixed across all sim experiments. ASPIRE learns/collects skills on a held-in seed range and evaluates one generated program per task on held-out seeds. The main comparison baseline, CaP-Agent0, is stronger than the ASPIRE evaluation protocol in one dimension: it regenerates a fresh program per seed with test-time reasoning and retries. ASPIRE commits to a single program per task.
- LIBERO-Pro (10 tasks × 50 held-out seeds under perturbations): ASPIRE improves by up to 77% over prior methods on manipulation under perturbation. Skills are learned on seeds 51–65 and evaluated on seeds 1–50.
- Robosuite (100 held-out trials per task, including 2A bimanual): up to 72% improvement on bimanual handover. Learning on seeds 101–125, evaluation on 1–100.
- BEHAVIOR-1K long-horizon mobile manipulation on two household tasks: up to 32% gain, with navigation and task success reported separately. Learning on 26–35, evaluation on 1–25 with incremental block execution.
Several ASPIRE results reportedly surpass programs written by human experts on these benchmarks.
Cross-embodiment transfer
The real-robot study (§3.6) is the most direct evidence that skills are not just per-task overfits. ASPIRE selects three skills learned in Franka-based sim — soda-can pickup, bowl-on-plate placement, drawer push/pull — and injects them as in-context guidance to a Codex GPT-5.5 coding agent on a bimanual YAM manipulation station with a different embodiment and API. The metric pair is tokens-to-first-success and held-out real-robot success rate, with and without the sim-discovered skill. The engine still exposes multimodal traces on the real robot so the agent can autonomously debug without task-specific human input. The direction of the results supports the transfer claim, though the sample of three skills is modest.
Limitations and open questions
- Evaluation uses frontier closed models (Claude Opus 4.6, GPT-5.5) with 1M-token contexts; the compute and latency envelope for open-loop debugging is substantial and not characterized in the excerpt.
- The one-program-per-task evaluation is elegant, but growth of the library across many tasks raises retrieval and interference questions that are not analyzed in the shown sections.
- Skills are stored as in-context text/code; there is no discussion of scaling behavior once the library exceeds context budgets, nor of skill deprecation when APIs or embodiments change.
- Real-robot transfer is demonstrated on three curated skills; whether arbitrary sim-discovered skills transfer, or only a filtered subset, is unclear.
- Failure modes of the trace-attribution heuristic (which keyframes and overlays to expose per call) are not quantified.
Why this matters
ASPIRE is a clean instantiation of the idea that agentic robot programming should treat the execution stack as a debuggable environment rather than a black box, with a persistent library of validated repairs replacing hand-authored primitives. The reported margins (up to 77% on LIBERO-Pro, 72% on Robosuite bimanual, 32% on BEHAVIOR-1K) and the sim-to-real skill transfer suggest that structured multimodal traces plus a compounding skill store may be the missing scaffolding between code-as-policy and durable robot competence.
Source: https://arxiv.org/abs/2607.00272
The State-Prediction Separation Hypothesis
In an autoregressive Transformer, each hidden state h_i^{(l)} serves two structurally distinct purposes: it produces the logits for x_{i+1}, and it emits the keys/values that every future position j>i will read from the KV cache. The authors formalize this by decomposing the total gradient over a length-T sequence into a prediction term and a state-preparation term:
\nabla_{\theta}\mathcal{L} = \sum_{i=1}^{T-1}\Big[\underbrace{\tfrac{1}{T-1}\nabla_{\theta_i}\ell_i}_{\text{Prediction}} + \underbrace{\tfrac{1}{T-1}\sum_{j=i+1}^{T-1}\nabla_{\theta_i}\ell_j}_{\text{State}}\Big].
Both terms flow through the same h_i^{(l)}, so a single activation is optimized to simultaneously encode “what to emit now” and “what future tokens will need from me.” The hypothesis is that this conflation is suboptimal and that architecturally routing the two gradient components through separate representations improves language modeling.
Two-stream architecture
The State-Prediction Separation (SPS) Transformer interleaves a learned <predict> token \rho_i after each input:
x \rightarrow (x_1,\rho_1,x_2,\rho_2,\ldots,x_T,\rho_T),
with x_i and \rho_i sharing a position encoding. The loss is applied only at \rho_i slots (Eq. 4). Crucially, the attention mask forces asymmetric persistence:
\mathcal{A}_{\mathrm{SPS}}(i,q) = \{x_k : k \leq i\} \cup \begin{cases}\{\rho_k : i-w \leq k < i\} & q = x_i \\ \{\rho_k : i-w \leq k \leq i\} & q = \rho_i\end{cases}.
Input entries live in the persistent KV cache and carry state across arbitrary distances; <predict> entries are evicted after a sliding window of size w (default w=64) and thus cannot contribute to state preparation for far-future tokens. Loss gradients at \rho_i propagate into prediction parameters through h_{\rho_i}; the state-preparation gradients \sum_{j>i}\nabla \ell_j flow into the input stream h_{x_i} via persistent KV reads. The two roles are mechanically separated.
Ablations that isolate the mechanism
Two baselines rule out obvious confounders. 2x Memory keeps both streams persistent — the model gets extra compute but re-conflates the roles, and doubles KV cache. Delayed State places both prediction and state at the <predict> slot with a sliding window over inputs — this preserves the extra compute step but not the separation. A Reverse SPS variant swaps the routing (state via <predict>, prediction via inputs) to test whether the direction of separation matters.
Results
Across five GPT-2-style scales (53M to 1.678B parameters, 4096-token context), SPS attains the lowest FineWeb-Edu validation NLL at every scale, with the gap over Standard widening monotonically: -0.042 at XS, -0.048 at S, -0.058 at M, -0.067 at L, -0.068 at XL.
At matched tokens SPS is uniformly ahead; at matched GPU-hours it overtakes Standard despite the interleaved sequence effectively doubling context length. Doubling Standard’s pre-decay budget from 18B to 36B tokens does not close the XL gap, implying SPS reaches Standard’s quality on roughly half the data with the ratio growing at scale.
Out-of-distribution corpus NLL drops by 0.09–0.11 across scales; the largest reduction is -0.149 at XL. Zero-shot task accuracy (5-benchmark average) improves by +2.5 at XS, +2.3 at S, +2.9 at M, +2.5 at L, and +3.1 at XL.
Critically, Delayed State recovers roughly half of SPS’s NLL gain but consistently trails SPS itself (e.g., XL: -0.048 vs -0.068), showing that the extra compute step alone is not the driver. 2x Memory matches Delayed State on NLL but pays a 1.75\times–1.81\times peak-memory cost during decode and up to a 0.64\times throughput drop at XL. SPS retains near-Standard throughput (0.94\times at XL) and memory (1.01\times).
The window sweep confirms the mechanism: quality is best at small but non-zero w, and Reverse SPS (state routed through <predict>, prediction through inputs) is strictly worse at every w.
The asymmetry — persistent inputs, ephemeral predictions — is doing the work, not merely the presence of two streams.
Limitations and open questions
The training-time attention pattern makes each sequence effectively 2T long, so training FLOPs per token are higher than Standard’s; the paper reports GPU-hour parity but not FLOP-matched curves. The sliding window w is a new hyperparameter and the paper does not explore whether the optimum shifts with scale or context length beyond 4096. There is no gradient-level measurement showing that prediction and state gradients are actually less aligned inside SPS than inside Standard — the “fundamental difference in gradients” claim is largely inferential from the ablation pattern. Finally, all experiments are decoder-only pretraining on FineWeb-Edu; behavior under long-context finetuning, RLHF, or retrieval settings is untested.
Why this matters
The result is a clean architectural intervention identifying, and separating, two functions that are usually invisibly entangled in every Transformer forward pass. If the 2–3 point downstream gains and \sim 2\times data efficiency hold under FLOP-matched comparisons and at longer contexts, this is the kind of change that composes with, rather than competes against, existing pretraining recipes.
Source: https://arxiv.org/abs/2607.01218
Autonomous Scientific Discovery via Iterative Meta-Reflection
Problem
Autonomous discovery pipelines built on LLMs typically operate under one of two constraints: (i) they search a restricted hypothesis space such as pairwise causal edges (PC, NOTEARS, GPT-4 BFS), or (ii) they require a full research question or seed hypothesis (AI Scientist, HeurekaBench, ExperiGen). Neither setting reflects the open-ended situation of a researcher handed a raw dataset and asked “what is here?”. A further gap: iterative proposal loops accumulate discoveries but rarely reason about that accumulation to identify structural gaps, confounds, or redundant coverage.
DiscoPER targets the \mathcal{P}=\emptyset, \mathcal{H}_{\text{code}}^{\text{open}} regime — no prior objectives, hypotheses expressible as arbitrary executable code — and adds an explicit meta-reflection step that treats prior claims as data.
Method
The authors formalize prior work as instances of a generic loop over a claim set \mathcal{C}_t and a guidance summary \mathcal{G}_t. Given data \mathcal{X} (optionally multimodal), the system iterates:
h_t^{(1)},\ldots,h_t^{(K)} \sim \textsc{Propose}(\mathcal{X},\mathcal{P},\mathcal{G}_{t-1},\mathcal{C}_{t-1}) b_t^{(k)}, e_t^{(k)} = \textsc{Evaluate}(h_t^{(k)}, \mathcal{X}) \mathcal{C}_t = \mathcal{C}_{t-1} \cup \{h_t^{(k)} : b_t^{(k)}=\text{supported}\} \mathcal{G}_t = \textsc{Reflect}(\mathcal{C}_t, \hat{\mathcal{C}}_t) \quad \text{every } \tau \text{ iterations}
Each hypothesis h is a program that consumes \mathcal{X} and returns b\in\{\text{supported},\text{rejected}\} with statistical evidence e. Acceptance criteria are strict: effect size |\delta|\geq 0.2, p\leq 0.05 on both a training and held-out validation split, plus an overfitting guard |\delta_{\text{val}}|\geq 0.6\cdot|\delta_{\text{train}}|. This bidirectional split test is what keeps LLM-fabricated “patterns” out of \mathcal{C}_t.
The three modules are shown in Figure 2. Propose emits natural-language claim plus code; Evaluate generates test code (and may invoke a VLM tied to the same base LLM to extract visual attributes from images, enabling multimodal hypotheses on iNaturalist observations); Reflect runs every 5 iterations and produces \mathcal{G}_t summarizing coverage, confounds, and unexplored directions.
The distinguishing feature relative to ExperiGen (which uses short-term memory) or HeurekaBench (no reflection, full prior info) is that \mathcal{G}_t is a second-order object derived from analyzing \mathcal{C}_t \cup \hat{\mathcal{C}}_t as empirical data, not merely a running scratchpad.
Results
Two new benchmarks are introduced from iNaturalist observations, since edge-recovery metrics (SHD, edge-F1) cannot score mediation/confound structures and QA benchmarks leak the objective:
- iNatDisco-800: 800 observations, 8 species, 9 peer-reviewed ecological patterns as ground truth.
- iNatDisco-50K: 50,000 observations, 9,776 species, 12 patterns.
On iNatDisco-800, with Claude Sonnet 4.6, 100 iterations, reflection every 5:
| Method | \mathcal{H} | Prior | Recall | Support rate |
|---|---|---|---|---|
| LLM+PC | edge | none | 0/9 | – |
| LLM+NOTEARS | edge | none | 0/9 | – |
| GPT-4 BFS | edge | Full | 1/9 | – |
| HeurekaBench | code (guided) | Partial | 3/9 | 62.2 ± 6% |
| ExperiGen | code (guided) | Partial | 3/9 | 56.6 ± 5% |
| DiscoPER w/o Reflect | code (open) | none | 7/9 | 70.0 ± 2% |
| DiscoPER | code (open) | none | 8/9 | — |
Two things stand out. First, moving from edge hypotheses to open code hypotheses is worth roughly 4/9 recall on its own — causal-graph methods essentially fail here (0–1/9) because the ground-truth patterns (e.g., seasonal latitude shifts in Monarch observations) are not pairwise CI relations. Second, Reflect adds +1/9 recall over the ablation while operating from zero prior information, beating methods that receive partial task descriptions.
Scaling behavior on iNatDisco-50K (Figure 3) is informative: (a) more data monotonically increases both recall and the number of supported claims; (b) more iterations increase recall but decrease support rate. The latter is the expected signature of a system that exhausts easy, high-effect-size patterns early and moves to speculative claims that fail the |\delta|\geq 0.2, held-out threshold — evidence that the acceptance filter is doing real work rather than rubber-stamping LLM output.
Limitations and open questions
- Ground truth is a small curated set of literature patterns (9 and 12); recall of 8/9 is on a low-cardinality reference. False negatives against the broader space of true patterns are not measured.
- Statistical acceptance rules out spurious effects but does not adjudicate causality; supported claims are associational unless the hypothesis code itself implements an identification strategy. There is no explicit multiple-testing correction reported for the open loop (contrast ExperiGen’s Bonferroni).
- Reflect quality depends on the base LLM’s ability to introspect over \mathcal{C}_t; degradation on smaller models (DeepSeek V4 Pro, etc.) is reported but not decomposed into Propose vs. Reflect failure modes.
- The VLM-in-the-loop for image attributes shares weights with the reasoning LLM, which risks correlated errors between hypothesis generation and its own evaluation of visual features.
Why this matters
Framing prior autonomous-discovery systems as restrictions of a single (\mathcal{H}, \textsc{Propose}, \textsc{Evaluate}, \textsc{Reflect}, \mathcal{P}) tuple clarifies where the leverage is: opening \mathcal{H} to executable code and adding second-order reflection each contribute measurable, additive gains, and together they outperform systems given partial prior information. The support-rate decay with iterations gives a principled stopping signal for open-ended agents.
Source: https://arxiv.org/abs/2607.01131
Graph-Native Reinforcement Learning Enables Traceable Scientific Hypothesis Generation through Conceptual Recombination
Problem
LLMs applied to open-ended scientific hypothesis generation (materials design, mechanics) produce fluent prose whose intermediate reasoning is either absent, post-hoc, or inconsistent with the final answer. For a PhD-level user who needs to audit whether a proposed mechanism (e.g., a hierarchical toughening pathway in a bio-inspired composite) actually follows from the intermediate causal claims, this untraceability is the binding constraint. The authors target the specific failure mode: reasoning that cannot be inspected, decomposed, or reused across problems.
Method
Graph-PRefLexOR is a family of models at 1.7B, 3B, and 8B parameters, initialized from Qwen3-1.7B, Llama-3.2-3B-Instruct, and Qwen3-8B. The core design forces the <think> region to decompose into five sentinel-delimited phases:
<brainstorm>— divergent candidate mechanisms and failure modes.<graph>— abstraction into entities (sequence, structure, processing, properties, …) and typed causal relations.<graph_json>— a strict machine-readable directed graph, e.g.{"source":"Sequence","relation":"encodes","target":"Structure"}, drawn from a fixed relational vocabulary.<patterns>— extraction of higher-order motifs: causal chains (\text{sequence} \to \text{structure} \to \text{properties} \to \text{failure}), scale-bridging edges, feedback loops.<synthesis>— integration into a testable hypothesis linking multi-scale mechanisms to predicted outcomes.
Training is two-stage. Stage 1 is ORPO on preference pairs where the chosen response is a full graph-native trace generated by a strong teacher (GPT-5.1) under a “reason using a graph-based latent structure” system prompt, and the rejected response is a deliberately shallow 1–3 sentence answer from GPT-5-nano. The behavioral gap is intentionally large so preference ordering saturates within one epoch — this is the “cold start” that installs the schema. Records with unparseable graph_json or empty answers are discarded, so every training example carries a valid parsed graph.
Stage 2 is Graph-GRPO. Only the prompt and gold answer are used; chosen/rejected pairs are dropped. The policy samples its own traces and receives a composite reward built on the same graph_json object, grading both answer correctness and graph quality (structural validity, node/edge typing, coherence with the extracted patterns). GRPO’s group-relative advantage estimator suits this regime because it dispenses with a value critic and normalizes rewards within sampled groups, which matters when the reward is a heterogeneous sum of format, structural, and semantic terms.
The training corpus mixes fineweb-edu (general educational web text) with bio-silk-mech-mix-80K (biological and mechanical-materials focus), so the graph schema is grounded in domain vocabulary rather than generic scaffolds. Notably, the 3B backbone (Llama-3.2-3B-Instruct) has no native thinking mode, so it must acquire the graph-native format from scratch; the authors flag that this backbone distinction — not scale — dominates training dynamics.
Results
Evaluation uses 100 open-ended questions derived from published materials science and mechanics literature, probing cross-disciplinary linkage, causal mapping, and hypothesis generation. Claude Opus-4.7 serves as judge (deliberately from a different family than the GPT-based dataset generator, to reduce family bias) scoring Reasoning Quality, Intellectual Depth, Reasoning Traceability, and an aggregate Overall Score on 0–10.
Across all three scales, Graph-PRefLexOR gains 40–65% over its base model on the aggregate score, with the largest deltas on Reasoning Traceability — the axis the method was designed to address. Embedding analyses of the generated traces show approximately 2–3× greater semantic diversity than the corresponding baselines, indicating that the <brainstorm> phase is not collapsing to a narrow mode after RL, and that graph-conditioned generation encourages broader conceptual recombination rather than repeating stock chains. Semantic backtracking and layer-wise hidden-state analyses (reported in the paper) further show that the graph structure is internally represented, not merely emitted as surface tokens.
Limitations and open questions
- The judge is a single LLM (Claude Opus-4.7). Absolute score levels are not comparable to human expert ratings, and “traceability” as scored by an LLM judge may reward format compliance more than genuine causal validity.
- The relational vocabulary is predefined; open-vocabulary edges or discovered relations are not evaluated.
- Only one epoch of ORPO on teacher-distilled traces makes the model a strong imitator of a specific graph style — how much of the gain is GRPO versus the ORPO cold start is not fully disentangled beyond the training-dynamics discussion.
- The benchmark is 100 open-ended items; there is no closed-form verifiable-answer set (e.g., property prediction) to check whether traceable reasoning produces numerically correct downstream predictions.
- Generalization outside the bio/mechanical-materials mixture is untested, and the 3B backbone comparison is against a non-thinking baseline, which inflates the traceability delta.
Why this matters
Enforcing a parseable graph inside the chain-of-thought converts “reasoning” from an opaque prose artifact into an object that can be audited, diffed, and recombined — a prerequisite for using LLMs as components in scientific discovery loops rather than as answer generators. The result that a 1.7B model can be pushed into this regime by ORPO cold start plus graph-conditioned GRPO suggests the format, not scale, is doing most of the work.
Source: https://arxiv.org/abs/2607.00924
TurboServe: Serving Streaming Video Generation Efficiently and Economically
Problem
Streaming video generation is a new serving workload distinct from both offline (one-shot) video generation and LLM serving. A user opens a long-lived session that produces video chunk-by-chunk, with interleaved active and idle periods; the server must maintain per-session state (temporal features, KV caches, latent buffers, control embeddings) across that entire lifetime while delivering each new chunk under a strict deadline.
Two workload properties, both visible in production traces from Shengshu Technology, break generic serving stacks:
- Session duration heterogeneity. Session lifetimes span orders of magnitude, so any initial placement decision becomes suboptimal as short sessions terminate and long ones persist.
- Temporal demand heterogeneity. The number of concurrently active sessions swings sharply over minutes.
The consequence, illustrated in the paper’s motivation, is that (a) uneven session activation across GPUs makes the most-loaded GPU dictate worst-case per-chunk latency, (b) low-demand periods leave a fixed allocation underutilized, and (c) high-demand periods cause SLO violations.
Method
TurboServe formulates serving as an event-driven online scheduling problem with two coupled controls: session placement \phi_i(t) and cluster size M(t) \in \{0,\dots,M_{\max}\}. At each event t (arrival, departure, active/idle transition), each session s_i \in \mathcal{S}(t) is either mapped to a GPU g_j \in \mathcal{G}(t) (execution), to \emptyset (suspend, state offloaded to host), or terminated. Each GPU hosts up to K concurrent sessions to preserve the per-chunk latency budget. The joint objective trades off worst-case per-chunk latency \mathcal{L}(t) against provisioning cost \mathcal{C}(t) with weight \lambda(t).
Runtime substrate (TurboServe_base). Each GPU runs a coalesced-chunk loop: gather sessions whose next chunk is ready, batch them, invoke the model once, write back per-session state. State is partitioned into (i) a shared model replica, (ii) isolated per-session state regions (descriptor, prompt/control embeddings, temporal/KV state, chunk history, latent buffers), and (iii) an ownership table. Sessions can be suspended by copying only their per-session region to host memory; recomputation-based rematerialization (common in LLM KV-cache management) is rejected because video generation is compute-bound rather than memory-bandwidth-bound.
Placement controller. A migration-aware controller rebalances sessions to reduce the maximum normalized GPU load \rho_{\max}(t), which drives worst-case per-chunk latency. Migration moves only the per-session state region between GPUs via RDMA/NIXL one-sided reads, avoiding CPU staging. Consistency is enforced at chunk boundaries: (i) source completes the current chunk and freezes state, (ii) target fetches and verifies buffers, (iii) ownership flips atomically. This scopes the transfer to what is needed for continuation and prevents duplicated chunk execution.
Autoscaling controller. A load-driven loop compares observed load and utilization against a target \hat{\rho}(t) to compute a target budget M_{\mathrm{tar}}(t), provisioning or releasing GPUs. Because sessions are migratable, scale-in is not blocked by long-lived residents: they are evacuated to remaining GPUs before shutdown.
Session manager. Tracks execution/suspend/terminate transitions and persists suspended state in host memory, decoupling session lifetime from GPU residency.
Results
Evaluation uses LongLive-style streaming video models (LongLive-1.3B and larger variants) on Shengshu’s clusters: 16 H20 GPUs (Cluster 1) and 64 B300 GPUs (Cluster 2), with 8-GPU NVLink islands at 956 GB/s and 50 GB/s RDMA InfiniBand across nodes. Six production traces are replayed. Baselines are TurboServe_base (round-robin + FCFS), TurboServe_base+LAG (least-loaded GPU by utilization), and TurboServe_base+MAG (least memory-utilized GPU).
The paper reports paired comparisons: worst-case per-chunk latency under matched cost, and cost under matched latency. Under matched cost, TurboServe reduces worst-case per-chunk latency relative to all three baselines across models, traces, and both clusters; under matched latency it reduces GPU operating cost. Ablations isolate the two mechanisms: removing autoscaling primarily inflates cost in low-demand regimes, while removing migration primarily inflates worst-case latency in periods with skewed activation.
(Exact per-trace numbers are contained in Figure 7 of the paper rather than in the extracted text; the qualitative ordering is TurboServe < MAG ≈ LAG < base on both axes.)
Limitations and open questions
- The paper’s contribution is a scheduler and runtime; it does not modify the generation model itself, so per-chunk compute latency at fixed batch size is unchanged. Improvements come entirely from placement, migration, and elasticity.
- K, the max concurrent sessions per GPU, is treated as a bound derived from the per-chunk latency SLO; the paper does not explore adaptive K or heterogeneous-model co-location.
- Migration is only performed at chunk boundaries. For very long chunks or models with expensive state (large temporal caches), the freeze-transfer-install window may dominate; the paper does not report distributions of migration overhead or its impact on tail latency.
- The autoscaling controller reacts to observed load; predictive scaling for known bursty patterns (e.g., diurnal traces) is not evaluated.
- Recomputation is dismissed on compute-bound grounds, but hybrid schemes (offload latents, recompute cheap conditioning) may still be attractive for the largest models.
Why this matters
Streaming, session-oriented generative video is a workload class that neither vLLM-style LLM serving nor batch diffusion pipelines target: it combines persistent state, active/idle cycling, and hard per-chunk deadlines. TurboServe articulates the scheduling problem cleanly and shows that migration-aware placement plus load-driven autoscaling, built on a per-session-region memory layout with RDMA transfer, is sufficient to close the gap between production streaming workloads and generic serving heuristics.
Source: https://arxiv.org/abs/2606.19271
Hacker News Signals
Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers
SWE-Bench tests whether agents can close GitHub issues, but the tasks skew toward self-contained, well-specified bug fixes. Senior SWE-Bench raises the bar by constructing tasks that reflect what a senior engineer actually does: multi-file refactors, API design decisions, adding features with downstream compatibility constraints, and resolving ambiguous specifications. The benchmark is open-source and built on top of Snorkel’s data-curation infrastructure.
Technically, each task is paired with a rubric that captures not just “does the test suite pass” but structural criteria: does the change respect existing abstractions, does it introduce unnecessary coupling, is the commit message coherent with the diff. This moves evaluation closer to code review rather than unit-test pass/fail, which is a meaningful distinction because passing tests is necessary but far from sufficient for production-quality code.
The harder engineering challenge here is rubric construction at scale without human annotation becoming the bottleneck. Snorkel’s approach uses programmatic labeling to bootstrap rubric components, then validates with human reviewers. The benchmark also tracks agent behavior on tasks with intentionally incomplete context — a closer analog to real tickets that omit relevant background.
Current frontier models score substantially lower on Senior SWE-Bench than on original SWE-Bench, which suggests the latter has become somewhat saturated as a signal. The benchmark is open-source, so the community can audit task construction and add domains beyond Python web frameworks.
Open questions: rubric validity (does rubric compliance actually correlate with senior engineer judgment?), language and framework coverage beyond the current Python-centric corpus, and whether the programmatic labeling introduces systematic blind spots in what counts as “senior” quality.
Source: https://senior-swe-bench.snorkel.ai/
Obfuscation: Building the final boss of cryptography (Part I)
Vitalik’s writeup covers indistinguishability obfuscation (iO), which is the cryptographic primitive that, if efficiently realizable, would subsume nearly every other cryptographic construction. The framing “final boss” is apt: iO lets you take any two functionally equivalent circuits and produce obfuscated versions that are computationally indistinguishable, even given the obfuscated code itself.
The post walks through the definitional hierarchy: virtual black-box (VBB) obfuscation — which would be ideal but is provably impossible for general circuits — versus iO, which weakens the guarantee to indistinguishability between equivalent programs. From iO plus one-way functions you can construct public-key encryption, fully homomorphic encryption, functional encryption, and witness encryption, all as special cases. This makes iO a “complete” primitive in a precise complexity-theoretic sense.
The current candidate constructions rely on multilinear maps or lattice-based assumptions (e.g., the Lin-Matt construction, subsequent work from Jain-Lin-Sahai). These are not yet practical: the polynomial overhead is large enough that even toy programs produce obfuscated circuits measured in gigabytes, and the security assumptions underlying multilinear map candidates have been attacked repeatedly. Lattice-based iO candidates are more robust assumption-wise but remain far from deployable.
Part I focuses on the definitional landscape and motivation rather than construction internals. The writing is technically accurate and does not oversell current feasibility. The connection to blockchain applications (smart contracts as obfuscated programs, trustless escrow without a trusted setup) is mentioned but not overemphasized.
For anyone working in applied cryptography or zero-knowledge systems, understanding where iO sits in the primitive hierarchy is genuinely useful background even if you never implement it.
Source: https://vitalik.eth.limo/general/2026/06/29/obfuscation1.html
Asymmetric Quantization: Near-Lossless Retrieval with 97% Storage Reduction
The core observation: in retrieval, queries and documents play asymmetric roles. Documents are stored at index time and retrieved millions of times; queries are generated at runtime and used once. This asymmetry justifies quantizing document embeddings aggressively while leaving query embeddings at full precision (float32 or bfloat16) during inference.
The method quantizes document vectors to 1-bit per dimension (binary quantization) or 2-bit, then compensates for the precision loss asymmetrically: the query side retains full precision and the similarity computation is adjusted. For dot-product similarity, a binary document vector b \in \{-1,+1\}^d interacts with a full-precision query q \in \mathbb{R}^d via \text{sim}(q, b) = q^\top b, which is computed as a popcount-based inner product after thresholding. The key insight is that retrieval quality degrades primarily when both sides are quantized; keeping one side full-precision recovers most of the loss.
Mixedbread reports that with 1-bit document quantization and full-precision queries, NDCG@10 on BEIR drops by roughly 1-2 points absolute compared to float32 baseline, while storage shrinks by 32x (float32 to 1-bit). The claimed 97% reduction combines quantization with dimensionality reduction. They also describe a calibration step that rescales quantized vectors to minimize the expected squared error of the inner product estimate, which is standard scalar quantization theory applied per-dimension.
Limitations: the approach is embedding-model-specific (calibration statistics are computed over a representative corpus), gains depend on corpus distribution, and extremely short documents with sparse semantic content lose more information under aggressive quantization than longer ones. The asymmetric framing is not new (Faiss has supported asymmetric distance computation for years) but applying it carefully to modern dense retrievers with reported numbers on standard benchmarks is useful.
Source: https://www.mixedbread.com/blog/asymmetric-quant
Opening up Zero-Knowledge Proof technology to promote privacy in age assurance
Google is open-sourcing its implementation of the Periodic Unlinkable Disclosure (PUD) protocol, which uses zero-knowledge proofs to let users prove an age threshold (e.g., “I am over 18”) without revealing a birthdate or any linkable identifier. The cryptographic core is a signature-based ZKP: a credential issuer signs an age attribute, and the user proves knowledge of a valid signature on a commitment to that attribute without revealing the attribute value or the signature itself.
The scheme uses BBS+ signatures (or a variant thereof), which support efficient ZK proofs of knowledge of a signature over a vector of messages. The “Periodic Unlinkable” property means that within a time window, successive presentations of the same credential produce unlinkable proofs — the verifier cannot correlate two presentations even if colluding with the issuer. This is achieved by binding each proof to a fresh randomness and a time epoch rather than to a fixed nullifier.
The open-sourcing matters because age verification is increasingly mandated by law across multiple jurisdictions, and the dominant current approaches (document upload, credit card checks) involve storing or transmitting sensitive PII. A ZKP approach that is cryptographically sound and auditable by third parties is a meaningful improvement over proprietary black-box solutions.
Technical caveats worth noting: the threat model still requires a trusted credential issuer (the government or a document authority); the ZKP only protects the presentation layer, not the issuance. Device binding and revocation remain open engineering problems. The library is in Go and is designed to integrate with the Android Identity Credential API.
How to corrupt an SQLite database file
This is the official SQLite documentation on corruption vectors, which is more useful than it sounds: it is a precise enumeration of the assumptions SQLite makes about the I/O stack, and where violating those assumptions produces silent data corruption rather than an error.
The main categories: (1) incomplete writes — SQLite requires that a write to a sector either completes fully or not at all after a power loss; filesystems or storage controllers that report success before flushing to persistent media violate this. (2) Out-of-order writes — the WAL (write-ahead log) protocol relies on the journal being durable before the main database is modified; if the OS reorders writes, atomicity breaks. (3) File locking failures — on network filesystems (NFS, SMB), advisory locks are unreliable or silently ignored, meaning concurrent writers can corrupt the file without detecting the conflict. (4) Partial sector writes — a torn write to a 4096-byte page on a drive with 512-byte atomic write granularity can leave a page half-written. (5) Filesystem bugs — specific known bugs in ext3 (fsync not guaranteeing directory entry durability) and others are cited with version numbers.
The document also covers incorrect use of the API: calling sqlite3_exec from multiple threads on the same connection without serialization, or closing a database while a prepared statement is still live.
What makes this valuable for systems engineers is that it maps each corruption mode to a specific invariant in the storage stack, so you can audit whether your deployment environment (cloud block storage, containerized overlayfs, network-mounted volumes) satisfies SQLite’s documented requirements. The answer for several common configurations is “not always.”
Source: https://www.sqlite.org/howtocorrupt.html
Fixing a kubelet memory leak in Kubernetes 1.36
The post documents a real production memory leak in kubelet introduced in Kubernetes 1.36, found via heap profiling with pprof. The leak was in the podResourcesServer component, which serves the PodResources gRPC API used by device plugins to query per-pod resource allocations (GPUs, FPGAs, etc.).
The root cause: a cache of ContainerDevices objects was populated on every gRPC call but never evicted. The cache key included pod UID and container name, and the eviction path (triggered on pod deletion) had a race where the cache entry was populated after the deletion event fired. On clusters with high pod churn — CI/CD runners, batch jobs, spot instance nodes — this produced steady unbounded growth because each completed pod left a stranded cache entry.
The fix was to add a finalizer-style cleanup that runs on kubelet’s existing pod lifecycle reconciler rather than relying on event ordering. The heap profile showed the ContainerDevices structs themselves as the dominant allocation, not metadata, which pointed the investigation toward the cache rather than logging or metrics subsystems.
The debugging methodology is worth noting: the author used kubectl exec to hit kubelet’s pprof endpoint (exposed on localhost:10248 by default), compared heap snapshots across a time window, and cross-referenced with the allocation site in the source using go tool pprof’s list command. This is standard Go memory debugging but the walkthrough is concrete.
The fix was merged upstream. Clusters running 1.36 with GPU workloads or high pod churn should check kubelet RSS growth over multi-day windows as a canary.
Source: https://heyoncall.com/blog/fixing-kubernetes-kubelet-memory-leak
A complete ClickHouse OLAP engine, compiled to WebAssembly
chDB is an embeddable version of ClickHouse that runs in-process, and this deployment compiles it fully to WebAssembly so it runs in a browser with no server backend. The entire ClickHouse query execution pipeline — columnar storage engine, vectorized expression evaluation, MergeTree indexing, aggregate functions — runs client-side via WASM.
The engineering constraints are significant. ClickHouse’s core is C++ with heavy use of SIMD intrinsics (SSE4.2, AVX2) for string operations, hash aggregation, and compression. WASM’s SIMD support (128-bit only, via the fixed-width SIMD proposal) means the SIMD paths are either disabled or rewritten; the build uses Emscripten with -msimd128 but cannot use AVX2 paths directly. Memory is limited to the WASM linear memory model (currently up to 4 GB in most browsers with the memory64 proposal not yet universally available), which constrains query working set.
Data loading is handled by transferring Parquet or CSV files from the browser’s local filesystem via the File System Access API into WASM memory, where ClickHouse parses and queries them. The demo shows interactive SQL on local files without any data leaving the browser, which is the compelling use case for privacy-sensitive analytics.
Binary size after compression is around 30-40 MB, which is large but acceptable for a one-time load. Query latency on the demo datasets is competitive with server-side ClickHouse on small-to-medium data because there is no network round-trip.
Open limitations: no persistent storage across sessions (IndexedDB integration is not yet implemented), memory ceiling means multi-gigabyte datasets are impractical, and the threading model uses SharedArrayBuffer, which requires specific COOP/COEP HTTP headers that not all hosting environments provide.
Source: https://wasm.chdb.io/
Asahi Linux 7.1 Progress Report
The Asahi Linux 7.1 report covers the kernel driver work required to support Apple Silicon features that landed or matured in this cycle, with the focus on the M4-series chips (M4, M4 Pro, M4 Max, M4 Ultra).
The most technically dense section covers the AGX GPU driver (the DRM driver for Apple’s GPU, drm/asahi). Apple changed the firmware interface in M4 in ways that required updates to the command submission path; specifically, the compute command structure gained new fields and the vertex/fragment pipeline scheduling semantics changed. The driver is written in Rust, and the report notes that the Rust-for-Linux abstractions for DRM required additions to support the new submission model.
The display controller work (DCPext) is also notable: M4 Pro and Max have a separate display controller die that communicates with the AP over PCIe rather than the internal fabric used in M1-M3. This required a new transport layer in the DCP (Display Controller Processor) driver. The DCP is a coprocessor running Apple’s proprietary firmware; the Linux driver reverse-engineers the RTKit IPC protocol to drive it.
CPU frequency scaling on M4 Ultra (which tiles two M4 Max dies) needed per-die P-state management because the two dies can operate at independent frequencies. The existing cpufreq driver assumed a single frequency domain per chip.
The neural engine and media encode/decode blocks remain unsupported; these require additional firmware reverse engineering. The status table in the report is a precise hardware-feature-by-driver-status matrix, which is more informative than narrative summaries.
Noteworthy New Repositories
john-rocky/coreai-model-zoo
A community-maintained model zoo and knowledge base targeting Apple’s Core AI stack (iOS/macOS 26+). The project ships end-to-end conversion pipelines for Qwen3.5 and Gemma 4, verified on iPhone 17 Pro hardware against both the GPU and Apple Neural Engine (ANE). Beyond model weights, the repo documents conversion gotchas that bite practitioners — quantization edge cases, operator coverage gaps in Core ML, and activation function remapping — alongside custom Metal Performance Shaders kernels for operations not natively accelerated. A Swift runner provides a minimal inference harness so contributors can benchmark directly on-device without wiring up an Xcode project from scratch. The “knowledge base” component is arguably the higher-value artifact: it catalogs which model families survive the coremltools → ANE pipeline with acceptable accuracy degradation versus which require fallback to CPU/GPU. For anyone shipping on-device inference on Apple silicon without access to internal Apple tooling, this fills a real documentation vacuum that the official docs leave mostly empty. Useful both for practitioners deploying production apps and researchers benchmarking edge inference.
Source: https://github.com/john-rocky/coreai-model-zoo
raiyanyahya/recall
Recall solves the stateless-session problem for Claude Code: every new terminal session loses all project context, forcing repeated re-explanation. Recall persists that context locally as structured plain-text files — no cloud, no external service, no token round-trips to a remote memory store. The implementation is entirely offline; state lives on disk in a format that Claude Code can ingest via its context window at session start. Architecturally this is simpler than vector-store-based memory systems: there is no embedding, no retrieval ranking, and no approximate nearest-neighbor index. The tradeoff is that recall scales linearly with project complexity rather than sublinearly, and very large projects will eventually hit context-window limits. The value proposition is the zero-infrastructure setup — a single install puts persistent memory into whatever Claude Code workflow already exists. Compared to brain.md (see below), recall appears to target session-level continuity rather than decision/constraint documentation, though the boundary is fuzzy. Useful for solo developers who want persistence without operating a sidecar service.
Source: https://github.com/raiyanyahya/recall
gykim80/perfectpixel-studio
A desktop application for generating game-ready character sprite sheets from text prompts, targeting the 8-direction, multi-action format conventional in 2D RPGs and strategy games. The stack is Wails (Go backend + React frontend compiled to a single native binary), which avoids the Electron memory overhead common in cross-platform dev tooling. The generation pipeline takes a single prompt and produces sprite variants across directional angles (N, NE, E, SE, S, SW, W, NW) for 100+ action categories — idle, walk, attack, death, etc. — maintaining visual consistency across the full matrix. Consistency across this many frames is technically the hard part; naive per-frame generation produces incoherent character identity. The repo does not expose implementation details of the consistency mechanism in public documentation, so whether it uses ControlNet-style conditioning, reference-image inpainting, or a custom fine-tuned model is unclear from the repository alone. Output targets standard sprite sheet layouts importable into Unity, Godot, or RPGMaker. For indie developers without a pixel artist, the tooling addresses a genuine production bottleneck.
Source: https://github.com/gykim80/perfectpixel-studio
mindmuxai/brain.md
brain.md is a file-based, version-control-friendly memory layer for coding agents. The mental model is a single Markdown file per project — brain.md — that accumulates durable decisions, architectural constraints, and requirements that agents like Claude Code or OpenAI Codex should respect across sessions. The CLI is zero-dependency (no npm package tree, no Python virtualenv) and operates by writing and reading structured Markdown sections. The key design insight is that Markdown is already in the context window of every major coding agent, so no retrieval layer is needed: the agent reads the file directly. This is semantically richer than raw session logs (recall) because it is curated and human-editable — developers can annotate why a decision was made, not just what was decided. The limitation is manual curation overhead: the file does not self-update without explicit agent or human writes. Compared to vector-store memory systems, there is no semantic search, which means the full file enters the context on every session, making it unsuitable for very large decision histories. Best suited for projects where the constraint set is bounded and stable.
Source: https://github.com/mindmuxai/brain.md
sums001/Windows-Copilot-API
This repo reverse-engineers the Windows Copilot client protocol and exposes it as an OpenAI-compatible REST API, allowing callers to reach GPT-4 and GPT-5 models without API keys or billing accounts. The interface mirrors the OpenAI /v1/chat/completions schema, so existing OpenAI SDK integrations drop in with a base URL swap. Implementation relies on intercepted session tokens from the Windows Copilot desktop client, meaning it depends on an authenticated Windows session and is structurally fragile against client-side protocol changes or server-side bot detection. There is no rate-limit documentation because the actual rate limits are those enforced by Microsoft’s backend, not by this proxy. The legal standing is ambiguous — this consumes a paid service without payment, which puts it in the same category as other “free tier extraction” proxies. Technically, the interesting artifact is the protocol reverse-engineering work itself: request signing, session token lifecycle, and response streaming format. For researchers who want to study the API surface or developers in regions without direct OpenAI access, it provides a functional interface, but production reliance would be inadvisable.
Source: https://github.com/sums001/Windows-Copilot-API
lidge-jun/opencodex
OpenCodex is a provider proxy that sits between the OpenAI Codex CLI/App/SDK and arbitrary LLM backends, translating requests so that any OpenAI-compatible endpoint can serve Codex tooling. The practical motivation is that Codex CLI hard-codes assumptions about the OpenAI endpoint and model family; opencodex abstracts those assumptions behind a routing layer. Supported backends include local models via Ollama, Anthropic Claude, and other OpenAI-compatible APIs. The proxy handles request/response schema normalization, including tool-call formatting differences across providers, which is where most naive base-URL substitutions break. For teams that want Codex’s agentic file-editing and shell-execution capabilities but need to route through a self-hosted model for data-residency or cost reasons, this is a direct solution. The architecture is a lightweight HTTP proxy rather than an SDK wrapper, which means it is language-agnostic — any Codex client works regardless of whether it uses the Python SDK, the Node SDK, or raw HTTP. Maintenance risk is high given how rapidly Codex’s protocol evolves.
Source: https://github.com/lidge-jun/opencodex
aresyn/codex-control-plane-mcp
Codex Desktop tasks are inherently ephemeral: close the window, lose the task state. This repo implements a durable control plane over Codex Desktop via the Model Context Protocol (MCP), allowing long-running agentic tasks to survive session interruptions and be monitored or resumed externally. The MCP server exposes task lifecycle operations — create, pause, resume, inspect, cancel — as MCP tool calls that any MCP-compatible client (Claude Desktop, custom agents) can invoke. Durability is achieved by persisting task state to disk between Codex Desktop interactions rather than relying on in-process memory. The control plane pattern is standard distributed systems practice applied to a single-node desktop agent context: separate the task definition and state from the execution environment so the executor can crash and restart without losing work. Practically useful for multi-hour coding tasks where Codex needs to do substantial file manipulation and the developer cannot keep the application foregrounded. The MCP interface also enables chaining: an orchestrating agent can spawn Codex subtasks and wait on their completion programmatically.
Source: https://github.com/aresyn/codex-control-plane-mcp
agentteamhq/agentteam-email
AgentTeam-email provides open-source email infrastructure purpose-built for AI agents rather than human users. Standard email libraries assume interactive authentication flows, human-readable error handling, and low message volumes — none of which match agent usage patterns. This project targets the inverse: programmatic auth, structured error surfaces, high-throughput send/receive, and webhook-driven inbound processing that feeds directly into agent pipelines. The architecture appears to wrap SMTP/IMAP with an agent-friendly API layer, adding primitives like mailbox polling loops, structured attachment parsing, and reply-chain threading that agents can consume without bespoke parsing logic. Inbound email as an agent trigger is an underserved pattern: most agent frameworks focus on tool calls and REST webhooks, but email remains a primary integration surface for enterprise workflows. The open-source positioning differentiates it from managed transactional email services (SendGrid, Postmark) by giving operators full control over routing, storage, and the agent integration layer. Relevant for anyone building agents that need to participate in email-driven business processes — approval flows, notification handling, customer communication automation.