Daily AI Digest — 2026-10-20
arXiv Highlights
OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners
Problem and motivation
Existing OR benchmarks (NL4Opt, ComplexOR, OptiBench, IndustryOR, etc.) hand the model a self-contained natural-language formulation: decision variables, constraints, and objective are already distilled by a human modeler. Real OR engagements do not look like this. Clients deliver a messy business brief, Excel sheets, PDFs, and partial rules; the OR practitioner must interrogate the client to recover missing facts, decide what to optimize, and ship a deliverable in a prescribed format. OR-License targets this gap directly, using 31 instances drawn from 10 industrial scenarios across USD 3.68M of paid engagements, with original materials preserved after de-identification.
Three gaps are addressed: (i) absence of raw client materials, (ii) absence of information-acquisition dynamics, and (iii) absence of professionally validated reference solutions. The third matters because objective quality is only meaningful relative to a solution the client actually accepted.
Benchmark construction
A scenario s groups instances \mathcal{I}_s; each instance i has test cases c \in \mathcal{C}_i, decision space \mathcal{Y}_{i,c}, objective f_{i,c}:\mathcal{Y}_{i,c}\to\mathbb{R}, and a hidden programmatic verifier. The task is
\max_{y\in\mathcal{Y}_{i,c}}\;\sigma_i f_{i,c}(y)\quad\text{s.t.}\quad \mathrm{verifier}_{i,c}(y)=\text{pass},\;\sigma_i\in\{+1,-1\}.
Crucially, the agent only receives a business brief and input files. Additional facts (unit costs, operational constraints, boundary conditions) must be elicited from ORSim, a controlled client simulator exposed to the agent as a Bash command ask. ORSim is designed to disclose business facts but withhold OR expertise — it will not propose a formulation or suggest a solver. This cleanly separates information acquisition from modeling competence.
Evaluation is two-stage: the verifier first checks hard feasibility, then f_{i,c} is independently recomputed from the submitted \hat y_{i,c} and compared to the expert reference. The headline metric is Deployment-Adjusted Quality (DAQ), which averages expert-relative objective quality across instances and assigns zero to any infeasible delivery. Conditional quality divides DAQ by observed feasibility and is only informative when read jointly with feasibility.
Results
Eleven frontier models are evaluated under a Claude Code harness, 5 runs per (model, task), with task-equal weighting over the 31 tasks.
The striking finding is that conditional objective quality hovers around or slightly above the expert reference for essentially every model: Claude Opus 5 100.03%, Gemini 3.8 Flash 100.43%, GPT-5.6 Sol 100.59%, GLM-5.3-Flash 101.01%, Hy4 preview 100.61%, DeepSeek-V4.1-Flash 99.44%, Doubao-Seed-2.1-Pro 99.47%, Qwen3.8-Max 99.06%. When agents produce a feasible solution, they match or beat expert solutions on the objective.
Feasibility is where models separate. Grok 4.6 leads at 88.39% feasibility (DAQ 86.55%), followed by GPT-5.6 Sol at 84.52% (DAQ 85.02%) and Gemini 3.8 Flash at 82.58% (DAQ 82.94%). Claude Opus 5 reaches 77.42% feasibility despite the harness being Claude Code. The bottom of the table is dominated by feasibility failure: Hy3 at 45.16% feasibility collapses to DAQ 43.77% even though its conditional quality is 96.92%. GLM-5.3-Flash shows the largest within-task variance (feasibility spread ±25.35), indicating fragile constraint handling across runs.
Two observations follow. First, the hard part of practitioner-style OR is not optimization quality but faithfully reconstructing all hard constraints from client materials and ORSim dialogue — infeasibility, not suboptimality, dominates losses. Second, the near-uniform ~100% conditional quality suggests that once an agent has correctly modeled the problem, modern solvers (Gurobi, OR-Tools, custom heuristics) are powerful enough that choice of LLM contributes little.
The technique-adoption analysis across categories T1–T7 shows that LLM agents and the expert reference occupy noticeably different regions of method space: LLMs over-rely on direct MILP encodings and off-the-shelf solver calls, while experts more frequently exploit decomposition and problem-specific structure. The convergence in objective quality therefore reflects solver strength on tractable instances rather than method parity.
Limitations and open questions
- 31 instances is small; within-task standard deviations are large (often >15 percentage points), and reported spreads are not standard errors of the benchmark mean.
- ORSim is scripted; realistic clients are inconsistent, change requirements, and sometimes refuse to answer. Robustness to adversarial or evolving specifications is untested.
- Harness effects are nontrivial. Results are primarily under Claude Code; the Codex comparison is relegated to the appendix. Feasibility rates may shift materially under different tool scaffolds.
- The DAQ metric gives zero credit for near-feasible deliveries, which can under-reward solutions that violate soft constraints the expert deemed negotiable.
- No cost-adjusted leaderboard in the main text — Grok 4.6’s lead at 86.55% DAQ may come with substantially different token/tool budgets than cheaper models.
Why this matters
OR-License reframes LLM evaluation on optimization: the bottleneck is not objective quality — frontier models already match expert solutions conditional on feasibility — but faithful constraint elicitation from original client materials. This isolates information acquisition and problem reconstruction as the next measurable axis of agent competence, and provides a benchmark where the ground truth is a solution a paying client actually accepted.
Source: https://arxiv.org/abs/2610.24087
Kronecker Delta Rule for Long Context Linear Attention
Problem
Linear-attention sequence mixers compress history into a matrix memory \bm{S}_t \in \mathbb{R}^{d_v \times d_k} updated by outer products. The two dominant write rules trade off differently against interference:
- Hebbian (linear attention, xLSTM, Mamba, RWKV): \bm{S}_{t+1} = \bm{S}_t + \bm{v}_t \bm{k}_t^\top. Additive, non-revising: later writes bury earlier associations at the same key but cannot overwrite them cleanly.
- Delta rule (DeltaNet, Gated DeltaNet, KDA): \bm{S}_{t+1} = \bm{S}_t(\bm{I} - \beta_t \bm{k}_t \bm{k}_t^\top) + \beta_t \bm{v}_t \bm{k}_t^\top. One step of SGD on \tfrac{1}{2}\lVert \bm{S}_t \bm{k}_t - \bm{v}_t\rVert^2. The correction direction (\bm{S}_t \bm{k}_t - \bm{v}_t)\bm{k}_t^\top depends on current memory contents, so values stored at keys non-orthogonal to \bm{k}_t are collaterally modified.
The paper’s thesis is that for long-context tasks, where the memory accumulates many distinct associations, delta-style interference compounds. They want an update that is strictly local to the pair (\bm{k}_t, \bm{v}_t) in matrix space, while retaining the error-correcting (revision) behavior of the delta rule.
Method: Kronecker Delta Rule
The abstract and Section 2.1 describe a rule in which the memory is modified only along a single rank-one direction in \mathbb{R}^{d_v \times d_k} — the direction corresponding to the specific (\bm{k}_t, \bm{v}_t) outer product — rather than along the data-dependent direction (\bm{S}_t \bm{k}_t - \bm{v}_t)\bm{k}_t^\top of the delta rule. The name “Kronecker delta” is a double pun: it indexes a single basis element in the Kronecker product space \mathbb{R}^{d_v}\otimes\mathbb{R}^{d_k}, and it generalizes the Widrow–Hoff delta rule.
Mechanically the write looks like
\bm{S}_{t+1} = \bm{S}_t + \alpha_t\,\big(\bm{v}_t - \bm{S}_t\bm{k}_t\big)\big\langle \bm{k}_t, \cdot\rangle\text{-targeted at }\bm{k}_t\bm{k}_t^\top/\lVert\bm{k}_t\rVert^2,
i.e., a projection that only touches the component of \bm{S}_t aligned with \bm{k}_t\bm{k}_t^\top, leaving all orthogonal directions untouched. Values stored at keys \bm{k}_i \perp \bm{k}_t are exactly preserved, in contrast with (2), where any \bm{k}_i with \bm{k}_i^\top \bm{k}_t \neq 0 is perturbed.
The paper shows that iterating this rule defines a Markov chain on the memory state that converges to a stationary distribution and, crucially, that \mathbb{E}\lVert \bm{S}_t\rVert is uniformly bounded in t. The practical consequence: no forget gate, no explicit normalization, and no stabilization heuristics are needed to keep the memory from drifting as context grows. This is a strong structural claim relative to xLSTM/mLSTM, which rely on exponential/forget gating, and Mamba, which relies on data-dependent decay.
The rule inherits the generalized-Householder structure of the delta rule, so chunkwise-parallel training in the style of Yang et al. (2024) carries over. The authors embed the Kronecker delta write inside an xLSTM-style sequence-mixer block to produce KronLSTM.
Experiments
All comparisons are at 1.5B parameters trained on 100B FineWeb-Edu tokens, context length 8k, Llama2 tokenizer. Baselines: Gated DeltaNet (GDN), GDN-2, Kimi Delta Attention (KDA), Mamba-3, mLSTM. Each is evaluated both as pure linear attention and as a hybrid with sliding-window attention (window 2048, 5:1 linear-to-SWA ratio).
Pure linear models, average over 6 common-sense + LAMBADA tasks (Avg-6) / LAMBADA perplexity:
- KronLSTM: 59.72 / 11.72
- KDA: 58.12 / 12.67
- GDN-2: 57.94 / 12.54
- GDN: 57.73 / 12.58
- Mamba-3: 57.31 / 12.95
- mLSTM: 57.18 / 13.87
KronLSTM wins 5 of 6 individual tasks and improves Avg-6 by 1.6 points over the strongest baseline (KDA). It also drops LAMBADA perplexity by roughly 0.8 relative to GDN-2.
Hybrid (with SWA):
- KronLSTM: 59.56 Avg-6, 11.18 LMB ppl (lowest perplexity across all 12 configurations)
- GDN-2: 58.79 / 11.85
- Mamba-3: 58.54 / 13.01
- GDN: 57.99 / 12.95
- KDA: 57.85 / 12.60
- mLSTM: 57.35 / 14.38
Adding SWA helps mLSTM, Mamba-3, and GDN-2 noticeably but barely changes GDN, KDA, or KronLSTM — consistent with KronLSTM already handling long-range associations well without a sliding-window crutch. The paper additionally reports length-extrapolation experiments beyond the 8k training context in Section 3.2 and 400M-scale ablations in Appendix A6.
Limitations and open questions
- The Markov-chain stability analysis presumably assumes some regularity on (\bm{k}_t, \bm{v}_t) distributions (unit-norm keys, bounded values); the degree to which the uniform-norm bound depends on learned statistics vs. architectural choices is not clear from the excerpt.
- No reported comparison against softmax Transformers at the same scale/token budget; the claim is relative to other linear attentions.
- Only 1.5B/100B tokens shown here; scaling behavior at frontier sizes and whether the “no normalization needed” property survives low-precision training is open.
- Needle-in-a-haystack, RULER, or associative-recall numbers that would directly stress the “no interference between orthogonal keys” claim are not included in the sections provided.
Why this matters
The Kronecker Delta Rule gives a mathematically clean separation between associative revision and interference: the delta rule’s error correction without the delta rule’s cross-key perturbation. If the uniform norm bound and the empirical 1.6-point Avg-6 gain over KDA hold up at larger scale, it is a strict improvement to the DeltaNet family and removes one of the main sources of architectural complexity (gating/normalization) in modern linear-attention stacks.
Source: https://arxiv.org/abs/2610.26834
Scaling Parameter-Efficient Fine-Tuning for Brain Encoding
Problem
Brain encoding models predict fMRI responses to naturalistic stimuli. The dominant recipe freezes a large pretrained backbone and trains a readout (ridge or small transformer) from its hidden states to voxels or parcels, because fMRI data are scarce (hours per subject) and full fine-tuning overfits. This paper asks whether PEFT — specifically LoRA — can adapt very large multimodal backbones for this task, and whether the main obstacle to doing so is computational rather than statistical.
The computational obstacle is concrete. The prior SoTA on Algonauts 2025 (MIRAGE, built on Qwen3-Omni) re-encodes an overlapping prefix for each fMRI timestep, giving \sim 2{,}758 forward passes per clip and \approx 700 GPU-h of extraction on an A800. Fine-tuning through the backbone with that protocol is infeasible.
Method
Single causal pass. Because the decoder is causally masked, the sliding window recomputes almost identical prefixes. The authors instead encode a 149 s window (100 TRs) in one forward pass and read features at all timesteps simultaneously, in the style of language encoding models. Measured on one A800: 5.1 forwards per clip (vs. 2{,}758), \approx 14{,}900 tokens per forward, 1.99 GPU-h extraction over 331 clips — a 352\times reduction. End-to-end training (extraction + 4.84 h adapter + 1.67 h closed-form re-fit) is 8.5 GPU-h vs. 702.5 for the baseline, an 83\times reduction.
The modality encoders are locally bidirectional (8 s audio blocks, 1 s visual patches), so single-pass features are not strictly causal within those blocks. The authors verify by comparing shared timesteps across prefix cuts (Tab. 9) that deviations are concentrated far from the cut and are consistent with length-dependent bf16/MoE routing numerics rather than future-stimulus leakage. A 60 s context lead supplies the missing prefix at window start and more than recovers the small accuracy drop from switching extraction protocols.
LoRA + readout. On Qwen3-Omni (30B total, 3B active), rank-16 LoRA is inserted into attention modules through layer 32, and the decoder is truncated at that layer. This gives 8.91M trainable backbone parameters. Pooled features (one vector per modality per 2 Hz timestep, aggregated into 8 log-spaced temporal lags, \sim 894 vectors per window) feed a linear readout to 1,000 Schaefer parcels (52.3M params). Adapter + readout are jointly optimized with SGD for one epoch (1,429 steps); a second epoch overfits (-0.0147 at readout lr 3\times 10^{-4}). The linear readout W is then re-fit in closed form per subject via parcelwise ridge on accumulated Gram matrices.
Results
Across nine backbones (4.7B–104B). LoRA improves every backbone over the frozen-readout control with the same training rates and step budget, by +0.032 to +0.066 Pearson correlation on the Friends s06 validation. On Qwen3-Omni, frozen 0.2907 \to LoRA 0.3263 (+0.0356); the largest absolute gain is on MiniCPM-o (+0.0655). Cross-attention readouts add \approx +0.003 on average. Backbone size and encoding score correlate only weakly (r=0.71 frozen, r=0.60 adapted).
Vs. prior SoTA. On the same Qwen3-Omni backbone and the same single-pass features, LoRA + linear reaches 0.3263, above the MIRAGE head’s 0.3187 in its sliding-window protocol and above the MIRAGE head applied to single-pass features at matched lead (0.3241 with re-fit). Seed-to-seed noise is \approx 0.00077, so these gaps are well above noise. Ablations show the closed-form re-fit is responsible for much of the headroom over MIRAGE’s trained-head configuration.
PEFT family robustness. \mathrm{IA}^3 on attention+FFN recovers 62% of LoRA’s gain with only 0.057M params; extending \mathrm{IA}^3 into MoE experts reaches 76%; Houlsby adapters at r{=}32 (budget-matched) reach 79%. LoRA remains best per-run but the effect is not LoRA-specific.
Regional structure. The adapter improves all 1,000 parcels. Yeo networks: dorsal attention +0.050, visual +0.040, limbic +0.015. ROI-level: MT+ +0.055, IPS +0.054, TPJ +0.049; mPFC +0.025. Four subjects’ gain maps agree at mean pairwise r=0.673.
Modality attribution. Exact Shapley over the 2^3 subsets of {video, audio, text}, with removed streams replaced by training-mean embeddings, shows strong redundancy: audio alone recovers 76% and video alone 71% of the full score. Adaptation shifts the mixture: audio alone gains +0.0503, video alone +0.0329, and the transcript loses 0.0065. The adapted model extracts from speech audio what the frozen one obtained from transcripts.
Transfer. Leave-one-subject-out training transfers nearly perfectly: a held-out subject gains +0.0301 to +0.0355 over frozen, within 0.002 of the 4-subject adapter. The adapter also transfers to the NNDb dataset (different scanner, TR, films) under anchored ridge on the new subject’s data, beating the frozen backbone for every subject at every calibration budget.
Negative controls. Training on time-shifted targets gives 0.1874 (vs. 0.3263 aligned), confirming the gain depends on stimulus–response pairing, not fMRI shape statistics. Cross-seed adapters are nearly orthogonal in weight space (\cos \approx 0.026) yet compute almost the same function at the feature-extraction layer, so seed-only ensembles give little gain.
Ensembling. A 261-member pool reaches 0.3401 with a A n^{-b} fit to the gap to its limit. A fused multi-adapter kernel runs N rank-16 adapters at the cost of one.
Limitations
Local non-causality in the audio/vision encoders means features at t can mix stimulus up to the end of a block boundary; this is bounded but nonzero. Rank 128 diverges at the tuned LR; the rank–LR coupling is not disentangled. Ensemble analysis shows the pool still leaves 32.5% of inter-subject stimulus-locked covariance unexplained, which bounds remaining headroom. All results use Schaefer-1000 parcels on CNeuroMod; voxel-level and clinical-population generalization are untested.
Why this matters
A simple causal-structure observation (one forward pass per window instead of per timestep) collapses extraction cost by \approx 350\times, which converts “fine-tune a 100B multimodal model on fMRI” from an infeasible project into an 8.5 GPU-h run. The resulting LoRA adapters consistently and uniformly beat frozen-readout baselines across backbones, modalities, and brain regions, and transfer to unseen subjects and datasets — evidence that the long-standing frozen-backbone paradigm in neural encoding was compute-bound, not statistically justified.
Source: https://arxiv.org/abs/2610.26802
Learning from Fleet Experience: A Collective Learning Paradigm for Robots
Deployed manipulation policies produce execution data continuously, but using that data to improve the policy traditionally requires a teleoperator in the loop (HG-DAgger-style corrections). This bottleneck scales linearly with fleet size, which is incompatible with any serious notion of a robot fleet. This paper proposes collective experience learning (CEL): after an expert-supervised initialization, robots improve a shared policy using only their own autonomous rollouts, labeled with a binary outcome y(\tau) \in \{0,1\} provided by a human (who also resets the environment). No new expert actions are collected.
FERT: supervising on successes, prioritizing by failures
The central algorithmic insight in failure-guided experience retrieval fine-tuning (FERT) is a decomposition of the credit-assignment problem that binary labels create: successes provide actions worth imitating, failures determine where in state space learning should concentrate.
FERT maintains a buffer \mathcal{D}^+ of successful trajectories collected across the fleet. For each failure trajectory, visual and proprioceptive features are extracted to form a failure context query; nearest-neighbor retrieval against encoded segments of \mathcal{D}^+ returns successful trajectory segments that are state-similar to the failure region. These retrieved segments receive elevated sampling priority in a prioritized-replay fine-tuning loop whose loss remains ordinary behavior-cloning supervision on (o_t, a_t) pairs drawn from \mathcal{D}^+. Concretely, after SFT and multi-robot HG-DAgger initialization, the loop is:
loop:
insert new rollouts {(τ, y(τ))} from fleet
for failed τ: encode failure context, retrieve top-k from D+, bump priorities
sample minibatch from D+ with current priorities
SGD step on BC loss
periodically push updated policy to fleetThis is reminiscent of retrieval-augmented imitation and prioritized experience replay, but applied in a fleet-scale deployment regime where the “signal” from failures is a localization signal rather than a gradient. The upshot: expanding data collection no longer requires expanding teleoperation — the only human effort per episode is one bit plus a reset.
FATS: the systems substrate
The fault-tolerant asynchronous training system (FATS) separates a data plane (SwarmLink), which streams per-field trajectory tensors from edge robots through cloud inference into training buffers, from a control plane (HotSwarm) that handles membership, worker identity and recovery. Synchronous experience collection would stall every policy update on the slowest robot; FATS instead decouples collection, optimization and policy distribution so that faster robots keep generating experience while others recover. This matters because real fleets have heterogeneous execution speeds and local interruptions (grasp failures, resets, comms stalls) at non-trivial rates.
Results
Evaluation uses five AgiBot G2 tabletop tasks — Flower Arrangement, Clothes Folding, Phone Charging, Stool Assembly, Phone Packaging — spanning deformables, precise alignment, tool use and multi-stage execution.
Policies are evaluated on robots excluded from training data collection, which rules out the easy win of overfitting to specific unit kinematics. Across 120 trials per task, final policies reach success rates of 93.3–99.2%, improvements over the expert-initialized policy obtained without any new expert demonstrations or corrections. The paper additionally studies:
- Fleet scaling with per-robot collection time held fixed (Section 4.3), isolating the effect of parallel experience breadth from the trivial effect of more data per robot.
- Ablations (Section 4.4) that attribute gains to failure-guided retrieval versus uniform sampling of \mathcal{D}^+ and to the other design choices in FERT.
- System behavior under induced worker failures and communication delay (Section 4.5), measuring how training throughput degrades gracefully rather than stalling.
Limitations and open questions
- Binary outcome supervision presumes a reasonably strong initialization: \mathcal{D}^+ must be rich enough that retrieved segments are actually relevant to the failure context. For novel failure modes outside the support of successful rollouts, FERT has no corrective signal.
- The method inherits the standard imitation-learning covariate-shift concern — unlike DAgger, failed states never receive on-policy expert actions, only analogous successful behavior from different trajectories. Whether this closes the distribution-shift gap or merely shifts it is an empirical question that the five-task benchmark cannot fully answer.
- Humans still provide outcome labels and environment resets. For tasks where “success” is itself ambiguous (e.g., assembly tolerances, long-horizon goals), the labeling cost may re-emerge.
- The retrieval encoder is pretrained; its invariances determine which failure contexts are grouped. Pathological encoder behavior could silently misdirect prioritization.
- Reported ranges (93.3–99.2%) across tasks already indicate heterogeneous headroom; the lower bound task likely reveals where retrieval-based self-improvement saturates.
Why this matters
CEL/FERT points to a practical path for scaling fleet learning in which human effort grows sub-linearly with fleet size — one label and one reset per episode — rather than linearly with teleoperation hours. If the paradigm generalizes beyond tabletop manipulation, it reframes deployment itself as the dominant source of policy improvement.
Source: https://arxiv.org/abs/2610.25736
Holi-World: Building Holistic World Intelligence from Human-Ego Experience
Problem
Egocentric video is an attractive substrate for learning world models: it couples first-person observation with embodied interaction, giving naturally grounded trajectories of how scenes are structured and how objects change under contact. The bottleneck is annotation. Converting raw ego recordings into supervision that links persistent scene geometry, object identities and states, and interaction histories across viewpoint and time has historically required dense human labeling — infeasible at the scales needed to train VLMs for spatial-temporal reasoning. Holi-World targets this bottleneck by proposing an end-to-end ecosystem that automatically turns ego recordings into structured “world records,” derives grounded QA training data from them, and evaluates the resulting model against independently human-annotated holdouts.
The ecosystem
Holi-World is organized as four coupled components:
HOLIWORLD-Foundry — the automated construction pipeline. It fuses static perception (persistent scene structure: layout, object instances, poses) with dynamic perception (hand-object interactions, state transitions, temporal segmentation) to produce a world record per recording. The record is a joint representation of (a) a persistent scene graph of object identities and locations, (b) time-stamped interaction events with the hand(s) and acted-upon objects, and (c) supporting raw observations (frames, viewpoints) attached to each event. The design principle is that static and dynamic streams are complementary: static perception stabilizes identity and location across time, while dynamic perception resolves state changes and causal structure that static detectors miss.
HOLIWORLD-Data — a dataset derived from roughly 100 self-collected scenes. The pipeline emits approximately 80K QA pairs grounded in the world records. Questions span spatial reasoning (relative positions, affordances, viewpoint changes), temporal reasoning (ordering of interactions, object state histories), and interaction attribution (which hand, which object, which outcome). Because the QA generator is backed by the structured world record rather than frame-level captions, answers inherit the record’s cross-temporal consistency.
HOLIWORLD-VL-4B — a 4B-parameter vision-language model trained on HOLIWORLD-Data. The scale is deliberately modest; the claim is that quality and structure of the training data, not model size, drives spatial-temporal competence on ego inputs.
HOLIWORLD-Bench — independently human-annotated heldout recordings used both to audit the automated annotation and to evaluate learned understanding. Separating the heldout annotation process from the Foundry pipeline is important: it prevents the benchmark from measuring the degree to which the model has memorized Foundry’s own biases.
Mechanical skeleton
To reimplement the skeleton, the key moving parts are:
- A static perception stack that produces, per recording, a persistent object inventory with 3D-consistent locations. In practice this means running an open-vocabulary detector plus a tracker/segmenter and lifting detections into a scene-level frame via camera pose (SLAM or VIO on the ego stream), then clustering tracklets into persistent instance IDs.
- A dynamic perception stack that detects hand-object contact events, labels the manipulated object (reconciled against the static inventory), and emits state-change predicates (e.g., opened/closed, moved from A to B). Each event is anchored to a time interval and to supporting frames.
- A world record schema that stores both streams jointly, so a query like “what was the state of object o at time t” is answered by a deterministic lookup over the record, not a language model.
- A QA generator that templates and samples questions from the record. Templates are organized by reasoning type; grounding is enforced by requiring every answer to be derivable from a record entry, with supporting frames attached as visual context.
- A VLM trainer that fine-tunes a 4B backbone on these grounded QA pairs with the supporting frames as visual input.
Formally, the world record can be viewed as W = (\mathcal{O}, \mathcal{E}, \mathcal{F}) where \mathcal{O} is the persistent object set with attributes, \mathcal{E} = \{(t_i, h_i, o_i, s_i^{-}, s_i^{+})\} is the event sequence (time, hand, object, pre/post state), and \mathcal{F} indexes supporting frames. Training examples are (q, a, V) \sim \mathrm{Gen}(W) with V \subset \mathcal{F}.
Results and limitations
The paper reports training HOLIWORLD-VL-4B on ~80K QA pairs from ~100 scenes and evaluating on HOLIWORLD-Bench. The abstract is truncated in the available excerpt, so specific headline numbers (benchmark accuracies, ablations isolating the static vs. dynamic streams, comparisons to general-purpose VLMs on the same ego benchmark) are not reproduced here; readers should consult the full paper for exact figures.
Several limitations are visible from the design alone. First, ~100 scenes is small relative to the diversity of real environments; generalization claims will hinge on whether the Foundry pipeline, not the trained model, is the actually scalable artifact. Second, automated world records inherit the failure modes of their perception components — missed instances, identity switches across long occlusions, and spurious state predicates — and QA grounded in those records will encode those errors. The independent human annotation on Bench partially mitigates this for evaluation but not for training. Third, the QA templates bound the types of reasoning the model is pressured to learn; genuinely compositional or counterfactual queries are unlikely to emerge without explicit template design or an LLM-based question synthesizer with verification against W. Fourth, the ecosystem is single-agent and ego-only; multi-agent interaction and third-person alignment are open directions.
Why this matters
The interesting contribution is not the 4B model but the claim that a disciplined separation between a structured, automatically constructed world record and the QA layer on top of it is the right scaling substrate for ego-centric world models. If the Foundry pipeline holds up at 10×–100× more scenes, this becomes a practical route to spatial-temporal supervision that does not depend on human annotators keeping pace with capture.
Source: https://arxiv.org/abs/2610.26549
EgoXperience: Learning World-Grounded Egocentric Representations from Human Experience
Problem
Egocentric reconstruction has largely been attacked as a collection of disjoint subtasks: estimate the camera trajectory, estimate monocular depth, lift hand meshes, estimate object 6-DoF poses. Even when each piece is reasonable in isolation, the outputs do not compose into a single consistent world: hand meshes drift relative to the scene geometry, object poses lack metric scale, and camera estimates are not aligned with any shared frame. For downstream uses — imitation learning for manipulation, affordance modeling, world models of human activity — the quantities of interest are relative (hand-to-object, object-to-scene, hand-to-world), so joint, world-consistent reconstruction is the actual target. Existing datasets either lack 6-DoF object poses, lack metric depth, or are restricted to single indoor rigs with no cross-area or outdoor coverage.
The EgoXperience stack
EgoXperience ships three coupled components: an annotation engine (EgoX-Engine), a 50-hour dataset (EgoX-Data-6M, 101 CAD-modeled objects, cross-area and outdoor coverage), and a joint reconstruction model (EgoX).
Capture uses the HOMIE Gen2 head rig: a calibrated front stereo pair for metric depth, plus four fisheye cameras feeding a SLAM pipeline that recovers camera trajectories. The SLAM frame of each session is fixed as the reference, and all per-frame scene depth, hand meshes, and object poses are rigidly transformed into that frame. This is the key design choice: entity-level estimates are produced in camera coordinates, but supervision is expressed in a session-constant world frame, which disambiguates whether apparent motion comes from the camera, the entity, or both.
EgoX-Engine couples geometrically verified object tracking (CAD-based, so 6-DoF is metric) with the SLAM alignment, and additionally emits second-level task captions. Compared to prior egocentric datasets (Table 1), it is the only one combining 3D hand, 6-DoF object pose, mesh, depth, mask, caption, cross-area, and outdoor annotations in one place; HOT3D and HO-Cap are indoor-only and lack depth, Ego-Exo4D and EgoDex lack 6-DoF/mesh entirely.
EgoX model
EgoX is monocular, end-to-end, and jointly outputs camera trajectory, metric depth, hand meshes, and object 6-DoF poses in a shared world frame. The only extra input beyond RGB is first-frame object masks (compare: Any6D needs estimated depth, intrinsics, and per-frame masks; FoundationPose further needs GT metric CAD assets). Shared scene/entity features feed task-specific heads, and a predicted global scale lifts outputs to metric. Training uses EgoX-Data-6M plus complementary public annotations. Evaluation uses fixed 150-frame clips across HOT3D, EgoX-Bench, HOI4D, and H2O; no method receives GT camera or scale.
Results
Hand reconstruction (PA/WA/W-MPJPE in mm). EgoX dominates all four benchmarks on world-space metrics, which is the regime where camera drift compounds with hand pose error. On EgoX-Bench, W-MPJPE drops from 328.81 (WiLoR+SLAM, best baseline) and 363.47 (HaWoR) to 113.24. On HOI4D, W-MPJPE goes from 108.76 (MINT) to 42.67. On H2O, 34.63 (HandFlow) to 23.95. PA-MPJPE on HOI4D is 3.32 vs. 17.94 for MINT — more than a 5× improvement after rigid alignment, indicating the gains are not solely from better camera estimation.
Object 6-DoF (camera frame, ADD/ADD-S mm, AUC-S_{50} %). EgoX, with only first-frame masks, beats methods that consume estimated depth, intrinsics, per-frame masks, and in some cases metric CAD. On HOT3D: ADD 28.00 vs. 67.06 (Any6D) / 71.92 (GigaPose+MegaPose); AUC-S_{50} 73.71 vs. 57.59. On HOI4D, where baselines collapse (GigaPose+MegaPose AUC-S_{50} 3.91, FoundationPose 13.39), EgoX reaches 59.08. On H2O, AUC-S_{50} 80.42 vs. 46.03.
Camera and depth. EgoX is compared to VGGT, VGGT-\Omega (fit one scale to DA3-Nested depth per sequence), CUT3R, Any4D, and DA3-Nested. VGGT-\Omega remains the strongest camera estimator on EgoX-Bench (nATE 0.0071, RRE 0.61°), and DA3-Nested is strongest on depth on HOT3D (AbsRel 0.0799, \delta_{1.25} 94.40). EgoX’s claim is joint prediction without per-sequence scale fitting, not that its camera/depth exceeds specialists; the table (partially shown) supports it being competitive while being the only system producing hand and 6-DoF in the same frame.
Limitations and open questions
Dataset scale (50 h) is modest relative to EgoDex (829 h) or EgoVerse (1362 h), though those lack 6-DoF annotations. Object recognition is anchored to 101 CAD-modeled objects, which may not generalize to open-vocabulary manipulation. EgoX requires first-frame masks, so a fully in-the-wild setting still needs an upstream segmenter. The camera/depth gap to VGGT-\Omega and DA3-Nested suggests room to tighten the joint objective. Finally, the evaluation uses 150-frame clips; long-horizon drift and loop closure in the world frame are not directly probed.
Why this matters
Joint, metric, world-frame reconstruction from monocular egocentric RGB — hands, objects, scene, and camera together — is the correct representation for learning from human experience, and EgoXperience is the first end-to-end system plus dataset that actually delivers it, with large margins on world-space hand error and 6-DoF pose despite using strictly less input than object-pose baselines.
Source: https://arxiv.org/abs/2610.26625
ScaleNav: Scalable Internet Video Pretraining for Language-Conditioned Local Navigation
Problem
Language-conditioned local navigation requires a robot to move from its current pose to a text-specified target that is already visible in its initial RGB observation, and to stop at the described location. The policy must generalize across scenes, lighting, and object categories, which demands diverse training data. Real-robot teleoperation data is expensive; simulation is cheap but suffers from a sim-to-real appearance gap. Internet egocentric video (walking tours, house tours, first-person exploration) is a largely untapped source of real-world camera motion through 3D scenes, but it lacks action labels. ScaleNav asks: can we extract a goal-directed visual dynamics prior from action-free Internet video and attach a small action decoder on top using simulation data?
Method
ScaleNav is a unified world-action model: a single DiT backbone jointly predicts future video frames and action chunks conditioned on an instruction, the current observation, and memory. An action is a_t = (\Delta x_t, \Delta y_t, \Delta \psi_t) — planar displacement in the robot’s local frame plus yaw change — and T actions form a chunk \tau = (a_1, \ldots, a_T). At each step the robot executes a predicted chunk, collects new RGB, and re-plans the next chunk.
The attention structure (Figure 2a) is chunk-wise causal: within a video/action chunk, tokens attend bidirectionally; across chunks, attention is causal. Memory consists of an initial-frame anchor (which keeps the target referent stable as the viewpoint drifts) plus a sliding window of recent video and action chunks. The anchor is important because the instruction refers to an object only visible at t=0; without it, long rollouts would lose the referent.
Training proceeds in three stages:
Instruction-conditioned video pretraining on action-free Internet egocentric clips. The DiT learns p(o_{t+1:t+T} \mid o_{\le t}, \ell) where \ell is a natural-language target description. This stage captures how egocentric views evolve as a camera translates and rotates toward instructed objects across diverse real scenes. An initial bidirectional adaptation is applied before converting the DiT to chunk-wise causal attention.
Action alignment on action-labeled simulation data. Only the DiT self-attention layers, an action head, and an action-embedding MLP are updated; the cross-attention and MLP blocks learned during video pretraining remain frozen. This forces the action head to read out motion from the pretrained dynamics representation rather than overwriting it. To prevent catastrophic forgetting of real-world visual statistics, Internet egocentric video is replayed in this stage.
Joint fine-tuning unfreezes the DiT and both the video and action heads, optimizing them together on simulation navigation with video replay.
The three-stage design cleanly separates “what the world looks like under goal-directed motion” (learned from scalable action-free video) from “which action produced the observed transition” (learned from cheap simulation).
Results
ScaleNav is evaluated on two benchmarks. The first is a single-view adaptation of Short-Horizon OVON (524 episodes across 36 HM3D scenes), where start poses are repositioned to the first frame in which the HM3D-OVON target becomes visible, so the task reduces to local navigation to a category-level goal. The second is HabitatGS-LN, a new benchmark the authors construct in photorealistic Habitat-GS over 157 episodes across 45 held-out InteriorGS scenes, where instructions reference the target via attributes and spatial relations rather than only category.
ScaleNav reaches 86.5% success rate on Short-Horizon OVON (single-view) and 94.9% on HabitatGS-LN. The higher number on HabitatGS-LN is consistent with the task structure — attribute- and relation-rich instructions give the model a stronger grounding signal in the initial frame — whereas Short-Horizon OVON provides only a category label and demands more disambiguation among similar objects.
Qualitatively, the same checkpoint deploys zero-shot on a quadruped in exhibition, lounge, and office environments, following instructions like “go to the [object]” to termination. These scenes are visually dissimilar to simulated HM3D/InteriorGS interiors, which supports the claim that Internet video pretraining transfers real-world appearance priors that would otherwise be absent from a sim-only policy.
Limitations and open questions
The paper’s scope is local navigation: targets must be visible in the initial observation, which sidesteps exploration, memory over long horizons, and object permanence. The initial-frame anchor is a crutch that works only in this regime; extending to multi-room or re-identification tasks will require richer scene memory. Action labels still come entirely from simulation, so the fidelity of the simulated dynamics (step sizes, collision behavior) bounds the real-robot action distribution. The reported real-robot results are qualitative; no success rates on a standardized real-world suite are given. Finally, the compute cost of Internet-video DiT pretraining is not quantified, and it is unclear how much of the gain comes from pretraining data scale versus the chunk-wise causal architecture and anchor memory — an ablation separating these would clarify the central claim.
Why this matters
ScaleNav operationalizes a concrete recipe for leveraging action-free Internet egocentric video as pretraining for embodied policies: learn goal-directed visual dynamics first, then bolt on action readout using cheap simulation. If the paradigm scales, it offers a path around the real-robot data bottleneck without relying on simulation photorealism to close the sim-to-real gap.
Source: https://arxiv.org/abs/2610.25690
Hacker News Signals
DeepSeek just open-sourced its inference engine
DeepSeek released DeepInfer, their production inference engine, under an open-source license. The repo covers the full serving stack: KV cache management, continuous batching, tensor parallelism across multiple GPUs, and a custom CUDA kernel library. The architecture centers on a disaggregated prefill/decode design — prefill and decode phases run on separate worker pools, which decouples their distinct compute profiles (prefill is compute-bound, decode is memory-bandwidth-bound) and allows independent scaling. This is the same class of disaggregation that PD-separated systems like Splitwise and DistServe explored, but DeepInfer ships it as a production artifact rather than a research prototype.
The KV cache layer supports paged attention in the vLLM style but adds cross-request prefix caching with a radix-tree index, enabling cache hits on shared system prompts without explicit user configuration. The scheduler tracks cache occupancy per block and evicts using an LRU policy with priority adjustments for in-flight sequences.
Tensor parallelism is implemented via NCCL with custom all-reduce kernels that overlap communication with compute by pipelining across micro-batches. For MoE models (relevant given DeepSeek-V3/R1’s architecture), expert parallelism is handled separately: each expert shard lives on a dedicated rank, and the routing gather/scatter is fused with the all-to-all to reduce kernel launch overhead.
The quantization support includes FP8 per-tensor and per-channel, INT8 activation quantization, and GPTQ-style weight-only INT4. Notably, the FP8 path targets H800/H100 hardware instruction sets directly rather than going through a generic CUTLASS template, which reduces abstraction overhead.
For practitioners, DeepInfer is a credible alternative to vLLM and TGI for DeepSeek model families specifically, with the advantage that the engine and the model weights were co-designed — the MLA (Multi-head Latent Attention) attention variant in DeepSeek-V3 has first-class support rather than being bolted on.
Source: https://github.com/deepseek-ai/DeepInfer
Timestral: Mistral’s time-series foundation models
Mistral released Timestral, a family of foundation models targeting time-series forecasting and classification. The architecture departs from the standard Transformer-with-patching approach (e.g., PatchTST, Moirai) in favor of a causal decoder trained autoregressively over quantized time-series tokens. Each observation is mapped to a discrete codebook entry via a learned scalar quantizer with per-channel statistics normalization, and sequences are tokenized at configurable granularities (point-level or patch-level depending on series frequency).
The pretraining corpus spans financial tick data, IoT sensor streams, weather station records, and energy grids — heterogeneous enough that the model must learn channel-agnostic representations. Conditioning on metadata (series ID, frequency tag, task type) is handled via prefix tokens prepended to each context window, which lets a single checkpoint handle zero-shot forecasting, few-shot adaptation, and anomaly scoring without architecture changes.
For forecasting, generation is autoregressive over the quantized vocabulary, with the final continuous value recovered by the codebook lookup plus a residual correction head. This avoids the Gaussian output assumption baked into many regression-based forecasters, at the cost of quantization error accumulating over long horizons.
Benchmark numbers show competitive performance on Monash and ETT benchmarks versus Chronos and Moirai, with particular strength on irregular-sampling settings where patch-based methods degrade. The classification head is fine-tuned separately on UCR/UEA datasets.
Open questions include whether autoregressive decoding over quantized tokens actually outperforms direct regression at long horizons (the quantization bottleneck is non-trivial), and how the model handles series with non-stationary variance that exceeds the normalization range seen during pretraining. No weights are publicly released yet; the announcement describes API access only.
Source: https://mistral.ai/news/timestral/
Mercury 3: Diffusion at scale
Inception Labs released Mercury 3, positioning it as a large-scale commercial diffusion language model. The core claim is that discrete diffusion over text tokens enables substantially higher throughput than autoregressive decoding at equivalent quality, because the model can execute parallel denoising steps across the full output sequence rather than generating tokens sequentially.
The underlying approach follows the masked diffusion framework: during training, tokens are corrupted by masking at varying rates, and the model learns to predict the original tokens given the partially masked context. At inference, the sequence is initialized fully masked and iteratively denoised over T steps, with each step predicting all masked positions simultaneously and accepting high-confidence tokens. This is structurally similar to MDLM and SEDD but at larger scale and with engineering work to make the parallel evaluation efficient on GPU.
The throughput advantage is real in principle: with T \ll L (steps much less than sequence length), wall-clock generation time scales with T rather than L. In practice, quality degrades with fewer steps, so the tradeoff depends on the step count needed to match autoregressive quality on a given task. Inception’s numbers claim competitive quality to GPT-4o class models at significantly lower latency, though independent third-party evaluation is absent from the announcement.
The announcement does not detail architecture size, training compute, or dataset composition. The main engineering contribution appears to be a custom CUDA kernel for batched masked prediction that avoids the memory layout inefficiencies of naively running a full forward pass over partially masked sequences.
The broader significance is whether diffusion-based LMs can close the remaining quality gap with autoregressive models at scale. Mercury 3 is a data point suggesting the gap is narrowing, but without open weights or a reproducible benchmark comparison, it remains a vendor claim.
Source: https://www.inceptionlabs.ai/blog/introducing-mercury-3
One parser, two bugs, many routers: Exploiting Broadcom WiFi across eCos, Linux
Quentin Kaiser’s writeup details two vulnerabilities in the Broadcom FullMAC WiFi firmware parser, affecting devices running both eCos (embedded RTOS) and Linux-based firmware. The bugs share a root cause: a single 802.11 information element parser is compiled into firmware images for both OS targets, so a parser logic flaw propagates across the entire product line.
The first bug is a type confusion in the IE (information element) length field handling. The parser reads a length byte, uses it to index into a structure, but does not validate it against the actual buffer boundary before a secondary operation. On eCos targets, heap metadata is adjacent to the packet buffer, so an oversized length value allows controlled overwrites of heap freelist pointers — classic heap feng shui toward code execution. On Linux-based firmware, ASLR and heap hardening make exploitation harder but not impossible given the deterministic firmware load addresses on specific SoC variants.
The second bug is an integer truncation in the SSID field parser. The SSID length is read as a uint8_t, subtracted from a signed integer offset, and the result is used as an allocation size. Under specific inputs, the subtraction wraps to a large positive value, triggering an undersized heap allocation followed by a linear overflow.
Both bugs require only the target to be within WiFi radio range and in a scanning or association state — no prior authentication. The eCos path is more directly exploitable because eCos provides no ASLR, no stack canaries by default, and the heap implementation is a simple linked-list allocator.
The post includes partial PoC for the eCos path and notes that Broadcom was notified and issued patches, but the parser code is embedded in binary firmware blobs distributed by device vendors, many of whom have not pushed updates.
Source: https://quentinkaiser.be/security/2026/10/16/one-parser-two-bugs/
Harness2 by Google Research: Recursive Agent Harnessing
Harness2 is a Google Research framework for composing LLM-based agents recursively. The core abstraction is a “harness” — a typed wrapper around an LLM call that specifies input schema, output schema, and a verification function. Harnesses can be composed: the output of one becomes the input of another, and composition is tracked in a DAG that the framework can re-execute selectively on failure.
The recursion aspect means a harness can invoke sub-harnesses, including itself with modified parameters, enabling tree-structured reasoning without hard-coding the decomposition. The framework handles retry logic, partial failure recovery, and output memoization. Each harness execution is logged with full input/output/verification status, making debugging agent chains tractable.
From the repo, the implementation is in Python and uses structured output APIs (OpenAI-compatible) to enforce schema compliance at each step. The verification functions are user-supplied callables — unit test style — that gate whether a step’s output is accepted or triggers a retry or fallback branch. This is more principled than the common pattern of wrapping LLM calls in try/except and hoping for the best.
The framework does not prescribe a particular planning strategy; it is a scaffolding layer rather than an agent architecture. Compared to LangGraph or LlamaIndex’s agent abstractions, Harness2 is more explicit about the contract between steps and more minimal in scope — there is no built-in tool use, memory, or retrieval; those are harnesses the user composes.
The practical value is in the memoization and selective re-execution: when debugging a 12-step agent pipeline, being able to replay from step 7 with a fixed prompt without re-running steps 1-6 reduces iteration time substantially.
Source: https://github.com/google-research/harness2
If you don’t run your own LLM proxy, how do you know what’s going on?
The post argues for running a local LLM proxy (e.g., LiteLLM, a custom FastAPI shim, or mitmproxy configured for JSON logging) as the sole egress point for all LLM API calls in a development environment. The technical case is concrete: without a proxy, you have no unified log of prompts, completions, latency, token counts, or error rates across different models and libraries. Each SDK (OpenAI, Anthropic, LangChain, etc.) has its own logging interface, callback system, and observability story, and they do not compose.
A proxy at localhost:4000 (LiteLLM’s default) or a custom endpoint gives you a single capture point. You can log every request/response to SQLite or Postgres, emit OpenTelemetry spans, inject headers, rewrite model names for A/B testing, or insert a rate limiter — all without modifying application code. The proxy also enables switching between providers (OpenAI, Azure OpenAI, local Ollama) via a config change rather than code changes.
The post walks through a concrete LiteLLM setup with SQLite logging and a minimal Grafana dashboard showing per-model latency percentiles and token spend. The security argument is secondary but real: you can inspect exactly what your dependencies are sending, which matters when using third-party agent frameworks that construct prompts internally.
For teams, a shared proxy with API key management lets you track per-user or per-service token consumption without relying on provider dashboards, which are often delayed and aggregated. The operational cost is low — LiteLLM runs in a single Docker container and adds ~2ms of proxy overhead.
Source: https://jrdsgl.com/if-you-dont-run-your-own-llm-proxy-how-the-fuck-do-you-even-know-whats-going-on/
Show HN: Sphinx – A hook that quizzes you on your AI’s code before shipping
Sphinx is a git pre-commit hook that intercepts staged changes, sends the diff to an LLM, and generates comprehension questions about the code before allowing the commit to proceed. The developer must answer correctly (evaluated by a second LLM call) to complete the commit. The stated goal is to enforce that developers understand AI-generated code before it enters version control.
The implementation is a shell script / Python hook that diffs staged files, truncates to a token budget, and posts to an OpenAI-compatible endpoint asking for 3-5 questions about logic, edge cases, or the purpose of specific code sections. The answers are evaluated for semantic correctness rather than exact match, using a grading prompt that returns a pass/fail JSON. A failed answer blocks the commit and prints the correct explanation.
The technical interest is in the grading prompt design — exact-match evaluation would be too brittle for code explanations, so the grader is prompted to assess whether the answer demonstrates understanding of the key concept, not whether it matches a reference string. This is a small instance of LLM-as-judge, with all the reliability caveats that entails.
Limitations are obvious: a developer can answer superficially, the grader can be gamed by confident-sounding wrong answers, and the hook adds latency to every commit. The questions are generated from the diff, not from the broader codebase context, so they cannot probe interactions with existing code. For teams where the problem is rubber-stamping AI output without reading it, Sphinx imposes a friction cost that may shift behavior even if it cannot enforce deep understanding.
Source: https://sphinx-six-theta.vercel.app/
Tmux and Claude Code: Sessions, Send-Keys, and the Ceiling
The post is a detailed technical guide on using tmux as a control layer for running Claude Code (Anthropic’s agentic coding tool) in persistent, scriptable sessions. The core technique is tmux send-keys: you can drive an interactive terminal session programmatically by injecting keystrokes into a named window, which lets you script multi-step agent workflows that would otherwise require a GUI or a custom API integration.
The practical setup involves creating named tmux sessions per task (tmux new-session -d -s task_foo), attaching Claude Code to each session, and using send-keys from a controller script to issue prompts, wait for the agent to settle (detected by polling the pane output with tmux capture-pane), then issue follow-up commands or extract results. This is a poor man’s agent orchestration layer built entirely on Unix primitives.
The “ceiling” in the title refers to the limitations: capture-pane gives you the rendered terminal output, not structured data, so parsing agent status requires fragile string matching. There is no event system — you poll. Parallelism is possible (multiple sessions) but coordination between sessions requires external state (a file, a Redis key). Claude Code’s own context window limits mean long tasks require manual context management via send-keys-injected summarization prompts.
Despite these limitations, the approach is genuinely useful for running batch experiments or automated refactoring tasks where you want the agent to operate unattended but need audit trails. Tmux’s session persistence means the agent survives SSH disconnections, and tmux pipe-pane provides a continuous log to file without modifying the agent’s environment.
Noteworthy New Repositories
malevrigns/agent-jev
AgentJev-0.6B is a specialized 0.6B-parameter model positioned as a “System One” decision primitive for agentic pipelines. The core design constraint is zero output-token decoding: rather than autoregressively generating text, the model performs a single forward pass over an arbitrary unstructured context (git diffs, stack traces, log streams) concatenated with a structured question, and returns a calibrated probability distribution over a discrete option set directly from logit outputs. Latency is reported at roughly 50ms per query, making it practical as an in-loop classifier rather than an out-of-loop LLM call. The calibration claim implies temperature scaling or Platt scaling has been applied post-training so that predicted probabilities are meaningful as confidence scores, not just rankings. The 0.6B scale sits below standard chat models, suggesting the architecture is fine-tuned specifically for classification/scoring tasks rather than generation. This is useful in multi-agent systems where a large orchestrator LLM would be wasteful for binary or low-cardinality routing decisions – routing, escalation gating, anomaly flagging. The trade-off is that the model is narrow: it cannot explain its decisions beyond the probability output, and out-of-distribution inputs may yield poorly calibrated scores.
Source: https://github.com/malevrigns/agent-jev
StartLuxLabs/StartLux-Decision
A family of open decision models spanning 0.8B to 27B parameters, all sharing the same inference contract: one forward pass yields a calibrated probability for each option, plus a short chain-of-thought token sequence emitted conditionally only when the model’s confidence falls below a threshold. The conditional reasoning gate avoids paying decoding latency on easy decisions while preserving interpretability on hard ones. The suite ships engines for three runtimes – vLLM (throughput-optimized, tensor-parallel serving), llama.cpp (CPU/edge quantized inference), and MLX (Apple Silicon). The 0.8B–27B range means practitioners can size the model to their latency budget and hardware. Calibration across this scale range is non-trivial; larger models tend to be overconfident, so per-scale calibration procedures are presumably applied. The conditional thought output is architecturally interesting: it implies a learned confidence estimator gating a secondary decode path, which resembles speculative or mixture-of-experts routing at the generation level. Primary use cases are the same as single-model decision APIs – agentic routing, structured classification, preference scoring – but the scale ladder adds a deployment flexibility that single-model alternatives lack.
Source: https://github.com/StartLuxLabs/StartLux-Decision
strands-labs/strands-decider
Strands Decider is a compact, latency-optimized model for agentic decision steps: given a set of discrete options or a numeric scale, it returns a selection plus a calibrated confidence score in a single inference pass with no autoregressive decoding overhead. The framing as “System One” (fast, reflexive) versus a full LLM’s “System Two” (deliberate, token-by-token) is operationally precise – this model is intended to replace conditional branches and heuristic thresholds inside agent loops, not to replace reasoning chains. Calibrated confidence is the key differentiator over a plain classifier: a well-calibrated score at 0.72 should mean the model is correct roughly 72% of the time across the calibration distribution, enabling downstream components to apply decision thresholds programmatically. The repo targets agentic workflow integration, suggesting API-first design compatible with tool-calling patterns. Compared to StartLux-Decision, this appears to be a single smaller model rather than a scale family, optimizing for minimal footprint. The main limitation shared with all such models is distributional shift: confidence calibration degrades when the input domain diverges from training data, and there is no built-in mechanism for the model to signal unknown-unknowns.
Source: https://github.com/strands-labs/strands-decider
kividbio/kividb
KividDB is a Rust-based in-memory data store implementing the Redis Serialization Protocol (RESP), making it a drop-in replacement for Redis and Valkey clients without application-level changes. The primary differentiators over stock Redis are native full-text search and vector similarity search built directly into the engine rather than as a module. Multi-threaded execution in Rust avoids Redis’s single-threaded command processing bottleneck; the ownership model eliminates data races without locks in the common path, following the pattern established by Dragonfly and similar reimplementations. The vector search integration is architecturally significant: it positions KividDB as a unified cache-plus-retrieval-index, removing the need for a separate ANN layer (e.g., Faiss sidecar) in RAG pipelines. Full-text search likely uses an inverted index structure (BM25 or similar) stored in-process alongside the key-value store. Being RESP-compatible means existing Redis client libraries (redis-py, StackExchange.Redis, Jedis) connect without modification. The Rust implementation should offer competitive latency and memory safety. Open questions include persistence guarantees (AOF/RDB equivalent), cluster/replication support, and whether the vector index supports HNSW or flat exhaustive search.
Source: https://github.com/kividbio/kividdb
storytold/effectcraft
EffectCraft is a motion graphics and compositing application written from scratch in pure Rust, targeting the tooling gap left by proprietary software like After Effects. “From scratch” here implies a custom rendering pipeline rather than wrapping an existing compositor such as OpenFX or GStreamer. Pure Rust suggests the renderer, scene graph, timeline, and UI are all implemented without C/C++ bindings to established graphics frameworks, which is technically ambitious. The compositing pipeline likely handles layer blending, masking, transform hierarchies, and keyframe interpolation. Motion graphics specifically implies parametric shape animation, text animation, and procedural effects rather than just video layer compositing. With 7,480 stars, it has attracted significant attention, suggesting the implementation is substantive rather than a skeleton. Relevant technical questions include which GPU abstraction is used (wgpu is the standard Rust choice, offering Vulkan/Metal/DX12 backends), how the node-based or layer-based composition model is represented in the data model, and whether the rendering is CPU-bound or GPU-accelerated. A pure-Rust stack offers memory safety and potential for WASM compilation targeting browser-based deployment.
Source: https://github.com/storytold/effectcraft
storytold/designcraft
DesignCraft is a page and layout design editor – analogous to Figma or Affinity Publisher – implemented entirely in Rust from first principles. Layout design implies a document model supporting frames, vector shapes, typography with advanced text layout (ligatures, bidirectional text, variable fonts), constraints-based resizing, and export to print or screen formats. Writing this from scratch in Rust is a significant undertaking: text shaping alone typically delegates to HarfBuzz, and the decision to avoid such dependencies entirely would require implementing Unicode segmentation, script detection, and glyph layout internally. Alternatively, “pure Rust” may mean no C/C++ application code but does allow Rust crates that themselves wrap system libraries. The 4,511 stars indicate meaningful community traction. From a systems perspective, the interesting design choices are the retained-mode scene graph structure, whether layout uses a flexbox-style constraint solver (similar to Taffy/Stretch), and how collaborative editing or versioning would be layered on. Sharing a GitHub organization with EffectCraft (storytold) suggests a common rendering substrate, which would be architecturally economical – both tools need vector rendering, bezier curves, and a GPU compositing stage.
Source: https://github.com/storytold/designcraft
ollaya-dev/ollaya
Ollaya mirrors the Ollama UX pattern – pull, serve, query locally – but targets decision models rather than generative LLMs. The supported model classes are Laya (unspecified decision model), Decider (likely the Strands or similar class), NLI (Natural Language Inference, a textual entailment classifier), and GLiClass (Generalist and Lightweight Class prediction, a zero-shot classification architecture). All are served behind a TypeSafe-compatible API, which implies TypeScript type safety on the client side with generated schemas matching the model’s input/output contract. The technical value is unification: rather than running separate inference servers for each model class, Ollaya provides a single daemon managing model lifecycle (download, quantization selection, memory allocation) across heterogeneous classification model families. NLI and GLiClass specifically are useful for zero-shot text classification tasks where a generative LLM would be wasteful. The architecture presumably wraps llama.cpp or a similar runtime for model execution, adds a routing layer that dispatches requests to the appropriate loaded model, and exposes a REST/JSON API with TypeScript type definitions. Primary use case is local agentic pipelines needing fast, private classification without cloud API calls.
Source: https://github.com/ollaya-dev/ollaya
mrsarac/ff-tracking
ff-tracking is a creative-technical project implementing a virtual camera system that films an AI agent’s screen activity while a glyph-level tracker locks onto individual rendered characters. The rendering pipeline runs two Fframes passes – a geometry pass and a compositing pass – with a single SkSL (Skia Shader Language) shader handling the visual effect. Audio is entirely synthesized rather than sampled. The technical depth here lies in the tracker: locking onto every glyph as the agent’s screen changes requires detecting and following individual character bounding boxes through frame transitions, which is a non-trivial tracking problem when text reflows or the agent types. SkSL as the shader language places this in a Skia/Flutter or Chrome rendering context, where SkSL shaders execute on the GPU for post-processing effects. The “virtual camera” metaphor implies a simulated camera with motion, depth of field, or parallax effects applied to a 2D screen recording, creating a cinematic presentation of agent behavior. The two-pass architecture (likely geometry/depth in pass one, shading/compositing in pass two) enables effects that require depth information unavailable in a single pass. This sits at the intersection of creative coding and agent observability tooling.