/en/research · what we measured

Research

One question per entry, a measurement against controls, and what did not work. Code and data are open: our numbers can be recomputed with one command.

6
entries
37
topics
2026-11-17
updated

Agents debated and voted, but a single model proved more accurate. Why we switched to state exchange

Three experts' vote yielded 0.848 vs. 0.893 for a single model, a debate between two models performed 7.7 p.p. [−13.0; −2.3] worse than one judge, and an ideal exchange would add no more than 2 p.p. We explain why opinion exchange doesn't work and describe a method where agents exchange not verdicts, but states: what they know, what they've done, and what they're unsure about. The method itself has not been measured yet, and we say so plainly.

Authors: slavb18

  • AI
  • LLM
  • Agents
  • Multi-Agent
  • Research
  • HRTech

90% Guarantee Fulfilled, and Every Seventh Promoted Candidate Is Not a Senior: Conformal Escalation on Skill Shift

We tested conformal prediction as a 'when to involve a human' rule in senior screening. Coverage on unfamiliar skills is 0.913, 4% of interviews are escalated to a human, but among automatically promoted candidates, there are 14.9% errors. A simple probability threshold of 0.9 from the model did not violate coverage in any of the five shifts: the preregistered hypothesis was disproven.

Authors: slavb18

  • AI
  • LLM
  • ML
  • Research
  • Conformal Prediction
  • HRTech

Expert Yielded, but Probe Says They Stood Their Ground. Model Without an Opinion Reads the Same

The LoRA-expert yielded to a stooge colleague in 36.7% of cases, and a linear probe of the hidden state recovered their original verdict with an AUC of 0.74. After deducting transcript features, it remained 0.63, while the model without an adapter gave 0.59: a difference of +0.04, with the CI including zero. We also calculated that an ideal exchange between three experts would add no more than 2 p.p.

Authors: slavb18

  • AI
  • LLM
  • LoRA
  • ML
  • Research
  • Interpretability

A Cheap Skill-Overlap Score Beat GPU Rerankers: How We Filter Garbage Candidates Before the LLM

In our candidate matching, nearly half of what reaches the expensive LLM stage is garbage. We tried GPU rerankers, CrossEncoder fine-tuning and feature combos to filter it cheaper. The simplest signal won — skill overlap. Here are the numbers, what worked, and why text lost to skills.

Authors: slavb18

  • AI
  • HR Tech
  • LLM
  • Information Retrieval
  • Reranking
  • Cross-Encoder

Latent Debates Between LoRA Experts: How We Made Models Argue in Hidden Space — and Learned to Read Those Arguments

An engineering deep-dive: a council of LoRA experts that exchange hidden states instead of text, and a decoder that reads those thoughts back into text. With the war stories, the measurements, and the honest negative results.

Authors: slavb18

  • AI
  • LLM
  • LoRA
  • ML
  • Research
  • Latent Space

Compress Four Tokens into One Vector: Running CALM Autoencoder on Domain-Specific Data (and on a Single CPU)

We took the autoencoder from the recent CALM (Continuous Autoregressive Language Models) work, which learns to pack a chunk of K=4 tokens into a single continuous vector and unpack it back, and trained it not on 15 billion Pile tokens on 8 GPUs, as in the original, but on 18 thousand short strings of requirements from IT job postings - on a regular machine without a graphics card. Along the way, we encountered three classic pitfalls (flash-attn without CUDA, deepspeed that doesn't import under NumPy 2.x, and a silent OOM on 33 GB of logits). Below is a detailed analysis of the architecture, configurations, and honest round-trip reconstruction results.

Authors: slavb18

  • CALM
  • Autoencoder
  • Machine Learning
  • Deep Learning
  • NLP
  • LLM