/en/research · what we measured
One question per entry, a measurement against controls, and what did not work. Code and data are open: our numbers can be recomputed with one command.
Three experts' vote yielded 0.848 vs. 0.893 for a single model, a debate between two models performed 7.7 p.p. [−13.0; −2.3] worse than one judge, and an ideal exchange would add no more than 2 p.p. We explain why opinion exchange doesn't work and describe a method where agents exchange not verdicts, but states: what they know, what they've done, and what they're unsure about. The method itself has not been measured yet, and we say so plainly.
Authors: slavb18
We tested conformal prediction as a 'when to involve a human' rule in senior screening. Coverage on unfamiliar skills is 0.913, 4% of interviews are escalated to a human, but among automatically promoted candidates, there are 14.9% errors. A simple probability threshold of 0.9 from the model did not violate coverage in any of the five shifts: the preregistered hypothesis was disproven.
Authors: slavb18
The LoRA-expert yielded to a stooge colleague in 36.7% of cases, and a linear probe of the hidden state recovered their original verdict with an AUC of 0.74. After deducting transcript features, it remained 0.63, while the model without an adapter gave 0.59: a difference of +0.04, with the CI including zero. We also calculated that an ideal exchange between three experts would add no more than 2 p.p.
Authors: slavb18
In our candidate matching, nearly half of what reaches the expensive LLM stage is garbage. We tried GPU rerankers, CrossEncoder fine-tuning and feature combos to filter it cheaper. The simplest signal won — skill overlap. Here are the numbers, what worked, and why text lost to skills.
Authors: slavb18
An engineering deep-dive: a council of LoRA experts that exchange hidden states instead of text, and a decoder that reads those thoughts back into text. With the war stories, the measurements, and the honest negative results.
Authors: slavb18
We took the autoencoder from the recent CALM (Continuous Autoregressive Language Models) work, which learns to pack a chunk of K=4 tokens into a single continuous vector and unpack it back, and trained it not on 15 billion Pile tokens on 8 GPUs, as in the original, but on 18 thousand short strings of requirements from IT job postings - on a regular machine without a graphics card. Along the way, we encountered three classic pitfalls (flash-attn without CUDA, deepspeed that doesn't import under NumPy 2.x, and a silent OOM on 33 GB of logits). Below is a detailed analysis of the architecture, configurations, and honest round-trip reconstruction results.
Authors: slavb18