Public benchmark · winnex-madhava · pip install from PyPI

Five datasets, one truth. What search engines do when the data is corrupt — and what we do instead.

Most search engines either break or quietly lie when they meet degraded data. This page is the deep analysis of one public benchmark — the five-datasets run — where the most instructive result is not the clean ones. It is the dataset where 94.7% of the vectors were NaN, and where the engine refused to pretend otherwise. Every number on this page is a measured output of a public, re-runnable notebook. Where we cannot measure, we do not claim.

94.7%

of Word2Vec raw vectors are NaNdiagnosed before indexing

0.7333

R@10 on degraded datasafe preset, honest ceiling

0.4833

R@10 if PCA is forcedthe number we refuse to report as 1.0

bound violationsacross all datasets and runs

100/100

certificate checksgenuine + deterministic

32/32

thread-safe queriesbit-identical serial vs 8 threads

Kaggle — five-datasets benchmark Back to the stack

01What we are actually claiming

We do not claim that Winnex-Madhava "heals" corrupt data. No engine can. We claim something narrower, and we can prove it:

What the engine does with degraded data

Diagnoses it. Before indexing, it measures the corruption in float32 — the exact dtype the engine runs on — and reports the fraction of non-finite and zero-norm rows.
Isolates the cause. The nan_policy blocks PCA when the corpus has NaN/inf, because a non-finite row silently corrupts the projection basis. That is the failure class it refuses to let through.
Applies the safe preset. It routes the configuration that maximizes provable fidelity on the rows that are real — instead of the configuration that would look better on paper.
Reports the honest number. On the degraded Word2Vec row it publishes 0.7333, not a rounded-up 1.0.
Certifies what it prunes. For every document the engine discards, it records a Cauchy–Schwarz bound proving that document could not be in the true top-K. Zero bound violations across every run.

What a search engine without a quality layer does

Indexes the garbage silently. FAISS has no input firewall. Feed it a NaN-poisoned corpus and it builds an index over it as if nothing were wrong.
Reports internally-consistent nonsense. "0 bound violations" stays true on corrupt data, because the bound is about internal consistency — not about whether the input means anything. The engine has no way to know. That is exactly why the gate exists.
Looks correct while being useless. The silent failure this benchmark exists to expose: a naive pipeline indexes the file, reports 0 violations, and nobody notices the results are garbage.

The one-line summary

We do not sell speed. We sell the truth about your data — including the truth you did not ask for. When the data is corrupt, the maximum fidelity is a refusal to pretend: diagnose the corruption, isolate its cause, route the safe configuration, and hand you a certificate of exactly what was proven.

02The five datasets — real, raw, and public

Five real Kaggle embedding datasets, loaded raw (no numpy pre-processing, no re-ordering), run through the production path: normalize (L2 + quality gate + per-dataset preset) → Madhava (search with proof). Recall is measured against the motor's own exact scan on the same query — a valid ceiling that is mathematically independent of any external ground truth file.

Dataset (raw, real)	d	N	Raw diagnosis	Routed config	Recall@10	Bound-pruned	Latency	Build
GloVe-Twitter	25	19,900	clean	default	1.0000	99.9%	0.76 ms	0.1 s
ProtBERT-CAFA5	1024	19,900	clean	word2vec preset	1.0000	33.4%	6.56 ms	2.9 s
SBERT-Forum	384	19,900	clean	default	0.998	99.7%	1.58 ms	2.9 s
FastText-Crawl	300	19,900	clean	default	1.0000	93.1%	1.39 ms	2.3 s
Word2Vec GoogleNews	300	986*	degraded	word2vec preset	0.7333	0.0%	0.23 ms	0.1 s

*After the quality gate removed the 18,914 non-finite / zero-norm rows from Word2Vec, 986 real vectors remained for the benchmark. The degradation was measured on the raw file before removal. Full per-dataset JSON (with audit commitment and flags) is linked from the notebook.

What "clean" means here — and the honest limit

Recall@10 = 1.0 on the clean rows means the engine returns the exact top-10 of the same embedding space, verified against a brute-force scan on the same vectors. It does not mean the embeddings capture human relevance. Embedding quality is a separate question this benchmark does not — and cannot — answer. What the benchmark does show is that, on real embeddings with real structure, the bound engine recovers 99.8–100% of the exact top-10 with a per-document proof on every exclusion — and that on the one genuinely corrupt file, it says so.

03The degraded dataset — Word2Vec GoogleNews, 94.7% NaN

Kernel: winnex-madhava-1913-five-datasets-fusoes. The raw file is GoogleNews-vectors-negative300.bin. It is corrupt: 94.7% of its raw vectors are non-finite — and a further 0.32% have zero norm. A naive pipeline would index it, report 0 bound violations (true — the bound is internally consistent), and return garbage with a clean face.

# the engine's own raw diagnosis, before indexing (measured in float32 — the dtype the motor runs on) [diagnóstico] dataset DEGRADADO (bruto): nan_fraction=94.7236% zero_norm_fraction=0.3216% | nan_policy=block_pca (pca_corpus blocked) [quality gate] removed 18,914 degenerate rows from the corpus, 94 from the queries

Two details matter here. First, the diagnosis is computed in float32, not element-wise isfinite on the raw numbers. Large-but-finite values overflow the sum-of-squares and become inf norms — the exact failure the engine would hit. Measuring in the engine's dtype catches what a per-element check misses. Second, the gate removes the degenerate rows and says it did. The number 986 (what survives) is reported as openly as the number 19,900 (what was loaded).

Why PCA is blocked — the corruption class

A single non-finite row entering a PCA basis silently corrupts the whole projection. On this file, the "forced-PCA" path produces recall 0.4833 — worse than the safe preset's 0.7333. Blocking PCA is not a limitation; it is the mechanism that keeps one corrupt row from poisoning every other query. This is the same failure class as the reordered BigANN ground-truth: input corruption that no index can detect on its own.

04Isolate the cause, then apply the preset — maximum fidelity, honestly stated

This is the step the marketing would love to overstate, so let us be precise. The engine does not restore the 94.7% of NaN rows — they are gone, and no algorithm brings back information that was never there. What the preset does is maximize the provable fidelity of the rows that are real, and refuse to trade that for a prettier number.

Configuration on the degraded corpus	Why it is used	Recall@10 (same ceiling)
word2vec preset — `random` basis, `k1=0.20`, `stage1=64`, `early_exit=false`	The widest k1 pool protects recall when the bound is loose; PCA is blocked by nan_policy; no early-exit that would cut the true top-K in high dimension.	0.7333
`pca_corpus` forced (the counterfactual)	What happens if you override the gate and use the "recommended" basis anyway on a NaN-poisoned corpus.	0.4833
`random` basis, `k1=0.05` (the aggressive default)	The default prefilter — on a corpus this damaged, too aggressive.	0.6333

Read the three rows together. The safe preset is not the fastest, and it is not the number a benchmark-fixer would publish. It is the configuration that loses the least real signal: 0.7333 is 51% better than forcing PCA (0.4833) and 16% better than the aggressive default (0.6333). And all three rows carry the same 0 bound violations — which is precisely the point: on corrupt data, 0 violations and 0.48 recall can coexist. That is why the quality gate runs before the engine, as a separate layer.

The honest reading of 0.7333

It is not a failure and it is not a hidden win. It is the measured truth about a corpus in which 95% of the rows were meaningless: of the ~1,000 real rows that survived, the engine recovers 73% of the exact top-10 with zero bound violations. The alternative — claiming 1.0 by indexing the NaN silently — is the lie we built this benchmark to make impossible.

05The certificate — what the 0 violations actually are

"0 bound violations" is the property no ANN index (HNSW, IVF, ScaNN) offers. For every document the engine prunes, it records a witness at the moment of pruning — an upper bound proving that document could not be in the true top-K. The certificate is not a post-hoc judge; it is the search itself, audited.

The mathematics, in one line

For a query q and a document v, with P an orthonormal projection, Cauchy–Schwarz gives an upper bound on the true cosine:

cos(v, q) ≤ ⟨Pv, Pq⟩ + ‖v−PᵀPv‖·‖q−PᵀPq‖ = UB(v, q)

If UB(v,q) < threshold — the exact score of the K-th result — then v mathematically cannot be in the true top-K. The engine records this at prune time. This is what a regulator can audit and what a court can re-check.

Measured across the five datasets (audit commitment, 100 queries each)

Dataset	Queries	Sample bounded	Exact count match	Deterministic	Genuine exclusions	Docs provably excluded
GloVe-Twitter	100	100/100	100/100	100/100	100/100	1,988,609
ProtBERT-CAFA5	100	100/100	100/100	100/100	100/100	664,361
SBERT-Forum	100	100/100	100/100	100/100	100/100	1,983,150
FastText-Crawl	100	100/100	100/100	100/100	100/100	1,853,327
Word2Vec (degraded)	6*	6/6	6/6	6/6	6/6	0 — nothing to prove outside a 986-row pool

*Word2Vec reports only 6 audited queries: after the gate removed 94 of the 100 raw queries (they were NaN too), the surviving query set is 6. That shrinkage is itself a finding — reported, not hidden. On those 6 real queries, every certificate check passes. The row also honestly shows 0 provable exclusions: on a 986-row pool where the bound is loose, the engine does not pretend to have proven more than it proved.

What the fields mean — and why they matter for audit

Sample bounded — every sampled exclusion record stayed under the max sample. The certificate is bounded, not open-ended.
Exact count match — the compact commitment's total equals a full independent audit's count. The summary is not lying about the detail.
Deterministic — two calls with the same query return the identical sample. Reproducible for audit.
Genuine exclusions — every sampled excluded doc has upper_bound < threshold. No false exclusions are reported.

06Bonus, not decoration: concurrent search is bit-identical

The same kernel verifies that search() is safe under concurrent calls on the same engine — a data race in the pruning buffer would make audit records non-reproducible. 32 queries run serially and under 8 threads produce bit-identical results: same indices, same audit ids, same pruned_by_bound, same recall_guarantee. A proof that is not reproducible under load is not a proof.

# thread-safety check (winnex-madhava 1.9.x) 32/32 queries bit-identical serial vs 8 threads PASS

07What this benchmark does not prove — read before you repeat it

1. Ground truth is the embedding space

The recall and NDCG are measured against a brute-force scan over the same vectors. The engine retrieves the geometrically-correct neighbors. It does not prove those neighbors are semantically relevant to a human. Two documents near in vector space but unrelated in meaning will still be "correctly" retrieved.

2. "0 violations" ≠ "recall 1.0" ≠ "data is good"

The bound proves every pruned document was outside the top-K of the scored pool. It says nothing about whether the input vectors mean anything. On the NaN corpus you can have 0 violations and 0.48 recall simultaneously — which is exactly why the quality gate exists as a separate, earlier layer, and why we report both numbers side by side.

3. No healing happens

The engine does not reconstruct the 94.7% of NaN rows. That information does not exist. "Maximum fidelity" means: of the rows that are real, recover as much of the true top-K as the geometry allows, and say precisely how much that is. Anyone who promises you a search engine that "heals" corrupt data is selling you the lie this benchmark exists to expose.

4. Latency is not the selling point

HNSW is faster at raw query latency, and we say so plainly. The trade this benchmark makes explicit is proof: a per-document certificate that a fast approximate index cannot offer. When the question is "how fast can you answer?", HNSW wins. When the question is "can you prove the answer is complete — and can you prove what you did not return?", the fast index has no cargo bay.

The reproducibility contract

Everything here is public and re-runnable: the PyPI packages, the notebook source in the Kaggle kernel, and the full execution logs. If a number looks surprising, run the notebook — it will reproduce. We would rather publish a 0.7333 than claim a 1.0.