Exact lexical scoring and bounded retrieval

Dataset evaluation prepares lexical token streams and Winnowing fingerprints once per distinct prepared text and configuration. GST indexes exact token prefixes up to its effective minimum match length (capped at eight tokens for bounded index storage), verifies extensions against the full minimum, and skips starts that cannot tie the current longest match. Longest-tile selection, overlap handling, tie order, short-input behavior, and normalization retain their existing semantics. The optimization does not truncate inputs or apply approximate cutoffs.

Two-stage experiments

Use an existing cheap scorer to select candidates, then an existing expensive scorer to rerank them. This reduces expensive comparisons; it does not turn the first stage into an approximate index. First-stage work remains O(QD), while only O(QN) candidate rows and a size-N heap per query are retained. Multivector stages preserve query/document roles without constructing a full Q*D score matrix.

from matheel import evaluate_two_stage_retrieval_dataset

scored, metrics = evaluate_two_stage_retrieval_dataset(
    "normalized_retrieval", candidate_count=5, k=10,
    candidate_similarity_options={"feature_weights": {"winnowing": 1.0}},
    similarity_options={"feature_weights": {"gst": 1.0}},
    exclude_self_matches=True,
)
print(metrics["candidate_recall"], metrics["mean_average_precision"])

The default first stage is LCS and the default reranker is Levenshtein; both run offline. Options files or dictionaries can choose other existing scorers. Python also accepts custom candidate_scorer and reranking scorer callbacks, and a custom reranking algorithm with the existing preparation hook. Candidate callbacks and reranker callbacks receive identifiers, without relevance labels. In this pipeline, custom preparation receives source IDs, file locators, source metadata, prepared texts, and empty qrels. Extra manifest columns and label metadata are omitted. The general custom algorithm API retains its existing preparation contract.

Every declared query and all original qrels contribute to end-to-end metrics. Missed relevant documents lower AP/MAP and recall. Omitted candidates remain absent from the ranking; they are never inserted as zero-score ties. Candidate ties use ascending document IDs. Self exclusion follows file identity and rejects positive self judgments. A query with no remaining candidates still contributes zero retrieval metrics. Queries without positive judgments remain in evaluation; candidate recall is undefined for them and reports their count separately.

The report includes macro candidate recall over queries with positives, micro recall over all positive judgments, per-query diagnostics, selected/full candidate counts, both stage comparison counts and times, and scorer configurations. Candidate recall measures the supplied judgments, whose annotation limits still apply.

matheel evaluate-two-stage-retrieval normalized_retrieval \
  --candidate-count 5 --candidate-feature-weight winnowing=1 \
  --feature-weight gst=1 --k 10 --exclude-self-matches \
  --scores-out experiment/scores.csv --metrics-out experiment/metrics.json

--candidate-options and --reranker-options accept JSON scorer-options files, including language, tokenizer, preparation, and model settings. Feature-weight flags override those files' feature weights. The scores CSV contains both candidate score/rank and final similarity; the JSON contains metrics and full diagnostics. Run examples/evaluation/two_stage_retrieval_demo.py for a complete synthetic example that needs neither downloads nor models.

Custom leaderboard methods can declare retrieval_options:

{
  "name": "Winnowing top5 then GST",
  "feature_weights": {"gst": 1.0},
  "retrieval_options": {
    "candidate_count": 5,
    "candidate_similarity_options": {"feature_weights": {"winnowing": 1.0}}
  }
}

The complete pipeline policy participates in method identity. Pipeline reports are separate experiments. The frozen exhaustive v1 runner rejects shortlist options; existing v1 protocols, scores, timeouts, and published artifacts remain unchanged. Use a new artifact directory for every new experiment.

Predeclare applicability

Methods can declare task exclusions before scoring:

{
  "name": "GST for pairs",
  "feature_weights": {"gst": 1.0},
  "task_eligibility": {
    "retrieval": {"status": "not_applicable", "reason": "Exhaustive retrieval is outside this experiment's runtime scope."}
  },
  "dataset_exclusions": {"large_pairs": "This experiment includes small pair tasks only."}
}

Task keys are pair or retrieval; status is eligible or not_applicable, with a nonempty reason required for exclusions. Dataset exclusions use declared dataset names. Omitted fields retain existing applicability and method identities. The declarations themselves participate in new method identities.

Excluded rows have status not_applicable, the declared reason, zero comparisons, and null score/rank. Runtime errors remain failed. Any exclusion in an aggregate's declared scope suppresses its mean and rank, even when legacy partial averaging is enabled. Other applicable task families can still rank normally. Frozen-result validation checks exclusions against the recorded method configuration; a failed run cannot be relabeled N/A after execution. Older published rows without status remain valid through their existing score/error/eligibility fields.

examples/evaluation/retrieval_methods.example.json provides a conservative new experiment policy for the five methods that exceeded v1's 1,800-second method budget: GST, Lexical Only, Partial Ratio, Partial Token Sort Ratio, and WRatio. It is an example scope declaration, not a global restriction or a claim that pair scoring will always fit the budget. Review runtime evidence before widening scope.

Measured feasibility example

Bounded reference runs used the complete SOCO14 C++ A2 judged-endpoint view: 145 queries and 145 corpus documents, with 20,880 allowed comparisons after self exclusion. Inputs came from the frozen v1 preparation recipe. The reference environment was CPython 3.12 on macOS arm64, with numerical-library threads limited to one. These are separate experiments with their own output directories.

Configuration	Budget	Outcome	First-stage comparisons / seconds	Completed reranks / seconds	Macro candidate recall
Exhaustive GST	60 s	Timed out	—	Not fully measured	—
Exhaustive Lexical Only	60 s	Timed out	—	Not fully measured	—
Exhaustive Partial Ratio	60 s	Timed out	—	Not fully measured	—
LCS top 10 → GST	90 s	Timed out	Not fully reported	Not fully measured	Not reported
LCS top 1 → GST	60 s	Completed in 31.75 s	20,880 / 0.901 s	145 / 29.30 s	0.7713
LCS top 10 → Partial Ratio	90 s	Timed out	Not fully reported	Not fully measured	Not reported
LCS top 1 → Partial Ratio	60 s	Completed in 10.47 s	20,880 / 0.924 s	145 / 7.93 s	0.7713
Winnowing top 5 → GST	90 s	Completed in 34.67 s	20,880 / 0.417 s	725 / 32.66 s	1.0
Winnowing top 5 → Partial Ratio	90 s	Completed in 26.75 s	20,880 / 0.424 s	725 / 24.72 s	1.0

The completed top-1 experiments retained 122 of 172 positive judgments (micro candidate recall 0.7093). All 145 queries and 172 positives remained in the final metric denominators; MAP was 0.7713. A top-1 shortlist demonstrates a runtime/recall tradeoff and supplies fewer than the requested k=10 results. Choose and record N and the time budget before evaluation. Candidate recall is diagnostic and does not control admission, thresholds, retained methods, or ranks. Runtime choices were bounded in advance; these observations do not alter the original 1,800-second v1 run or establish feasibility outside the tested configurations. The predeclared Winnowing top-5 experiments retained all 172 positive judgments in this view; MAP was 0.9671 with GST and 0.9327 with Partial Ratio. The candidate count and budget were chosen before observing those recall values. Both top-1 and top-5 shortlists supply fewer than ten results at k=10; precision retains its declared denominator. Inspect query-level diagnostics and repeat on other data before making broader performance or quality claims.

Compare presets across the existing judged views

The 0.7.1 retrieval-preset experiment declares shortlist sizes and deadlines before scoring. It covers every existing retrieval view: IRPlag Java and the ten nonempty SOCO14 C++/Java judged-endpoint views. The source query/corpus/qrel tables and labels are retained. The seven configurations compare an exhaustive Winnowing control, Winnowing top 1/5/10 followed by Levenshtein, LCS top 5 followed by Levenshtein, and Winnowing top 5 followed by GST or RapidFuzz Partial Ratio.

To replay with locally prepared data, use a new output directory:

OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 \
  python scripts/validate_retrieval_presets.py \
    --manifest prepared/prepared-manifest.json \
    --plan benchmarks/retrieval-presets/plagiarism-071/plan.json \
    --source-root ../matheel-0.7.1 --release-wheel dist/matheel-0.7.1-py3-none-any.whl \
    --cpu-lock /private/tmp/matheel-071-benchmark-cpu.lock \
    --output-dir benchmark_outputs/retrieval-071

Create the source checkout with git worktree add --detach ../matheel-0.7.1 v0.7.1 and obtain the published wheel with python -m pip download --no-deps matheel==0.7.1 --dest dist. This POSIX runner checks the exact release checkout and recursively matches its Python modules to that wheel. It hashes each normalized table, metadata and source input, then checks those hashes again in each worker. Each configuration/view gets its own 45-second process deadline. Public outputs contain source-free scores, per-query candidate diagnostics, input/configuration identities and an environment snapshot. Timeouts retain null quality metrics and remain timeouts. They cannot become N/A after execution. The exhaustive control also exports grouped uncertainty and its relevant results below rank ten through the error-analysis API. The recorded macOS runs use the explicit shared lock above; on another POSIX machine, choose one writable absolute lock path and use it for every benchmark process. Provenance records wait/acquisition/finish timestamps after acquisition. Deadline cleanup stops each worker's complete process group.

The recorded run retained all 77 cells: 76 completed and Partial Ratio timed out on Java A1 at 45 seconds. GST top 5 completed all eleven views, with a maximum scoring time of 33.05 seconds. Exhaustive Winnowing's maximum was 0.49 seconds; its MAP ranged from 0.9530 to 1.0 over the eleven views. These single-observation times describe the recorded hardware and input pools.

Winnowing top 5 and top 10 both missed a judged positive on C++ A1 (macro candidate recall 0.992063); they retained all positives in the other ten views. On IRPlag, Levenshtein's final MAP decreased from 0.929953 at N=5 to 0.924842 at N=10 while candidate recall stayed 1.0. Keep the full cheap scorer as a control: candidate coverage and final ordering answer different questions. Per-view results and a verified complete evidence ZIP are linked from the experiment.

scoring_seconds times scoring and metric computation; wall_seconds also includes worker startup, validation, exports and the control's error analysis. Stage times include the stage's representation preparation. These are single observations on one machine, rather than stable latency percentiles or cross-hardware guarantees.

Choose a preset by the constraints of the new experiment:

Need	Starting configuration	What to inspect
Cheap reference over the complete corpus	Exhaustive Winnowing	Full-ranking MAP, missed positives and score ties
A complete top-ten reranking window	Winnowing top 10 → Levenshtein	Candidate recall per query and final MAP; increase N in a new declared run if required
A smaller expensive-stage budget	Winnowing top 5 → Levenshtein, GST or Partial Ratio	Runtime, missed judgments and fewer-than-ten returned rows
Sensitivity to the first-stage choice	LCS top 5 → Levenshtein	Compare the same N/reranker against Winnowing top 5
A deliberately severe budget diagnostic	Winnowing top 1 → Levenshtein	The recall ceiling and missing ranks; this cannot supply a ten-result list

All relevant judgments remain in the final denominators. Candidate recall alone does not establish good final ordering, and reranking need not improve a cheap scorer's MAP. A shortlist cannot exceed the number of eligible documents: IRPlag has seven originals, so its N=10 run scans and reranks those seven. Precision@10 keeps its declared denominator even when a corpus or shortlist supplies fewer than ten results. Treat the SOCO views as their recorded judged pools, rather than full competition corpora. Select N and a budget for the intended workload before measuring that workload; the recorded experiments do not automatically select or admit a method based on its observed recall.

Reproduce a bounded GST profile

From a checkout containing the trusted v0.7.0 tag:

python scripts/profile_lexical_scoring.py --baseline-ref v0.7.0 --seconds 5
python scripts/profile_lexical_scoring.py --pair-dataset normalized_pairs --seconds 5

Each before/after case runs in a bounded subprocess and records scores, token counts, input hashes, wall time, and cProfile call counts. Optional real inputs select the smallest, middle, and largest supplied pair by character length. Source text is not exported. A deadline is recorded as a timeout, and parity is asserted only when both implementations finish. Exactness is also covered by an independent exhaustive GST oracle and deterministic adversarial/random cases.

An initial synthetic profile with 750 punctuation-heavy tokens retained score 0 and reduced cProfile calls from 630,017 to 4,489 (0.115 s to 0.00128 s in one observation). An identical 1,200-token case retained score 1 and measured 0.125 s before versus 0.00110 s after. These fixtures exercise exact pruning; the real feasibility runs above show why a synthetic speedup alone does not establish corpus-scale suitability. A separate cached-score check compared 20 full-source SOCO14 C++ A2 observations each for Levenshtein, LCS, Winnowing, and Partial Token Ratio: all 80 floats matched the published CSV values exactly.