[林SO (Linxiushen)]

Describe your changes

Classification metrics currently cast predictions to integers before Torchmetrics applies its threshold or argmax. For example, binary scores [0.9, 0.8, 0.1, 0.2] with targets [1, 1, 0, 0] report accuracy 0.5 and F1/precision/recall 0.0 instead of 1.0.

Preserve floating scores in the four classification metrics while keeping integer conversion for hard labels, including multiclass class-label vectors. Add real scoring regressions for probabilities/logits, custom thresholds, multiclass/multilabel tasks, input representations, fractional scores and averaging, plus AUROC/perplexity controls.

Validation: all 41 new fixed-array CPU scoring regressions pass; the same file gives 18 failures and 23 passes on the untouched base. The combined neighboring tests give 76 passed, 1 failed. The failure is an existing test_metrics_config test requiring the unavailable optional huggingface/evaluate package; it also fails on a fresh archive of the untouched base. Tests used --noconftest to avoid repository-wide model-export fixtures; no model inference or GPU validation. Changed-file lintrunner -a, Ruff 0.15.11 check/format and Pylint 3.3.6 passed.

AI assistance: implementation, regression design and automated review were assisted by OpenAI Codex.

Checklist before requesting a review

• Add unit tests for this change.
• Make sure all tests can pass. The optional-dependency limitation above remains.
• Update documents if necessary. No separate documentation change is needed.
• Lint and apply fixes to your code by running lintrunner -a
• Is this a user-facing change? Release note: fix accuracy/F1/precision/recall evaluation for floating classification probabilities and logits.

(Optional) Issue link

No existing issue; reproduced directly on current main.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟢 Approval recommended

No active review issues are identified for the changes preserving floating-point classification scores while retaining integer conversion for hard labels.

No open issues 🎉

What changed in this PR

This PR fixes Olive’s classification metric evaluation by preserving floating-point probabilities and logits until Torchmetrics applies thresholding or argmax, while retaining integer conversion for hard labels.

Changes:

• Share prediction preparation across accuracy, F1, precision, and recall.
• Add scoring regressions covering input formats, classification tasks, thresholds, averaging, and AUROC/perplexity controls.
• Update mocked tests to expect prediction dtype preservation.

File	Description
test/​evaluator/​test_classification_probabilities.py	Adds real scoring regressions for floating predictions and hard-label compatibility.
test/​evaluator/​test_accuracy.py	Updates prediction tensor-conversion expectations.
olive/​evaluator/​accuracy.py	Preserves floating classification scores while converting hard labels to integers.

_{Review effort: Balanced}