Describe the bug
Two edge-case defects in deeptab/metrics/classification.py, both verified by execution:
ExpectedCalibrationError ignores samples with confidence exactly 1.0 (classification.py:220-221). Every bin is half-open [lo, hi), including the last [0.9, 1.0), so confidence-1.0 predictions (saturated softmax, or hard labels passed as probabilities) fall into no bin and contribute 0. A model that predicts the wrong class with probability 1.0 on every sample — worst possible calibration — scores ECE = 0.0 (perfect).Accuracy and F1Score binarize 1-D input at 0.5 (classification.py:79, 104): (y_pred.ravel() >= 0.5).astype(int) is applied to any 1-D input, although the docstrings promise 1-D integer label support. Multiclass labels get mangled (label 2 → 1): Accuracy()([0,1,2,2], [0,1,2,2]) — perfect predictions — returns 0.5. Binary 0/1 labels happen to survive the threshold, hiding the bug.
To Reproduce
importnumpyasnpfromdeeptab.metrics.classificationimportExpectedCalibrationError, Accuracy# (1) wrong class with confidence 1.0 everywhere -> ECE says perfectly calibratedy_true=np.array([0, 0, 0, 0])
proba=np.array([[0.0, 1.0]] *4)
print(ExpectedCalibrationError()(y_true, proba)) # 0.0, should be 1.0# (2) perfect multiclass label predictions -> 50% accuracyprint(Accuracy()(np.array([0, 1, 2, 2]), np.array([0, 1, 2, 2]))) # 0.5, should be 1.0
Expected behavior
The final ECE bin must be right-inclusive ([0.9, 1.0]); 1-D inputs to Accuracy/F1 should be treated as class labels when integer-valued (thresholding only makes sense for 1-D probability scores, which could be detected or made explicit via a parameter).
Screenshots
n/a
Desktop (please complete the following information):
- OS: macOS (Darwin 25.5.0, arm64)
- Python version: 3.11.15
- deeptab Version: 2.0.0 (main @ 4e6a359)
Additional context
Bug 2 bites the natural composition Accuracy()(y, model.predict(X)), since predict returns 1-D labels. Related registry hazard (not a bug yet): get_default_metrics returns the registry's own mutable list of shared metric instances (metrics/registry.py:75) — a caller mutating it corrupts the global registry.
Describe the bug
Two edge-case defects in
deeptab/metrics/classification.py, both verified by execution:ExpectedCalibrationErrorignores samples with confidence exactly 1.0 (classification.py:220-221). Every bin is half-open[lo, hi), including the last[0.9, 1.0), so confidence-1.0 predictions (saturated softmax, or hard labels passed as probabilities) fall into no bin and contribute 0. A model that predicts the wrong class with probability 1.0 on every sample — worst possible calibration — scores ECE = 0.0 (perfect).AccuracyandF1Scorebinarize 1-D input at 0.5 (classification.py:79, 104):(y_pred.ravel() >= 0.5).astype(int)is applied to any 1-D input, although the docstrings promise 1-D integer label support. Multiclass labels get mangled (label 2 → 1):Accuracy()([0,1,2,2], [0,1,2,2])— perfect predictions — returns 0.5. Binary 0/1 labels happen to survive the threshold, hiding the bug.To Reproduce
Expected behavior
The final ECE bin must be right-inclusive (
[0.9, 1.0]); 1-D inputs to Accuracy/F1 should be treated as class labels when integer-valued (thresholding only makes sense for 1-D probability scores, which could be detected or made explicit via a parameter).Screenshots
n/a
Desktop (please complete the following information):
Additional context
Bug 2 bites the natural composition
Accuracy()(y, model.predict(X)), sincepredictreturns 1-D labels. Related registry hazard (not a bug yet):get_default_metricsreturns the registry's own mutable list of shared metric instances (metrics/registry.py:75) — a caller mutating it corrupts the global registry.