Metric names and icons

[ecomodeller]

The skore project is designed to make evaluation of ML models easy.
One feature is nice skill tables were the direction of better is indicated for clarity.

skore uses a simple dict to store this info.
https://github.com/probabl-ai/skore/blob/b5f84adb03cfb659233d3085739b33ab086710d9/skore/src/skore/sklearn/_estimator/metrics_accessor.py#L27-L38

Should we add a similar feature to ModelSkill?

[jsmariegaard]

defmetric(best=None):
"""Decorator to attach a 'best' attribute to metric functions."""defdecorator(func):
func.best=best# Set the best valuereturnfuncreturndecorator@metric(best=0)defmean_error(obs, model):
return (model-obs).mean()
@metric(best="+")defr2_score(obs, model):
return1- ((obs-model)**2).sum() / ((obs-obs.mean())**2).sum()
# User-defined metric without 'best'defuser_metric(obs, model):
return (obs-model).std()
# Function to get best value safelydefget_best(metric_func):
"""Returns the 'best' attribute if it exists, otherwise None."""returngetattr(metric_func, "best", None)
# Usageprint(get_best(mean_error)) # Output: 0print(get_best(r2_score)) # Output: "+"print(get_best(user_metric)) # Output: None (no best defined)

[ecomodeller]

Is this how you envision this decorator to be used for user defined metrics?

frommodelskill.metricsimportmetric, add_metric@metric(best="-")defsilly(obs,model):
returnnp.random.uniform()
add_metric(silly) # this step seems unnecessary, could this be part of the decorator?cc.skill(metrics=[silly]).style()

[jsmariegaard]

Nice. I think the add_metric could be part of the decorator too and potentially more behaviour in the future. Seems like a nice way to register functions as metrics :-)

[ecomodeller]

The metric decorator was introduced in #505 but we didn't add helpful icons so far.