credible.metrics

Common interface for Scikit-Learn compatible measures with uncertainity estimates.

This module provides wrappers around sklearn.metrics that augment standard performance measures with uncertainty quantification through either Bayesian credible intervals or frequentist confidence intervals, depending on the selected method argument.

Functions

accuracy_score(y_true, y_pred[, coverage, ...])

Accuracy binary classification score.

average_precision_score(y_true, y_score[, ...])

Compute average precision (AP) from prediction scores.

det_curve(y_true, y_score[, coverage, method])

Compute the Detection Error-Tradeoff (DET) curve.

f1_score(y_true, y_pred[, coverage, method])

Return the mean, mode, upper and lower bounds of the credible region of the F1 score.

jaccard_score(y_true, y_pred[, coverage, method])

Jaccard binary classification score.

precision_recall_curve(y_true, y_score[, ...])

Compute Precision-Recall (PR) curve.

precision_score(y_true, y_pred[, coverage, ...])

Precision binary classification score.

recall_score(y_true, y_pred[, coverage, method])

Recall binary classification score.

roc_auc_score(y_true, y_score[, coverage, ...])

Calculate the area under the ROC (FPR vs TPR) curve.

roc_curve(y_true, y_score[, coverage, method])

Compute Receiver operating characteristic (ROC).

specificity_score(y_true, y_pred[, ...])

Specificity binary classification score.
credible.metrics.precision_score(y_true, y_pred, coverage=0.95, method='bayesian', **kwargs)[source]

Precision binary classification score.

AKA positive predictive value (PPV). It corresponds arithmetically to tp/(tp+fp). This function only supports binary classification problems.

Parameters:

  • y_true (Iterable[int]) – Ground truth (correct) labels.

  • y_pred (Iterable[int]) – Predicted labels, as returned by a classifier.

  • coverage (float) –

    A floating-point number between 0 and 1 indicating the desired interval coverage.

    • For the bayesian method, a value of 0.95 will ensure 95% of the area under the probability density of the posterior is covered by the returned equal-tailed interval.

    • For the frequentist method, a value of 0.95 returns a percentile bootstrap confidence interval containing the central 95% of the bootstrap distribution.

  • method (str) – The method used to compute the results, either frequentist or bayesian.

  • **kwargs –

    Additional arguments to be passed. In particular:

    • lambda_ (bayesian): The parameterisation of the Beta prior to consider.

    • n_bootstraps (frequentist): Number of bootstrap replicates.

    • require_all_classes (frequentist): Whether each accepted bootstrap sample must contain all classes present in y_true.

    • max_resample_attempts (frequentist): Maximum number of resampling attempts. If require_all_classes is True, this parameter limits the number of times the function will attempt to resample the data to obtain a bootstrap sample that contains all classes.

    • rng (frequentist): An initialized numpy random number generator.

    Any remaining keyword arguments are passed to sklearn.metrics.precision_score. For additional details on the supported arguments and their behavior, refer to the scikit-learn documentation.

Returns:

Tuple with 4 floating-point numbers:

  • The actual precision, as would be returned by scikit-learn

  • The mode of the estimated distribution. For the Bayesian method, this is the mode of the posterior distribution. For the frequentist method, this is the peak of a Gaussian KDE fitted to the bootstrap samples.

  • The lower value of the credible region/confidence interval

  • The upper value of the credible region/confidence interval

Return type:

tuple[float, float, float, float]

credible.metrics.recall_score(y_true, y_pred, coverage=0.95, method='bayesian', **kwargs)[source]

Recall binary classification score.

AKA sensitivity, hit rate, or true positive rate (TPR). It corresponds arithmetically to tp/(tp+fn).

Parameters:

  • y_true (Iterable[int]) – Ground truth (correct) labels.

  • y_pred (Iterable[int]) – Predicted labels, as returned by a classifier.

  • coverage (float) –

    A floating-point number between 0 and 1 indicating the desired interval coverage.

    • For the bayesian method, a value of 0.95 will ensure 95% of the area under the probability density of the posterior is covered by the returned equal-tailed interval.

    • For the frequentist method, a value of 0.95 returns a percentile bootstrap confidence interval containing the central 95% of the bootstrap distribution.

  • method (str) – The method used to compute the results, either frequentist or bayesian.

  • **kwargs –

    Additional arguments to be passed. In particular:

    • lambda_ (bayesian): The parameterisation of the Beta prior to consider.

    • n_bootstraps (frequentist): Number of bootstrap replicates.

    • require_all_classes (frequentist): Whether each accepted bootstrap sample must contain all classes present in y_true.

    • max_resample_attempts (frequentist): Maximum number of resampling attempts. If require_all_classes is True, this parameter limits the number of times the function will attempt to resample the data to obtain a bootstrap sample that contains all classes.

    • rng (frequentist): An initialized numpy random number generator.

    Any remaining keyword arguments are passed to sklearn.metrics.recall_score. For additional details on the supported arguments and their behavior, refer to the scikit-learn documentation.

Returns:

Tuple with 4 floating-point numbers:

  • The actual recall, as would be returned by scikit-learn

  • The mode of the estimated distribution. For the Bayesian method, this is the mode of the posterior distribution. For the frequentist method, this is the peak of a Gaussian KDE fitted to the bootstrap samples.

  • The lower value of the credible region/confidence interval

  • The upper value of the credible region/confidence interval

Return type:

tuple[float, float, float, float]

credible.metrics.specificity_score(y_true, y_pred, coverage=0.95, method='bayesian', **kwargs)[source]

Specificity binary classification score.

AKA selectivity or true negative rate (TNR). It corresponds arithmetically to tn/(tn+fp).

Parameters:

  • y_true (Iterable[int]) – Ground truth (correct) labels.

  • y_pred (Iterable[int]) – Predicted labels, as returned by a classifier.

  • coverage (float) –

    A floating-point number between 0 and 1 indicating the desired interval coverage.

    • For the bayesian method, a value of 0.95 will ensure 95% of the area under the probability density of the posterior is covered by the returned equal-tailed interval.

    • For the frequentist method, a value of 0.95 returns a percentile bootstrap confidence interval containing the central 95% of the bootstrap distribution.

  • method (str) – The method used to compute the results, either frequentist or bayesian.

  • **kwargs –

    Additional arguments to be passed. In particular:

    • lambda_ (bayesian): The parameterisation of the Beta prior to consider.

    • n_bootstraps (frequentist): Number of bootstrap replicates.

    • require_all_classes (frequentist): Whether each accepted bootstrap sample must contain all classes present in y_true.

    • max_resample_attempts (frequentist): Maximum number of resampling attempts. If require_all_classes is True, this parameter limits the number of times the function will attempt to resample the data to obtain a bootstrap sample that contains all classes.

    • rng (frequentist): An initialized numpy random number generator.

    Any remaining keyword arguments are passed to sklearn.metrics.recall_score (this because the specificity is computed as the recall of the negative class). For additional details on the supported arguments and their behavior, refer to the scikit-learn documentation.

Returns:

Tuple with 4 floating-point numbers:

  • The actual specificity score computed via scikit-learn recall_score setting pos_label=0

  • The mode of the estimated distribution. For the Bayesian method, this is the mode of the posterior distribution. For the frequentist method, this is the peak of a Gaussian KDE fitted to the bootstrap samples.

  • The lower value of the credible region/confidence interval

  • The upper value of the credible region/confidence interval

Return type:

tuple[float, float, float, float]

credible.metrics.accuracy_score(y_true, y_pred, coverage=0.95, method='bayesian', **kwargs)[source]

Accuracy binary classification score.

See Accuracy. is the proportion of correct predictions (both true positives and true negatives) among the total number of pixels examined. It corresponds arithmetically to (tp+tn)/(tp+tn+fp+fn). This measure includes both true-negatives and positives in the numerator, what makes it sensitive to data or regions without annotations. AKA selectivity or true negative rate (TNR). It corresponds arithmetically to tn/(tn+fp).

Parameters:

  • y_true (Iterable[int]) – Ground truth (correct) labels.

  • y_pred (Iterable[int]) – Predicted labels, as returned by a classifier.

  • coverage (float) –

    A floating-point number between 0 and 1 indicating the desired interval coverage.

    • For the bayesian method, a value of 0.95 will ensure 95% of the area under the probability density of the posterior is covered by the returned equal-tailed interval.

    • For the frequentist method, a value of 0.95 returns a percentile bootstrap confidence interval containing the central 95% of the bootstrap distribution.

  • method (str) – The method used to compute the results, either frequentist or bayesian.

  • **kwargs –

    Additional arguments to be passed. In particular:

    • lambda_ (bayesian): The parameterisation of the Beta prior to consider.

    • n_bootstraps (frequentist): Number of bootstrap replicates.

    • require_all_classes (frequentist): Whether each accepted bootstrap sample must contain all classes present in y_true.

    • max_resample_attempts (frequentist): Maximum number of resampling attempts. If require_all_classes is True, this parameter limits the number of times the function will attempt to resample the data to obtain a bootstrap sample that contains all classes.

    • rng (frequentist): An initialized numpy random number generator.

    Any remaining keyword arguments are passed to sklearn.metrics.accuracy_score. For additional details on the supported arguments and their behavior, refer to the scikit-learn documentation.

Returns:

Tuple with 4 floating-point numbers:

  • The actual accuracy, as would be returned by scikit-learn

  • The mode of the estimated distribution. For the Bayesian method, this is the mode of the posterior distribution. For the frequentist method, this is the peak of a Gaussian KDE fitted to the bootstrap samples.

  • The lower value of the credible region/confidence interval

  • The upper value of the credible region/confidence interval

Return type:

tuple[float, float, float, float]

credible.metrics.jaccard_score(y_true, y_pred, coverage=0.95, method='bayesian', **kwargs)[source]

Jaccard binary classification score.

See Jaccard Index or Similarity. It corresponds arithmetically to tp/(tp+fp+fn). The Jaccard index depends on a TP-only numerator, similarly to the F1 score. For regions where there are no annotations, the Jaccard index will always be zero, irrespective of the model output. Accuracy may be a better proxy if one needs to consider the true abscence of annotations in a region as part of the measure.

Parameters:

  • y_true (Iterable[int]) – Ground truth (correct) labels.

  • y_pred (Iterable[int]) – Predicted labels, as returned by a classifier.

  • coverage (float) –

    A floating-point number between 0 and 1 indicating the desired interval coverage.

    • For the bayesian method, a value of 0.95 will ensure 95% of the area under the probability density of the posterior is covered by the returned equal-tailed interval.

    • For the frequentist method, a value of 0.95 returns a percentile bootstrap confidence interval containing the central 95% of the bootstrap distribution.

  • method (str) – The method used to compute the results, either frequentist or bayesian.

  • **kwargs –

    Additional arguments to be passed. In particular:

    • lambda_ (bayesian): The parameterisation of the Beta prior to consider.

    • n_bootstraps (frequentist): Number of bootstrap replicates.

    • require_all_classes (frequentist): Whether each accepted bootstrap sample must contain all classes present in y_true.

    • max_resample_attempts (frequentist): Maximum number of resampling attempts. If require_all_classes is True, this parameter limits the number of times the function will attempt to resample the data to obtain a bootstrap sample that contains all classes.

    • rng (frequentist): An initialized numpy random number generator.

    Any remaining keyword arguments are passed to sklearn.metrics.jaccard_score. For additional details on the supported arguments and their behavior, refer to the scikit-learn documentation.

Returns:

Tuple with 4 floating-point numbers:

  • The actual jaccard score, as would be returned by scikit-learn

  • The mode of the estimated distribution. For the Bayesian method, this is the mode of the posterior distribution. For the frequentist method, this is the peak of a Gaussian KDE fitted to the bootstrap samples.

  • The lower value of the credible region/confidence interval

  • The upper value of the credible region/confidence interval

Return type:

tuple[float, float, float, float]

credible.metrics.f1_score(y_true, y_pred, coverage=0.95, method='bayesian', **kwargs)[source]

Return the mean, mode, upper and lower bounds of the credible region of the F1 score.

See F1-score. It corresponds arithmetically to 2*P*R/(P+R) or 2*tp/(2*tp+fp+fn). The F1 or Dice score depends on a TP-only numerator, similarly to the Jaccard index. For regions where there are no annotations, the F1-score will always be zero, irrespective of the model output. Accuracy may be a better proxy if one needs to consider the true abscence of annotations in a region as part of the measure.

This implementation is based on [GOUTTE-2005].

Parameters:

  • y_true (Iterable[int]) – Ground truth (correct) labels.

  • y_pred (Iterable[int]) – Predicted labels, as returned by a classifier.

  • coverage (float) –

    A floating-point number between 0 and 1 indicating the desired interval coverage.

    • For the bayesian method, a value of 0.95 will ensure 95% of the area under the probability density of the posterior is covered by the returned equal-tailed interval.

    • For the frequentist method, a value of 0.95 returns a percentile bootstrap confidence interval containing the central 95% of the bootstrap distribution.

  • method (str) – The method used to compute the results, either frequentist or bayesian.

  • **kwargs –

    Additional arguments to be passed. In particular:

    • lambda_ (bayesian): The parameterisation of the Beta prior to consider.

    • n_bootstraps (frequentist): Number of bootstrap replicates.

    • require_all_classes (frequentist): Whether each accepted bootstrap sample must contain all classes present in y_true.

    • max_resample_attempts (frequentist): Maximum number of resampling attempts. If require_all_classes is True, this parameter limits the number of times the function will attempt to resample the data to obtain a bootstrap sample that contains all classes.

    • rng (frequentist): An initialized numpy random number generator.

    Any remaining keyword arguments are passed to sklearn.metrics.f1_score. For additional details on the supported arguments and their behavior, refer to the scikit-learn documentation.

Returns:

Tuple with 4 floating-point numbers:

  • The actual f1 score, as would be returned by scikit-learn

  • The mode of the estimated distribution. For the Bayesian method, this is the mode of the posterior distribution. For the frequentist method, this is the peak of a Gaussian KDE fitted to the bootstrap samples.

  • The lower value of the credible region/confidence interval

  • The upper value of the credible region/confidence interval

Return type:

tuple[float, float, float, float]

credible.metrics.roc_curve(y_true, y_score, coverage=0.95, method='bayesian', **kwargs)[source]

Compute Receiver operating characteristic (ROC).

Approximately follows API of sklearn.metrics.roc_curve().

Note

The frequentist implementation is currently not available for this metric. At present, only the bayesian method is supported.

Important

The returned credible regions are not immediately usable for plots or the evaluation of the area under the curve, only as point estimates for individual thresholds. To plot, feed the output of this funtion to curves.curve_ci_hull() and use the lower and upper estimates provided by that function instead.

Parameters:

  • y_true (Iterable[int]) – Ground truth (correct) labels.

  • y_score (Iterable[float]) – Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).

  • coverage (float) – A floating-point number between 0 and 1.0 indicating the coverage you are expecting. A value of 0.95 will ensure 95% of the area under the probability density of the posterior is covered by the returned equal-tailed interval.

  • method (str) – The method used to compute the results, either frequentist or bayesian.

  • **kwargs –

    Additional arguments to be passed. In particular:

    • lambda_ (bayesian): The parameterisation of the Beta prior to consider.

    • rng (frequentist): An initialized numpy random number generator.

Return type:

tuple[GenericAlias[float64], GenericAlias[float64], GenericAlias[float64], GenericAlias[float64], GenericAlias[float64], GenericAlias[float64], GenericAlias[float64]]

Returns:

Seven 1-D floating point arrays corresponding to:

  • FPR (false positive rates)

  • TPR (true positive rates)

  • The thresholds used to evaluated the selected metrics

  • The lower confidence interval for the FPR

  • The lower confidence interval for the TPR

  • The upper confidence interval for the FPR

  • The upper confidence interval for the TPR

credible.metrics.roc_auc_score(y_true, y_score, coverage=0.95, method='bayesian', **kwargs)[source]

Calculate the area under the ROC (FPR vs TPR) curve.

This function mimics the scikit-learn API, except it also returns lower and upper bounds of uncertainity.

For method="bayesian", the returned bounds corrspond to credible regions defined in each threshold. For method="frequentist", they correspond to confidence intervals computed via percentile bootstrap.

Parameters:

  • y_true (Iterable[int]) – Ground truth (correct) labels.

  • y_score (Iterable[float]) – Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).

  • coverage (float) –

    A floating-point number between 0 and 1 indicating the desired interval coverage.

    • For the bayesian method, a value of 0.95 will ensure 95% of the area under the probability density of the posterior is covered by the returned equal-tailed interval.

    • For the frequentist method, a value of 0.95 returns a percentile bootstrap confidence interval containing the central 95% of the bootstrap distribution.

  • method (str) – The method used to compute the results, either frequentist or bayesian.

  • **kwargs –

    Additional arguments to be passed. In particular:

    • lambda_ (bayesian): The parameterisation of the Beta prior to consider.

    • n_bootstraps (frequentist): Number of bootstrap replicates.

    • require_all_classes (frequentist): Whether each accepted bootstrap sample must contain all classes present in y_true.

    • max_resample_attempts (frequentist): Maximum number of resampling attempts. If require_all_classes is True, this parameter limits the number of times the function will attempt to resample the data to obtain a bootstrap sample that contains all classes.

    • rng (frequentist): An initialized numpy random number generator.

    Any remaining keyword arguments are passed to sklearn.metrics.roc_auc_score. For additional details on the supported arguments and their behavior, refer to the scikit-learn documentation.

Return type:

tuple[float, float, float]

Returns:

A tuple with 3 floats:

  • The area under the ROC (FPR vs. TPR) curve

  • The lower bound of the estimated interval. For method="bayesian", this corresponds to the lower bound of the credible interval. For method="frequentist", this corresponds to the lower bound of the bootstrap confidence interval.

  • The upper bound of the estimated interval. For method="bayesian", this corresponds to the upper bound of the credible interval. For method="frequentist", this corresponds to the upper bound of the bootstrap confidence interval.

credible.metrics.det_curve(y_true, y_score, coverage=0.95, method='bayesian', **kwargs)[source]

Compute the Detection Error-Tradeoff (DET) curve.

Approximately follows API of sklearn.metrics.det_curve().

Note

The frequentist implementation is currently not available for this metric. At present, only the bayesian method is supported.

Important

The returned credible regions are not immediately usable for plots or the evaluation of the area under the curve, only as point estimates for individual thresholds. To plot, feed the output of this funtion to curves.curve_ci_hull() and use the lower and upper estimates provided by that function instead.

Parameters:

  • y_true (Iterable[int]) – Ground truth (correct) labels.

  • y_score (Iterable[float]) – Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).

  • coverage (float) – A floating-point number between 0 and 1.0 indicating the coverage you are expecting. A value of 0.95 will ensure 95% of the area under the probability density of the posterior is covered by the returned equal-tailed interval.

  • method (str) – The method used to compute the results, either frequentist or bayesian.

  • **kwargs –

    Additional arguments to be passed. In particular:

    • lambda_ (bayesian): The parameterisation of the Beta prior to consider.

    • rng (frequentist): An initialized numpy random number generator.

    The rest are passed to the scikit-learn metric function. For additional details, see the corresponding methods in the respective modules.

Return type:

tuple[GenericAlias[float64], GenericAlias[float64], GenericAlias[float64], GenericAlias[float64], GenericAlias[float64], GenericAlias[float64], GenericAlias[float64]]

Returns:

Seven 1-D floating point arrays corresponding to:

  • FPR (false positive rates)

  • FNR (false negative rates)

  • The thresholds used to evaluated the selected metrics

  • The lower confidence interval for the FPR

  • The lower confidence interval for the FNR

  • The upper confidence interval for the FPR

  • The upper confidence interval for the FNR

credible.metrics.precision_recall_curve(y_true, y_score, coverage=0.95, method='bayesian', **kwargs)[source]

Compute Precision-Recall (PR) curve.

Approximately follows API of sklearn.metrics.precision_recall_curve().

Note

The frequentist implementation is currently not available for this metric. At present, only the bayesian method is supported.

Note

This package computes the precision-recall curve in a similar, but slightly different way than scikit-learn. It does not add an extra (1.0, 0.0) at the end of the PR curve. (c.f.: documentation for sklearn.metrics.precision_recall_curve()).

Important

The returned credible regions are not immediately usable for plots or the evaluation of the area under the curve, only as point estimates for individual thresholds. To plot, feed the output of this funtion to curves.curve_ci_hull() and use the lower and upper estimates provided by that function instead.

Parameters:

  • y_true (Iterable[int]) – Ground truth (correct) labels.

  • y_score (Iterable[float]) – Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).

  • coverage (float) – A floating-point number between 0 and 1.0 indicating the coverage you are expecting. A value of 0.95 will ensure 95% of the area under the probability density of the posterior is covered by the returned equal-tailed interval.

  • method (str) – The method used to compute the results, either frequentist or bayesian.

  • **kwargs –

    Additional arguments to be passed. In particular:

    • lambda_ (bayesian): The parameterisation of the Beta prior to consider.

    • rng (frequentist): An initialized numpy random number generator.

    The rest are passed to the scikit-learn metric function. For additional details, see the corresponding methods in the respective modules.

Return type:

tuple[GenericAlias[float64], GenericAlias[float64], GenericAlias[float64], GenericAlias[float64], GenericAlias[float64], GenericAlias[float64], GenericAlias[float64]]

Returns:

Seven 1-D floating point arrays corresponding to:

  • Precision

  • Recall

  • The thresholds used to evaluated the selected metrics

  • The lower confidence interval for the Precision

  • The lower confidence interval for the Recall

  • The upper confidence interval for the Precision

  • The upper confidence interval for the Recall

credible.metrics.average_precision_score(y_true, y_score, coverage=0.95, method='bayesian', **kwargs)[source]

Compute average precision (AP) from prediction scores.

This function mimics the scikit-learn API, except it also returns lower and upper bounds of uncertainity.

For method="bayesian", the returned bounds corrspond to credible regions defined in each threshold. For method="frequentist", they correspond to confidence intervals computed via percentile bootstrap.

Parameters:

  • y_true (Iterable[int]) – Ground truth (correct) labels.

  • y_score (Iterable[float]) – Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).

  • coverage (float) –

    A floating-point number between 0 and 1 indicating the desired interval coverage.

    • For the bayesian method, a value of 0.95 will ensure 95% of the area under the probability density of the posterior is covered by the returned equal-tailed interval.

    • For the frequentist method, a value of 0.95 returns a percentile bootstrap confidence interval containing the central 95% of the bootstrap distribution.

  • method (str) – The method used to compute the results, either frequentist or bayesian.

  • **kwargs –

    Additional arguments to be passed. In particular:

    • lambda_ (bayesian): The parameterisation of the Beta prior to consider.

    • n_bootstraps (frequentist): Number of bootstrap replicates.

    • require_all_classes (frequentist): Whether each accepted bootstrap sample must contain all classes present in y_true.

    • max_resample_attempts (frequentist): Maximum number of resampling attempts. If require_all_classes is True, this parameter limits the number of times the function will attempt to resample the data to obtain a bootstrap sample that contains all classes.

    • rng (frequentist): An initialized numpy random number generator.

    Any remaining keyword arguments are passed to sklearn.metrics.average_precision_score. For additional details on the supported arguments and their behavior, refer to the scikit-learn documentation.

Return type:

tuple[float, float, float]

Returns:

A tuple with 3 floats:

  • The area under the Precision-Recall curve.

  • The lower bound of the estimated interval. For method="bayesian", this corresponds to the lower bound of the credible interval. For method="frequentist", this corresponds to the lower bound of the bootstrap confidence interval.

  • The upper bound of the estimated interval. For method="bayesian", this corresponds to the upper bound of the credible interval. For method="frequentist", this corresponds to the upper bound of the bootstrap confidence interval.