credible.frequentist.metrics

Implementation of Scikit-Learn compatible measures with confidence intervals estimated via non-parametric bootstrapping.

Module Attributes

NUMBER_BOOTSTRAPS

Default number of bootstrap replicates used to estimate confidence intervals throughout the package.

Functions

accuracy_score(y_true, y_pred[, coverage, ...])	Accuracy binary classification score.
average_precision_score(y_true, y_score[, ...])	Compute average precision (AP) from prediction scores.
det_curve()	Compute the Detection Error-Tradeoff (DET) curve.
f1_score(y_true, y_pred[, coverage, ...])	Return the mean, mode, upper and lower bounds of the credible region of the F1 score.
jaccard_score(y_true, y_pred[, coverage, ...])	Jaccard binary classification score.
precision_recall_curve()	Compute Precision-Recall (PR) curve.
precision_score(y_true, y_pred[, coverage, ...])	Precision binary classification score.
recall_score(y_true, y_pred[, coverage, ...])	Recall binary classification score.
roc_auc_score(y_true, y_score[, coverage, ...])	Calculate the area under the ROC (FPR vs TPR) curve.
roc_curve()	Compute Receiver operating characteristic (ROC).
specificity_score(y_true, y_pred[, ...])	Specificity binary classification score.

credible.frequentist.metrics.NUMBER_BOOTSTRAPS = 1000

Default number of bootstrap replicates used to estimate confidence intervals throughout the package. A value of 1000 generally provides a good trade-off between statistical stability and computational cost for most applications.

credible.frequentist.metrics.precision_score(y_true, y_pred, coverage=0.95, n_bootstraps=1000, require_all_classes=True, max_resample_attempts=100, rng=Generator(PCG64) at 0x7DD0621C0660, **kwargs)

Precision binary classification score.

AKA positive predictive value (PPV). It corresponds arithmetically to tp/(tp+fp). This function only supports binary classification problems.

Parameters:

• y_true (Iterable[int]) – Ground truth (correct) labels.

• y_pred (Iterable[int]) – Predicted labels, as returned by a classifier.

• coverage (float) – A floating-point number between 0 and 1 indicating the desired confidence level. A value of 0.95 will compute a bootstrap confidence interval containing the central 95% of the bootstrap distribution.

• n_bootstraps (int) – Number of bootstrapping steps to evaluate.

• require_all_classes (bool) – If set to True, each accepted bootstrap sample must contain all classes present in y_true. This is required for metrics such as ROC AUC.

• max_resample_attempts (int) – Maximum number of redraw attempts for a given bootstrap replicate when require_all_classes=True.

• rng (Generator) – An initialized numpy random number generator.

• **kwargs – Additional parameters for sklearn.metrics.precision_score.

Returns:

Tuple with 4 floating-point numbers:

• The actual precision, as would be returned by scikit-learn

• The mode of the bootstrap distribution, estimated as the peak of a Gaussian kernel density estimate (KDE) fitted to the bootstrap samples.

• The lower bound of the percentile bootstrap confidence interval defined by the coverage parameter.

• The upper bound of the percentile bootstrap confidence interval defined by the coverage parameter.

Return type:

tuple[float, float, float, float]

credible.frequentist.metrics.recall_score(y_true, y_pred, coverage=0.95, n_bootstraps=1000, require_all_classes=True, max_resample_attempts=100, rng=Generator(PCG64) at 0x7DD0621C0580, **kwargs)

Recall binary classification score.

AKA sensitivity, hit rate, or true positive rate (TPR). It corresponds arithmetically to tp/(tp+fn).

Parameters:

• y_true (Iterable[int]) – Ground truth (correct) labels.

• y_pred (Iterable[int]) – Predicted labels, as returned by a classifier.

• coverage (float) – A floating-point number between 0 and 1 indicating the desired confidence level. A value of 0.95 will compute a bootstrap confidence interval containing the central 95% of the bootstrap distribution.

• n_bootstraps (int) – Number of bootstrapping steps to evaluate.

• require_all_classes (bool) – If set to True, each accepted bootstrap sample must contain all classes present in y_true. This is required for metrics such as ROC AUC.

• max_resample_attempts (int) – Maximum number of redraw attempts for a given bootstrap replicate when require_all_classes=True.

• rng (Generator) – An initialized numpy random number generator.

• **kwargs – Additional parameters for sklearn.metrics.recall_score.

Returns:

Tuple with 4 floating-point numbers:

• The actual recall, as would be returned by scikit-learn

• The mode of the bootstrap distribution, estimated as the peak of a Gaussian kernel density estimate (KDE) fitted to the bootstrap samples.

• The lower bound of the percentile bootstrap confidence interval defined by the coverage parameter.

• The upper bound of the percentile bootstrap confidence interval defined by the coverage parameter.

Return type:

tuple[float, float, float, float]

credible.frequentist.metrics.specificity_score(y_true, y_pred, coverage=0.95, n_bootstraps=1000, require_all_classes=True, max_resample_attempts=100, rng=Generator(PCG64) at 0x7DD0621C10E0, **kwargs)

Specificity binary classification score.

AKA selectivity or true negative rate (TNR). It corresponds arithmetically to tn/(tn+fp).

Parameters:

• y_true (Iterable[int]) – Ground truth (correct) labels.

• y_pred (Iterable[int]) – Predicted labels, as returned by a classifier.

• coverage (float) – A floating-point number between 0 and 1 indicating the desired confidence level. A value of 0.95 will compute a bootstrap confidence interval containing the central 95% of the bootstrap distribution.

• n_bootstraps (int) – Number of bootstrapping steps to evaluate.

• require_all_classes (bool) – If set to True, each accepted bootstrap sample must contain all classes present in y_true. This is required for metrics such as ROC AUC.

• max_resample_attempts (int) – Maximum number of redraw attempts for a given bootstrap replicate when require_all_classes=True.

• rng (Generator) – An initialized numpy random number generator.

• **kwargs – Additional parameters for sklearn.metrics.recall_score (this because the specificity is computed as the recall of the negative class).

Returns:

Tuple with 4 floating-point numbers:

• The actual specificity, as would be returned by scikit-learn

• The mode of the bootstrap distribution, estimated as the peak of a Gaussian kernel density estimate (KDE) fitted to the bootstrap samples.

• The lower bound of the percentile bootstrap confidence interval defined by the coverage parameter.

• The upper bound of the percentile bootstrap confidence interval defined by the coverage parameter.

Return type:

tuple[float, float, float, float]

credible.frequentist.metrics.accuracy_score(y_true, y_pred, coverage=0.95, n_bootstraps=1000, require_all_classes=True, max_resample_attempts=100, rng=Generator(PCG64) at 0x7DD0621C11C0, **kwargs)

Accuracy binary classification score.

See Accuracy. is the proportion of correct predictions (both true positives and true negatives) among the total number of pixels examined. It corresponds arithmetically to (tp+tn)/(tp+tn+fp+fn). This measure includes both true-negatives and positives in the numerator, what makes it sensitive to data or regions without annotations. AKA selectivity or true negative rate (TNR), mean, mode and credible intervals. It corresponds arithmetically to tn/(tn+fp).

Parameters:

• y_true (Iterable[int]) – Ground truth (correct) labels.

• y_pred (Iterable[int]) – Predicted labels, as returned by a classifier.

• coverage (float) – A floating-point number between 0 and 1 indicating the desired confidence level. A value of 0.95 will compute a bootstrap confidence interval containing the central 95% of the bootstrap distribution.

• n_bootstraps (int) – Number of bootstrapping steps to evaluate.

• require_all_classes (bool) – If set to True, each accepted bootstrap sample must contain all classes present in y_true. This is required for metrics such as ROC AUC.

• max_resample_attempts (int) – Maximum number of redraw attempts for a given bootstrap replicate when require_all_classes=True.

• rng (Generator) – An initialized numpy random number generator.

• **kwargs – Additional parameters for sklearn.metrics.accuracy_score.

Returns:

Tuple with 4 floating-point numbers:

• The actual accuracy, as would be returned by scikit-learn

• The mode of the bootstrap distribution, estimated as the peak of a Gaussian kernel density estimate (KDE) fitted to the bootstrap samples.

• The lower bound of the percentile bootstrap confidence interval defined by the coverage parameter.

• The upper bound of the percentile bootstrap confidence interval defined by the coverage parameter.

Return type:

tuple[float, float, float, float]

credible.frequentist.metrics.jaccard_score(y_true, y_pred, coverage=0.95, n_bootstraps=1000, require_all_classes=True, max_resample_attempts=100, rng=Generator(PCG64) at 0x7DD0621C12A0, **kwargs)

Jaccard binary classification score.

See Jaccard Index or Similarity. It corresponds arithmetically to tp/(tp+fp+fn). The Jaccard index depends on a TP-only numerator, similarly to the F1 score. For regions where there are no annotations, the Jaccard index will always be zero, irrespective of the model output. Accuracy may be a better proxy if one needs to consider the true abscence of annotations in a region as part of the measure.

Parameters:

• y_true (Iterable[int]) – Ground truth (correct) labels.

• y_pred (Iterable[int]) – Predicted labels, as returned by a classifier.

• coverage (float) – A floating-point number between 0 and 1 indicating the desired confidence level. A value of 0.95 will compute a bootstrap confidence interval containing the central 95% of the bootstrap distribution.

• n_bootstraps (int) – Number of bootstrapping steps to evaluate.

• require_all_classes (bool) – If set to True, each accepted bootstrap sample must contain all classes present in y_true. This is required for metrics such as ROC AUC.

• max_resample_attempts (int) – Maximum number of redraw attempts for a given bootstrap replicate when require_all_classes=True.

• rng (Generator) – An initialized numpy random number generator.

• **kwargs – Additional parameters for sklearn.metrics.jaccard_score.

Returns:

Tuple with 4 floating-point numbers:

• The actual jaccard score, as would be returned by scikit-learn

• The mode of the bootstrap distribution, estimated as the peak of a Gaussian kernel density estimate (KDE) fitted to the bootstrap samples.

• The lower bound of the percentile bootstrap confidence interval defined by the coverage parameter.

• The upper bound of the percentile bootstrap confidence interval defined by the coverage parameter.

Return type:

tuple[float, float, float, float]

credible.frequentist.metrics.f1_score(y_true, y_pred, coverage=0.95, n_bootstraps=1000, require_all_classes=True, max_resample_attempts=100, rng=Generator(PCG64) at 0x7DD0621C1380, **kwargs)

Return the mean, mode, upper and lower bounds of the credible region of the F1 score.

See F1-score. It corresponds arithmetically to 2*P*R/(P+R) or 2*tp/(2*tp+fp+fn). The F1 or Dice score depends on a TP-only numerator, similarly to the Jaccard index. For regions where there are no annotations, the F1-score will always be zero, irrespective of the model output. Accuracy may be a better proxy if one needs to consider the true abscence of annotations in a region as part of the measure.

This implementation is based on [GOUTTE-2005].

Parameters:

• y_true (Iterable[int]) – Ground truth (correct) labels.

• y_pred (Iterable[int]) – Predicted labels, as returned by a classifier.

• coverage (float) – A floating-point number between 0 and 1 indicating the desired confidence level. A value of 0.95 will compute a bootstrap confidence interval containing the central 95% of the bootstrap distribution.

• n_bootstraps (int) – Number of bootstrapping steps to evaluate.

• require_all_classes (bool) – If set to True, each accepted bootstrap sample must contain all classes present in y_true. This is required for metrics such as ROC AUC.

• max_resample_attempts (int) – Maximum number of redraw attempts for a given bootstrap replicate when require_all_classes=True.

• rng (Generator) – An initialized numpy random number generator.

• **kwargs – Additional parameters for sklearn.metrics.f1_score.

Returns:

Tuple with 4 floating-point numbers:

• The actual F1 score, as would be returned by scikit-learn

• The mode of the bootstrap distribution, estimated as the peak of a Gaussian kernel density estimate (KDE) fitted to the bootstrap samples.

• The lower bound of the percentile bootstrap confidence interval defined by the coverage parameter.

• The upper bound of the percentile bootstrap confidence interval defined by the coverage parameter.

Return type:

tuple[float, float, float, float]

credible.frequentist.metrics.roc_curve()

Compute Receiver operating characteristic (ROC).

credible.frequentist.metrics.roc_auc_score(y_true, y_score, coverage=0.95, n_bootstraps=1000, require_all_classes=True, max_resample_attempts=100, rng=Generator(PCG64) at 0x7DD0621C1460, **kwargs)

Calculate the area under the ROC (FPR vs TPR) curve.

This function mimics the scikit-learn API, except it also returns lower and upper bounds computed from empirical quantiles of the bootstrap distribution (e.g., the 2.5th and 97.5th percentiles for a 95% confidence interval).

Parameters:

• y_true (Iterable[int]) – Ground truth (correct) labels.

• y_score (Iterable[float]) – Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).

• coverage (float) – A floating-point number between 0 and 1 indicating the desired confidence level. A value of 0.95 will compute a bootstrap confidence interval containing the central 95% of the bootstrap distribution.

• n_bootstraps (int) – Number of bootstrapping steps to evaluate.

• require_all_classes (bool) – If set to True, each accepted bootstrap sample must contain all classes present in y_true. This is required for metrics such as ROC AUC.

• max_resample_attempts (int) – Maximum number of redraw attempts for a given bootstrap replicate when require_all_classes=True.

• rng (Generator) – An initialized numpy random number generator.

• **kwargs – Additional parameters for sklearn.metrics.roc_auc_score.

Returns:

A tuple with 3 floats:

• The area under the ROC (FPR vs. TPR) curve

• The lower bound of the percentile bootstrap confidence interval defined by the coverage parameter.

• The upper bound of the percentile bootstrap confidence interval defined by the coverage parameter.

Return type:

tuple[float, float, float]

credible.frequentist.metrics.det_curve()

Compute the Detection Error-Tradeoff (DET) curve.

credible.frequentist.metrics.precision_recall_curve()

Compute Precision-Recall (PR) curve.

credible.frequentist.metrics.average_precision_score(y_true, y_score, coverage=0.95, n_bootstraps=1000, require_all_classes=True, max_resample_attempts=100, rng=Generator(PCG64) at 0x7DD0621C1540, **kwargs)

Compute average precision (AP) from prediction scores.

This function mimics the scikit-learn API, except it also returns lower and upper bounds computed from empirical quantiles of the bootstrap distribution (e.g., the 2.5th and 97.5th percentiles for a 95% confidence interval).

Parameters:

• y_true (Iterable[int]) – Ground truth (correct) labels.

• y_score (Iterable[float]) – Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).

• coverage (float) – A floating-point number between 0 and 1 indicating the desired confidence level. A value of 0.95 will compute a bootstrap confidence interval containing the central 95% of the bootstrap distribution.

• n_bootstraps (int) – Number of bootstrapping steps to evaluate.

• require_all_classes (bool) – If set to True, each accepted bootstrap sample must contain all classes present in y_true. This is required for metrics such as ROC AUC.

• max_resample_attempts (int) – Maximum number of redraw attempts for a given bootstrap replicate when require_all_classes=True.

• rng (Generator) – An initialized numpy random number generator.

• **kwargs – Additional parameters for sklearn.metrics.average_precision_score.

Returns:

A tuple with 3 floats:

• The area under the Precision-Recall curve.

• The lower bound of the percentile bootstrap confidence interval defined by the coverage parameter.

• The upper bound of the percentile bootstrap confidence interval defined by the coverage parameter.

Return type:

tuple[float, float, float]