credible.bayesian.kfold¶

Implementation of Scikit-Learn compatible measures with bayesian credible regions for k-folding experiments.

Functions

`accuracy_score`(y_true, y_pred, rng[, ...])	Accuracy binary classification score.
`f1_score`(y_true, y_pred, rng[, lambda_, ...])	Return the mean, mode, upper and lower bounds of the credible region of the F1 score.
`jaccard_score`(y_true, y_pred, rng[, ...])	Jaccard binary classification score.
`precision_score`(y_true, y_pred, rng[, ...])	Precision binary classification score.
`recall_score`(y_true, y_pred, rng[, lambda_, ...])	Recall binary classification score.
`specificity_score`(y_true, y_pred, rng[, ...])	Specificity binary classification score.

credible.bayesian.kfold.precision_score(y_true, y_pred, rng, lambda_=1.0, coverage=0.95, nb_samples=100000)[source]¶

Precision binary classification score.

AKA positive predictive value (PPV), mean, mode and credible intervals. It corresponds arithmetically to tp/(tp+fp). This function only supports binary classification problems.

Parameters:

y_true (Iterable[Iterable[int]]) – Ground truth (correct) labels.
y_pred (Iterable[Iterable[int]]) – Predicted labels, as returned by a classifier.
rng (Generator) – An initialized numpy random number generator.
lambda – The parameterisation of the Beta prior to consider. Use \(\lambda=1\) for a flat prior. Use \(\lambda=0.5\) for Jeffrey’s prior. Changes in this value do not significantly affect the outcome, unless tp or fp are very small (close to 1).
coverage (float) – A floating-point number between 0 and 1.0 indicating the coverage you’re expecting. A value of 0.95 will ensure 95% of the area under the probability density of the posterior is covered by the returned equal-tailed interval.
nb_samples (int) – Number of generated variates for the M-C simulation.

Return type:

tuple[float, float, float, float]

Returns:

A tuple with 4 floating-point numbers:

The average precision, as would be returned by scikit-learn
The mode of the posterior distribution
The lower value of the credible region/confidence interval
The upper value of the credible region/confidence interval

credible.bayesian.kfold.recall_score(y_true, y_pred, rng, lambda_=1.0, coverage=0.95, nb_samples=100000)[source]¶

Recall binary classification score.

AKA sensitivity, hit rate, or true positive rate (TPR), mean, mode and credible intervals. It corresponds arithmetically to tp/(tp+fn).

Parameters:

y_true (Iterable[Iterable[int]]) – Ground truth (correct) labels.
y_pred (Iterable[Iterable[int]]) – Predicted labels, as returned by a classifier.
rng (Generator) – An initialized numpy random number generator.
lambda – The parameterisation of the Beta prior to consider. Use \(\lambda=1\) for a flat prior. Use \(\lambda=0.5\) for Jeffrey’s prior. Changes in this value do not significantly affect the outcome, unless tp or fp are very small (close to 1).
coverage (float) – A floating-point number between 0 and 1.0 indicating the coverage you’re expecting. A value of 0.95 will ensure 95% of the area under the probability density of the posterior is covered by the returned equal-tailed interval.
nb_samples (int) – Number of generated variates for the M-C simulation.

Return type:

tuple[float, float, float, float]

Returns:

A tuple with 4 floating-point numbers:

The average recall, as would be returned by scikit-learn
The mode of the posterior distribution
The lower value of the credible region/confidence interval
The upper value of the credible region/confidence interval

credible.bayesian.kfold.specificity_score(y_true, y_pred, rng, lambda_=1.0, coverage=0.95, nb_samples=100000)[source]¶

Specificity binary classification score.

AKA selectivity or true negative rate (TNR), mean, mode and credible intervals. It corresponds arithmetically to tn/(tn+fp).

Parameters:

y_true (Iterable[Iterable[int]]) – Ground truth (correct) labels.
y_pred (Iterable[Iterable[int]]) – Predicted labels, as returned by a classifier.
rng (Generator) – An initialized numpy random number generator.
lambda – The parameterisation of the Beta prior to consider. Use \(\lambda=1\) for a flat prior. Use \(\lambda=0.5\) for Jeffrey’s prior. Changes in this value do not significantly affect the outcome, unless tp or fp are very small (close to 1).
coverage (float) – A floating-point number between 0 and 1.0 indicating the coverage you’re expecting. A value of 0.95 will ensure 95% of the area under the probability density of the posterior is covered by the returned equal-tailed interval.
nb_samples (int) – Number of generated variates for the M-C simulation.

Return type:

tuple[float, float, float, float]

Returns:

A tuple with 4 floating-point numbers:

The average specificity, as would be returned by scikit-learn
The mode of the posterior distribution
The lower value of the credible region/confidence interval
The upper value of the credible region/confidence interval

credible.bayesian.kfold.accuracy_score(y_true, y_pred, rng, lambda_=1.0, coverage=0.95, nb_samples=100000)[source]¶

Accuracy binary classification score.

See Accuracy. is the proportion of correct predictions (both true positives and true negatives) among the total number of pixels examined. It corresponds arithmetically to (tp+tn)/(tp+tn+fp+fn). This measure includes both true-negatives and positives in the numerator, what makes it sensitive to data or regions without annotations. AKA selectivity or true negative rate (TNR), mean, mode and credible intervals. It corresponds arithmetically to tn/(tn+fp).

Parameters:

y_true (Iterable[Iterable[int]]) – Ground truth (correct) labels.
y_pred (Iterable[Iterable[int]]) – Predicted labels, as returned by a classifier.
rng (Generator) – An initialized numpy random number generator.
lambda – The parameterisation of the Beta prior to consider. Use \(\lambda=1\) for a flat prior. Use \(\lambda=0.5\) for Jeffrey’s prior. Changes in this value do not significantly affect the outcome, unless tp or fp are very small (close to 1).
coverage (float) – A floating-point number between 0 and 1.0 indicating the coverage you’re expecting. A value of 0.95 will ensure 95% of the area under the probability density of the posterior is covered by the returned equal-tailed interval.
nb_samples (int) – Number of generated variates for the M-C simulation.

Return type:

tuple[float, float, float, float]

Returns:

A tuple with 4 floating-point numbers:

The average accuracy, as would be returned by scikit-learn
The mode of the posterior distribution
The lower value of the credible region/confidence interval
The upper value of the credible region/confidence interval

credible.bayesian.kfold.jaccard_score(y_true, y_pred, rng, lambda_=1.0, coverage=0.95, nb_samples=100000)[source]¶

Jaccard binary classification score.

See Jaccard Index or Similarity. It corresponds arithmetically to tp/(tp+fp+fn). The Jaccard index depends on a TP-only numerator, similarly to the F1 score. For regions where there are no annotations, the Jaccard index will always be zero, irrespective of the model output. Accuracy may be a better proxy if one needs to consider the true abscence of annotations in a region as part of the measure.

Parameters:

y_true (Iterable[Iterable[int]]) – Ground truth (correct) labels.
y_pred (Iterable[Iterable[int]]) – Predicted labels, as returned by a classifier.
rng (Generator) – An initialized numpy random number generator.
lambda – The parameterisation of the Beta prior to consider. Use \(\lambda=1\) for a flat prior. Use \(\lambda=0.5\) for Jeffrey’s prior. Changes in this value do not significantly affect the outcome, unless tp or fp are very small (close to 1).
coverage (float) – A floating-point number between 0 and 1.0 indicating the coverage you’re expecting. A value of 0.95 will ensure 95% of the area under the probability density of the posterior is covered by the returned equal-tailed interval.
nb_samples (int) – Number of generated variates for the M-C simulation.

Return type:

tuple[float, float, float, float]

Returns:

A tuple with 4 floating-point numbers:

The average jaccard score, as would be returned by scikit-learn
The mode of the posterior distribution
The lower value of the credible region/confidence interval
The upper value of the credible region/confidence interval

credible.bayesian.kfold.f1_score(y_true, y_pred, rng, lambda_=1.0, coverage=0.95, nb_samples=100000)[source]¶

Return the mean, mode, upper and lower bounds of the credible region of the F1 score.

See F1-score. It corresponds arithmetically to 2*P*R/(P+R) or 2*tp/(2*tp+fp+fn). The F1 or Dice score depends on a TP-only numerator, similarly to the Jaccard index. For regions where there are no annotations, the F1-score will always be zero, irrespective of the model output. Accuracy may be a better proxy if one needs to consider the true abscence of annotations in a region as part of the measure.

This implementation is based on [GOUTTE-2005].

Parameters:

y_true (Iterable[Iterable[int]]) – Ground truth (correct) labels.
y_pred (Iterable[Iterable[int]]) – Predicted labels, as returned by a classifier.
rng (Generator) – An initialized numpy random number generator.
lambda – The parameterisation of the Beta prior to consider. Use \(\lambda=1\) for a flat prior. Use \(\lambda=0.5\) for Jeffrey’s prior.
coverage (float) – A floating-point number between 0 and 1.0 indicating the coverage you are expecting. A value of 0.95 will ensure 95% of the area under the probability density of the posterior is covered by the returned equal-tailed interval.
nb_samples (int) – Number of generated variates for the M-C simulation.

Return type:

tuple[float, float, float, float]

Returns:

A tuple with 4 floating-point numbers:

The average F1-score, as would be returned by scikit-learn
The mode of the posterior distribution
The lower value of the credible region/confidence interval
The upper value of the credible region/confidence interval