credible.frequentist.utils¶

Frequentist confidence interval estimation.

(Frequentist) confidence interval interpretation, with 95% coverage: If we are to take several independent random samples from the population and construct confidence intervals from each of the sample data, then 95 out of 100 confidence intervals will contain the true mean (true proportion, in this context of proportion).

See a discussion in Five Confidence Intervals for Proportions That You Should Know About.

Functions

`agresti_coull`(successes, failures[, coverage])	Calculate the confidence interval for proportion estimates.
`agresti_coull_array`(successes, failures, ...)	`agresti_coull()`, for multiple systems.
`bootstrap_metric`(y_true, y_score_or_pred, ...)	Compute the confidence interval of `metric_func` via non-parametric bootstrapping.
`bootstrap_metric_distribution`(y_true, ...[, ...])	Build the empirical bootstrap distribution of a metric.
`clopper_pearson`(successes, failures[, coverage])	Calculate the "exact" confidence interval for proportion estimates.
`clopper_pearson_array`(successes, failures, ...)	`clopper_pearson()`, for multiple systems.
`compare_systems`(y_true, output_a, output_b, ...)	Compare 2 system outputs using a paired permutation test.
`percentile_interval`([distribution, coverage])	Derive confidence intervals.
`wilson`(successes, failures[, coverage])	Calculate the confidence interval for proportion estimates.
`wilson_array`(successes, failures, coverage)	`wilson()`, for multiple systems.

credible.frequentist.utils.clopper_pearson_array(successes, failures, coverage)[source]¶

clopper_pearson(), for multiple systems.

Parameters:

successes (Iterable[int]) – Number of successes observed on the experiment.
failures (Iterable[int]) – Number of failures observed on the experiment.
coverage (float) – A floating-point number between 0 and 1.0 indicating the coverage you’re expecting. A value of 0.95 will ensure 95% coverage.

Return type:

tuple[GenericAlias[float64], GenericAlias[float64], GenericAlias[float64]]

Returns:

The estimated ratio between successes, and total trials (successes plus failures), lower and upper bounds of the confidence interval, in this order.

Raises:

TypeError – If the dimensions of successes and failures do not match, or in case the input types are unsupported.

credible.frequentist.utils.clopper_pearson(successes, failures, coverage=0.95)[source]¶

Calculate the “exact” confidence interval for proportion estimates.

The Clopper-Pearson interval method is used for estimating the confidence intervals. This implementation is based on [CLOPPER-1934]. This technique is very conservative - in most of the cases, coverage is greater than the required value, which may imply in too large confidence intervals.

Parameters:

successes (int) – Number of successes observed on the experiment.
failures (int) – Number of failures observed on the experiment.
coverage (float) – A floating-point number between 0 and 1.0 indicating the coverage you’re expecting. A value of 0.95 will ensure 95% coverage.

Return type:

tuple[float, float, float]

Returns:

The estimated ratio between successes, and total trials (successes plus failures), lower and upper bounds of the confidence interval, in this order.

Raises:

TypeError – If the dimensions of successes and failures do not match, or in case the input types are unsupported.

credible.frequentist.utils.agresti_coull_array(successes, failures, coverage)[source]¶

agresti_coull(), for multiple systems.

Parameters:

successes (Iterable[int]) – Number of successes observed on the experiment.
failures (Iterable[int]) – Number of failures observed on the experiment.
coverage (float) – A floating-point number between 0 and 1.0 indicating the coverage you’re expecting. A value of 0.95 will ensure 95% coverage.

Return type:

tuple[GenericAlias[float64], GenericAlias[float64], GenericAlias[float64]]

Returns:

The estimated ratio between successes, and total trials (successes plus failures), lower and upper bounds of the confidence interval, in this order.

Raises:

TypeError – If the dimensions of successes and failures do not match, or in case the input types are unsupported.

credible.frequentist.utils.agresti_coull(successes, failures, coverage=0.95)[source]¶

Calculate the confidence interval for proportion estimates.

The Agresti-Coull interval method is used for estimating the confidence intervals. This implementation is based on [AGRESTI-1998]. This technique is conservative - in most of the cases, coverage is greater than the required value, which may imply a larger confidence interval that required.

This function is considered a good choice for the frequentist approach, if you cannot use clopper_pearson().

Parameters:

successes (int) – Number of successes observed on the experiment.
failures (int) – Number of failures observed on the experiment.
coverage (float) – A floating-point number between 0 and 1.0 indicating the coverage you’re expecting. A value of 0.95 will ensure 95% coverage.

Return type:

tuple[float, float, float]

Returns:

The estimated ratio between successes, and total trials (successes plus failures), lower and upper bounds of the confidence interval, in this order.

Raises:

TypeError – If the dimensions of successes and failures do not match, or in case the input types are unsupported.

credible.frequentist.utils.wilson_array(successes, failures, coverage)[source]¶

wilson(), for multiple systems.

Parameters:

successes (Iterable[int]) – Number of successes observed on the experiment.
failures (Iterable[int]) – Number of failures observed on the experiment.
coverage (float) – A floating-point number between 0 and 1.0 indicating the coverage you’re expecting. A value of 0.95 will ensure 95% coverage.

Return type:

tuple[GenericAlias[float64], GenericAlias[float64], GenericAlias[float64]]

Returns:

The estimated ratio between successes, and total trials (successes plus failures), lower and upper bounds of the confidence interval, in this order.

Raises:

TypeError – If the dimensions of successes and failures do not match, or in case the input types are unsupported.

credible.frequentist.utils.wilson(successes, failures, coverage=0.95)[source]¶

Calculate the confidence interval for proportion estimates.

The Wilson interval method is used for estimating the confidence intervals. This implementation is based on [WILSON-1927]. This implementation does not contain the continuity correction. It is as conservative in the extremes of the domain as the bayesian approach and can be a good default, if clopper_pearson() cannot be used.

This function is considered the best “default” for the frequentist approach as it is not too conservative and assumes a resonable value through out the range.

Parameters:

successes (int) – Number of successes observed on the experiment.
failures (int) – Number of failures observed on the experiment.
coverage (float) – A floating-point number between 0 and 1.0 indicating the coverage you’re expecting. A value of 0.95 will ensure 95% coverage.

Return type:

tuple[float, float, float]

Returns:

The estimated ratio between successes, and total trials (successes plus failures), lower and upper bounds of the confidence interval, in this order.

Raises:

TypeError – If the dimensions of successes and failures do not match, or in case the input types are unsupported.

credible.frequentist.utils.percentile_interval(distribution=list[float], coverage=0.95)[source]¶

Derive confidence intervals.

Parameters:

distribution – Distribution from which to compute the confidence interval.
coverage (float) – A floating-point number between 0 and 1.0 indicating the coverage you’re expecting. A value of 0.95 will ensure 95% coverage.

Returns:

Tuple with 2 floating-point numbers:

The lower value of the confidence interval
The higher value of the confidence interval

Return type:

tuple[float, float]

credible.frequentist.utils.bootstrap_metric_distribution(y_true, y_score_or_pred, metric_func, rng, n_bootstraps=1000, require_all_classes=True, max_resample_attempts=100, **kwargs)[source]¶

Build the empirical bootstrap distribution of a metric.

This function repeatedly resamples the dataset with replacement and evaluates a metric on each bootstrap sample. The resulting distribution can be used to estimate confidence intervals for the metric.

Parameters:

y_true (GenericAlias[integer]) – Ground truth (correct) labels.
y_score_or_pred (GenericAlias[floating] | GenericAlias[integer]) – Scores or predicted labels, as returned by a classifier.
metric_func (Callable) – The sklearn metric function for which to compute the confidence interval.
rng (Generator) – An initialized numpy random number generator.
n_bootstraps (int) – Number of bootstrapping steps to evaluate.
require_all_classes (bool) – If set to True, each accepted bootstrap sample must contain all classes present in y_true. This is required for metrics such as ROC AUC.
max_resample_attempts (int) – Maximum number of redraw attempts for a given bootstrap replicate when require_all_classes=True.
**kwargs – Parameters for the given metric_func.

Returns:

A list of bootsrapped results from y_score_or_pred evaluated on metric_func. If require_all_classes`was set to `True, there is a chance the number of returned values is less than n_bootstraps.

Return type:

list[float]

credible.frequentist.utils.bootstrap_metric(y_true, y_score_or_pred, metric_func, rng, n_bootstraps=1000, coverage=0.95, require_all_classes=True, max_resample_attempts=100, **kwargs)[source]¶

Compute the confidence interval of metric_func via non-parametric bootstrapping.

Parameters:

y_true (GenericAlias[integer]) – Ground truth (correct) labels.
y_score_or_pred (GenericAlias[floating] | GenericAlias[integer]) – Scores or predicted labels, as returned by a classifier.
metric_func (Callable) – The sklearn metric function for which to compute the confidence interval.
rng (Generator) – An initialized numpy random number generator.
n_bootstraps (int) – Number of bootstrapping steps to evaluate.
coverage (float) – A floating-point number between 0 and 1.0 indicating the coverage you’re expecting. A value of 0.95 will ensure 95% coverage.
require_all_classes (bool) – If set to True, each accepted bootstrap sample must contain all classes present in y_true. This is required for metrics such as ROC AUC.
max_resample_attempts (int) – Maximum number of redraw attempts for a given bootstrap replicate when require_all_classes=True.
**kwargs – Parameters for the given metric_func.

Returns:

Tuple with 4 floating-point numbers:

The output of the metric function
The mode of the bootstrapped distribution
The lower value of the confidence interval
The higher value of the confidence interval

Return type:

tuple[float, float, float, float]

credible.frequentist.utils.compare_systems(y_true, output_a, output_b, metric_func, rng, n_resamples=9999, **kwargs)[source]¶

Compare 2 system outputs using a paired permutation test.

This function returns the observed difference in performance between system A and system B, together with the p-value obtained from a paired permutation test.

The comparison assumes both systems were evaluated on the same samples, in the same order. Under the null hypothesis, the two systems are exchangeable within each sample, so permutations are generated by swapping system outputs pairwise.

Parameters:

y_true (Sequence[int]) – Ground truth (correct) labels. These are the same for both systems.
output_a (Sequence[float]) – Outputs produced by the first system. These may be predicted labels, scores, or any other per-sample output accepted by metric_func as its second argument.
output_b (Sequence[float]) – Outputs produced by the second system. These may be predicted labels, scores, or any other per-sample output accepted by metric_func as its second argument.
metric_func (Callable) – Metric function used to compare the two systems. It must accept y_true as its first argument and a system output as its second argument. Ususally this is a sklearn metric function for which to compare the two systems.
rng (Generator) – An initialized numpy random number generator.
n_resamples (int) – Number of random permutations used to approximate the null distribution.
**kwargs – Additional parameters for the given metric_func.

Returns:

tuple[float, float]

Tuple with 2 floating-point numbers:

The observed difference metric_func(y_true, output_a) - metric_func(y_true, output_b).
The p-value from the paired permutation test.