measures Package

measures Package

Define standard reliability and agreement measures.

agreement Module

pyanno.measures.agreement.cohens_kappa(annotations1, annotations2, nclasses=None)

Compute Cohen’s kappa for two annotators.

Assumes that the annotators draw annotations at random with different but constant frequencies.

See also pairwise_matrix().

References:

• Cohen, Jacob (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46.
• Wikipedia entry

Parameters:

• annotations1 (ndarray, shape = (n_items, )) – Array of annotations for a single annotator. Missing values should be indicated by pyanno.util.MISSING_VALUE
• annotations2 (ndarray, shape = (n_items, )) – Array of annotations for a single annotator. Missing values should be indicated by pyanno.util.MISSING_VALUE
• nclasses (int) – Number of annotation classes. If None, nclasses is inferred from the values in the annotations

Returns:

stat (float) - The value of the statistics

pyanno.measures.agreement.cohens_weighted_kappa(annotations1, annotations2, weights_func=<function diagonal_distance at 0x238f730>, nclasses=None)

Compute Cohen’s weighted kappa for two annotators.

Assumes that the annotators draw annotations at random with different but constant frequencies. Disagreements are weighted by a weights w_ij representing the “seriousness” of disagreement. For ordered codes, it is often set to the distance from the diagonal, i.e. w_ij = |i-j|.

When w_ij is 0.0 on the diagonal and 1.0 elsewhere, Cohen’s weighted kappa is equivalent to Cohen’s kappa.

See also: diagonal_distance(), binary_distance(), cohens_kappa(), pairwise_matrix()

References:

• Cohen, J. (1968). “Weighed kappa: Nominal scale agreement with provision for scaled disagreement or partial credit”. Psychological Bulletin 70 (4): 213-220.
• Wikipedia entry

Parameters:

• annotations1 (ndarray, shape = (n_items, )) – Array of annotations for a single annotator. Missing values should be indicated by pyanno.util.MISSING_VALUE
• annotations2 (ndarray, shape = (n_items, )) – Array of annotations for a single annotator. Missing values should be indicated by pyanno.util.MISSING_VALUE
• weights_func (function(m_i, m_j)) – Weights function that receives two matrices of indices i, j and returns the matrix of weights between them. Default is diagonal_distance()
• nclasses (int) – Number of annotation classes. If None, nclasses is inferred from the values in the annotations

Returns:

stat (float) - The value of the statistics

pyanno.measures.agreement.fleiss_kappa(annotations, nclasses=None)

Compute Fleiss’ kappa for multiple annotators.

References:

• Fleiss, J. L. (1971). “Measuring nominal scale agreement among many raters.”. Psychological Bulletin, Vol 76(5), 378-382
• Wikipedia entry

Parameters:

• annotations (ndarray, shape = (n_items, n_annotators)) – Array of annotations for multiple annotators. Missing values should be indicated by pyanno.util.MISSING_VALUE
• nclasses (int) – Number of annotation classes. If None, nclasses is inferred from the values in the annotations

Returns:

stat (float) - The value of the statistics

pyanno.measures.agreement.krippendorffs_alpha(annotations, metric_func=<function diagonal_distance at 0x238f730>, nclasses=None)

Compute Krippendorff’s alpha for multiple annotators.

References:

• Klaus Krippendorff (2004). “Content Analysis, an Introduction to Its Methodology”, 2nd Edition. Thousand Oaks, CA: Sage Publications. In particular, Chapter 11, pages 219–250.
• Wikipedia entry

See also: diagonal_distance(), binary_distance(),

Parameters:

• annotations (ndarray, shape = (n_items, n_annotators)) – Array of annotations for multiple annotators. Missing values should be indicated by pyanno.util.MISSING_VALUE
• weights_func (function(m_i, m_j)) – Weights function that receives two matrices of indices i, j and returns the matrix of weights between them. Default is diagonal_distance()
• nclasses (int) – Number of annotation classes. If None, nclasses is inferred from the values in the annotations

Returns:

stat (float) - The value of the statistics

pyanno.measures.agreement.scotts_pi(annotations1, annotations2, nclasses=None)

Return Scott’s pi statistic for two annotators.

Assumes that the annotators draw random annotations with the same frequency as the combined observed annotations.

See also pairwise_matrix().

References:

• Scott, W. (1955). “Reliability of content analysis: The case of nominal scale coding.” Public Opinion Quarterly, 19(3), 321-325.
• Wikipedia entry

Parameters:

• annotations1 (ndarray, shape = (n_items, )) – Array of annotations for a single annotator. Missing values should be indicated by pyanno.util.MISSING_VALUE
• annotations2 (ndarray, shape = (n_items, )) – Array of annotations for a single annotator. Missing values should be indicated by pyanno.util.MISSING_VALUE
• nclasses (int) – Number of annotation classes. If None, nclasses is inferred from the values in the annotations

Returns:

stat (float) - The value of the statistics

covariation Module

Standard reliability and covariation measures.

pyanno.measures.covariation.cronbachs_alpha(annotations, nclasses=None)

Compute Cronbach’s alpha.

References:

• Cronbach, L. J. (1951). “Coefficient alpha and the internal structure of tests.” Psychometrika, 16(3), 297-334.
• Wikipedia entry

Parameters:

• annotations (ndarray, shape = (n_items, n_annotators)) – Array of annotations for multiple annotators. Missing values should be indicated by pyanno.util.MISSING_VALUE
• nclasses (int) – Number of annotation classes. If None, nclasses is inferred from the values in the annotations

Returns:

stat (float) - The value of the statistics

pyanno.measures.covariation.pearsons_rho(annotations1, annotations2, nclasses=None)

Compute Pearson’s product-moment correlation coefficient.

See also pairwise_matrix().

References:

• Wikipedia entry

Parameters:

• annotations1 (ndarray, shape = (n_items, )) – Array of annotations for a single annotator. Missing values should be indicated by pyanno.util.MISSING_VALUE
• annotations2 (ndarray, shape = (n_items, )) – Array of annotations for a single annotator. Missing values should be indicated by pyanno.util.MISSING_VALUE
• nclasses (int) – Number of annotation classes. If None, nclasses is inferred from the values in the annotations

Returns:

stat (float) - The value of the statistics

pyanno.measures.covariation.spearmans_rho(annotations1, annotations2, nclasses=None)

Compute Spearman’s rank correlation coefficient.

See also pairwise_matrix().

References:

• Wikipedia entry

Parameters:

• annotations1 (ndarray, shape = (n_items, )) – Array of annotations for a single annotator. Missing values should be indicated by pyanno.util.MISSING_VALUE
• annotations2 (ndarray, shape = (n_items, )) – Array of annotations for a single annotator. Missing values should be indicated by pyanno.util.MISSING_VALUE
• nclasses (int) – Number of annotation classes. If None, nclasses is inferred from the values in the annotations

Returns:

stat (float) - The value of the statistics

distances Module

Definition of distance measures between classes.

pyanno.measures.distances.binary_distance(i, j)

Binary weight function returning 0 if i==j, else 1.

pyanno.measures.distances.diagonal_distance(i, j)

Weight function returning |i-j|.

helpers Module

pyanno.measures.helpers.all_invalid(*annotations)

Return True if all annotations are invalid.

pyanno.measures.helpers.chance_adjusted_agreement(observed_agreement, chance_agreement)

Return the chance-adjusted agreement given the specified agreement and expected agreement.

Defined by (observed_agreement - chance_agreement)/(1.0 - chance_agreement)

Parameters:

• observed_agreement (float) – Agreement computed from the data
• chance_agreement (float) – Agreement expected by chance give the assumptions of the statistics

result : float

Chance adjusted agreement value

pyanno.measures.helpers.chance_agreement_different_frequency(annotations1, annotations2, nclasses)

Expected frequency of agreement by random annotations.

Assumes that the annotators draw annotations at random with different but constant frequencies.

Parameters:

• annotations1 (ndarray, shape = (n_items, )) – Array of annotations for a single annotator. Missing values should be indicated by pyanno.util.MISSING_VALUE
• annotations2 (ndarray, shape = (n_items, )) – Array of annotations for a single annotator. Missing values should be indicated by pyanno.util.MISSING_VALUE
• weights_func (function(m_i, m_j)) – Weights function that receives two matrices of indices i, j and returns the matrix of weights between them. Default is diagonal_distance()

result : float

Chance agreement value

pyanno.measures.helpers.chance_agreement_same_frequency(annotations1, annotations2, nclasses)

Expected frequency of agreement by random annotations.

Assumes that the annotators draw random annotations with the same frequency as the combined observed annotations.

Parameters:

• annotations1 (ndarray, shape = (n_items, )) – Array of annotations for a single annotator. Missing values should be indicated by pyanno.util.MISSING_VALUE
• annotations2 (ndarray, shape = (n_items, )) – Array of annotations for a single annotator. Missing values should be indicated by pyanno.util.MISSING_VALUE
• weights_func (function(m_i, m_j)) – Weights function that receives two matrices of indices i, j and returns the matrix of weights between them. Default is diagonal_distance()

result : float

Chance agreement value

pyanno.measures.helpers.coincidence_matrix(annotations, nclasses)

Build coincidence matrix.

The element c,k of the coincidence matrix contains the number of c-k pairs in the data (across annotators), over the total number of observed pairs.

Reference

• Wikipedia entry

Parameters:

• annotations (ndarray, shape = (n_items, n_annotators)) – Array of annotations for multiple annotators. Missing values should be indicated by pyanno.util.MISSING_VALUE
• nclasses (int) – Number of annotation classes. If None, nclasses is inferred from the values in the annotations

Returns:

coinc_mat (ndarray, shape = (n_classes, n_classes)) - Coincidence matrix

pyanno.measures.helpers.compute_nclasses(*annotations)

Infer the number of label classes from the data.

pyanno.measures.helpers.confusion_matrix(annotations1, annotations2, nclasses)

Compute confusion matrix from pairs of annotations.

References

• Wikipedia entry

Parameters:

• annotations1 (ndarray, shape = (n_items, )) – Array of annotations for a single annotator. Missing values should be indicated by pyanno.util.MISSING_VALUE
• annotations2 (ndarray, shape = (n_items, )) – Array of annotations for a single annotator. Missing values should be indicated by pyanno.util.MISSING_VALUE
• nclasses (int) – Number of annotation classes. If None, nclasses is inferred from the values in the annotations

Returns:

conf_mat (ndarray, shape = (n_classes, n_classes)) - Confusion matrix; conf_mat[i,j] = number of observations that was annotated as category i by annotator 1 and as j by annotator 2

pyanno.measures.helpers.observed_agreement_frequency(annotations1, annotations2, nclasses)

Observed frequency of agreement by two annotators.

If a category is never observed, the frequency for that category is set to 0.0 .

Only count entries where both annotators responded toward observed frequency.

Parameters:

• annotations1 (ndarray, shape = (n_items, )) – Array of annotations for a single annotator. Missing values should be indicated by pyanno.util.MISSING_VALUE
• annotations2 (ndarray, shape = (n_items, )) – Array of annotations for a single annotator. Missing values should be indicated by pyanno.util.MISSING_VALUE
• weights_func (function(m_i, m_j)) – Weights function that receives two matrices of indices i, j and returns the matrix of weights between them. Default is diagonal_distance()

result : float

Observed agreement frequency value

pyanno.measures.helpers.pairwise_matrix(pairwise_statistic, annotations, *args, **kwargs)

Compute the matrix of all combinations of a pairwise statistics.

This function applies an agreement or covariation statistic that is only defined for pairs of annotators to all combinations of annotators pairs, and returns a matrix of the result.

Example

>>> from pyanno.measures import pairwise_matrix, cohens_kappa
>>> stat_matrix = pairwise_matrix(cohens_kappa, annotations, nclasses=4)

Parameters:

• pairwise_statistics (function) – Function accepting as first two arguments two 1D array of annotations, and returning a single scalar measuring some annotations statistics.
• annotations (ndarray, shape = (n_items, n_annotators)) – Annotations in pyanno format.
• args (any) – Additional arguments passed to pairwise_statistics.
• kwargs (any) – Additional keyword arguments passed to pairwise_statistics.

Returns:

stat_matrix (ndarray, shape = (n_annotators, n_annotators)) - stat_matrix[i,j] is the value of pairwise_statistics applied to the annotations of annotators i and j