Multiclass IMV

MulticlassIMV fits an intercept-only null classifier and a feature-based classifier in each fold, then reduces their multiclass probabilities to binary contrasts in two ways.

End-to-end evaluation

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

from imvpy import MulticlassIMV

iris = load_iris(as_frame=True)
features = list(iris.data.columns)
data = iris.data.assign(target=iris.target)

def model_creator():
    return make_pipeline(
        StandardScaler(),
        LogisticRegression(max_iter=2000, random_state=42),
    )

evaluator = MulticlassIMV(
    data=data,
    outcome_variable="target",
    model_creator=model_creator,
    n_splits=5,
    optional_explanatory_variables=features,
    random_state=42,
    stratified=True,
    verbose=False,
)

fold_ova, mean_ova = evaluator.k_fold_one_vs_all()
fold_matrices, mean_matrix = evaluator.k_fold_imv_matrix()

The model factory must return a fresh classifier with fit, predict_proba, and classes_ after fitting. The two fitted models must expose the same probability column order.

One versus rest

For each class c, outcomes become 1 for c and 0 for all other classes. The class's own probability column is scored directly for the null and enhanced models.

k_fold_one_vs_all() returns a list containing one (n_classes,) NumPy array per fold and a (n_classes,) NumPy array containing the NaN-aware fold mean. A class absent from a non-stratified test fold receives NaN for that fold. Use stratified=True for new i.i.d. analyses when class counts permit it.

Pairwise matrix

For every class pair (i, j), observations outside the pair are removed and the two corresponding probability columns are renormalized:

p(i | i or j) = p(i) / (p(i) + p(j))

The resulting binary probabilities and labels are passed through the same core IMV transformation. k_fold_imv_matrix() returns a list of fold-level NumPy matrices and a labeled pandas DataFrame containing their NaN-aware mean.

The matrix is exactly symmetric. Swapping i and j complements both the binary label and probability, and ll(y, p) == ll(1-y, 1-p). This matrix does not encode model direction in its two triangles.

Low-level fold methods

Use the low-level methods when probabilities were generated by a custom validation design:

pairwise = evaluator.multinominal_imv_matrix(
    test_frame,
    outcome_variable="target",
    p_base=baseline_probabilities,
    p_enhanced=enhanced_probabilities,
    classes=fitted_model.classes_,
)

ova = evaluator.one_vs_all_single_fold(
    test_frame,
    outcome_variable="target",
    p_base=baseline_probabilities,
    p_enhanced=enhanced_probabilities,
    classes=fitted_model.classes_,
)

classes names probability columns in order. Pass it whenever a test fold may omit a trained class. If omitted, the implementation assumes the columns match the sorted labels present in test_frame; that assumption is unsafe for an incomplete fold.

multinominal_imv_matrix retains its historical misspelling for compatibility.

Plotting

figure, axis = evaluator.multinomial_IMV_heatmap(mean_matrix)
figure_ova, axis_ova = evaluator.multinomial_IMV_boxplot(fold_ova)

The class plotting methods use generic Outcome1, Outcome2, and so on. For explicit labels and shared styling, prefer the package-level helpers:

from imvpy.utils import plot_imv_heatmap, plot_ova_boxplot

figure, axis = plot_imv_heatmap(mean_matrix, title="Iris pairwise IMV")
figure_ova, axis_ova = plot_ova_boxplot(
    fold_ova,
    labels=[str(label) for label in mean_matrix.columns],
)