Model Selection - part 2

We want to predict who will receive a vaccination. Although the outcome is binary (the patient did or did not receive a vaccination), we can build a confusion matrix if for each class, we pretend it's a binary problem. So, if we started with the booked_vaccinated class, our confusion matrix looks like:

true positive:  prediction == booked_vaccinated AND label == booked_vaccinated

false positive: prediction == booked_vaccinated AND label != booked_vaccinated

true negative:  prediction != booked_vaccinated AND label != booked_vaccinated

false negative: prediction != booked_vaccinated AND label == booked_vaccinated

Now we have a confusion matrix, we can look at AUCs to test our model. But, is it the only metric?

AUC or Log-Likelihood

Here are some miscellaneous notes from a data scientist who spent a good chunk of his PhD optimizing models:

1. AUC is good for ascertaining whether the model is well specified and realistic. It can tell you which choice of features are better than others.
2. Log-likelihood tells us how well the model fits the data. If the data is constant and the model structure is the same, the LL can tell us which choice of hyperparameters are better than others.

Note that the hypersurface of log-likelihood for a linear model is always convex. That is, there are no local maxima and they're guaranteed to give a global maxima.

Because log-likelihoods are logs of probabilities, even small relative differences are important. Since delta log likelihood is the ratio between probabilities, differences of a few units can be significant.

Guarding again Regressions

By having a stage in our pipeline that invokes the AUC calculations, we can keep an eye on our models as the code is developed. For instance, I have a Palantir @transform that just prints out a single row like this:

AUC calculations for an ML pipeline

If this changes radically, I know I've done something wrong.