Model Selection

Area Under the Precision-Recall Curve 

Unlike ROC curves, PR curves have space that it's impossible to reach. There are techniques therefore to calculate the area under the curve (AUC) by interpolating. [2]

"One can easily show that, like in ROC space, each valid confusion matrix, where TP > 0, defines a single and unique point in PR space. In PR space, both recall and precision depend on the TP cell of the confusion matrix, in contrast to the true positive rate and false positive rate used in ROC space. This dependence, together with the fact that a specific data set contains a fixed number of negative and positive examples, imposes limitations on what precisions are possible for a particular recall." [1]

This isn't just an interesting fact. It has implications for calculating the AUC. Calculating the AUC requires some approximations. For instance, if the model predicts no positives (either true or false), the precision goes off to infinity.

Apache Spark "computes the area under the curve (AUC) using the trapezoidal rule." [Spark docs] However, "the trapezoidal rule ... uses linear interpolation and can be too optimistic." [SKLearn docs]

SKLearn's average_precision_score (see above) calulates the AUC by taking the x-deltas by the y-value (that is, not using trapezoids but rectangles). Since the deltas on the x-axis add up to 1.0, this result of this calculation is also the average. The y-value for x=0 can be estimated from the first data point (see [2]).

Properties of the PR-curve

Another feature of a PR curve is the monkey score line. In a classifier that is wildly guessing, the PR curve will look like a flat line as the precision (=TP/FP) will be proportional to E[P/N] (StackExchange)

The perfect plot of a PR curve is a perfect ROC curve rotated 90-degrees clockwise. Furthermore, there are 3 interesting properties of PR curves [2]:

1. Interpolation between two precision-recall points is non-linear.
2. The ratio of positives and negatives defines the baseline.
3. A ROC point and a precision-recall point always have a one-to-one relationship.

[1] Unachievable Region in Precision-Recall Space and Its Effect on Empirical Evaluation

[2] The blog of Takaya Saito and Marc Rehmsmeier