FPI is annoying

Before I get into this post, I want to make clear that I don't have any animus toward Espn or their analytics people. I like them! I've defended them! (Though they've gone backwards a bit since I wrote that "In defense of BPI, etc." post.)

With that throat-clearing out of the way...

Espn has a predictive metric for college football called FPI. It is relatively good* at predicting the results of college football games. So if you want to lose a little (instead of a lot) of money, use FPI to pick games against the spread. Or you could just toss a coin and do slightly better.

For the most part, the analytics people at Espn are admirably clear that FPI is a predictive tool. But lately they've been doing two things with FPI that annoy me: First, they've been touting it as an accurate and objective arbiter of team quality. Second, they've been openly clamoring for an FPI-based metric—Strength of Record—to be used in selecting the teams for the college football playoffs.

Here's why I'm annoyed.

FPI is not an accurate measure of team quality.

I said above that FPI is relatively good at predicting the results of games. I say relatively good because college football is a crazy sport where crazy things happen on the regular. People say "Vegas knows, man" but Vegas don't know much about college football: the mean absolute error for closing betting lines in college football is about 12.2 points (and FPI is at 12.4). In the NFL, the mean error is 10.5 points—significantly better.

Why is this? It's because there is just not enough good data in a college football season to produce a highly accurate measure of team quality. Each team plays just 12 games, and only 8-10 of those are games that we can extract much useful information. (The other 2-4 games are mismatches that are likely to be all but useless for discerning differences among quality teams.)

So this is not a knock on FPI in particular. There simply does not exist a highly accurate measure of team quality in college football. What annoys me is that Espn pretends that there does, and that it is FPI.

We can see this hubris in an article today where they purport to show that the playoff selection committee is not selecting the "best" teams because it is not selecting the teams rated highest in the FPI and is considering teams rated poorly by the FPI (poster boy: Wisconsin). In other words, FPI is the arbiter of "best." Bullshit! If the question is: "Who are the best teams?" the only intellectually honest answer is:

"We don't know."

FPI-based "Strength of Record" should not and cannot be used in playoff selection 

As I mentioned above, there is not enough good data in a single college football season to construct an accurate enough measure of team quality to reliably discern between the best teams. To help make up for this, Espn incorporates its preseason prior — which is based on things like historical performance and recruiting ranks — into the FPI all year long. It never goes away. And while they refuse to say how much it still, at this point, influences the ratings, it is pretty clear that is still doing a lot of work. There is no other explanation for how Florida State (preseason No. 3, but 5-6 with bad losses) is still ranked 21st.

There is nothing wrong with keeping the preseason prior part of the formula. Espn's guys say that using the prior makes the predictions more accurate, and that's what the FPI is supposed to be maximizing. Fine!

But keeping the preseason prior means that FPI cannot be used in any way for selection to the playoffs. No one would argue that selection should be based on recruiting ranks or last year's performance. But if you rely on a strength of record measure based on FPI, that's what you'd be doing. You're giving Alabama extra credit for beating a bad Florida State team because FSU has five-star recruits and used to be good. Completely unacceptable.

Who benefits?

Obviously, a system that maintains a significant prior based on program strength and recruiting ranks will boost teams like Ohio State and Alabama. It makes sense if you are trying to predict games to give those teams a boost, because even in the middle of a down season they probably still have a lot of good football players could put it together at any time.

And obviously it hurts Wisconsin, which has not ever had a recruiting class better than 30th in the country. Which is why it is particularly annoying to see Espn touting Wisconsin's low FPI as a reason that it can't truly be considered among the best teams. It's crazy-making.