Friday, October 2, 2015

In defense of the BPI, etc.

In the past couple of years, ESPN has come out with some proprietary "advanced metrics." These include the "Total QBR" for rating QB play in football, the FPI, or "football power index," for rating football teams, and the BPI, or basketball power index, for rating college basketball teams.

A lot of the folks I engage with on Twitter like to lampoon these metrics, mainly on the notion that they are "black boxes"—that is, we don't know how they're calculated, so we can't really evaluate them. Thus, they are entitled to no respect.

I certainly agree with the notion that power ratings should tell us how they work, at least in general terms. But both the FPI and the BPI are actually pretty open, or at least can be figured out rather easily.  And it's not like other rating systems, such as Ken Pomeroy's ratings, are completely open about how their calculations are made. As with the BPI, Kenpom tells us what he considers in general terms, but the exact weights and calculations are his special formula.

Looking specifically at BPI, it is very clear that it is a Kenpom-style per-possession-efficiency rating system with basically one twist: it adjusts the weight of individual games based on whether one or both of the teams was at less than full strength. So, for example, when Rutgers beat Wisconsin with Frank Kaminsky sitting on the bench, they didn't get "full credit" for the win. Instead, the BPI only counted that game as .832 of a game because of Frank's injury. This is all open and published.

ESPN doesn't explain exactly how they came up with that number of .832 to account for Frank sitting out. But that's fine. Kenpom doesn't explain exactly how he discounts blowout mismatches, or exactly how much more weight he gives to recent games.

In the end, BPI and Kenpom (and T-Rank) come up with very similar results. For example, here's the top 20 of last year's BPI, with their T-Rank and Kenpom rank also listed:


BPI TeamName KenPom Trank
1 Kentucky 1 1
2 Wisconsin 3 3
3 Arizona 2 2
4 Virginia 5 4
5 Villanova 6 6
6 Duke 4 5
7 Gonzaga 7 8
8 Kansas 14 12
9 Notre Dame 9 11
10 Utah 8 7
11 North Carolina 11 9
12 Wichita St. 10 16
13 Oklahoma 13 10
14 Northern Iowa 12 22
15 Louisville 16 17
16 Iowa St. 18 18
17 Ohio St. 21 14
18 Michigan St. 15 15
19 Baylor 17 13
20 Butler 20 19

A lot of agreement there, and this holds true for most teams.

The fun thing about a power rating like BPI (or T-Rank) is that it allows one to give probabilities of various events happening, and estimate points spreads for hypothetical or far-off matchups. I think it's cool that T-Rank Pure can currently give you a prediction of about 5,000 upcoming college basketball games, and ESPN can do the same thing with BPI. Given that we basically know what the BPI is, except for minor details, I think the lampooning is unwarranted.

No comments:

Post a Comment