In defense of the BPI, etc.

In the past couple of years, ESPN has come out with some proprietary "advanced metrics." These include the "Total QBR" for rating QB play in football, the FPI, or "football power index," for rating football teams, and the BPI, or basketball power index, for rating college basketball teams.

A lot of the folks I engage with on Twitter like to lampoon these metrics, mainly on the notion that they are "black boxes"—that is, we don't know how they're calculated, so we can't really evaluate them. Thus, they are entitled to no respect.

I certainly agree with the notion that power ratings should tell us how they work, at least in general terms. But both the FPI and the BPI are actually pretty open, or at least can be figured out rather easily.  And it's not like other rating systems, such as Ken Pomeroy's ratings, are completely open about how their calculations are made. As with the BPI, Kenpom tells us what he considers in general terms, but the exact weights and calculations are his special formula.

Looking specifically at BPI, it is very clear that it is a Kenpom-style per-possession-efficiency rating system with basically one twist: it adjusts the weight of individual games based on whether one or both of the teams was at less than full strength. So, for example, when Rutgers beat Wisconsin with Frank Kaminsky sitting on the bench, they didn't get "full credit" for the win. Instead, the BPI only counted that game as .832 of a game because of Frank's injury. This is all open and published.

ESPN doesn't explain exactly how they came up with that number of .832 to account for Frank sitting out. But that's fine. Kenpom doesn't explain exactly how he discounts blowout mismatches, or exactly how much more weight he gives to recent games.

In the end, BPI and Kenpom (and T-Rank) come up with very similar results. For example, here's the top 20 of last year's BPI, with their T-Rank and Kenpom rank also listed:

BPI	TeamName	KenPom	Trank
1	Kentucky	1	1
2	Wisconsin	3	3
3	Arizona	2	2
4	Virginia	5	4
5	Villanova	6	6
6	Duke	4	5
7	Gonzaga	7	8
8	Kansas	14	12
9	Notre Dame	9	11
10	Utah	8	7
11	North Carolina	11	9
12	Wichita St.	10	16
13	Oklahoma	13	10
14	Northern Iowa	12	22
15	Louisville	16	17
16	Iowa St.	18	18
17	Ohio St.	21	14
18	Michigan St.	15	15
19	Baylor	17	13
20	Butler	20	19

A lot of agreement there, and this holds true for most teams.

The fun thing about a power rating like BPI (or T-Rank) is that it allows one to give probabilities of various events happening, and estimate points spreads for hypothetical or far-off matchups. I think it's cool that T-Rank Pure can currently give you a prediction of about 5,000 upcoming college basketball games, and ESPN can do the same thing with BPI. Given that we basically know what the BPI is, except for minor details, I think the lampooning is unwarranted.