Using performance analysis to determine how I dropped the ball

Making predictions is easy. Making accurate predictions is hard.

Back in September, I built a storyboard in Pyramid Analytics BI Office to help me draft a winning fantasy football team. Unfortunately, my team hasn't done a whole lot of that "winning" thing so far this season. I would like to pin my team's woeful performance squarely on the shoulders of Lady Luck, but in reality, it was my own doing: I spent more time tweaking my data reports than actually using those reports to make decisions.

What did I expect, though? My passion lies in building the tools that others can use to analyze data and make informed business decisions. For instance, an NFL general manager might hire dozens of people with my skillset to build tools for him, but those employees wouldn't actually make the decisions (though they could make a data report to analyze their own performance!).

I created the Team Performance storyboard with several goals in mind. First, I wanted to quickly identify players who were under-performing so that I could replace them through free agency or trades. Second, I hoped to evaluate the effectiveness of my draft storyboard so I could adjust it for next year. Finally, I wanted to showcase some tools that may not have actually helped me from a fantasy perspective, but might be useful in business use cases (and which provide some interesting insights into player performance beyond the narrow scope of fantasy football).

Figure 1. Team Performance Story Board
Team performance storyboard

The first report I generated for the Team Performance storyboard was the Team - Actual vs. Projected chart. I can see from this report that most of my players have underperformed compared to their projections. In particular, Jacksonville Jaguars running back Chris Ivory and Seattle Seahawks wide receiver Tyler Lockett have underwhelmed in terms of their fantasy production. Had I spent more time analyzing the data rather than fine tuning it, I would have dropped Ivory and Lockett long ago and likely replaced them with one of the top available players, as indicated by the orange "Top 3 Avail Avg." bar.

My lack of action is actually a common business problem when dealing with analytics. I call this phenomenon "KP-Irony." It occurs when a manager fails to act on data reports because they aren't exactly how the manager wants them to be. In my case, I had perfectly acceptable tools to help me make decisions, but I kept reworking the reports without consulting them to make data-driven decisions. It's crucial to understand that no report will ever be perfect, but that doesn't mean they can't be incredibly useful for making business decisions.

Figure 2. Team Performance - Actual vs Projected Bar Chart
Team actual vs. projected

For instance, the sunburst in the lower-right hand corner labeled "Season Actual vs. Projected Next Week" makes it easy for me quickly assess the current health of my team. Darker colors indicate how well a player is expected to perform next week, and larger segments represent how well a player has performed to this point relative to the rest of my team. Advanced visualizations like this one are designed to speed the decision-making process and minimize mental effort.

Video 1. Season Actual vs. Projected Next Week

The trend chart in the upper-right hand corner (see figure 1) shows how my team performed in several KPIs each week, like passing touchdowns or field goals. I can use this chart to see how my managerial decisions impact not only my team's overall performance, but the derivative factors that led to that performance. As a sales manager, I might use a chart like this one to track how an incentive program drove various aspects of performance (sales calls, client visits, etc.), while a product manager might discover seasonality of sales for a particular product line or quickly assess the results of a recent marketing campaign. Whatever the KPI, viewing the trend over time is useful for future forecasting.

Figure 3. Trend Analysis Chart for Team Performance
Trend chart

Sometimes I want to view the trend for just one particular player. For instance, if I have identified a player who seems to be under-performing, it would be beneficial to know in what direction his performance is trending so I don't accidentally drop a player who is heating up. BI Office makes it easy to interact with my visualizations and reports on the fly. If I want to filter view the trends for my underperforming running back Chris Ivory, I just click on his name in the Team table and choose "Interaction." The other reports are updated automatically. Being able to effortlessly navigate to the data I need to make real-time decisions is invaluable in a competitive market.

What did I learn from measuring my team's performance with this storyboard? Primarily, I need to take better advantage of the reports I've generated while making iterative improvements to them throughout the season. All of my indicators told me to bail on certain players. Second, I would like to figure out a way to do a better job of putting my findings into context. It's possible that rostering half of the Broncos' offense wasn't the best idea after all. But at least they still have a shot at making the playoffs.