What Madden NFL Can Teach Us About Cybersecurity Expectation Management

For nearly two decades there has been a steady drumbeat around Artificial Intelligence (AI) and Machine Learning (ML) solving all the world’s ills. From military intelligence to cybersecurity, experts told us that machines will replace human decision-making. But while there is great value in these tools as enablers, we are still far from automating away our problems. Many people think AI will lead us to identify the “unknown unknowns.” Most people I’ve spoken with in cyber intelligence don’t believe this is possible, including vendors whose messaging implies otherwise. In fact, the U.S. government has been trying to automate Intelligence for decades while simultaneously increasing the number of personnel across the Intelligence Community.

As someone with a passing knowledge of AI and ML, but not the scientific background to understand all the nuances, I’ve spoken with experts in the field who appear to agree that these technologies have incredible promise but are still more hype than reality today. Knowing this, I wondered if there was a tool filled with a robust amount of data used for modeling future events (that I could understand) to judge its performance. I found that tool in the popular Electronic Arts (EA) Sports video game franchise Madden NFL. It annually predicts the entire NFL season and is often used as a predictor for the outcome of the Super Bowl once the two competing teams have earned their spots in the game.

Madden NFL Data and Predictions

To get a baseline for performance going into an NFL season, EA Sports creates annual ratings for 2,600+ players across 53 criteria (up from 40) and applies significant weighting to the ratings to create overall player ratings. These ratings are based on hundreds (if not thousands) of hours of analysis that span up to several years of players’ careers. The data is impressive, and for the millions of fans of the game, it is the foundation for what is considered the pinnacle of sports-based video game franchises. Even so, the Madden NFL engine has rarely come close to accurately predicting an entire NFL season. Focusing on the last four seasons, Madden NFL predicted two of the eight conference champions (25%) and one Super Bowl champion (25%) in that time. While those numbers may not sound bad, consider that Madden NFL also predicted three of the cumulative eight Super Bowl slots (37.5%) would go to teams that did not even qualify for the postseason. At best, those would be considered mixed results that don’t offer a great deal of confidence in the ability to predict the future.

For the recently concluded 2021 season, Madden NFL predicted that the Kansas City Chiefs would defeat the Carolina Panthers to win Super Bowl LVI. While the Chiefs had a strong record at 12-5, they lost in the AFC Championship game. Meanwhile, the Panthers (5-12) were one of the worst teams in the league.

Coincidentally, Madden NFL predicted that the Cincinnati Bengals would fire Head Coach Zak Taylor after also finishing with a 5-12 record. In reality, the Bengals finished the season 10-7 and won two playoff games before defeating the aforementioned Chiefs to advance to the Super Bowl where they lost to the Los Angeles Rams.

You may be asking, “How accurate was Madden NFL in predicting the season for the World Champion Los Angeles Rams?” Well…ummm…that didn’t go well either. Madden NFL predicted the Rams would finish with the league’s worst record (4-13) and also fire their Head Coach, Sean McVay. Of course, we now know that the Rams went 12-5 in the regular season and won three playoff games before defeating the Bengals 23-20 in Super Bowl LVI.

Adding insult to injury for the folks at EA Sports, Madden NFL predicted that Urban Meyer would lead the Jacksonville Jaguars to an 11-6 record en route to being named NFL Coach of the Year in his first season on the job. Unfortunately, Urban Meyer had a scandal-plagued season that saw him coach the Jags to a 2-11 record before being fired with four games left in the season. Meyer is already widely considered to be one of the worst coaching hires in NFL history.

Madden NFL and the Super Bowl

For those who argue that predicting an entire NFL season is too much to ask of technology, let’s analyze Madden NFL just on the basis of predicting one game per year: the Super Bowl. Over the past 10 seasons, Madden NFL has accurately chosen the winner of the Super Bowl four times (40%). Was Madden NFL’s algorithm weighted to favor underdogs? Not really. Out of their four correct predictions, two were the team favored to win and two were underdogs. Out of the six incorrect predictions, Madden NFL again split evenly between the favored and underdog team (3-3). Ten Super Bowl predictions. Four correct and six wrong. A coin toss would have had better odds. Don’t get me wrong. The game is great! It just isn’t a crystal ball.

What Does This Have to Do with Cybersecurity?

Madden NFL is a sophisticated system that has a remarkable amount of data available as a baseline for each player and scheme (offensive and defensive) in today’s NFL. Based on that input, Madden NFL is expected to predict the future activity of thousands of people against one another. As difficult as that task may sound, it is no more complex than the cybersecurity landscape. We track hundreds of threat actors and groups using at least as many Tactics, Techniques, and Procedures (TTPs) — when we consider variants — against a nearly endless list of potential security stacks and configurations.

Machine Learning: Hype vs. Hyperbole

So why do we continue to hear the drumbeat of AI and ML as the solution to all of our cybersecurity woes? Why do so many security pros, technologists, and vendors proclaim that  with enough machines and enough data, we can automate ourselves to safety without the need for many (any?) people? It’s not because the claims are true today. Rather, I think we hear about these technologies in such grandiose terms because people desperately want to believe in an easy future compared to the stress of constantly protecting against unseen foes… and some vendors get swept up in capitalizing on that. Some even claim to sell ML or AI when, in fact, they just have a lot of people doing data entry. Besides, when was the last time a customer asked to see the ML or AI algorithms? Excitement is understandable given the potential, but today’s exaggeration of ML and AI capabilities undermines security.

The truth is that ML is incredibly powerful as a human enabler while AI is still more myth than reality. Ask anyone working on self-driving cars. Machine learning can speed up research and process incredible amounts of data to give humans a fighting chance to prioritize and act on the most significant threats. Given the right training data, and enough time, ML is impressive. Today, it helps us identify malicious objects and images, process more data than any amount of humans could, disrupt threat actors who bypass text analysis, and much more. BUT it’s worth reinforcing that ML cannot accurately predict the future. The key to proactive security continues to be a combination of superior access to data, information, and intelligence; powerful tools (including ML) to normalize, deduplicate, categorize, and prioritize that content; and skilled personnel to apply what can only be accomplished by human beings: Intelligence tradecraft, experience, and intuition.

Would you bet your cybersecurity solely on technology that, even with the best data and decades of tuning, is less than 50% accurate at predicting a single football game? I wouldn’t.

Tags: Artificial Intelligence, Threat Intelligence