Chief Innovation Officer at Tidal Cyber
Frank Duff is a distinguished thought leader in threat-informed defense, specializing in the assessment of organizations and security capabilities. Prior to Tidal, Frank spent his entire 18-year professional career at The MITRE Corporation in a variety of roles. Frank is most well-known as the General Manager of MITRE ATT&CK® Evaluations where he conceptualized, stood up, and oversaw the program. He spent the early years of ATT&CK on the front lines, transitioning it to the private sector, working with solution providers to understand the importance of the burgeoning knowledge base, as well as advising in its integration into their products and workflows. Recognizing a gap in current evaluation processes, he devised a threat-informed evaluation methodology that would leverage ATT&CK as the common language and would revolutionize how solution provider testing was performed. He oversaw nearly 100 evaluations, including over 90% of Forrester and Gartner endpoint security analyzes.
Prior to ATT&CK Evaluations, Frank helped advance the concept of post-exploit detection by exploring the benefits of host-based data, on the project that inspired the creation of the ATT&CK knowledge base. Needing a way to provably and repeatably measure progress, he then transitioned to managing red teamers where he advanced the concepts of Adversary Emulation. He also worked with a variety of government customers as a specialist in growing work programs, where he worked with them to embrace threat-informed defense concepts, including advancing malware analysis, ATT&CK-based analytics, and purple teaming. He oversaw another 30 evaluations, across a broad range of capabilities to ensure they addressed the threat, while meeting mission needs.
Frank started at MITRE in 2003 as an intern in Rome, NY, while obtaining his bachelor’s degree in Computer Engineering from Syracuse University. After graduation, he would start his full-time career in 2005. During his early years, he worked with radar data processing. As he projected a change in the work program, he decided to pursue a master’s degree in Computer Engineering, Information Assurance from Syracuse University. He received this degree in 2008, and shortly after became the face of the new local cyber work program, expanding and evolving MITRE’s presence at the site.
In MITRE ATT&CK, techniques describe the means by which adversaries achieve tactical goals, sub-techniques describe the same means but a more specific level, and procedures describe the variations that are precise implementations of those techniques. This precision in many ways is what enables adversary emulation, and makes it, well, emulation. It allows us to confidently and accurately call something “in the spirit of APT29”. In many cases, in an effort to try to be precise, we narrow the focus of our evaluations and only implement the limited procedures an adversary is known to perform. But what happens if procedural information is not available for a specific adversary? We have to make an assumption about them. We do our best to get in their mindset. We consider what we believe to be their end goals, but in the end, we are left with a couple choices. We can make an educated guess, but in this case we fall into the same trapping of above - a narrowed focus that might not even be accurate. The alternate is to implement a variety of procedures and hope that we effectively cover our bases.
Procedural variation looks at a single technique or sub-technique, and implements them in different ways, ideally to trigger different data sources, and thus potentially different defensive capabilities. It is for this reason that over the past year, there has been an increased awareness and advocacy for procedural variation. Procedural variation gives us greater confidence that when we say we have a defensive for the technique under test, the defense will actually work. Procedural variation comes with its own challenges; increased development costs and potentially reducing the accuracy of our emulations are only the start of that conversation. So how do we balance the benefits of procedure variation with the challenges? In this talk, we will present the key considerations to make when designing your ATT&CK test plans so that you can maximize your test plan’s bang-for-the-buck, gaining the key confidence that procedural variation offers while staying true to threat intelligence, and doing all of this while keeping budget in the back of our minds.
