At DEFCON 23 this weekend, it was announced that next year's DEFCON CTF will feature the winning cyber reasoning system (CRS) from the Cyber Grand Challenge finals playing alongside the human teams. This will be a watershed moment, but I'm not convinced it's a positive one for the CTF scene (and, specifically, the DEFCON CTF).
Don't get me wrong, I love the Cyber Grand Challenge. I'm incredibly excited for CGC finals - and not just because of my involvement with my employer's entry or my teammates' and former co-workers' work on the event itself. I think CGC has huge potential for advancing "cybersecurity" and Mike Walker should be extremely proud of what he's put together at DARPA.
Since the initial announcement in 2013, Mike has been drawing a lot of parallels between CGC and the World Computer Chess Championship (WCCC). The hope for CGC finalists, just like IBM and "Deep Blue" before them, is to see their automated system(s) one day defeat humans at their own game. Only, this time, winning the game will have a very real application to very real problems (and translate into very real amounts of "cash money"). Thus, it's perhaps inevitable that a CRS will compete at what is, effectively, the Hacker Olympics.
Chess and CTFs have a few key differences, however:
In chess, you block on the opponent. Each player only gets to make one move at a time. In a CTF, that constraint doesn't really exist.
In chess, despite an incredibly simple ruleset, it's hard to prepare for an opponent's strategy. You can analyze a particular player's past performances and there are accepted lines of play, but you're still ultimately dependent on input from the opponent. Not so in a CTF. Each CTF service has at least one provably correct "solution" for each vulnerability. Discovering, exploiting, and patching these requires no input from the opponent at all. Thus, the only way to get ahead is if you're faster, if the opponent's patch was incomplete, or if you devise and install a side-channel for obtaining flags.
In chess, the humans aren't aided by computers. In a CTF, each team has its own automation (mostly for throwing exploits, but also for some aspects of reverse engineering, vulnerability assessment, and initial exploitation).
The first problem is obvious: Theoretically, a CRS could burn through all the challenge binaries in seconds and effectively "finish" the competition immediately after it starts. This is, in my opinion, pretty unlikely for now. I'm confident Legitimate Business Syndicate (LegitBS) will create extremely challenging services and I'm not confident any CRS is that far ahead at the moment.
The second problem is less obvious: Having the game on the CGC architecture changes the very nature of the game we've been playing for the past 11 years.
In past DEFCON CTF Finals, the organizers created custom services that could't be understood beforehand. They also provided you with a server and prevented you from having kernel access (though you did, generally, have root). You could still deploy your own system if you were good, even up to DDTEK's last year (DEFCON 20). I'm confident at least one team did this in DEFCON 19 as a "perfect defense". But, since protections couldn't be turned on and services were custom, defenders had to actually understand and patch their services (or, at least, this was the intended design - the implementation missed the mark a bit).
Starting in DEFCON 21, LegitBS took it one step further: You now don't even have root on your own server (so you can't mess with the network like we did in DEFCON 19). Both attack and defense have to be performed at the binary level on services that aren't known ahead of time. LegitBS also turned on a number of protections like DEP and ASLR for everyone by default, but made their custom services still exploitable under those conditions.
I'm familiar with Kenshoto's competitions and I played in both DDTEK's and LegitBS's. I have to say, LegitBS has done a wonderful job keeping DEFCON CTF focused on the skill of the players, rather than the tools they have available. Sure, teams still have tools: PPP had Qira, Gallopsled had Pwntools, and we had Binary Ninja. But, these are general tools meant to enhance, rather than replace, skills possessed by the team.
The reason these tools were more general is because teams couldn't count on knowing the architecture or platform before the game started. To be fair: We mostly had it figured out. The game ran on x86 FreeBSD for 3 of the 4 Kenshoto games (it was x86 Solaris at DEFCON 14) and for all 4 of DDTEK's games (though it was IPv6 and not IPv4 for DEFCON 19 and 20). And LegitBS hosted their game on ARMv7 Linux for both DEFCON 21 and 22.
But, if you created all your tools for something, and you guessed wrong, a lot of your preparation went out the window. As an example, the switch to IPv6 in DEFCON 19 required us to re-write all our defenses on the first day of the competition. This caused us (and many other teams) to focus on multi-platform support for our tools so we could cover all the possible bases. During DDTEK's years, for example, we had everything working on x86 and x64, FreeBSD and Linux. We had considered ARMv7 and MIPS support for our tools as well.
With the announcement that next year's DEFCON CTF will be on the CGC architecture (so that a CRS may compete), teams now know it will be x86 DECREE. As a result, any team looking to compete must now either create their own CRS or figure out how to leverage an existing one (or pieces of one like McSema or Angr). Why? Because you have to. Everyone else competing will have access to something because everyone knows what the game will look like in advance. There's no mystery to cause teams to "waste" development time or deter them.
Individual skill will undoubtedly be a factor next year. But, I'm left wondering whether next year's DEFCON CTF will tell us anything more than how well-developed each team's tools are (and how well they can interpret the results). I believe we've already seen a slight shift toward a tools-focused trend at this year's DEFCON CTF Qualifier, and not just because of the CGC challenge that was fielded.
If a future DEFCON CTF is run on a non-standard architecture or platform, it's entirely possible we'll see a full return to the game as we know it. But, I don't think it's too far of a stretch to say that certain configurations (x86 Linux being the prime example) could soon not be viable due to available tools. And this would likely extend to other CTFs as well (such as DEFCON CTF qualifying events).
Are we witnessing an end of an era in CTFs? Am I crazy? Was this inevitable? Is this even a bad thing? I'm interested to know what the community thinks.
UPDATE (2016-08-13): It's been a year since I wrote this and I don't think I was off-the-mark by much. Although I was not fortunate enough to qualify, I've talked with a number of participants and their experiences seem to validate my concerns. Heck, half of Zachary Wade's retrospective on PPP winning is him talking about their tools.
On top of that, LegitBS announced at DEFCON (can't find a link, so you'll have to take my word for it) that they will be running next year's game on a custom architecture and OS. Seems they feel similarly that the only way to keep the game focused on individual skill is to break all the tools (or at least require people to automate for optimizing human analysis time, rather than automating for the solution itself).
Ultimately, I'm still not sure what to make of the current CTF landscape and whether it's moving in a positive direction or not. But, I think it's important we all recognize that it is shifting.