DARPA's Cyber Grand Challenge took place this year right before DEFCON 24 and was a watershed moment for automated security analysis. All of the raw data from the competition has been out for awhile now, but I haven't seen anyone analyze it yet. So, I took a stab at it myself.

I was specifically interested in looking at how well teams actually performed. The fact is that the winning Cyber Reasoning System (CRS), ForAllSecure's Mayhem, didn't do anything for the last 30 or so rounds of the game due to Mayhem taking too long to receive data. So, I set out to try and answer what I felt was the most pressing question: Was the only winning move not to play?

A strange game @DARPACGC it seems the only winning move is not to play.

— Mayhem CRS (@MayhemCRS) August 5, 2016

Scoring Formula

To figure this out, we need to first understand exactly how the scores were calculated. This was described during the CGC stream, but they displayed two different, contradicting formulas (check out 20:35 and 22:35)! Since I clearly couldn't trust them, I wanted to find it in print. Turns out it's harder to find than you'd think.

It's not in the Finalist Event Information Sheet, it's not in the latest version of the CGC Rules (section 3.2.3 at least outlines some of the components of the scoring algorithm, though), it's not in the Event FAQ or the overall CGC FAQ, and I don't see it posted on either their website or GitHub. There is, however, a little graphic on the bottom of page 4 of the CFE Brochure that states the overall formula:

score = availability * security * evaluation * 100

...where score is the score for an individual challenge set (CS). Each of the scores (one per CS) is then added together to calculate the final score for that round. Each of the scores for each round is added together to calculate the final total score. Hilariously, all of this contradicts everything shown during the event.

How each of the component scores themselves are calculated is never explained. I'm guessing they kept the same overall framework from the CQE Scoring Document. The full explanation is sorta lengthy (see the document), but here's the gist:

• The availability metric ensures that the service is available (not disconnected or using too many resources) and functional (not missing required features). It is scored on a scale of 0 to 1. If your service isn't available at all, you receive no points for anything. If your service is perfectly available, you get 1 point. If there are problems, you receive something in-between (see formula in document).
• The security metric ensures that the service is not vulnerable. It is scored on a scale of 0 to 1. If your service is vulnerable to all reported proofs of vulnerability for that service, you receive no points. If your service is no longer vulnerable after patching you receive 1 point. If your service is partially vulnerable, you receive something in-between (see formula in document).
• The evaluation metric grants bonus points for finding vulnerabilities. It is scored on a scale of 1 to 2. If you don't discover any vulnerabilities, you simply get 1 point. If you prove that a vulnerability exists, you get 2 points.

I believe there are only two marked differences between this explanation and what's published for the CFE (aside from the CQE using challenge binaries and the CFE using challenge sets). The first is that the security metric is now scored entirely based on PoVs from other teams and doesn't check if you've patched vulnerabilities the organizers intended (it's also now scored between 1 and 2). The second is that the evaluation metric is no longer simply 1 or 2 - it can be in-between if your PoV only works on some of the teams, rather than all of them.

Assuming the above formulas and explanations are correct, we can say that, for each challenge set in a round, a CRS doing everything right would receive 400 points (1 * 2 * 2 * 100 = 400). A CRS doing nothing, by contrast, would receive either 100 points (1 * 1 * 1 * 100 = 100) if an exploit exists, or 200 points (1 * 2 * 1 * 100 = 200) if no exploit exists. Multiplying each of these by the number of challenge sets in a given round will give us the score for that round, and adding all those together will give us theoretical total scores.

UPDATE (2017-02-19): While I'd been waiting for confirmation from my employer that I was clear to post this (since I had small involvement in our CGC effort prior to CQE), Shellphish released a write-up of their CGC experiences in Phrack. They confirmed everything I said above (and also provided some insights into why the availability score was so difficult for teams during the competition). It's a great paper and I recommend reading it after this post if you're interested in more information.

Scoring Data

Alright, so, let's move on to the actual scoring data. The data for each scoring round is provided as a 13GB tarball by DARPA. This tarball contains folders named cfe-export-bundle-N-T where N appears to be the round number and T appears to be a timestamp in epoch time. There are 96 folders in total, one for each round (not 95, like some of these articles would have you expect - the first round is round 0).

Inside each folder are two files: A blank file called flag and another tarball with the name cfe-export-bundle-N (where N is the round number again). Inside the tarball are two folders:

• files, which contains a number of .ids and .rcb files along with .zips named csid_X-round_N (where N is the round and I have no idea what X is)
• N (where N is the round number yet again), which contains a crs_data.csv and a score_data.json.

I wasn't sure what to do with the stuff inside of files, so I started with the other folder. Each crs_data.csv file contains a bunch of lines that look like this:

2016-08-04 15:56:02,lcp-red,lcp-cooling-power,8129.0
2016-08-04 15:56:02,lcp-red,lcp-water-temp-in,57.0
2016-08-04 15:56:02,lcp-red,lcp-water-temp-out,63.0
2016-08-04 15:56:02,lcp-red,lcp-water-valve,52.0

These appear to be the logs of what was happening with each system's hardware as the game was running. Makes sense, given that a hardware failure would obviously have ruined the game. I skipped over these since they didn't contain any scoring data.

The score_data.json files are really what we want to look at. They contain a huge dump of data from each scoring round with the following top-level keys:

• round - The round number.
• challenges - The IDs of the challenge binaries (CBs) scored during this round.
• csid_map - The full mapping of CB IDs to CB name for the entire competition.
• rank - The score for each team during this round.
• teams - A full dump of all actions from each team.

I un-packed all the files with this BASH one-liner (it's 37GB of data when decompressed, if you were wondering):

cd cgc-submissions; for d in *; do cd $d; tar xf *.tar; cd ..; done

I then used Ruby to load the JSON data and do some analysis. By inspecting the challenges data, we can see that every round (with exceptions) had 15 challenge sets that were to be scored. The exceptions to this rule were:

• Round 0 - 10 challenge sets
• Round 1 - 13 challenge sets
• Round 12 - 14 challenge sets
• Round 92 - 14 challenge sets
• Round 93 - 13 challenge sets
• Round 94 - 12 challenge sets
• Round 95 - 11 challenge sets

By inspecting the rank key, we can see what each team's score was in a given round. The mapping of team ID to team appears to be:

• Team 1 - Galactica
• Team 2 - Jima
• Team 3 - Rubeus
• Team 4 - Crspy
• Team 5 - Mayhem
• Team 6 - Mechaphish
• Team 7 - Xandra

At first, I thought this data was what each team scored that round. That doesn't make any sense when you look at all the data. Instead, it appears to be what each team's total score was as of that round. To get a complete list of scores per round, you have to subtract out the score from the previous round.

Now that we know how many challenge sets there were and what each team's score was per round, the last piece of data we need is how many exploits existed in each round. For simplicity's sake, we'll assume that any successful proof of vulnerability (PoV) against any other CRS for a particular challenge set would work against a CRS doing nothing. To get this data, we need to loop through each team's pov_results. A successful PoV will be marked with a result of success (unsuccessful PoVs say failure).

Final Result

I implemented everything above in this Ruby script:

#!/usr/bin/env ruby

require "json"

scores = []
challenges = []
Dir["cfe-submissions/*"].each do |dir|
    File.open("#{dir}/#{dir.split("-")[4]}/score_data.json", "r") do |f|
        round = []
        json = JSON.load(f.read())
        i = json["round"]

        # get challenges
        challenges[i] = {}
        json["challenges"].each do |cs|
            challenges[i][cs] = false
        end

        # get scores
        json["rank"].each do |rank|
            round[rank["team"]-1] = rank["score"]
        end
        scores[i] = round

        # get successful exploits
        json["teams"].each do |team, data|
            data["pov_results"].each do |_, pov|
                if pov.keys.include?("result") and pov["result"] == "success"
                    challenges[i][pov["csid"]] = true
                end
            end
        end
    end
end

last = nil
wopr = []
perfect = []
puts "| Round | Galactica | Jima | Rubeus | Crspy | Mayhem | Mechaphish | Xandra | WOPR | Perfection |"
puts "| -------------------------------------------------------------------------------------------- |"
scores.each_with_index do |round, i|
    this = round.dup
    this.length.times { |j| this[j] -= last[j] } if last
    wopr[i] = 0
    perfect[i] = 0
    challenges[i].each do |id, pwned|
        if pwned then wopr[i] += 100 else wopr[i] += 200 end
        perfect[i] += 400
    end
    puts "| #{i} | #{this[0]} | #{this[1]} | #{this[2]} | #{this[3]} | #{this[4]} | #{this[5]} | #{this[6]} | #{wopr[i]} | #{perfect[i]} |"
    last = round
end
puts "| TOTAL | #{last[0]} | #{last[1]} | #{last[2]} | #{last[3]} | #{last[4]} | #{last[5]} | #{last[6]} | #{wopr.reduce(0, :+)} | #{perfect.reduce(0, :+)} |"

If you run it, it'll give you the following table (the "WOPR" column is our do-nothing CRS, while the "Perfection" column shows the theoretical maximum score for that round):

Yes. You are reading that correctly. A CRS that literally did nothing could have placed 3rd. Setting up absolutely no software in the CFE could have won you $750,000. It would seem that, while not playing wasn't a winning move, it could still have been worth some serious money.

I do have to mention two caveats, though:

1. Although I tried to figure out what the scoring algorithm was, I haven't been able to completely audit all the scores for the entire event. All the data above simply trusts that each round was scored appropriately by the organizers.
2. I believe it's possible that a given CRS may have submitted a PoV after all other CRSs had patched. As a result, there may be PoVs that didn't work on anyone (and, thus, weren't scored), but that would have worked on a system doing absolutely no patching. One could probably determine definitively if this is or isn't the case by taking all submitted PoVs and re-throwing them. Unfortunately, that would take a fair amount of effort and I don't have time for it right now.

Regardless, I think it's still pretty obvious just how far these systems have to go before they're taking my job or making the world a safer place.

Design Thoughts

To wrap up, I thought a little bit about the design of the competition and how this might be avoided. I think the problem here is that, unlike the CQE, there was no "oracle" whose score counted. The CQE had teams losing points if any of the "known" vulnerabilities weren't patched in addition to losing points for vulnerabilities other teams uncovered. Here, the only way to lose points is if other teams find vulnerabilities.

Given that the stated purpose of CGC was to "create automatic defensive systems capable of reasoning about flaws, formulating patches, and deploying them on a network in real time", this might seem odd. How can you appropriately test if a system was capable of these things if you don't at least score them on flaws you already know about?

I think the answer lies in what the CGC organizers intended the competition to be about. We already have generic mitigations. CGC was about building systems that could, in the most ideal case, fix every flaw with a targeted patch instead. In order to develop those targeted patches, you have to find and understand the vulnerability. Demonstrating true understanding requires not only a targeted patch that works, but also a proof of that vulnerability. Not having an oracle means the only teams that consistently score are those with both a patch and a proof.

I still feel the design of the game was solid. The fact that a do-nothing system would have scored well should speak to the difficulty of the problem space and the incompleteness of each system. Not poor game design. I also have a feeling that teams didn't place as much emphasis on offense and availability (where it seems most teams missed or lost points) given the uncertainty about the CFE rules.

I'm not sure if there will be another Cyber Grand Challenge. If there is, I would expect to see this become less and less of a problem as time goes on and systems improve.