[Dailydave] CGC Wrapup Video

[dave aitel]

So I wanted to type up some notes on the CGC Wrapup

video, which was
excellent. I mean, a part of what you want to do, while you watch it, is
strip out all the parts of the thing that are about "playing the game".
I know Jordan loves CTFs as some sort of e-sport and also there's a
whole community who for whatever reason plays CTFs instead of playing
corewars on helpless Chinese networks like of yore, but that stuff is
100% distraction when it comes to the CGC.


As you can see, the tiny red lines on the right are supposed to be some
combination of "could hack and could secure a service". I can't find
anywhere something that has a simple spreadsheet of which samples
<http://www.lungetech.com/cgc-corpus/challenges/NRFIN_00080/> (and even
which vulns in which samples) were able to be attacked by which teams.
So much of the game was weighted towards performance characteristics
that it's hard to determine the information you really need from the
scores, although the video goes over some anecdotal examples where
RUBEUS and MECHAPHISH were able to attack particular historically
interesting programs. It's telling that Mayhem won despite being
basically off for half the contest. ;)

Does anyone have better data on this?

-dave

P.S. Holy cow the visualizations on program execution are next gen!
Worth a close watch just to see them.

[Jordan Wiens]

For what it's worth, the CTF-as-esport dream takes more work than I have
time these days unfortunately. I'm still convinced it will happen, just not
sure who will do it. That said, visualizations like this are going to be
key to pulling it off.

A really nice interface to the raw data is available from Lunge:
http://www.lungetech.com/cgc-corpus/

The modified Qemu and API to produce execution traces for visualization
(along with yet another web interface to pick through the raw data which is
what we used during the live event to try to make sense of the chaos) from
Rusty and I is online at: https://github.com/Vector35/trace-api

And most importantly, the software to visualize the traces is open sourced
here: https://github.com/voidALPHA/cgc_viz

Generating trace-files that are capable of visualization isn't hard. At
their simplest form you just need an instruction pointer trace over an
execution. Adding on disassembly, register contents, data flow, and memory
read/writes makes for a much more useful visualization, but there's a
pretty surprising amount of value just in instruction pointer "shapes".

Bonus related links:
Raw data: https://github.com/lungetech/cgc-corpus
https://github.com/lungetech/cgc-cfe-submission-corpus
https://github.com/lungetech/cgc-cqe-submission-corpus
Trail of Bits is maintaining a patched version of the binaries and build
configurations (https://github.com/trailofbits/cb-multios/) with the goal
of being able to be built across lots of platforms. In many ways, one of
the best legacies of CGC is the very convenient and large sample corpus
with included exploits, patches, and functionality polls

Post by dave aitel
So I wanted to type up some notes on the CGC Wrapup
http://youtu.be/SYYZjTx92KU video, which was excellent.
I mean, a part of what you want to do, while you watch it, is strip out all
the parts of the thing that are about "playing the game". I know Jordan
loves CTFs as some sort of e-sport and also there's a whole community who
for whatever reason plays CTFs instead of playing corewars on helpless
Chinese networks like of yore, but that stuff is 100% distraction when it
comes to the CGC.
As you can see, the tiny red lines on the right are supposed to be some
combination of "could hack and could secure a service". I can't find
anywhere something that has a simple spreadsheet of which samples
<http://www.lungetech.com/cgc-corpus/challenges/NRFIN_00080/> (and even
which vulns in which samples) were able to be attacked by which teams. So
much of the game was weighted towards performance characteristics that it's
hard to determine the information you really need from the scores, although
the video goes over some anecdotal examples where RUBEUS and MECHAPHISH
were able to attack particular historically interesting programs. It's
telling that Mayhem won despite being basically off for half the contest. ;)
Does anyone have better data on this?
-dave
P.S. Holy cow the visualizations on program execution are next gen! Worth
a close watch just to see them.
_______________________________________________
Dailydave mailing list
https://lists.immunityinc.com/mailman/listinfo/dailydave

[Tyler Nighswander]

I think I posted a link to this on here before, but
http://www.lungetech.com/cgc-corpus has some information about each
challenge, including whether there was a successful POV on it during the
contest (though it's not the easiest thing to navigate).
Most of the challenges have no successful POVs against them. In my
totally-neutral-not-biased-at-all-objective-opinion, that is because Mayhem
was borked for a large portion of the contest ;). Mayhem exploited 11
unique services for however long it was working (it started degrading
around round 30), Mecaphish exploited the most of any competitor during the
game 15 total. That's out of around 100 or so total challenges, so not a
very high percentage. I didn't spend much time looking to see how hard the
CFE challenges were, but they are not buffer overflow 101 type of things,
I'd say.

Post by dave aitel
So I wanted to type up some notes on the CGC Wrapup
http://youtu.be/SYYZjTx92KU video, which was excellent.
I mean, a part of what you want to do, while you watch it, is strip out all
the parts of the thing that are about "playing the game". I know Jordan
loves CTFs as some sort of e-sport and also there's a whole community who
for whatever reason plays CTFs instead of playing corewars on helpless
Chinese networks like of yore, but that stuff is 100% distraction when it
comes to the CGC.
As you can see, the tiny red lines on the right are supposed to be some
combination of "could hack and could secure a service". I can't find
anywhere something that has a simple spreadsheet of which samples
<http://www.lungetech.com/cgc-corpus/challenges/NRFIN_00080/> (and even
which vulns in which samples) were able to be attacked by which teams. So
much of the game was weighted towards performance characteristics that it's
hard to determine the information you really need from the scores, although
the video goes over some anecdotal examples where RUBEUS and MECHAPHISH
were able to attack particular historically interesting programs. It's
telling that Mayhem won despite being basically off for half the contest. ;)
Does anyone have better data on this?
-dave
P.S. Holy cow the visualizations on program execution are next gen! Worth
a close watch just to see them.
_______________________________________________
Dailydave mailing list
https://lists.immunityinc.com/mailman/listinfo/dailydave

[Chris Eagle]

Dave,

You may find some of what you want here: http://www.lungetech.com/cgc-corpus/cfe/

I have all the raw data from the event including the answers to some of your questions below. If I can format then in some useful manner I will post some of those answers.

Chris

So I wanted to type up some notes on the CGC Wrapup http://youtu.be/SYYZjTx92KU video, which was excellent. I mean, a part of what you want to do, while you watch it, is strip out all the parts of the thing that are about "playing the game". I know Jordan loves CTFs as some sort of e-sport and also there's a whole community who for whatever reason plays CTFs instead of playing corewars on helpless Chinese networks like of yore, but that stuff is 100% distraction when it comes to the CGC.
As you can see, the tiny red lines on the right are supposed to be some combination of "could hack and could secure a service". I can't find anywhere something that has a simple spreadsheet of which samples <http://www.lungetech.com/cgc-corpus/challenges/NRFIN_00080/> (and even which vulns in which samples) were able to be attacked by which teams. So much of the game was weighted towards performance characteristics that it's hard to determine the information you really need from the scores, although the video goes over some anecdotal examples where RUBEUS and MECHAPHISH were able to attack particular historically interesting programs. It's telling that Mayhem won despite being basically off for half the contest. ;)
Does anyone have better data on this?
-dave
P.S. Holy cow the visualizations on program execution are next gen! Worth a close watch just to see them.
_______________________________________________
Dailydave mailing list
https://lists.immunityinc.com/mailman/listinfo/dailydave

[dave aitel]

Ah, it's there for sure, although you're not sure which bug they
exploited. Interesting to draw some corrolations. For example DeepRed
(Raytheon) got two weird heap overflows exploited, and then a lot of
stack overflows...did that heap overflow come from a replay of someone
else's bug? Is that a thing?

Heap Overflows:

1. http://www.lungetech.com/cgc-corpus/challenges/CROMU_00055/
2. *http://www.lungetech.com/cgc-corpus/challenges/NRFIN_00052/*

Hmm. Lots of interesting information here, although somewhat hard to dig
through I guess?

-dave

Post by Chris Eagle
Dave,
You may find some of what you want here: http://www.lungetech.com/cgc-corpus/cfe/
I have all the raw data from the event including the answers to some of your questions below. If I can format then in some useful manner I will post some of those answers.
Chris

So I wanted to type up some notes on the CGC Wrapup http://youtu.be/SYYZjTx92KU video, which was excellent. I mean, a part of what you want to do, while you watch it, is strip out all the parts of the thing that are about "playing the game". I know Jordan loves CTFs as some sort of e-sport and also there's a whole community who for whatever reason plays CTFs instead of playing corewars on helpless Chinese networks like of yore, but that stuff is 100% distraction when it comes to the CGC.
As you can see, the tiny red lines on the right are supposed to be some combination of "could hack and could secure a service". I can't find anywhere something that has a simple spreadsheet of which samples <http://www.lungetech.com/cgc-corpus/challenges/NRFIN_00080/> (and even which vulns in which samples) were able to be attacked by which teams. So much of the game was weighted towards performance characteristics that it's hard to determine the information you really need from the scores, although the video goes over some anecdotal examples where RUBEUS and MECHAPHISH were able to attack particular historically interesting programs. It's telling that Mayhem won despite being basically off for half the contest. ;)
Does anyone have better data on this?
-dave
P.S. Holy cow the visualizations on program execution are next gen! Worth a close watch just to see them.
_______________________________________________
Dailydave mailing list
https://lists.immunityinc.com/mailman/listinfo/dailydave

[Jordan Wiens]

Replaying someone's bug was absolutely a thing.

Each team was given what amounts to a direct feed of all network traffic to
their server. If they had good instrumentation they could replay it locally
and automatically detect which flows represented successful exploits and
which didn't.

There are some interesting ideas though on how you might ensure that an
automated system can't do such a thing. Rubeus, for example, fingerprinted
the cpuid output of the target infrastructure and introduced divergent
behavior based on that cpuid. I don't know if it made the final cut of the
video (still watching it now!) but we did find teams biting on their
honeypot on multiple occasions. A team would be successfully exploit a
vulnerability, Rubeus would replace the service with one similar except
adding a fake vuln only reachable with a non CGC infrastructure cpuid and
the team would now target that vulnerability, losing out on the points they
were getting before and netting rubeus some free defense points when they
were still vulnerable.

Post by dave aitel
Ah, it's there for sure, although you're not sure which bug they
exploited. Interesting to draw some corrolations. For example DeepRed
(Raytheon) got two weird heap overflows exploited, and then a lot of stack
overflows...did that heap overflow come from a replay of someone else's
bug? Is that a thing?
1. http://www.lungetech.com/cgc-corpus/challenges/CROMU_00055/
2. *http://www.lungetech.com/cgc-corpus/challenges/NRFIN_00052/
<http://www.lungetech.com/cgc-corpus/challenges/NRFIN_00052/>*
Hmm. Lots of interesting information here, although somewhat hard to dig
through I guess?
-dave
Dave,
You may find some of what you want here: http://www.lungetech.com/cgc-corpus/cfe/
I have all the raw data from the event including the answers to some of your questions below. If I can format then in some useful manner I will post some of those answers.
Chris
So I wanted to type up some notes on the CGC Wrapup http://youtu.be/SYYZjTx92KU http://youtu.be/SYYZjTx92KU video, which was excellent. I mean, a part of what you want to do, while you watch it, is strip out all the parts of the thing that are about "playing the game". I know Jordan loves CTFs as some sort of e-sport and also there's a whole community who for whatever reason plays CTFs instead of playing corewars on helpless Chinese networks like of yore, but that stuff is 100% distraction when it comes to the CGC.
As you can see, the tiny red lines on the right are supposed to be some combination of "could hack and could secure a service". I can't find anywhere something that has a simple spreadsheet of which samples <http://www.lungetech.com/cgc-corpus/challenges/NRFIN_00080/> <http://www.lungetech.com/cgc-corpus/challenges/NRFIN_00080/> (and even which vulns in which samples) were able to be attacked by which teams. So much of the game was weighted towards performance characteristics that it's hard to determine the information you really need from the scores, although the video goes over some anecdotal examples where RUBEUS and MECHAPHISH were able to attack particular historically interesting programs. It's telling that Mayhem won despite being basically off for half the contest. ;)
Does anyone have better data on this?
-dave
P.S. Holy cow the visualizations on program execution are next gen! Worth a close watch just to see them.
_______________________________________________
_______________________________________________
Dailydave mailing list
https://lists.immunityinc.com/mailman/listinfo/dailydave

[Dave Aitel]

So basically what we REALLY want is to know which team found a POV first?
That CPUID thing did make it into the video btw. That really complicates
the analysis. If you ran this again, maybe there is another way?


https://github.com/lungetech/cgc-challenge-corpus/blob/master/CROMU_00055/src/proto.c
<--would
def. expect simple static analysis to find this. (Shellphish found it
first, I think, but you would expect every time to find this?)

How many vulns did Shellphish find that no one else found? What's the
overlap rate? I see a lot of stack corruption bugs in the corpus - do we
have statistics for what the types of vulns solved were?

This one is interesting:
http://www.lungetech.com/cgc-corpus/challenges/CROMU_00058/

Also, did ForAllSecure or any other teams fix and rerun their engines on
the corpus?
-dave

Post by Jordan Wiens
Replaying someone's bug was absolutely a thing.
Each team was given what amounts to a direct feed of all network traffic
to their server. If they had good instrumentation they could replay it
locally and automatically detect which flows represented successful
exploits and which didn't.
There are some interesting ideas though on how you might ensure that an
automated system can't do such a thing. Rubeus, for example, fingerprinted
the cpuid output of the target infrastructure and introduced divergent
behavior based on that cpuid. I don't know if it made the final cut of the
video (still watching it now!) but we did find teams biting on their
honeypot on multiple occasions. A team would be successfully exploit a
vulnerability, Rubeus would replace the service with one similar except
adding a fake vuln only reachable with a non CGC infrastructure cpuid and
the team would now target that vulnerability, losing out on the points they
were getting before and netting rubeus some free defense points when they
were still vulnerable.

Post by dave aitel
Ah, it's there for sure, although you're not sure which bug they
exploited. Interesting to draw some corrolations. For example DeepRed
(Raytheon) got two weird heap overflows exploited, and then a lot of stack
overflows...did that heap overflow come from a replay of someone else's
bug? Is that a thing?
1. http://www.lungetech.com/cgc-corpus/challenges/CROMU_00055/
2. *http://www.lungetech.com/cgc-corpus/challenges/NRFIN_00052/
<http://www.lungetech.com/cgc-corpus/challenges/NRFIN_00052/>*
Hmm. Lots of interesting information here, although somewhat hard to dig
through I guess?
-dave
Dave,
You may find some of what you want here: http://www.lungetech.com/cgc-corpus/cfe/
I have all the raw data from the event including the answers to some of your questions below. If I can format then in some useful manner I will post some of those answers.
Chris
So I wanted to type up some notes on the CGC Wrapup http://youtu.be/SYYZjTx92KU http://youtu.be/SYYZjTx92KU video, which was excellent. I mean, a part of what you want to do, while you watch it, is strip out all the parts of the thing that are about "playing the game". I know Jordan loves CTFs as some sort of e-sport and also there's a whole community who for whatever reason plays CTFs instead of playing corewars on helpless Chinese networks like of yore, but that stuff is 100% distraction when it comes to the CGC.
As you can see, the tiny red lines on the right are supposed to be some combination of "could hack and could secure a service". I can't find anywhere something that has a simple spreadsheet of which samples <http://www.lungetech.com/cgc-corpus/challenges/NRFIN_00080/> <http://www.lungetech.com/cgc-corpus/challenges/NRFIN_00080/> (and even which vulns in which samples) were able to be attacked by which teams. So much of the game was weighted towards performance characteristics that it's hard to determine the information you really need from the scores, although the video goes over some anecdotal examples where RUBEUS and MECHAPHISH were able to attack particular historically interesting programs. It's telling that Mayhem won despite being basically off for half the contest. ;)
Does anyone have better data on this?
-dave
P.S. Holy cow the visualizations on program execution are next gen! Worth a close watch just to see them.
_______________________________________________
_______________________________________________
Dailydave mailing list
https://lists.immunityinc.com/mailman/listinfo/dailydave

_______________________________________________
Dailydave mailing list
https://lists.immunityinc.com/mailman/listinfo/dailydave

[Dave Aitel]

[image: image.png]

Like - this is fascinating.
Shellphish gets it first, then later ForAllSecure gets it. But are they
replaying it? Why doesn't Shellphish patch their version (or if they do,
why doesn't it work?) ? Why doesn't ForAllSecure's work against Diseckt
later?

I still have so many questions about the CGC.

Bug is interesting too...
[image: image.png]
-dave

Post by Dave Aitel
So basically what we REALLY want is to know which team found a POV first?
That CPUID thing did make it into the video btw. That really complicates
the analysis. If you ran this again, maybe there is another way?
https://github.com/lungetech/cgc-challenge-corpus/blob/master/CROMU_00055/src/proto.c <--would
def. expect simple static analysis to find this. (Shellphish found it
first, I think, but you would expect every time to find this?)
How many vulns did Shellphish find that no one else found? What's the
overlap rate? I see a lot of stack corruption bugs in the corpus - do we
have statistics for what the types of vulns solved were?
http://www.lungetech.com/cgc-corpus/challenges/CROMU_00058/
Also, did ForAllSecure or any other teams fix and rerun their engines on
the corpus?
-dave

Post by Jordan Wiens
Replaying someone's bug was absolutely a thing.
Each team was given what amounts to a direct feed of all network traffic
to their server. If they had good instrumentation they could replay it
locally and automatically detect which flows represented successful
exploits and which didn't.
There are some interesting ideas though on how you might ensure that an
automated system can't do such a thing. Rubeus, for example, fingerprinted
the cpuid output of the target infrastructure and introduced divergent
behavior based on that cpuid. I don't know if it made the final cut of the
video (still watching it now!) but we did find teams biting on their
honeypot on multiple occasions. A team would be successfully exploit a
vulnerability, Rubeus would replace the service with one similar except
adding a fake vuln only reachable with a non CGC infrastructure cpuid and
the team would now target that vulnerability, losing out on the points they
were getting before and netting rubeus some free defense points when they
were still vulnerable.

Post by dave aitel
Ah, it's there for sure, although you're not sure which bug they
exploited. Interesting to draw some corrolations. For example DeepRed
(Raytheon) got two weird heap overflows exploited, and then a lot of stack
overflows...did that heap overflow come from a replay of someone else's
bug? Is that a thing?
1. http://www.lungetech.com/cgc-corpus/challenges/CROMU_00055/
2. *http://www.lungetech.com/cgc-corpus/challenges/NRFIN_00052/
<http://www.lungetech.com/cgc-corpus/challenges/NRFIN_00052/>*
Hmm. Lots of interesting information here, although somewhat hard to dig
through I guess?
-dave
Dave,
You may find some of what you want here: http://www.lungetech.com/cgc-corpus/cfe/
I have all the raw data from the event including the answers to some of your questions below. If I can format then in some useful manner I will post some of those answers.
Chris
So I wanted to type up some notes on the CGC Wrapup http://youtu.be/SYYZjTx92KU http://youtu.be/SYYZjTx92KU video, which was excellent. I mean, a part of what you want to do, while you watch it, is strip out all the parts of the thing that are about "playing the game". I know Jordan loves CTFs as some sort of e-sport and also there's a whole community who for whatever reason plays CTFs instead of playing corewars on helpless Chinese networks like of yore, but that stuff is 100% distraction when it comes to the CGC.
As you can see, the tiny red lines on the right are supposed to be some combination of "could hack and could secure a service". I can't find anywhere something that has a simple spreadsheet of which samples <http://www.lungetech.com/cgc-corpus/challenges/NRFIN_00080/> <http://www.lungetech.com/cgc-corpus/challenges/NRFIN_00080/> (and even which vulns in which samples) were able to be attacked by which teams. So much of the game was weighted towards performance characteristics that it's hard to determine the information you really need from the scores, although the video goes over some anecdotal examples where RUBEUS and MECHAPHISH were able to attack particular historically interesting programs. It's telling that Mayhem won despite being basically off for half the contest. ;)
Does anyone have better data on this?
-dave
P.S. Holy cow the visualizations on program execution are next gen! Worth a close watch just to see them.
_______________________________________________
_______________________________________________
Dailydave mailing list
https://lists.immunityinc.com/mailman/listinfo/dailydave

_______________________________________________
Dailydave mailing list
https://lists.immunityinc.com/mailman/listinfo/dailydave