~/blog/defcon34-owasp-ctf

The final numbers from the OWASP Foundation CTF at DEF CON 34:

• 2,317 pull requests
• 5,799 workflow runs
• 5,827 solves across 321 challenges
• 60 players, 23 teams
• 0 pull requests merged

That last one is not a failure mode. It’s the design.

The Inversion #

Most CTFs pay you for breaking things. Find the injection, pull the flag, submit the string, move on.

This one pays you for closing them.

Contestants forked one of six deliberately vulnerable OWASP apps, found a vulnerability, and then wrote the patch. The submission was a pull request against their own fork. A GitHub Action booted the patched app, ran the rubric for that challenge, and posted a score. Nobody typed a flag into a box. Nobody graded by hand.

A challenge only counted when the fix actually blocked the exploit.

That single rule is what makes the format worth the trouble. Exploiting a known bug in a training app is a skill you can rent for an afternoon. Understanding it well enough to close it, without breaking the application around it, is the part that transfers to Monday morning.

The pull requests were never meant to land. The app has to stay broken for the next player. The scoreboard is the artifact, not the merge.

One Scorer, Six Targets #

The six targets were VAmPI, WebGoat, VulnerableApp, Juice Shop, DVWA and Security Shepherd. Projects a lot of us learned on, maintained for years by people who never asked for anything back.

Six apps means six languages, six build systems, six ways to stand up a database. Juice Shop is Node. VulnerableApp is Spring Boot. Security Shepherd is Tomcat with MariaDB and MongoDB. VAmPI is a Flask container you seed with a call to /createdb.

The thing that kept this manageable was refusing to write six scorers.

There is one TypeScript scorer. It embeds every rubric for every target, ships as a private container image, and picks a rubric at runtime:

score --target juice-shop|dvwa|vulnerableapp|webgoat|securityshepherd|vampi

Each target contributes only a bring-up entrypoint: the containers to start, the health check to wait on, the seed to run. The scoring logic, the leaderboard write, the PR comment, all of it is shared. Adding a target became a rubric plus an entrypoint instead of a new pipeline.

The rubric itself is plain node:test files, one per challenge, zero dependencies. Each test performs the exploit. On an unpatched app the exploit fires and the test fails. On a patched app it doesn’t, and the test passes.

Pass on Patch, and the Bug It Exposed #

That inversion is the entire scoring model, and it has one invariant that has to hold everywhere:

A stock, unpatched app must score exactly zero.

Any point above zero on a stock app is a point paid for work nobody did.

When I first ran the whole thing against a stock Juice Shop image, it scored 4 out of 38. Two separate causes, both instructive.

The first one was embarrassing in a useful way. The repo carried a vendored copy of the Juice Shop source tree, and that copy was the fully solved reference: twenty four patch markers across seventeen files. Three challenges are static source analysis, and the scorer pointed them at that tree. They were reading the answer key. They passed before a contestant touched anything.

The second was upstream doing its job. One challenge, a zip slip that overwrites a file on upload, no longer reproduced on the newer image. Upstream had fixed it. On a deliberately vulnerable app, a silent version bump is a scoring bug: it quietly hands out points for a vulnerability that isn’t there anymore.

The fixes were to delete the vendored tree, pin every target to an exact version or digest rather than latest, and then add the backstop that should have existed first:

A stock-scores-zero workflow that boots each target’s stock image, scores it with the real rubric, and fails the build if anything scores above zero, naming the offenders.

The guard is worth more than either fix. Both causes were one-time bugs. Upstream drift is forever. Six targets pinned by hand will drift the moment someone bumps a tag in a hurry, and the failure is silent: the scores just quietly inflate. A test that asserts “broken app earns nothing” catches every future instance of that, loudly, in CI, before a contestant ever sees it.

The Forks Are Public, and They Hold the Answer Key #

The scoring boundary is the part I’d want someone to copy.

The workflow runs on pull_request_target, which means it runs with the base repo’s token against code an attacker wrote. Assume the pull request is hostile, because in a room full of DEF CON attendees it eventually will be.

Three properties held it together:

The patched app runs on an internet-less container network. It gets built and booted, but it cannot phone home. The registry credential lives on the host job, never inside the app.

The rubric is baked into the binary, never mounted. The tests are the answer key. Code in the pull request cannot read a file that was compiled into an image it never gets to open.

Secrets only exist in jobs that run no contestant code. Scoring runs untrusted code with read scopes. Writing to the leaderboard and commenting on the pull request happen in separate jobs, with no checkout and no build.

There was a fourth leak I found by acting like a contestant. The forks are public, so their Actions logs and build artifacts are readable by anyone with a GitHub account. The scorer was printing a full per challenge table into the log and uploading the results file as an artifact. That’s the complete answer key, downloadable by any player who thought to look. I confirmed it by downloading one from a live run.

The fix was a --quiet mode that reports totals and nothing else, plus passing results between jobs as a base64 job output instead of an artifact. There are now tests that assert the redacted output leaks no challenge name, no flag id, no status, no OWASP category.

Slow Is a Correctness Bug, Not a Comfort Problem #

I went into this expecting performance work to be about patience. It wasn’t.

Security Shepherd took about 21 minutes to score, roughly 30 seconds per challenge, against 4 to 72 seconds total for every other target. The scorer ran each challenge as a child process with a 30 second timeout, and against an HTTPS target a keep alive TLS socket kept every child alive until that timeout killed it. Not one test was slow. Every test was waiting to be executed. Streaming the reporter output and killing each child the moment its result was known took it under 90 seconds.

VulnerableApp was 110 independent test files run one at a time, about 15 minutes. Those files are stateless, so they got a per target concurrency default of 8, which brought it to about 2 minutes. Security Shepherd stayed serial because its tests mutate shared server state and would race.

Then the real cost showed up.

The consumer workflow minted a GitHub OIDC token, then built the app, then scored, then posted. An Actions id token lives about five minutes. The slow targets were spending more than that on the build and the scoring, so by the time the result was posted the token was expired. The API rejected it with a 401, the score was discarded, and the run still finished green. The comment on the pull request said the result was on the leaderboard. It wasn’t.

Measured across all six forks, two of them could not record a score at all. On the live board that showed up as me sitting at 214 of 321 patched with every challenge scored green.

The fix was to mint the token microseconds before the POST rather than at the top of the job, so build duration stops mattering, and to make a dropped score loud: a run annotation and a banner in the comment instead of silent success.

A slow pipeline didn’t just waste CI minutes. It corrupted the scoreboard while reporting success. That is the sort of failure worth designing against, and I would not have found it without re-solving every fork from a saved patch set and checking the board afterward.

The Weekend #

We burned roughly 293 hours of CI compute over the weekend, and Juice Shop alone accounted for close to half of it.

Sixty players opened 2,317 pull requests. The top of the board patched 318 of 321 challenges. Nothing merged.

The thing I did not expect was the question I kept getting at the booth. OWASP chapter leaders, university lecturers, a couple of high school teachers. All variations of the same one: how do I run this for my own students?

So that’s where the work goes next. Making the platform self hostable, so a chapter or a university or a school can stand it up and run their own event without needing us in the loop.

A CTF that runs once a year in Vegas reaches 60 people. One that a teacher in any country can deploy reaches a lot more.

None of this was a solo build. I owned the scorer, the rubrics and the CI pipeline described above. Christian Nuss wrote most of score-action, the piece each fork actually calls to build, score, record and comment. Chris Maenner built ctf.owasp.org, the contestant site with the sign in, the leaderboard, the teams and the hints. The scoring model only works because all three parts line up.

Thanks to both of them, to the OWASP community for the opportunity, and to everyone who sat down and played. The target forks are public under OWASP-CTF.