Your EDR Saw 3 Events. The Network Saw 20,000. Find the Beacon.
A walkthrough of a Signal9 incident-response lab — and a demonstration of why we grade the state of the machine instead of the string you typed.
Your netflow sensor screams overnight. One production box — the one running the corp storefront — made roughly 20,000 outbound connection attempts to a single external IP. You pull the EDR logs for the same window, expecting a wall of alerts.
Three events. That's it.
Something is beaconing out of a revenue-generating server, the security tool that's supposed to catch it is nearly blind, and the storefront still has to be serving customers when you're done. First responders grabbed a process snapshot before handing you the box. Now it's yours.
This is a Signal9 lab called Beacon Hunt. You do it on a real root terminal, against a real filesystem. Here's how the investigation actually goes — and then the part that matters for how you're scored.
1. Hunt: trust ground truth, not names
Start with the captured snapshot:
$ cat /root/triage/ps.txt
root 8 2 ... [kworker/0:1-events]
root 12 2 ... [kworker/u4:0-flush-8:0]
root 842 1 ... bash /usr/local/sbin/kworkerd --c2 198.51.100.7:443
Two things named like kernel workers. Only one is lying. Real kernel threads show up bracketed ([kworker/*]), parented to kthreadd (PID 2), with no file on disk. The impostor at PID 842 has a real path (/usr/local/sbin/kworkerd), a parent of PID 1, and a C2 address sitting right there in its arguments.
Confirm it's not supposed to be there by asking the package database who owns it:
$ dpkg -S /usr/local/sbin/kworkerd
dpkg-query: no path found matching pattern /usr/local/sbin/kworkerd
No owning package. A legitimate system binary answers with a package name. This one was dropped, not installed. That's your tamper signal.
2. Prove: make independent sources agree
Good responders don't attribute a C2 off one artifact. This lab makes you triangulate. The destination 198.51.100.7 should appear in three places that have no reason to agree unless it's real:
$ grep kworkerd /root/triage/ps.txt # the process arguments
$ cat /usr/local/sbin/kworkerd # the beacon script itself
$ grep -c 198.51.100.7 /var/log/netwatch/flows.log # the netflow log
20000
Argv, binary, and network telemetry all point at the same IP. Now you can write it down with confidence.
3. Explain the telemetry gap
Why did the EDR see ~3 events while netwatch saw ~20,000 flows? Because the EDR hooks process creation (execve). The beacon is a single long-lived process that re-opens a socket in a loop — no new process per beacon, so the hook fired once at launch and went quiet. The network layer, which counts every flow, saw all 20,000. That gap is the whole lesson: a quiet agent is not a clean box.
4. Eradicate — without taking down the business
$ rm /usr/local/sbin/kworkerd
Precise. Note what you do not do: the legit storefront binary lives at /usr/local/bin/webapp. The tempting "just nuke /usr/local/sbin to be safe" move would take the money-maker offline. Responders get judged on the blast radius too.
5. Kill persistence — all of it
The intruder left two ways back in:
$ grep -rs kworkerd /etc/cron.d /etc/systemd/system
/etc/cron.d/system-health:*/2 * * * * root /usr/local/sbin/kworkerd --c2 ...
/etc/systemd/system/kworkerd.service:ExecStart=/usr/local/sbin/kworkerd ...
A cron.d job re-drops the beacon every two minutes. A systemd unit — enabled via a multi-user.target.wants symlink — revives it on reboot. You have to remove the cron file, the .service unit, and the wants symlink. Miss any one, and the box re-infects itself.
Here's the part other platforms can't do
On a flag-hunting or multiple-choice platform, the game ends when you name the C2 or paste a hash. You'd get a green checkmark and move on.
Signal9 doesn't grade your answer. It grades the state of the machine you leave behind. Watch what that catches:
Learner A finds the beacon, records the C2, deletes kworkerd, removes the cron job — and stops. The beacon is gone right now, so they feel done. But they missed the systemd wants-symlink. Our check runs grep -r kworkerd /etc/cron.d /etc/systemd/system, finds the leftover unit, and fails them — because a single reboot brings the C2 beacon straight back. A string-matching grader would have passed Learner A. The box is still compromised.
Learner B gets nervous during eradication and runs rm -rf /usr/local/sbin to be thorough. The beacon is definitely gone. So is nothing else… except the check test -e /usr/local/bin/webapp is fine, but had they reached one directory over, the storefront would be down. The lab is built so the "be safe, delete broadly" instinct has real consequences — because in production it does. You're scored on eradicating the threat and leaving the business running.
That's the difference. "I found it" is not the same as "the box is clean and the business is up." Real incident response is graded on the second one. So is Signal9.
Seven checks run against this box: the evidence you recorded, the C2 you attributed, the tamper signal you noted, the telemetry gap you explained, the payload you removed, both persistence mechanisms gone, and the storefront still standing. Get the beacon but leave a reboot-persistent unit? Not a pass. That's not us being harsh — that's the job.
Try it yourself. Beacon Hunt is free — real root terminal, real filesystem, graded on what you actually leave behind. Hunt a live C2 beacon — free lab →
Signal9 is where security engineers practice real Linux hardening, incident response, and forensics on live systems — graded by the state of the machine, not by hunting for a flag.