Incidents

Mid-contest server crash · debugged, rate-limited, and restored the platform

1. 1During the Prompt Engineering contest, our server crashed under contest traffic — participants were mid-submission and the clock was running.
2. 2Led end-to-end debugging: traced load spikes, stabilized services, and added rate limits so the API could survive burst traffic without falling over again.
3. 3Spun up redundant hosting across the university server room and a Hostinger VPS, routed traffic, and brought the platform back online so the segment could finish.
4. 4Coordinated with faculty and volunteers on status updates so teams knew what was happening instead of guessing.

Root compromise · mining malware removed · WireGuard VPN + SSH key access only

1. 1Our office VPS was hacked — an attacker had access and started crypto mining on production infrastructure.
2. 2Contained the breach: killed malicious processes, audited running services, rotated credentials, and verified no persistence mechanisms were left behind.
3. 3Disabled root SSH login, created a dedicated deploy user with least privilege, and locked down sudo to what operations actually need.
4. 4Stood up a WireGuard (wg) VPN server — team access to the VPS runs through a WireGuard tunnel only, with SSH public/private key auth and no password logins or wide-open port 22 on the public internet.

MongoDB port exposed to the internet · full data recovery

1. 1A junior developer accidentally left MongoDB publicly accessible — the database port was open to the world instead of bound to localhost or a private network.
2. 2Detected suspicious access patterns, locked down firewall rules immediately, and closed every path that shouldn't have been reachable from outside.
3. 3Recovered all data from backups and verified integrity before bringing services back — no silent corruption, no guessing about what was lost.
4. 4Documented the fix, added checklist gates for deploys, and walked the team through why exposed database ports are a production incident, not a small mistake.