Why bot scanners flood your error tracker with 404s, and how to stop it

What's happening

Any server with a public IP is scanned all day long. Automated tools walk through lists of well-known paths looking for exposed config files, admin panels and old WordPress installs. On a Node API, every one of those paths throws a not-found error.

Most error trackers group errors by message and stack trace. Cannot GET /wp-admin and Cannot GET /.env are different messages, so each probed path becomes a brand-new issue - and a brand-new issue is exactly what triggers an alert.

Nothing is broken. But your inbox is, and the one real bug that happened the same morning is buried.

Layer 1: don't report client errors

A 404 for a path that doesn't exist is the caller's mistake, not a bug in your code. The same goes for most 400s, 401s and 403s. The simplest fix is to never send them.

The BugCatch SDK does this by default: any error carrying a 4xx status - including NestJS's NotFoundException and BadRequestException - is skipped. With other tools, do the same in your framework's error filter: only report when the status is 500 or above.

const status = typeof err.getStatus === 'function' ? err.getStatus() : 500;
if (status >= 500) BugCatch.captureException(err);

Errors from your own outgoing HTTP calls are the exception: if your code calls an API and gets a 404, that's usually your bug, so those are still reported.

Layer 2: group unknown routes into one issue

Even with the SDK filtering, older app versions or other SDKs may still send these errors. So BugCatch also normalises them when grouping: Cannot GET /anything becomes Cannot GET <route>, and Fastify's Route GET:/anything not found gets the same treatment. A scan of 300 paths becomes one issue per HTTP method with a count of 300, and one alert at most.

On top of that, if one visitor IP opens 10 new issues within 10 seconds, it can't open any more for an hour. Events for issues that already exist are still counted. Push notifications are also capped at five per project per ten minutes, followed by a single "many alerts at once" message.

Layer 3: ban the scanner at the server

Filtering protects your inbox, but the scanner is still hammering your server. A fail2ban jail can ban any IP that collects a burst of 404s and 405s. A real user almost never hits 30 missing pages in 20 seconds; a scanner does it constantly.

Filter

[Definition]
failregex = ^<HOST> -[^"]*"[A-Z]+ [^"]*" (?:404|405)
ignoreregex = "(?:GET|HEAD) /(?:favicon\.ico|robots\.txt|apple-touch-icon[^ ]*)

Jail

[http-burst]
enabled   = true
port      = http,https
filter    = http-burst
logpath   = /var/log/apache2/*access.log
backend   = polling
maxretry  = 30
findtime  = 20
bantime   = 1d
bantime.increment = true
bantime.maxtime   = 4w
ignoreip  = 127.0.0.1/8 ::1

For nginx, point logpath at /var/log/nginx/access.log; the combined log format matches the same regex. Add your own servers and office IPs to ignoreip - especially any server that calls this one and might legitimately hit a burst of 404s.

The window slides: the ban fires when any 20-second span contains 30 hits, not 30 hits since midnight. Repeat offenders get longer bans, up to four weeks.

Apply it and check it's running:

fail2ban-client reload
fail2ban-client status http-burst

The short version

• Don't report 4xx errors from incoming requests - they're not your bugs.
• Group Cannot GET /… by method, not by path, so a scan is one issue.
• Cap alerts per project, so a burst can't turn into a hundred notifications.
• Ban IPs that produce a burst of 404s at the server.

BugCatch does the first three for you. How fingerprinting works.