PageFind vs CrowdSec: How I Locked Myself Out

Intro

I recently added Pagefind to my Hugo blog to provide local static search.

Shortly after that, I noticed I could no longer SSH to the server from my home network. SSH still worked over mobile data, so the server itself was fine.

My first suspect was Fail2ban, but its SSH jail showed no banned IPs.

Checking the nftables rules revealed my home IP was actually present in a CrowdSec blacklist set.

I inspected the CrowdSec alert:

cscli alerts inspect <alert-id>

The trigger was:

crowdsecurity/http-crawl-non_statics

CrowdSec had seen 43 HTTP requests in about 20 seconds, all for Pagefind search fragments:

/pagefind/fragment/*.pf_fragment

Pagefind generates a large number of static search assets with extensions such as:

.pf_fragment
.pf_index
.pf_meta
.pagefind

CrowdSec did not recognize these as normal static content, so normal Pagefind search traffic looked like HTTP crawling.

Fix

I created a local CrowdSec whitelist:

/etc/crowdsec/parsers/s02-enrich/pagefind-whitelist.yaml

with:

name: local/pagefind-whitelist
description: "Whitelist Pagefind generated search assets"
filter: "evt.Meta.service == 'http'"
whitelist:
  reason: "Pagefind generated static assets"
  expression:
    - evt.Meta.http_verb == 'GET' &&
      evt.Meta.http_status in ['200', '304'] &&
      evt.Meta.http_path matches '^/pagefind/.*\\.(pf_fragment|pf_index|pf_meta|pagefind)$'

Then restarted CrowdSec:

systemctl restart crowdsec

and removed the existing ban:

cscli decisions delete --ip <my-ip>

SSH access immediately started working again.

Lesson learned

The security tooling was doing exactly what it was supposed to do. The problem was that Pagefind introduced a request pattern that looked like aggressive crawling.

Instead of weakening the CrowdSec scenario globally, I added a narrowly scoped whitelist only for Pagefind-generated static assets.

See Also