audit-trail-rate-limiting-public-free-scan-endpoints-are-easily-tarpitted-at-2640a32d

THEN Implement IP + URL pair rate-limit middleware on public endpoints (e.g., /sync/onboard) with same-resource exemption. Allow re-onboarding the same URL freely; only count new URLs per IP per time window.

@app.middleware("http")
async def rate_limit_onboard(request, call_next):
    if request.url.path == "/api/v1/sync/onboard" and request.method == "POST":
        ip = ...
        body_bytes = await request.body()
        url = (json.loads(body_bytes or b"{}") or {}).get("url", "")
        if not _check_rate(ip, url):
            return JSONResponse(429, {"detail": (
                f"Rate limit exceeded — max {_RATE_MAX_UNIQUE_URLS} "
                "distinct site URLs per 60s per IP. Re-onboarding the "
                "same URL is free; try a different request shortly."
            )})