Revisit caching headers and etags for our endpoints

[brontolosone]

Once getodk/central-backend#1654 is in place we can use the counters as etags in the way analogous with getodk/central-backend#1657 at other endpoints.
Not for all endpoints of course. getodk/central-backend#1654 is not "one size fits all", it doesn't fit all use cases, but hopefully it'll fit many . For more info on that, see this elaboration of where it sits on a spectrum.

Anyway, we can:

• See where we can use these in place of the Express-default bodyhashing approach. Personally I think we should turn that default off. It's even etagging error responses.
• Use these etags in place of approaches that are prone to under-invalidation under concurrency (variants of max(some_timestamp)).
• Use them where we currently can't have any ETag at all since the body-hashing approach can't work on responses where the headers are sent out while the body isn't known yet. example: submissions.csv.
• In general scrutinize the way caching headers are currently set, as it currently results in errors also being cacheable. They're subject to revalidation, so while somewhat silly (likely unintended), they're not currently harmful, but for our future ideal front-end proxy caching setup, they result in cache flushes. See deliberations.

[matthew-white]

It's even etagging error responses.

I thought that was addressed with getodk/central-backend#1103.

[brontolosone]

Well, I get this in a 401 from staging upon a curl --compressed --verbose --silent --show-error -H 'Accept: application/json' -H 'Authorization: Bearer nopenope' https://staging.getodk.cloud/v1/projects/1/forms/geotest/submissions.geojson:

< HTTP/2 401 
< server: nginx
< date: Sat, 01 Nov 2025 22:02:22 GMT
< content-type: application/json; charset=utf-8
< content-length: 80
< cache-control: private, no-cache
< vary: Accept-Encoding, Origin
< etag: W/"50-ZRrLMzj8S67I1VD1Vnbql2PHY1Y"
< strict-transport-security: max-age=63072000

And, also from current staging, this response (minus body) in my browser dev console for a 404, with a valid session but addressing a nonexistent endpoint:

HTTP/2 404 
server: nginx
date: Sat, 01 Nov 2025 22:07:13 GMT
content-type: application/json; charset=utf-8
content-length: 76
cache-control: private, no-cache
vary: Accept-Encoding, Origin
etag: W/"4c-ShTrjQvXJDA49Wfxw7ES7C6cQFg"
strict-transport-security: max-age=63072000
X-Firefox-Spdy: h2

[matthew-white]

I see, that's interesting, seems like there's work to do in that area. 👍

[alxndrsn]

etagging error responses

it currently results in errors also being cacheable. They're subject to revalidation, so while somewhat silly (likely unintended), they're not currently harmful

I don't think there's theoretically anything wrong with 4xx headers having cache headers or ETags. However, I agree it's not particularly useful:

1. central's error response bodies are generally small, so there's little to be gained from etagging & caching them
2. requests generating 401 and 404 responses seem to ignore If-None-Match request headers anyway, so the ETags are useless in these cases

To me this is a separate issue to how and for which endpoints ETags etc. are added to success responses.

[brontolosone]

Random facts about the cacheable 4xx (probably 5xxes too) responses:

• I found a performance impact of 6% for the hashing of a simple 401. It's not a lot. But it's also not nothing. A bit like being asked for a dollar after getting your spotless windscreen surprise-washed when stopped at a red light.
• as @matthew-white pointed out there's been an attempt to remove them so presumably there has been a decision that we don't want them.

To me this is a separate issue to how and for which endpoints ETags etc. are added to success responses.

On the surface it looks like that, but the behaviours in Central are not so nicely composable, and together with idiosyncracies of nginx this leads to entanglement.

Let me try to reformulate the problem so it's more clear how they are related without having to read the background materials:

Pathology

1. Caching headers for revalidation (except for sensitive /session endpoints) are currently centrally set up at the point where the request is known, at that point nothing is known yet about the response.
2. Be that as it may, that then seems like the obvious place to add the magic header that conveys immediately-stale semantics to nginx.
3. But, lack of proper "undoing" of the caching headers (regardless of whether that's tasteful to begin with, or points to deeper-rooted problems of control) for errors make those errors flush out the nginx cache. And we don't want that.

So the options on the table are 🍽️ ):

1. Don't set Special Header For Nginx globally. Things continue to look and work like they do now. No controverse, for better or worse. Caching at the reverse proxy is then opt-in per endpoint. Express continues to hash everything in sight that we don't set our own etags on. Questionable outcomes persist, such as the two Vary headers I see on https://staging.getodk.cloud/v1/projects/112/forms/geotest/submissions.geojson when I go and take a look at https://staging.getodk.cloud/projects/112/forms/geotest/submissions?map=true:

   vary:	Accept-Encoding
   vary:	Accept-Encoding, Origin
   

   Probably there's multiple places in the code that have an opinion on when to set what header. Note that this is allowed by the standard, as Vary has a multivalued body, these two headers can be merged by a client. No harm done (in this particular case)!. Yet it gives me an uneasy sense that the right hand doesn't know what the left hand is doing. Maybe others are more at ease with that.
   The upside of this option is that then the work will be purely additive, which makes for a smooth PR review. 🥳
2. Set Special Header For Nginx globally so that everything that's currently transparently cacheable (deliberate or not), will be transparently cached by nginx, no changes to endpoints needed to make that happen. It then becomes crucial to unset cache headers for errors (attempted already, yet failed), lest we let those flush out the nginx cache.
3. Turn off Express' blithe auto-hasihng-etagging. Turn it on on a per-endpoint basis (on successful responses) where we feel it makes sense. Tag those endpoints in api.yaml. Use tests to assert appropriate caching behaviour of the whole chain (eg, directly test for what Nginx is doing, fortunately it can communicate cache hit/miss/revalidated in a response header).

I lean towards #3 but could work around things with #1. With #1 I wouldn't be surprised that in the end you end up with over time every endpoint worth using cache for, receiving deliberate handling of it to make it nginx-cacheable (be it by using audit event counters, content hashing, or something else). At which point we might as well then turn off Express' autohashing and remove all the header-twiddling and end up in the same place as when we'd have chosen #3 right away 🤷‍♂️ .