POST /api/v1/series: Incorrect match[] param set in the URL query #134
Description
Summary
I was testing the prom-label-proxy v0.6.0 + Grafana 9.1.8 and I noticed that when using HTTP POST method with the /api/v1/series endpoint, prom-label-proxy is passing two different value for the match[] parameter: one in the POST body and the other one in the URL query parameters.
This leads to incorrect results. For instance the thanos querier seems to read only the match[] value in the query parameter in this situation which is pretty much garbage.
For instance if we are doing:
curl 127.0.0.1:39090/api/v1/series -X POST -H 'X-Tenant-Id: foo' --data-urlencode "match[]=up"Then the resulting HTTP query sent by prom-label-proxy is (in JSON format as recorded by gohrec and URL-decoded) will be:
{
"Body": "match[]={__name__=\"up\",tenant_id=\"foo\"}",
"URI": "/api/v1/series?match[]={tenant_id=\"foo\"}"
}Depending on its implementation, the upstream may only read the match[] value set in the URI (which is the case for the Thanos querier). Even if it is true that the upstream should only read the match[] value in the request body with Content-Type: application/x-www-form-urlencoded, this is still an issue in the way prom-label-proxy is writing the resulting query.
Full response recorded by gohrec
{
"Protocol": "HTTP/1.1",
"Headers": [
"Accept-Encoding: gzip",
"Accept: */*",
"Content-Length: 61",
"Content-Type: application/x-www-form-urlencoded",
"User-Agent: curl/7.58.0",
"X-Forwarded-For: 172.29.0.1",
"X-Tenant-Id: foo"
],
"ContentLength": 61,
"Body": "match[]={__name__=\"up\",tenant_id=\"foo\"}",
"Trailers": [],
"TransferEncodings": null,
"RemoteAddr": "172.29.0.2:39070",
"Host": "127.0.0.1:39090",
"Method": "POST",
"Path": "/api/v1/series",
"Query": [
"match[]: {tenant_id=\"foo\"}"
],
"URI": "/api/v1/series?match[]={tenant_id=\"foo\"}"
}Steps to reproduce
- docker-compose.yml:
version: '3'
services:
http-recorder:
image: frxyt/gohrec:latest
command: ["record", "--listen=:8080"]
volumes:
- ./logs:/gohrec/log
prom-label-proxy:
image: quay.io/prometheuscommunity/prom-label-proxy:v0.6.0
ports:
- 127.0.0.1:39090:39090
command:
- "-header-name=X-Tenant-Id"
- "-label=tenant_id"
- "-upstream=http://http-recorder:8080"
- "-insecure-listen-address=0.0.0.0:39090"
- "-enable-label-apis"- Run:
curl 'http://127.0.0.1:39090/api/v1/series' -X POST -H 'X-Tenant-Id: foo' --data-urlencode "match[]=up" -o /dev/null -sS- See recorded HTTP query in
./logs/[date]/
Expected behavior
The HTTP request sent to the upstream should be:
POST /api/v1/series HTTP/1.1
X-Tenant-Id: foo
Content-Type: application/x-www-form-urlencoded
match[]={__name__="up",tenant_id="foo"}
instead of
POST /api/v1/series?match%5B%5D=%7Btenant_id%3D%22foo%22%7D HTTP/1.1
X-Tenant-Id: foo
Content-Type: application/x-www-form-urlencoded
match[]={__name__="up",tenant_id="foo"}