add tbs storage limit and disk-related metrics

[rubvs]

Motivation/summary

Expose a new set of metrics to enhance TBS observability. The metric fields in the index data are tested in this PR, while the mappings are tested in the corresponding linked PRs. Changes in the elastic/elasticsearch#138131 PR are tested in conjunction with the changed in this PR.

See #15533 (comment) for the detailed overview.

Depends on PR:

• apm-server: update monitor mappings to include new TBS metrics elasticsearch#138131
• metricbeats: update monitor mappings to include new TBS metrics #138131 beats#47709
• elastic-agent: update monitor fields to include new TBS metrics integrations#16029

Checklist

• Update CHANGELOG.asciidoc
• Documentation has been updated

For functional changes, consider:

• Is it observable through the addition of either logging or metrics?
• Is its use being published in telemetry to enable product improvement?
• Have system tests been added to avoid regression?

How to test these changes

Step 1: Ensure Elasticsearch & Kibana is running

Depends on elastic/elasticsearch#138131 with updates to monitoring-beats.json.

1. Build Docker image with changes

> cd elasticsearch

# Build the ES image with the added metric mappings.
> ./gradlew buildAarch64DockerImage --rerun-tasks

2. Edit apm-server/docker-compose.yml and change the ES image.

elasticsearch:
 image: docker.elastic.co/elasticsearch/elasticsearch:9.3.0-custom-SNAPSHOT
 # ... rest of config

3. Spin up the required services.

> cd apm-server
> docker-compose up elasticsearch kibana

Step 2: Create APM Server config

apm-server:
  host: "127.0.0.1:8200"

output.elasticsearch:
  enabled: true
  hosts: ["http://localhost:9200"]
  username: "admin"
  password: "changeme"

monitoring.enabled: true

monitoring.elasticsearch:
  protocol: "http"
  hosts: ["http://localhost:9200"]
  username: "admin"
  password: "changeme"

Step 3: Start APM Server

Run APM Server binary directly:

> cd apm-server
> ./apm-server -e -v -c apm-server.yml

Step 6: Verify Data

Verify data in index .monitoring-beats-7-*

GET .monitoring-beats-7-2025.11.14/_search
{
  "_source": ["beats_stats.metrics.apm-server.sampling.tail"],
  "query": {
    "exists": {
      "field": "beats_stats.metrics.apm-server.sampling.tail"
    }
  },
  "size": 1
}

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 31,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".monitoring-beats-7-2025.11.14",
        "_id": "ApzqgpoBYbFQC8_sQ5JQ",
        "_score": 1,
        "_source": {
          "beats_stats": {
            "metrics": {
              "apm-server": {
                "sampling": {
                  "tail": {
                    "storage": {
                      "value_log_size": 0,
                      "storage_limit": 0,
                      "disk_used": 309071659008,
                      "disk_total": 994662584320,
                      "disk_usage_threshold": 80,
                      "lsm_size": 8891
                    }
                  }
                }
              }
            }
          }
        }
      }
    ]
  }
}

Verify mappings for index monitoring-beats-7-*

GET .monitoring-beats-7-2025.11.18/_mapping?filter_path=**.storage

{
  ".monitoring-beats-7-2025.11.18": {
    "mappings": {
      "properties": {
        "beats_stats": {
          "properties": {
            "metrics": {
              "properties": {
                "apm-server": {
                  "properties": {
                    "sampling": {
                      "properties": {
                        "tail": {
                          "properties": {
                            "storage": {
                              "properties": {
                                "disk_total": {
                                  "type": "long"
                                },
                                "disk_usage_threshold": {
                                  "type": "long"
                                },
                                "disk_used": {
                                  "type": "long"
                                },
                                "lsm_size": {
                                  "type": "long"
                                },
                                "storage_limit": {
                                  "type": "long"
                                },
                                "value_log_size": {
                                  "type": "long"
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Related issues

Part of #15533

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label. Could you fix it @rubvs? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-7.17 is the label to automatically backport to the 7.17 branch.
• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
• backport-9./d is the label to automatically backport to the 9./d branch. /d is the digit.
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: bcd0882

History

• 💔 Build #7279 failed 082a2e7
• 💔 Build #7277 failed 71a5fbb

[ericywl]

I also ran the test following the steps posted by Ruben, and can confirm that the metrics mapping appears.

GET .monitoring-beats-7-2025.11.28/_mapping?filter_path=**.storage

{
  ".monitoring-beats-7-2025.11.28": {
    "mappings": {
      "properties": {
        "beats_stats": {
          "properties": {
            "metrics": {
              "properties": {
                "apm-server": {
                  "properties": {
                    "sampling": {
                      "properties": {
                        "tail": {
                          "properties": {
                            "storage": {
                              "properties": {
                                "disk_total": {
                                  "type": "long"
                                },
                                "disk_usage_threshold": {
                                  "type": "long"
                                },
                                "disk_used": {
                                  "type": "long"
                                },
                                "lsm_size": {
                                  "type": "long"
                                },
                                "storage_limit": {
                                  "type": "long"
                                },
                                "value_log_size": {
                                  "type": "long"
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

[carsonip]

implementation looks good, 1 nit on metric naming

[carsonip]

nit: I wonder if it will be more descriptive / self-explanatory with a suffix e.g. _pct. Also wonder if it better to range from 0 to 1 (correct to 2 d.p. maybe?) instead of 0-100.

[carsonip]

nit: remember to update comment if we change to 0-1

[carsonip]

ditto