Counter retention intervals

There are different retention intervals, such as "10 seconds" or "1 hour." These retention intervals are further divided into two kinds:

• Regular aggregates
• Seasonal averages

Regular aggregates

Regular aggregates roll up the data of the shorter retention intervals (say, 10 s samples) into larger intervals (say, 10 minute samples) to allow you to keep aggregate data around for longer without wasting a lot of disk space.

Seasonal averages

Seasonal averages keep a "rolling average" based on some particular time interval -- for example, daily, weekly, or yearly averages. A rolling average calculates the next value for a bucket based on some weighting between the previous bucket value, and the new bucket value. This weighting factor is called "lambda." An example might be in order:

Suppose it's currently September 8, 2013, at 16:30. There is a seasonal average retention interval with a season of one day, and a resolution of 1 minute. Any data collected for the 16:30 time slot will be aggregated, and once 16:31 rolls around, the data from September 7, 2013, at 16:30, will be read, and the lambda weighting applied, and the value written out for September 8, 2013, 16:30. (The value for September 7 is really the aggregate/weighted roll-up of all previous days.)

You configure a seasonal average like you configure regular (aggregate) retention intervals, by specifying them in the comma-separated list of retention intervals. In addition to the duration, you also need to configure the lambda value, and optionally a name for the season file (so it doesn't conflict with the names for the regular retention files.)

Example

--retention=10s:10d,10m:310d,10m:7d:ret-10m:0.9

The retention option is first split on commas. Thus, we are configuring three retention intervals. Each retention interval maps to one counter file on disk. This retention interval specification configures three files.

Each interval is then split on commas. The format for each interval is:

<interval>:<retention>[[:<name>]:lambda]

The first file will store counter values per ten-second intervals, and retain those values for ten days, after which older values will start getting overwritten by new values.

The second file will store counter values per ten-minute intervals, and retain those values for approximately ten months (310 days,) after which older values will start getting overwritten by new values.

The third and final file will store counter values per ten-minute intervals, and retain those values for seven days. When it rolls around, rather than overwriting the 7 day old values, it will average the new values with the older values, giving the older values a weight (lambda) of 0.9, and thus giving the new values a weight (1-labmda) of 0.1. This will calculate weekly averages of both the value, the min/max, and the average/standard deviations.