|
| 1 | +(longterm-store)= |
| 2 | +(timeseries-longterm)= |
| 3 | +(timeseries-long-term-storage)= |
| 4 | + |
| 5 | +# Long-term store |
| 6 | + |
| 7 | +:::{div} sd-text-muted |
| 8 | +Never retire data just because your other systems can't handle the cardinality. |
| 9 | +::: |
| 10 | + |
| 11 | +CrateDB stores large volumes of data, keeping it accessible for querying |
| 12 | +and insightful analysis, even considering historic data records. |
| 13 | + |
| 14 | +Many organizations need to retain data for years or decades to meet regulatory |
| 15 | +requirements, support historical analysis, or preserve valuable insights for |
| 16 | +future use. However, traditional storage systems force you to choose between |
| 17 | +accessibility and affordability, often leading to data exports, archival |
| 18 | +systems, or downsampling that sacrifice query capabilities. |
| 19 | + |
| 20 | +CrateDB eliminates this trade-off by storing large volumes of data efficiently |
| 21 | +while keeping it fully accessible for querying and analysis. Unlike systems |
| 22 | +that struggle with high cardinality or require expensive tiered architectures, |
| 23 | +CrateDB handles billions of unique records in a single platform, maintaining |
| 24 | +fast query performance even on historic datasets spanning years. |
| 25 | + |
| 26 | +By keeping all your data in one place, you avoid the complexity and costs of |
| 27 | +exporting to specialized long-term storage systems, data lakes, or cold storage |
| 28 | +tiers. Your historical data remains as queryable as your recent data, enabling |
| 29 | +seamless analysis across any time range without data movement, ETL pipelines, |
| 30 | +or rehydration processes. |
| 31 | + |
| 32 | +With CrateDB, compatible to PostgreSQL, you can do all of that using plain SQL. |
| 33 | +Other than integrating well with commodity systems using standard database |
| 34 | +access interfaces like ODBC or JDBC, it provides a proprietary HTTP interface |
| 35 | +on top. |
| 36 | + |
| 37 | +## Use cases |
| 38 | + |
| 39 | +:::{rubric} Metrics and monitoring |
| 40 | +::: |
| 41 | + |
| 42 | +::::{grid} 1 1 1 2 |
| 43 | +:gutter: 2 |
| 44 | +:padding: 0 |
| 45 | + |
| 46 | +:::{grid-item-card} Prometheus |
| 47 | +:link: prometheus |
| 48 | +:link-type: ref |
| 49 | +Prometheus and similar monitoring systems excel at real-time alerting but face challenges |
| 50 | +with long-term metric retention due to storage costs and query performance at scale. CrateDB |
| 51 | +addresses these challenges by providing: |
| 52 | +- **Scalable long-term storage**: Store years of metrics without compromising query performance. |
| 53 | +- **High cardinality support**: Handle millions of unique label combinations that would overwhelm traditional TSDBs. |
| 54 | +- **Rich SQL analytics**: Perform complex analytical queries on historic metrics using standard SQL. |
| 55 | +- **Seamless integration**: Use CrateDB's Prometheus Adapter for transparent remote write/read operations. |
| 56 | ++++ |
| 57 | +Set up CrateDB as a long-term metrics store for Prometheus. |
| 58 | +::: |
| 59 | + |
| 60 | +:::{grid-item-card} OpenTelemetry |
| 61 | +:link: opentelemetry |
| 62 | +:link-type: ref |
| 63 | +OpenTelemetry and similar observability frameworks excel at generating rich telemetry data |
| 64 | +but face challenges with long-term retention due to storage scale and query complexity. |
| 65 | +CrateDB addresses these challenges by providing: |
| 66 | +- **Scalable long-term storage**: Store large volumes of telemetry through CrateDB's distributed architecture. |
| 67 | +- **Vendor-neutral ingestion**: Use OpenTelemetry SDKs/agents and Telegraf to send telemetry into your CrateDB observability pipeline. |
| 68 | +- **Rich SQL analytics**: Run SQL/time-series queries, aggregations and joins on telemetry data for troubleshooting and analytics. |
| 69 | +- **Flexible attribute mapping**: Customize which span/log/profile attributes become columns/tags for dimensional queries. |
| 70 | ++++ |
| 71 | +Set up CrateDB as a long-term observability backend for OpenTelemetry. |
| 72 | +::: |
| 73 | + |
| 74 | +:::: |
| 75 | + |
| 76 | +## Related sections |
| 77 | + |
| 78 | +{ref}`metrics-store` includes information about how to |
| 79 | +store and analyze high volumes of system monitoring information |
| 80 | +like metrics and log data with CrateDB. |
| 81 | + |
| 82 | +{ref}`analytics` describes how |
| 83 | +CrateDB provides real-time analytics on raw data stored for the long term. |
| 84 | +Keep massive amounts of data ready in the hot zone for analytics purposes. |
| 85 | + |
| 86 | +[Optimizing storage efficiency for historic time series data] |
| 87 | +illustrates how to reduce table storage size by 80%, |
| 88 | +by using arrays for time-based bucketing, a historical table having |
| 89 | +a dedicated layout, and querying using the UNNEST table function. |
| 90 | + |
| 91 | +{ref}`Build a hot/cold storage data retention policy <airflow-data-retention-hot-cold>` |
| 92 | +describes how to manage aging data by leveraging CrateDB cluster |
| 93 | +features to mix nodes with different hardware setups, i.e. hot |
| 94 | +nodes using the latest generation of NVMe drives for responding |
| 95 | +to analytics queries quickly, and cold nodes that have access to |
| 96 | +cheap mass storage for retaining historic data. |
| 97 | + |
| 98 | +{ref}`weather-data-storage` provides information about how to |
| 99 | +use CrateDB for mass storage of synoptic weather observations, |
| 100 | +allowing you to query them efficiently. |
| 101 | + |
| 102 | + |
| 103 | +[Optimizing storage efficiency for historic time series data]: https://community.cratedb.com/t/optimizing-storage-for-historic-time-series-data/762 |
0 commit comments