Skip to content

Commit cfac549

Browse files
vankovapvankovap
authored
observability (#292)
* observability
1 parent bf7f391 commit cfac549

File tree

11 files changed

+5039
-4969
lines changed

11 files changed

+5039
-4969
lines changed

‎apps/docs/content/references/log-forwarding.mdx‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
---
2+
title: Log Forwarding
3+
description: Learn how to set up log forwarding in Zerops.
4+
---
5+
6+
For more advanced [log](/references/logging) analysis and centralized logging, you can forward all your Zerops logs to external logging services. The Zerops logger service uses **syslog-ng** to enable this functionality.
7+
8+
## Ready-Made Configurations
9+
10+
Zerops provides pre-configured setups for popular logging services:
11+
12+
- **[Better Stack](https://betterstack.com/)**
13+
- **[Papertrail](https://www.papertrail.com/)**
14+
- **Self-hosted [ELK Stack](https://www.elastic.co/elastic-stack) (Logstash)**
15+
16+
To set up one of these integrations, go to your project detail, select **Advanced Observability**, choose your preferred service, and follow the guided steps in the interface.
17+
18+
## Self-Hosted Logstash
19+
20+
To collect logs in a self-hosted ELK stack, a Logstash instance (deployed as the `logstash` service) is required. Logstash listens for incoming UDP syslog packets and indexes them to Elasticsearch.
21+
22+
The one-click GUI integration setup is all you need to get started. After setup completes, logs will appear in Kibana's "Logs > Explorer" section. You can find access information in the **Advanced Observability** section of your project in the GUI.
23+
24+
:::warning
25+
Setting up log forwarding to ELK will override any previously configured log forwarding.
26+
:::
27+
28+
### What Happens Behind the Scenes
29+
30+
1. The `elkstorage`, `kibana`, and `logstash` services are deployed and configured in the target project (if not already present)
31+
2. If logging is required across different projects, the `logstash` service is made publicly accessible through port routing with firewall whitelisting for source project IPs
32+
3. Log forwarding is configured on source projects to forward logs to the target Logstash instance
33+
34+
:::tip
35+
For the complete ELK stack setup including tracing capabilities, see the [Self-Hosted Observability](/references/metrics) guide.
36+
:::
37+
38+
## Custom Log Forwarding Configuration
39+
40+
You can set up forwarding to any syslog-ng compatible software. To do this, navigate to your project detail's **Advanced Observability** section, and choose the "Setup forwarding to any syslog-ng compatible software" option.
41+
42+
When configuring your custom syslog-ng setup, note the following important details:
43+
44+
### Certificate Configuration
45+
- Certificates are located in `/etc/ssl/certs`
46+
- If your configuration references `/etc/syslog-ng/ca.d` or `/etc/syslog-ng/cert.d`, change these paths to `/etc/ssl/certs`
47+
- For custom certificates, you can use: `ca-file("/etc/syslog-ng/user.crt")`
48+
- You can combine custom certificates with standard certificates using: `ca-dir("/etc/ssl/certs")`
49+
(This will verify both your custom certificate and standard certificates like those from LetsEncrypt)
50+
51+
### Source Configuration
52+
- Zerops uses `s_src` as the source configuration name
53+
- This differs from Papertrail, which might instruct you to "replace 's_sys' with the name you found" - in Zerops, always use `s_src` instead

‎apps/docs/content/references/logging.mdx‎

Lines changed: 4 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,15 @@
11
---
2-
title: Logging & Log Forwarding
3-
description: Learn how to access logs and set up log forwarding in Zerops.
2+
title: Logging
3+
description: Learn how to access logs in Zerops.
44
---
55

6-
## Overview
7-
86
Zerops automatically collects logs from all services in your project through a built-in logger service. These logs include runtime operations, database activities, build processes, and more.
97

108
## Project-Wide Logs
119

12-
To view all project logs, navigate to your project detail and select the **Log Forwarding & Logs Overview** page. In the **Project logs overview** section, you'll find a consolidated view of all logging activity from all services with multiple filtering options.
10+
To view all project logs, navigate to your project detail and select the **Logs overview** section, where you'll find a consolidated view of all logging activity from all services with multiple filtering options.
1311

14-
Additionally, from the same page, you can set up [log forwarding](#log-forwarding) to external logging services for more advanced analysis and long-term storage.
12+
Additionally, in the **Advanced Observability** section, you can set up [log forwarding](/references/log-forwarding) to external logging services for more advanced analysis and long-term storage.
1513

1614
## Service-Specific Logs
1715

@@ -52,33 +50,3 @@ To view logs from all containers of a service combined, you can either use the P
5250
:::important
5351
For severity levels to work properly in Zerops, your application must log to syslog.
5452
:::
55-
56-
## Log Forwarding
57-
58-
For more advanced log analysis and centralized logging, you can forward all your Zerops logs to external logging services. The Zerops logger service uses **syslog-ng** to enable this functionality.
59-
60-
### Ready-Made Configurations
61-
62-
Zerops provides pre-configured setups for popular logging services:
63-
64-
- **[Better Stack](https://betterstack.com/)**
65-
- **[Papertrail](https://www.papertrail.com/)**
66-
67-
To set up one of these integrations, go to your project detail, select **Log Forwarding & Logs Overview**, choose your preferred service, and follow the guided steps in the interface.
68-
69-
### Custom Log Forwarding Configuration
70-
71-
You can set up forwarding to any syslog-ng compatible software. To do this, navigate to your project detail, select **Log Forwarding & Logs Overview**, and choose the "Setup forwarding to any syslog-ng compatible software" option.
72-
73-
When configuring your custom syslog-ng setup, note the following important details:
74-
75-
#### Certificate Configuration
76-
- Certificates are located in `/etc/ssl/certs`
77-
- If your configuration references `/etc/syslog-ng/ca.d` or `/etc/syslog-ng/cert.d`, change these paths to `/etc/ssl/certs`
78-
- For custom certificates, you can use: `ca-file("/etc/syslog-ng/user.crt")`
79-
- You can combine custom certificates with standard certificates using: `ca-dir("/etc/ssl/certs")`
80-
(This will verify both your custom certificate and standard certificates like those from LetsEncrypt)
81-
82-
#### Source Configuration
83-
- Zerops uses `s_src` as the source configuration name
84-
- This differs from Papertrail, which might instruct you to "replace 's_sys' with the name you found" - in Zerops, always use `s_src` instead

‎apps/docs/content/references/metrics.mdx‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
---
2+
title: Advanced Observability
3+
description: Deploy self-hosted observability solutions with ELK stack or Prometheus & Grafana.
4+
---
5+
6+
Zerops Advanced Observability provides one-click, self-hosted, pre-configured deployments of the **[ELK stack](#elk-stack)** or **[Prometheus with Grafana](#prometheus-with-grafana)**. This enables comprehensive observability for your deployments while maintaining full control over your data and infrastructure.
7+
8+
## Deployment Modes
9+
10+
Both ELK and Prometheus can be deployed in two ways:
11+
12+
- **Local deployment**: Services are deployed within your target project
13+
- **Global deployment**: A dedicated project is created specifically for observability
14+
15+
## ELK Stack
16+
17+
The ELK observability setup deploys and configures two base services:
18+
19+
- **Elasticsearch** (as `elkstorage`)
20+
- **Kibana** (as `kibana`) - the observability access point
21+
22+
### Logging
23+
24+
For log collection with ELK, see the [Log Forwarding](/references/log-forwarding#self-hosted-logstash) guide which covers the Logstash setup and configuration.
25+
26+
### Tracing
27+
28+
To collect traces in the ELK stack, an APM Server instance (deployed as the `apmserver` service) is required. APM Server listens for incoming traces securely over HTTPS and indexes them to Elasticsearch.
29+
30+
First, follow the one-click GUI integration setup to deploy the required infrastructure. Then configure your application to send traces to APM Server.
31+
32+
#### Setting Up APM Agents
33+
34+
1. Update your code according to the [APM documentation](https://www.elastic.co/guide/en/apm/agent/index.html) for your application's language. See this [simple Go application](https://github.com/zerops-recipe-apps/go-hello-world-app/tree/with-apm-and-metrics) using APM libraries for reference.
35+
36+
2. Add the following environment variables to your application service (these are specific to the official Go APM Agent library and may differ for other agents):
37+
```
38+
ELASTIC_APM_ACTIVE=true
39+
ELASTIC_APM_SERVICE_NAME=recipe-go
40+
ELASTIC_APM_SERVER_URL=https://apmserver.url.copy.from.gui
41+
ELASTIC_APM_SECRET_TOKEN=secret_token_copy_from_gui
42+
```
43+
44+
:::note
45+
The `ELASTIC_APM_ACTIVE` variable is set to `false` by default on Zerops, so you must explicitly set it to `true` to enable APM for the official Go APM agent.
46+
:::
47+
48+
3. Restart or reload your application service.
49+
50+
You should start seeing traces appear in Kibana's "Applications > Traces" section.
51+
52+
#### What Happens Behind the Scenes
53+
54+
1. The `elkstorage`, `kibana`, and `apmserver` services are deployed and configured in the target project (if not already present)
55+
2. The `apmserver` is made publicly accessible over HTTPS via a Zerops subdomain and configured with a secret token for secure access
56+
57+
Access information for both Kibana and APM Server can be found in the **Advanced Observability** section of your project in the GUI.
58+
59+
### Next Steps
60+
61+
- **Security**: Set up custom domains for Kibana and APM Server
62+
- **Optimization**: Connect to APM Server locally via HTTP for improved performance
63+
- **Customization**: Fork the [ELK recipe](https://github.com/zeropsio/recipe-elk), customize the Logstash configuration, and redeploy the `logstash` service via `zcli`
64+
65+
## Prometheus with Grafana
66+
67+
The full Prometheus setup consists of:
68+
69+
- **Prometheus** (as `prometheus`)
70+
- **Grafana** (as `grafana`) - the observability access point
71+
- **PostgreSQL database** (as `grafanadb`) - for Grafana data storage
72+
- **S3 bucket** (as `prometheusbackups`) - for storing Prometheus backups
73+
74+
Metrics can be forwarded to the target project via a Prometheus exporter (single `prometheuslight` service configured [with remote write](https://grafana.com/docs/agent/latest/flow/reference/components/prometheus.remote_write/#prometheusremote_write)). When present in a source project, it collects metrics and forwards them to Prometheus in the target project.
75+
76+
Since Prometheus supports only filesystem storage and non-HA (single instance) setup, the `prometheus` service is configured to run a backup cron job that stores metric data snapshots in object storage.
77+
78+
### Metrics
79+
80+
After deploying the full or forwarder Prometheus setup, metrics are automatically scraped by Prometheus. After setup completes, log in to Grafana using the credentials found in the **Advanced Observability** section of your project in the GUI, then navigate to `/dashboards` to start exploring your metrics.
81+
82+
Zerops provides several metrics out of the box:
83+
84+
- Service scaling and resource metrics
85+
- PostgreSQL database service metrics (with HAProxy balancer metrics in HA setups)
86+
- MariaDB database service metrics
87+
- Valkey database metrics
88+
89+
:::note
90+
Some PostgreSQL database metrics are available only after enabling the `pg_stat_statements` extension. Run `CREATE EXTENSION IF NOT EXISTS pg_stat_statements;` as a superuser and restart the database service.
91+
:::
92+
93+
You can also expose custom metrics from your applications.
94+
95+
#### Exposing Custom Metrics
96+
97+
1. Configure your application to expose an HTTP `/metrics` endpoint on an arbitrary port. See this [simple Go application using `promauto`](https://github.com/zerops-recipe-apps/go-hello-world-app/tree/with-apm-and-metrics) for reference.
98+
99+
:::note
100+
Only the `/metrics` path is scraped and cannot currently be configured otherwise.
101+
:::
102+
103+
2. Add and commit the `ZEROPS_PROMETHEUS_PORT` environment variable to your application service with the port where you exposed the `/metrics` endpoint:
104+
```
105+
Single port:
106+
9090
107+
108+
Multiple ports can be defined using commas:
109+
9090,9091
110+
```
111+
112+
For example: `ZEROPS_PROMETHEUS_PORT=9090`. After setting the environment variable, your service's ports will be automatically added to metrics discovery and scraped by Prometheus.
113+
114+
#### What Happens Behind the Scenes
115+
116+
**Full setup:**
117+
118+
1. The `prometheus`, `grafana`, `grafanadb`, and `prometheusbackups` services are deployed and configured (if not already present)
119+
2. `grafana` is made publicly accessible over HTTPS, securely accessible via a generated password
120+
121+
After that, metrics are scraped and available for visualization in Grafana.
122+
123+
**Forwarder setup:**
124+
125+
1. The `prometheuslight` forwarder service is deployed in the source project
126+
2. The `prometheus`, `grafana`, `grafanadb`, and `prometheusbackups` services are deployed and configured in the target project (if not already present)
127+
3. `grafana` is made publicly accessible over HTTPS, securely accessible via a generated password
128+
4. `prometheus` is made publicly accessible over HTTPS via a Zerops zone domain and secured with basic access authentication (username and password)
129+
5. `prometheuslight` is configured to forward metrics to the target `prometheus` using the generated credentials over secure HTTPS
130+
131+
### Next Steps
132+
133+
- **Security**: Set up custom domains for Grafana and Prometheus
134+
- **Customization**: Fork the [Prometheus recipe](https://github.com/zeropsio/recipe-prometheus), customize dashboards and alerting rules, and redeploy via `zcli`
135+
- **Advanced monitoring**: Explore Grafana's alerting capabilities and create custom dashboards for your specific use cases

‎apps/docs/package.json‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -22,10 +22,10 @@
2222
"@babel/preset-react": "^7.18.6",
2323
"@cloudinary/react": "^1.11.2",
2424
"@cloudinary/url-gen": "^1.9.2",
25-
"@docusaurus/core": "latest",
26-
"@docusaurus/preset-classic": "latest",
27-
"@docusaurus/remark-plugin-npm2yarn": "latest",
28-
"@docusaurus/theme-mermaid": "latest",
25+
"@docusaurus/core": "3.4.0",
26+
"@docusaurus/preset-classic": "3.4.0",
27+
"@docusaurus/remark-plugin-npm2yarn": "3.4.0",
28+
"@docusaurus/theme-mermaid": "3.4.0",
2929
"@mdx-js/mdx": "3.0.0",
3030
"@mdx-js/react": "3",
3131
"@medusajs/icons": "1.2.1",
@@ -68,10 +68,10 @@
6868
},
6969
"devDependencies": {
7070
"@babel/plugin-proposal-decorators": "^7.22.15",
71-
"@docusaurus/eslint-plugin": "latest",
72-
"@docusaurus/module-type-aliases": "latest",
73-
"@docusaurus/tsconfig": "latest",
74-
"@docusaurus/types": "latest",
71+
"@docusaurus/eslint-plugin": "3.4.0",
72+
"@docusaurus/module-type-aliases": "3.4.0",
73+
"@docusaurus/tsconfig": "3.4.0",
74+
"@docusaurus/types": "3.4.0",
7575
"@types/react": "^18.2.0",
7676
"@types/react-dom": "^18.2.0",
7777
"@types/react-transition-group": "^4.4.6",

‎apps/docs/sidebars.js‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -721,12 +721,30 @@ module.exports = {
721721
{
722722
type: 'doc',
723723
id: 'references/logging',
724-
label: 'Logging & Log Forwarding',
724+
label: 'Logs',
725725
customProps: {
726726
sidebar_icon: 'list-bullet',
727727
},
728728
className: 'homepage-sidebar-item',
729729
},
730+
{
731+
type: 'doc',
732+
id: 'references/log-forwarding',
733+
label: 'Log Forwarding',
734+
customProps: {
735+
sidebar_icon: 'arrow-right-on-rectangle',
736+
},
737+
className: 'homepage-sidebar-item',
738+
},
739+
{
740+
type: 'doc',
741+
id: 'references/metrics',
742+
label: 'Metrics & Tracing',
743+
customProps: {
744+
sidebar_icon: 'chart-bar',
745+
},
746+
className: 'homepage-sidebar-item',
747+
},
730748
]
731749
},
732750
{

‎apps/docs/src/components/content/logs.mdx‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@ Zerops provides 3 different logs for your <Var name="servicePath" /> service:
66
- [Prepare runtime log](#prepare-runtime-log) - Custom runtime image creation
77
- [Runtime log](#runtime-log) - Operational output from your running service
88

9-
For comprehensive logging features including project-wide log views, filtering options, and log forwarding to external services, see the [Logging & Log Forwarding](/references/logging) documentation.
10-
119
## Quick access to logs
1210

1311
### Build log
@@ -26,6 +24,12 @@ For comprehensive logging features including project-wide log views, filtering o
2624
For severity levels to work properly in Zerops, your application must log to syslog.
2725
:::
2826

29-
:::tip
30-
For advanced log analysis, consider setting up [log forwarding](/references/logging#log-forwarding) to external logging services like Better Stack or Papertrail.
31-
:::
27+
## Beyond basic logging
28+
29+
Zerops offers additional observability options to help you monitor and debug your application:
30+
31+
### Log Forwarding
32+
Forward logs to [external services](/references/log-forwarding) for advanced analysis and long-term storage, including Better Stack, Papertrail, self-hosted ELK stack with Logstash, or any syslog-ng compatible software.
33+
34+
### Full Observability Stack
35+
Deploy [self-hosted observability solutions](/references/metrics) for comprehensive monitoring with ELK Stack (centralized logging with Kibana + application tracing with APM) or Prometheus & Grafana (metrics collection, visualization, and custom dashboards).

‎apps/docs/src/theme/Icon/index.tsx‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import {
55
ArrowDownLeftMini,
66
ArrowDownTray,
77
ArrowPath,
8+
ArrowRightOnRectangle,
89
ArrowUpRightOnBox,
910
ArrowUturnLeft,
1011
BarsThree,
@@ -172,6 +173,7 @@ export default {
172173
'arrow-down-left-mini': ArrowDownLeftMini,
173174
'arrow-down-tray': ArrowDownTray,
174175
'arrow-path': ArrowPath,
176+
'arrow-right-on-rectangle': ArrowRightOnRectangle,
175177
'back-arrow': ArrowUturnLeft,
176178
'bars-three': BarsThree,
177179
beaker: IconBeaker,

0 commit comments

Comments
 (0)