> For the complete documentation index, see [llms.txt](https://docs.rumi.systems/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.rumi.systems/rumi-management/guides/monitor/customizing-dashboards.md).

# Customizing Dashboards

This guide covers creating custom Grafana panels and dashboards for Rumi monitoring.

## Overview

While the canned dashboards cover common monitoring scenarios, you can create custom dashboards to focus on specific metrics, combine data from multiple sources, or add alerting.

## Creating Custom Panels

### Querying InfluxDB

All Rumi telemetry data is stored in InfluxDB under the `rumi_heartbeat_rp` retention policy. To create a custom panel:

1. Add a new panel to a dashboard.
2. Select the InfluxDB data source.
3. Write a query targeting the appropriate measurement.

### Query Example

To chart process CPU load for a specific container:

```sql
SELECT mean("process_cpu_load") FROM "rumi_heartbeat_rp"."application"
WHERE "system_name" =~ /^$system_name$/ AND "vm_name" =~ /^$vm_name$/
GROUP BY time($__interval) fill(null)
```

### Using Template Variables

Reference the existing template variables (`$system_name`, `$vm_name`, `$app_name`, `$host_name`) in your queries to make custom panels filterable using the same dropdowns as the canned dashboards.

## Creating Custom Dashboards

To create a new dashboard:

1. Navigate to **Dashboards** > **New Dashboard**.
2. Add panels querying the InfluxDB measurements.
3. Add template variables for filtering (see [Setting Up Dashboards](/rumi-management/guides/monitor/setting-up-dashboards.md) for variable query syntax).
4. Save the dashboard.

## Available Measurements

For a complete reference of all available InfluxDB measurements, tags, and fields, see the [InfluxDB Measurements Reference](/rumi-management/reference/agent/influxdb-measurements.md).

Key measurements for custom dashboards:

| Measurement                           | Content                                        |
| ------------------------------------- | ---------------------------------------------- |
| `application`                         | Container and application-level system metrics |
| `application.store`                   | State store operations and hit rates           |
| `application.messages`                | Message counts and throughput                  |
| `application.messages.type`           | Per-message-type counts                        |
| `application.messages.type.latencies` | Per-message-type latency statistics            |
| `application.transaction`             | Transaction counts and durations               |
| `application.transaction.latencies`   | Transaction latency percentiles                |
| `application.disruptor`               | Ring buffer utilization                        |
| `application.disruptor.latencies`     | Disruptor processing latencies                 |
| `application.user.gauge`              | Custom application gauges                      |
| `application.user.counter`            | Custom application counters                    |
| `application.bus`                     | Per-bus messaging statistics                   |
| `system`                              | Container-level system metrics                 |
| `system.gc`                           | Garbage collection statistics                  |
| `system.disk`                         | Disk I/O and space utilization                 |
| `system.threads`                      | Thread-level statistics                        |

## Alerting

Grafana supports alerting on panel queries. You can configure alerts on custom panels to notify when:

* Latency exceeds a threshold.
* Memory usage approaches limits.
* Error rates spike.
* Custom application counters cross thresholds.

See the [Grafana alerting documentation](https://grafana.com/docs/grafana/latest/alerting/) for details on configuring alert rules and notification channels.

## Best Practices

* **Save customizations separately** — Avoid modifying canned dashboards directly. Create new dashboards or copy panels to preserve the ability to upgrade canned dashboards.
* **Use consistent naming** — Follow the tag naming conventions used by the canned dashboards for consistency.
* **Leverage template variables** — Use the same template variable names so filtering works consistently across dashboards.