> For the complete documentation index, see [llms.txt](https://docs.rumi.systems/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.rumi.systems/rumi-management/reference/agent/influxdb-measurements.md). # InfluxDB Measurements This page is a reference for the InfluxDB measurements the Rumi Agent writes from each container heartbeat (and from trace records). Use it to build custom dashboards or write InfluxQL queries directly against the Agent's InfluxDB. All heartbeat-derived measurements are written under the `rumi_heartbeat_rp` retention policy. Trace records are written under `rumi_trace_rp`. {% hint style="info" %} **Note on nomenclature.** The measurement names and some tag names (for example, measurements starting with `application.*` and `xvm.*`, and the tags `app_name`, `app_state`, `app_role`, `app_ha_policy`, `vm_name`) retain legacy X Platform terminology. These names may be renamed ahead of the Rumi 4.0 release to align with Rumi nomenclature (service, container). The descriptions on this page use the Rumi nomenclature; the on-the-wire names are left unchanged so that existing dashboards and queries continue to work. {% endhint %} ## Common Tags Every heartbeat-derived measurement carries these tags: | Tag | Description | | ------------- | ---------------------------------------------------------------------------------------------- | | `system_name` | Name of the Rumi system (or `"unspecified"` if the container's heartbeat does not specify one) | | `vm_name` | Name of the container | | `host_name` | Hostname of the machine running the container | Service-scoped measurements (anything starting with `application.*`) additionally carry: | Tag | Description | | --------------- | ---------------------------------------------------------- | | `app_name` | Service name | | `app_state` | Current service state (e.g. `Active`, `Started`, `Loaded`) | | `app_role` | Service role (e.g. `Primary`, `Backup`) | | `app_ha_policy` | HA policy in effect for the service | Most measurements also set `system_version` as a field (not a tag), when the heartbeat reports one. ## Common Latency / Series Fields Any measurement whose name ends in `.latencies` (and a few others that wrap a histogram series) carries the same set of fields, emitted by the Agent's internal `addIntSeriesStats` / `addLongSeriesStats` helpers: | Field | Description | | -------------------------------------------------------------------------------------- | ------------------------------------------------------ | | `points` | Number of data points recorded in the current interval | | `skipped` | Number of skipped data points | | `sno` | Last sequence number | | `sample` | Interval histogram sample size | | `min`, `max`, `mean`, `median` | Interval histogram summary values | | `pct75`, `pct90`, `pct99`, `pct999`, `pct9999` | Interval histogram percentiles | | `running_sample` | Running (cumulative) histogram sample size | | `running_min`, `running_max`, `running_mean`, `running_median` | Running histogram summary values | | `running_pct75`, `running_pct90`, `running_pct99`, `running_pct999`, `running_pct9999` | Running histogram percentiles | Values are in microseconds for recent heartbeat formats (v4 and later); earlier formats stored them in nanoseconds and the Agent scales them up by 1000 so the stored fields are always in microseconds. Each latency sub-variant is distinguished by an additional `name` tag (e.g. `name=MsgProcessing`, `name=CommitStore`, `name=OfferToPoll`). The available `name` values per measurement are enumerated in each section below. ## System Measurements ### `system` Container-level system metrics. One point per heartbeat per container. **Tags:** common tags, plus optionally `jit_name` (when the container reports JIT compiler stats). **Selected fields:** | Field | Description | | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `xvm_loaded_time` | Container load timestamp (when the heartbeat reports it) | | `processors` | Number of available processors | | `load_average` | System load average | | `process_cpu_load` | Process CPU load (0.0–1.0), when available | | `system_cpu_load` | System-wide CPU load (0.0–1.0), when available | | `threads_count`, `threads_daemon_count`, `threads_peak_count` | JVM thread counts | | `jit_time` | Cumulative JIT compilation time (when JIT stats are reported) | | `gc_count`, `gc_time` | Aggregate GC counts and cumulative time across all collectors | | `memory_heap_init`, `memory_heap_used`, `memory_heap_committed`, `memory_heap_max` | Heap memory usage (bytes) | | `memory_non_heap_init`, `memory_non_heap_used`, `memory_non_heap_committed`, `memory_non_heap_max` | Non-heap memory usage (bytes) | | `memory_off_heap_used`, `memory_off_heap_live`, `memory_off_heap_pooled` | Off-heap memory usage (bytes) | | `memory_process_rss` | Process RSS in bytes (when available) | | `memory_physical_size_total`, `memory_physical_size_free` | Host physical memory (bytes) | | `memory_swap_size_total`, `memory_swap_size_free` | Host swap (bytes) | | `memory_virtual_size_committed` | Committed virtual memory (bytes) | | `memory_native_count_*`, `memory_native_bytes_*` | Native allocator counters and byte totals | | `memory_iobuf_count_*`, `memory_iobuf_bytes_*` | I/O buffer counters and byte totals | | `memory_embedded_entity_*`, `memory_entity_*`, `memory_message_*` | Aggregated allocator counters for embedded entities, state entities, and messages (counts, pooled/live, POJO/serializer/deserializer/data/metadata variants) | ### `system.disk` One point per disk root per heartbeat. **Tags:** common tags, plus `disk_root` (root path). | Field | Description | | ------------------- | ------------------------------------- | | `disk_total_space` | Total space on the disk root (bytes) | | `disk_usable_space` | Usable space on the disk root (bytes) | | `disk_free_space` | Free space on the disk root (bytes) | ### `system.threads` One point per tracked thread per heartbeat (written only when per-thread stats are available; requires the native library). **Tags:** common tags, plus `id` (OS thread ID as a string). | Field | Description | | ------------------------- | --------------------------------------------------------------- | | `id` | OS thread ID (numeric) | | `name` | Thread name | | `state` | Thread state (e.g. `RUNNING`, `SLEEPING`) | | `cpu_time` | Total CPU time (ns) | | `usr_time` | User-mode CPU time (ns) | | `collect_time` | Timestamp when the native sample was taken (ns) | | `affinity` | Affinity mask summary (when affinitization is enabled) | | `wait_count`, `wait_time` | Disruptor wait-count and wait-time, when wait stats are enabled | ### `system.gc` One point per GC collector per heartbeat. **Tags:** common tags, plus `collector_name`. | Field | Description | | ----------------- | -------------------------------------------- | | `num_collections` | Total collections by this collector | | `cum_time` | Cumulative time spent in this collector (ms) | ### `system.entity_memory` One point per entity type per heartbeat (broken out by `type`: embedded entity, state entity, message). **Tags:** common tags, plus: | Tag | Description | | ------ | --------------------------------------------------- | | `type` | One of `embedded_entity`, `state_entity`, `message` | | `name` | Fully qualified entity or message name | **Fields:** a `short_name` field plus the full set of per-type allocator counters from `addTypeSpecificEntityMemoryCounters` (alloc / pool-alloc / GC-dispose / pool-dispose / pool-leak / live / pooled counts and byte totals for the type, its POJO, its serializer, its deserializer, its data payload, and its metadata). ## Container-Level Measurements ### `xvm.heartbeats.logger` Heartbeat-logging subsystem statistics. **Tags:** common tags. | Field | Description | | ---------------- | ------------------------------------- | | `detached` | Whether detached persist is in effect | | `commits` | Number of commits | | `forks` | Number of forks | | `messages` | Number of heartbeat messages recorded | | `file_size` | Current log file size (bytes) | | `file_size_used` | Bytes used within the log file | ### `xvm.heartbeats.logger.disruptor` Disruptor ring-buffer state for the heartbeat logger (only when the logger runs on a disruptor). **Tags:** common tags. | Field | Description | | --------------------------------- | ------------------------------- | | `capacity`, `remaining`, `used` | Ring-buffer capacity and usage | | `claim_strategy`, `wait_strategy` | Configured disruptor strategies | ### `xvm.heartbeats.logger.latencies` Offer-to-poll latencies for the heartbeat-logger disruptor. **Tags:** common tags, plus `name` (currently only `OfferToPoll`). **Fields:** see [Common Latency / Series Fields](#common-latency--series-fields). ### `client.admin` Admin client (connection to the Rumi Admin) statistics. One point per admin client per heartbeat. **Tags:** common tags, plus `name` (admin client name). | Field | Description | | ------------------------------------------------------------------ | ------------------------- | | `lifecyclesSent` | Lifecycle messages sent | | `heartbeatsSent`, `heartbeatsDropped` | Heartbeats sent / dropped | | `tracesSent`, `tracesDropped` | Traces sent / dropped | | `outboundQueueCapacity`, `outboundQueueSize`, `outboundQueueCount` | Outbound queue state | ### `pools` Per-pool statistics. One point per pool per heartbeat. **Tags:** common tags, plus `name` (pool key). | Field | Description | | ------------------- | ------------------------------ | | `capacity`, `size` | Pool capacity and current size | | `hits`, `misses` | Hit/miss counts | | `gets`, `puts` | Get/put counts | | `growths`, `evicts` | Growth and eviction counts | | `preallocated` | Preallocated count | | `detached_washes` | Detached-wash count | ## Service Measurements ### `application` One point per service per heartbeat, carrying service **info** (not metrics). **Tags:** common tags plus service-scoped tags. | Field | Description | | ------------------------------ | ------------------------------------------ | | `version` | Service version | | `role` | Current role (e.g. `Primary`, `Backup`) | | `ha_policy` | HA policy | | `state` | Current state (e.g. `Active`, `Started`) | | `loaded_time` | Service load timestamp (when reported) | | `jar_version` | Jar version string (when reported) | | `bindings`, `clients`, `flows` | Number of bus bindings, clients, and flows | ### `application.events` **Fields:** `rcvd` (events received), `proc` (events processed). ### `application.flows` **Fields:** `count` (number of active flows). ### `application.multiplexer` Event multiplexer statistics. | Field | Description | | --------------------- | --------------------------- | | `last_decongested` | Last decongestion timestamp | | `max_concurrency` | Max concurrency | | `schedule_queue_size` | Current schedule queue size | ### `application.disruptor` Event multiplexer disruptor ring-buffer state. **Fields:** `capacity`, `remaining`, `used`, `claim_strategy`, `wait_strategy`. ### `application.disruptor.latencies` Event multiplexer offer-to-poll latencies. **Tags:** app-scoped tags; no additional `name` tag (single series). **Fields:** see [Common Latency / Series Fields](#common-latency--series-fields). ### `application.messages` Aggregate inbound and outbound messaging counters. | Field | Description | | ----------------------------------------------------- | -------------------------------------------------- | | `in_best_effort`, `in_guaranteed` | Inbound messages by QoS | | `in_proc_complete`, `in_proc_fail`, `in_proc_success` | Flow-event processing outcomes | | `in_dups` | Duplicate inbound messages | | `in_filtered` | Filtered inbound messages | | `in_sourced` | Inbound messages sourced internally | | `in_value` | Total inbound flow events | | `out_best_effort`, `out_guaranteed` | Outbound messages by QoS | | `out_resent` | Messages resent | | `out_sno`, `out_stable_sno` | Outbound and stable sequence numbers | | `out_value` | Total outbound messages (best-effort + guaranteed) | ### `application.messages.type` Per-message-type counts. One point per message type per service per heartbeat. **Tags:** app-scoped tags, plus `msg_id` (composite of type name, factory id, type id). | Field | Description | | ------------------------------------------------------------------------------------- | --------------------------------------- | | `msg_name`, `msg_factory_id`, `msg_type_id` | Message type identifiers | | `in_best_effort`, `in_guaranteed`, `in_sourced`, `in_dups`, `in_filtered`, `in_value` | Inbound counters for this message type | | `out_best_effort`, `out_guaranteed`, `out_resent`, `out_value` | Outbound counters for this message type | ### `application.messages.type.latencies` Per-message-type latency histograms. **Tags:** app-scoped tags, `msg_id`, plus `name` one of `CreateToOffer`, `OfferToPoll`, `MsgFiltering`, `MsgPreProc`, `MsgProcessing`. **Fields:** `msg_name`, `msg_factory_id`, `msg_type_id`, plus [Common Latency / Series Fields](#common-latency--series-fields). ### `application.transaction` Transaction-level counters. | Field | Description | | ---------------------------------------------------------------------------------------- | ----------------------------------------- | | `count` | Number of transactions | | `started`, `completed` | Commit starts/completions | | `rollbacks` | Rollback count | | `inflight` | In-flight commits (`started - completed`) | | `store_commits_started`, `store_commits_completed`, `store_commit_queue_completion_size` | Store-commit counters | | `send_commits_started`, `send_commits_completed`, `send_commit_queue_completion_size` | Send-commit counters | | `events_per_commit` | Flow events per commit (ratio) | ### `application.transaction.latencies` Transaction-path latency histograms. **Tags:** app-scoped tags, plus `name` one of: `CommitEpilogue`, `CommitFull`, `CommitPrologue`, `CommitSend`, `CommitStart`, `CommitStore`, `CommitTransition`, `InAck`, `InOut`, `MsgFiltering`, `MsgPreProc`, `MsgProcessing`, `MsgSendCore`, `MsgSend`, `TransactionLeg1`, `TransactionLeg2`, `TransactionLeg3`. **Fields:** see [Common Latency / Series Fields](#common-latency--series-fields). ### `application.backup` Backup replication queue depths. **Fields:** `outbound_queue_size`, `outbound_log_queue_size`. ## Service Message-Logger Measurements Each of the three message loggers (inbound, outbound, per-transaction-stats) writes three measurements: a general stats point, a disruptor ring-buffer point, and an offer-to-poll latency histogram. ### `application.messagelogger.inbound`, `application.messagelogger.outbound`, `application.pertransactionstats.logger` General message-logger stats. | Field | Description | | ----------------------------- | ---------------------------- | | `detached` | Detached-persist flag | | `commits` | Commit count | | `forks` | Fork count | | `messages` | Messages recorded | | `file_size`, `file_size_used` | Log file size and bytes used | ### `application.messagelogger.inbound.disruptor`, `application.messagelogger.outbound.disruptor`, `application.pertransactionstats.logger.disruptor` Disruptor ring-buffer state for the corresponding logger. **Fields:** `capacity`, `remaining`, `used`, `claim_strategy`, `wait_strategy`. ### `application.messagelogger.inbound.latencies`, `application.messagelogger.outbound.latencies`, `application.pertransactionstats.logger.latencies` Offer-to-poll latencies for the corresponding logger disruptor. **Tags:** app-scoped tags, plus `name=OfferToPoll`. **Fields:** see [Common Latency / Series Fields](#common-latency--series-fields). ## Microservice State Store Measurements ### `application.store` Microservice state store (ODS) top-level stats. | Field | Description | | -------------------------------------------------------- | ------------------------------ | | `state` | Store state | | `role` | Store role | | `icr_role` | Inter-cluster-replication role | | `size` | Store size | | `data_size` | Data size | | `commit_completions_received`, `commit_completions_sent` | Commit-completion counters | | `commit_entries_rcvd`, `commit_entries_sent` | Commit-entry counters | | `commit_commits_rcvd`, `commit_commits_sent` | Commit counters | ### `application.store.latencies` Store-path latency histograms. **Tags:** app-scoped tags, plus `name` one of: `CommitQueueSize`, `Commit`, `Deserialize`, `IcrSend`, `Index`, `Persist`, `Serialize`, `SerializeToPersist`, `SerializeToWire`, `Wire`, `WireToDeserialize`. **Fields:** see [Common Latency / Series Fields](#common-latency--series-fields). ### `application.store.persister` Persister stats (written only when the store is persisted). | Field | Description | | ------------------------------------- | -------------------------------------- | | `detached` | Detached-persist flag | | `commits`, `forks` | Commit / fork counts | | `puts`, `updates`, `removes`, `sends` | Recorded operation counts | | `file_size`, `file_size_used` | Persister log file size and bytes used | ### `application.store.persister.disruptor`, `application.store.persister.latencies` Disruptor ring-buffer and offer-to-poll latencies for the persister. Same field shapes as the other disruptor / latency pairs above. ### `application.store.icr_sender` Inter-cluster-replication sender stats. | Field | Description | | ------------------------------------- | ------------------------- | | `detached` | Detached-send flag | | `commits` | Commit count | | `puts`, `updates`, `removes`, `sends` | Recorded operation counts | ### `application.store.icr_sender.disruptor`, `application.store.icr_sender.latencies` Disruptor ring-buffer and offer-to-poll latencies for the ICR sender. Same field shapes as the other disruptor / latency pairs above. ## Service Bus Measurements ### `application.bus` Per-bus-binding stats. One point per external bus binding per service per heartbeat (internal bindings are skipped). **Tags:** app-scoped tags, plus `bus_name`. | Field | Description | | ----------------------------------------------------------------------- | ------------------------------------------------------------------------ | | `internal` | Whether the binding is internal (always `false` for points emitted here) | | `detached_commmit` | Detached-commit flag (spelled with three `m`s for historical reasons) | | `commits`, `rollbacks`, `failures` | Transaction counters | | `channels_up`, `clients` | Channel and client counts | | `msgs_rcvd`, `msgs_sent` | Message counters | | `stability_rcvd`, `stability_batches_rcvd`, `stability_rcvd_in_batches` | Stability-ack counters | | `msgs_enqueued` | Enqueued message count | | `rcvd_batches`, `rcvd_batch_msgs` | Received-batch counters | | `acks_sent`, `packets_rcvd` | Ack / packet counters | ### `application.bus.disruptor` Per-bus disruptor ring-buffer state. **Tags:** app-scoped tags, plus `bus_name`. **Fields:** `capacity`, `remaining`, `used`, `claim_strategy`, `wait_strategy`. ### `application.bus.disruptor.latencies` Per-bus disruptor offer-to-poll latencies. **Tags:** app-scoped tags, `bus_name`, plus `name=OfferToPoll`. **Fields:** see [Common Latency / Series Fields](#common-latency--series-fields). ### `application.bus.latencies` Per-bus messaging latency histograms. **Tags:** app-scoped tags, `bus_name`, plus `name` one of: `CreateToSend`, `Deserialize`, `DeserializeToReceive`, `OriginToReceive`, `SendToSerialize`, `Serialize`, `SerializeToWire`, `Wire`, `WireSend`, `WireToDeserialize`, `WireToWire`. **Fields:** see [Common Latency / Series Fields](#common-latency--series-fields). ## Service User-Defined Stats These measurements carry custom stats that service code defines via the Rumi stats API. See [Exposing Application Statistics](https://github.com/rumidata/nvx-rumi-docs/blob/develop/management/guides/developing-applications/authoring-user-code/monitoring/exposing-application-statistics.md). ### `application.user.gauge` One point per user-defined gauge per heartbeat. **Tags:** app-scoped tags, plus `name` (gauge name). Because InfluxDB cannot mix numeric and string values in the same field across series, the Agent writes one of two fields depending on the gauge type: | Gauge type | Field written | | ------------------------------------------------- | ------------------------------- | | `BYTE`, `SHORT`, `INT`, `LONG`, `FLOAT`, `DOUBLE` | `numeric_value` (as a `double`) | | `BOOLEAN`, `CHAR`, `STRING` | `string_value` | ### `application.user.counter` One point per user-defined counter per heartbeat. **Tags:** app-scoped tags, plus `name` (counter name). | Field | Description | | ------- | ------------- | | `value` | Counter value | ### `application.user.series` One point per user-defined series per heartbeat. Supports `INT` and `LONG` series (other types are skipped and logged as a warning). **Tags:** app-scoped tags, plus `name` (series name). **Fields:** see [Common Latency / Series Fields](#common-latency--series-fields). ## Trace Measurements ### `trace` Trace records from containers. Written under the `rumi_trace_rp` retention policy (not `rumi_heartbeat_rp`). Not emitted for internal stats/heartbeat tracer names. **Tags:** | Tag | Description | | ------------- | ----------------------------------------------------------- | | `host` | `hostname/address` of the originating container | | `xvm` | Container name | | `type` | Fixed value `xvm` | | `collector` | Agent service name | | `trace_level` | Trace level in lowercase (e.g. `info`, `warning`, `severe`) | **Fields:** | Field | Description | | ------------- | ---------------------------------- | | `trace` | The trace message text | | `trace_id` | Composite `host/address/xvm` key | | `tracer_name` | Source tracer name (when reported) | | `process_id` | Source process id (when reported) | | `thread_id` | Source thread id (when reported) | ## Example Queries All examples assume the `rumi_heartbeat_rp` retention policy on the `rumi` database (InfluxDB 2.x exposes this via the V1 compatibility API). ### Process CPU Load for a Specific Container ```sql SELECT mean("process_cpu_load") FROM "rumi_heartbeat_rp"."system" WHERE "system_name" = 'my-system' AND "vm_name" = 'processor-1' AND time > now() - 1h GROUP BY time(10s) fill(null) ``` ### Transaction Processing 99th-Percentile Latency ```sql SELECT mean("pct99") FROM "rumi_heartbeat_rp"."application.transaction.latencies" WHERE "system_name" = 'my-system' AND "app_name" = 'my-app' AND "name" = 'MsgProcessing' AND time > now() - 1h GROUP BY time(10s) fill(null) ``` ### Store Commit Latency, 99.9th Percentile ```sql SELECT mean("pct999") FROM "rumi_heartbeat_rp"."application.store.latencies" WHERE "system_name" = 'my-system' AND "app_name" = 'my-app' AND "name" = 'Commit' AND time > now() - 1h GROUP BY time(10s) fill(null) ``` ### Custom User Gauges ```sql SELECT last("numeric_value") FROM "rumi_heartbeat_rp"."application.user.gauge" WHERE "system_name" = 'my-system' AND "app_name" = 'my-app' AND time > now() - 1h GROUP BY "name" ``` ### Custom User Series, 99th Percentile by Series Name ```sql SELECT mean("pct99") FROM "rumi_heartbeat_rp"."application.user.series" WHERE "system_name" = 'my-system' AND "app_name" = 'my-app' AND time > now() - 1h GROUP BY time(10s), "name" fill(null) ``` ### Inbound vs. Outbound Message Rate ```sql SELECT non_negative_derivative(last("in_value"), 1s) AS inbound, non_negative_derivative(last("out_value"), 1s) AS outbound FROM "rumi_heartbeat_rp"."application.messages" WHERE "system_name" = 'my-system' AND "app_name" = 'my-app' AND time > now() - 1h GROUP BY time(10s) fill(null) ``` ### List All Systems Seen ```sql SHOW TAG VALUES FROM "rumi_heartbeat_rp"."system" WITH KEY = "system_name" ``` ### List Containers in a System ```sql SHOW TAG VALUES FROM "rumi_heartbeat_rp"."system" WITH KEY = "vm_name" WHERE "system_name" = 'my-system' ```