> For the complete documentation index, see [llms.txt](https://docs.rumi.systems/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.rumi.systems/rumi-core/guides/developing-applications/configuring-the-runtime/monitoring/xvm-heartbeats.md).

# Container Heartbeats

Configure periodic container-level statistics emission from your container.

## Overview

Container heartbeats provide periodic snapshots of container-level statistics including system metrics, thread statistics, pool usage, engine stats, and application-defined stats. Heartbeats can be emitted via messaging, logged to binary transaction logs, or traced for debugging.

Statistics collection must be balanced against performance impact. While many statistics have minimal overhead, enabling all collection options can impact latency-sensitive applications.

## Heartbeat Configuration

Heartbeats for a container can be enabled via DDL using the `<heartbeats>` element:

```xml
<containers>
  <container name="my-container">
    <heartbeats enabled="true" interval="5">
      <collectNonZGStats>true</collectNonZGStats>
      <collectIndividualThreadStats>true</collectIndividualThreadStats>
      <collectSeriesStats>true</collectSeriesStats>
      <collectSeriesDatapoints>false</collectSeriesDatapoints>
      <maxTrackableSeriesValue>100000000</maxTrackableSeriesValue>
      <includeMessageTypeStats>false</includeMessageTypeStats>
      <collectPoolStats>true</collectPoolStats>
      <poolDepletionThreshold>1.0</poolDepletionThreshold>
      <logging enabled="true"></logging>
      <tracing enabled="true"></tracing>
    </heartbeats>
  </container>
</containers>
```

## Configuration Settings

| Configuration Setting          | Default    | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| ------------------------------ | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `enabled`                      | false      | <p>Enable or disable container stats collection and heartbeat emission.<br><br></p><div data-gb-custom-block data-tag="hint" data-style="warning" class="hint hint-warning"><p><br><strong>Performance Impact</strong>: Collection of stats and emission of heartbeats can impact application performance from both a latency and throughput standpoint. For applications that are particularly sensitive to performance, it is a good idea to compare performance with and without heartbeats enabled to understand the overhead that is incurred by enabling heartbeats.<br></p></div>                                                                                                                                                                                                                                                                                      |
| `interval`                     | 1000       | The interval in seconds at which container stats will be collected and emitted                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `collectNonZGStats`            | true       | Some statistics collected by the stats collection thread require creating a small amount of garbage. This can be set to false to suppress collection of these stats                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `collectIndividualThreadStats` | true       | Indicates whether heartbeats will contain stats for each active thread in the JVM. Individual thread stats are useful for identifying busy threads and thread affinitization candidates                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `collectSeriesStats`           | true       | Indicates whether or not series stats should be included in heartbeats                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `collectSeriesDatapoints`      | false      | <p>Indicates whether or not series stats should report the data points captured for a series statistic.<br><br></p><div data-gb-custom-block data-tag="hint" data-style="warning" class="hint hint-warning"><p><br><strong>Warning</strong>: Enabling this value includes each datapoint collected in a series in heartbeats which can make emitted heartbeats very large and slow down their collection. It is not recommended that this be run in production.<br></p></div>                                                                                                                                                                                                                                                                                                                                                                                                 |
| `maxTrackableSeriesValue`      | 10 minutes | The maximum value (in **microseconds**) that can be tracked for reported series histogram timings. Datapoints above this value will be downsampled to this value, but will be reflected in the max value reported in an interval                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `includeMessageTypeStats`      | false      | <p>Sets whether or not message type stats are included in heartbeats (when enabled for the app).<br><br>When <code>captureMessageTypeStats</code> is enabled for an app, the AEP engine will record select statistics on a per message type basis. Because inclusion of per message type stats can significantly increase the size of heartbeats, inclusion in heartbeats is disabled by default.<br><br></p><div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p><br><strong>Tip</strong>: For message type stats to be included in heartbeats, both <code>captureMessageTypeStats</code> for the app must be set to true (capture is disabled by default because recording them is costly), and <code>includeMessageTypeStats</code> must be set to true (inclusion is disabled by default because emitting them is costly).<br></p></div> |

\| | `collectPoolStats` | true | Indicates whether or not pool stats are collected by the container | | `poolDepletionThreshold` | 1.0 | Configuration property used to set the percentage decrement at which a preallocated pool must drop to be included in a container heartbeat. Setting this to a value greater than 100 or less than or equal to 0 disables depletion threshold reporting.\
\
This gives monitoring applications advanced warning if it looks like a preallocated pool may soon be exhausted. By default the depletion threshold is set to trigger inclusion in heartbeats at every 1% depletion of the preallocated count. This can be changed by specifying the configuration property `nv.server.stats.pool.depletionThreshold` to a float value between 0 and 100.\
\
**Example**: If a pool is preallocated with 1000 items and this property is set to 10, pool stats will be emitted for the pool each time a heartbeat occurs and the pool has dropped below a 10% threshold of the preallocated size (e.g., at 900, 800, 700, until its size reaches 0). | | `logging` | *See below* | Configures binary logging of heartbeats.\
\
Binary heartbeat logging provides a means by which heartbeat data can be captured in a zero garbage fashion. Collection of such heartbeats can be useful in diagnosing performance issues in running apps | | `tracing` | *See below* | Configures trace logging of heartbeats.\
\
Enabling textual tracing of heartbeats is a useful way to quickly capture data from container heartbeats for applications that aren't monitoring container heartbeats remotely. Textual trace of heartbeats is not zero garbage and is therefore not suitable for applications that are latency sensitive |

## Trace Output Configuration

By default all container statistics tracers are disabled as trace logging is not zero garbage and introduces CPU overhead. While tracing heartbeats isn't recommended in production, enabling container statistics trace output can be useful for debugging and performance tuning.

```xml
<containers>
  <container name="my-container">
    <heartbeats enabled="true" interval="5">
      <tracing enabled="true">
        <traceSysStats>true</traceSysStats>
        <traceThreadStats>true</traceThreadStats>
        <tracePoolStats>true</tracePoolStats>
        <traceAppStats>true</traceAppStats>
        <traceUserStats>true</traceUserStats>
      </tracing>
    </heartbeats>
  </container>
</containers>
```

{% hint style="info" %}
**Note**: Memory statistics (heap, non-heap, off-heap, IOBuffer, and entity memory counters) are included as part of the system stats trace when `traceSysStats` is enabled. See [Memory Statistics](/rumi-core/guides/operating-applications/monitoring/memory-statistics.md) for a complete description of all memory stats fields and their meanings.
{% endhint %}

## Binary Logging Configuration

Applications that are latency sensitive might prefer to leave all tracers disabled to avoid unnecessary allocations and the associated GC activity. As an alternative, it's possible to enable logging of zero-garbage heartbeat messages to a binary transaction log:

```xml
<containers>
  <container name="my-container">
    <heartbeats enabled="true" interval="5">
      <logging enabled="true">
         <storeRoot>/path/to/heartbeat/log/directory</storeRoot>
      </logging>
    </heartbeats>
  </container>
</containers>
```

When a `storeRoot` is not set, a container will log heartbeats to `{Config.getDataDirectory}/container-heartbeats/<container-name>-heartbeats.log`, which can then be queried and traced from a separate process using the [Stats Dump Tool](/rumi-cli/commands/tools/hbdump.md).

{% hint style="warning" %}
**Warning**: At this time binary heartbeat logs do not support rolling collection. Consequently this mechanism is not suitable for long running application instances.
{% endhint %}

## Related Topics

* [Container Heartbeats](/rumi-core/guides/operating-applications/monitoring/container-heartbeats.md) - Viewing and interpreting heartbeat statistics at runtime
* [Engine Statistics](/rumi-core/guides/developing-applications/configuring-the-runtime/monitoring/engine-statistics.md) - Configure engine-level statistics collection
* [Stats Dump Tool](/rumi-cli/commands/tools/hbdump.md) - Analyzing binary heartbeat logs
* [Operating Model](/rumi-core/concepts/operating-model.md) - Monitoring architecture overview

## Next Steps

1. Enable heartbeats with appropriate collection settings
2. Test performance impact under load
3. Configure binary logging for zero-garbage collection or tracing for debugging
4. Set up monitoring tools to consume heartbeats
5. Balance operational visibility against performance requirements