Link Module

The Link module benchmarks cluster replication link latency, measuring the performance of the low-level transport used for replicating state between cluster members.

Overview

The Link layer provides the foundation for cluster replication in Rumi, offering:

• Direct point-to-point communication between cluster members

• Minimal overhead transport for state replication

• Support for various network protocols (TCP, UDP, InfiniBand, etc.)

This benchmark measures the raw performance capabilities of the replication link layer.

Test Programs

The Link module provides multiple test program variants:

Streaming Tests

Measure unidirectional throughput:

Blocking Variant:

• Sender: com.neeve.perf.link.BlockingStreamingSender

• Receiver: com.neeve.perf.link.BlockingStreamingReceiver

Non-Blocking Variant:

• Receiver: com.neeve.perf.link.NonBlockingStreamingReceiver (no sender variant)

RDMA Variant:

• Sender: com.neeve.perf.link.RdmaStreamingSender

• Receiver: com.neeve.perf.link.RdmaStreamingReceiver

The sender continuously sends messages to the receiver as fast as possible, measuring maximum sustained throughput.

Ping-Pong Tests

Measure round-trip latency:

Blocking Variant:

• Sender: com.neeve.perf.link.BlockingPingPongSender

• Receiver: com.neeve.perf.link.BlockingPingPongReceiver

Non-Blocking Variant:

• Sender: com.neeve.perf.link.NonBlockingPingPongSender

• Receiver: com.neeve.perf.link.NonBlockingPingPongReceiver

The sender sends a message and waits for a response from the receiver, measuring round-trip time.

Command-Line Parameters

Blocking Streaming Sender

Short	Long	Default	Description
-d	--descriptor	-	Connection descriptor (e.g., tcp://192.168.1.7:12000&tcpnodelay=true)
-m	--messageSize	256	Message size in bytes
-b	--bufferSize	256	Write buffer size
-t	--testCount	100000000	Number of messages to send
-r	--testRate	10000000	Send rate (messages/sec)
-w	--warmupTime	2	Warmup time in seconds
-c	--cpuAffinityMask	-	CPU affinity mask
-i	--printIntervalStats	false	Output periodic interval stats
-f	--dontWriteLatenciesToFile	false	Suppress latency file output

Blocking Streaming Receiver

Short	Long	Default	Description
-d	--descriptor	-	Connection descriptor
-m	--messageSize	256	Message size in bytes
-c	--cpuAffinityMask	-	CPU affinity mask
-s	--stats	false	Output incremental throughput stats

Blocking Ping-Pong Sender

Short	Long	Default	Description
-d	--descriptor	-	Connection descriptor
-m	--messageSize	256	Message size in bytes
-c	--testCount	300000	Number of messages to send
-r	--testRate	10000	Send rate (messages/sec)
-a	--cpuAffinityMask	-	CPU affinity mask
-p	--spinRead	false	Spin or block on read
-o	--oneWayLatency	false	Calculate one-way latency
-i	--printIntervalStats	false	Output periodic interval stats
-f	--dontWriteLatenciesToFile	false	Suppress latency file output

Blocking Ping-Pong Receiver

Short	Long	Default	Description
-d	--descriptor	-	Connection descriptor
-m	--messageSize	256	Message size in bytes
-c	--cpuAffinityMask	-	CPU affinity mask
-p	--spinRead	false	Spin or block on read
-s	--stats	false	Output incremental throughput stats

Use Cases

Streaming Test

Purpose: Measure maximum throughput Setup: Two machines connected via high-speed network Use Case: Validate network configuration and capacity

Ping-Pong Test

Purpose: Measure minimum latency Setup: Two machines connected via low-latency network Use Case: Validate network tuning and baseline latency

Running Benchmarks

Streaming Throughput Test

On Receiver Machine:

$JAVA_HOME/bin/java -cp "libs/*" com.neeve.perf.link.BlockingStreamingReceiver \
  -d "tcp://0.0.0.0:5000&tcpnodelay=true" \
  -m 256 \
  -s true

On Sender Machine:

$JAVA_HOME/bin/java -cp "libs/*" com.neeve.perf.link.BlockingStreamingSender \
  -d "tcp://receiver.example.com:5000&tcpnodelay=true" \
  -m 256 \
  -t 10000000 \
  -r 10000000

Ping-Pong Latency Test

On Receiver Machine:

$JAVA_HOME/bin/java -cp "libs/*" com.neeve.perf.link.BlockingPingPongReceiver \
  -d "tcp://0.0.0.0:5000&tcpnodelay=true" \
  -m 256 \
  -s true

On Sender Machine:

$JAVA_HOME/bin/java -cp "libs/*" com.neeve.perf.link.BlockingPingPongSender \
  -d "tcp://receiver.example.com:5000&tcpnodelay=true" \
  -m 256 \
  -c 100000 \
  -r 10000

Interpreting Results

Throughput Results

Messages sent: 10000000
Duration: 45.2 seconds
Throughput: 221,238 messages/second
Bandwidth: 452 Mbps

Latency Results

Round-trip latency statistics:
  50th percentile: 42.5 µs
  99th percentile: 58.3 µs
  99.9th percentile: 82.1 µs

Network Configurations

TCP over 10GbE

Typical Results:

• Throughput: 1-2M messages/second

• Latency: 15-25µs round-trip

TCP over InfiniBand

Typical Results:

• Throughput: 2-4M messages/second

• Latency: 8-15µs round-trip

RDMA over InfiniBand

Typical Results:

• Throughput: 5-10M messages/second

• Latency: 2-5µs round-trip

Comparison with Higher Layers

Link layer provides the foundation for cluster replication:

• Link Layer: ~10µs (raw replication transport)

• Messaging Layer: ~15µs (adds SMA abstractions)

• AEP Engine: ~27µs (adds transactions, persistence, full clustering with consensus)

Next Steps

• Review Messaging Module for SMA layer benchmarks

• Review AEP Module for end-to-end benchmarks

• Return to Benchmark Suite for other modules