Optimizing System Configuration
There are a virtually unlimited number of unique individual setups that cannot be covered in this document. As well, even similar builds and configurations can behave differently due to external factors, so your results may vary. Here are some general guidelines to use as a starting point. Be cautious, make incremental changes, testing and observing before moving forward. Always focus on only one specific area at a time - avoid making changes to memory, storage, and CPU configs all at once. Diagnosing potential problems becomes nearly impossible otherwise.Memory Management
The following settings in/etc/sysctl.conf can optimize memory usage and disk I/O patterns:
sudo sysctl -p
Network Stack
The following settings in/etc/sysctl.conf may improve network performance:
Storage Configuration
For NVMe drives, optimize I/O scheduling: Storage Optimization CommandsInfrastructure Monitoring
Monitoring is one of the most critical components of network infrastructure. performance tuning, and alerting configuration for Cosmos-SDK/Tendermint nodes.Prometheus Setup
First, install Prometheus:Grafana Integration
Install and configure Grafana:Sample Grafana Dashboard JSON
Sample Grafana Dashboard JSON
Alert Management
Install Alertmanager:Create Alert Rules Configuration
Create Alert Rules Configuration
Log Management
Loki Setup
Using Loki for log aggregation:Promtail Configuration
Promtail Configuration
Log Rotation
Configure logrotate to manage log files:Security Configuration
Network Security
UFW firewall configuration:Rate Limiting
Example Nginx Configuration with Rate Limiting
Example Nginx Configuration with Rate Limiting
Validator-Specific Monitoring
Status Query
Query validator status through SDK:Validator "Status" Query Script
Validator "Status" Query Script
Critical Metrics
Monitor these validator-specific metrics:Backup Management
Complete Automated Backup Script
Complete Automated Backup Script
Host System Monitoring
Resource Usage Tracking
Install and configure node_exporter:EVM RPC OpenTelemetry Metrics
The EVM RPC layer emits OpenTelemetry metrics through the process-wideMeterProvider (for example, a Prometheus exporter). These are emitted in parallel with the legacy sei_* metrics so you can migrate dashboards incrementally.
Available EVM RPC Metrics
The
evmrpc_request_latency_seconds histogram carries the following labels:
Migrating from Legacy EVM RPC Metrics
The following legacysei_* metrics remain available today but are deprecated and scheduled for removal once dashboards migrate to the evmrpc_* OpenTelemetry metrics:
Update your Prometheus and Grafana dashboards to consume the
evmrpc_* metrics before the legacy metrics are removed. Note that latency changed units from milliseconds (sei_rpc_request_latency_ms) to seconds (evmrpc_request_latency_seconds), so adjust any thresholds and panel formatting accordingly.
FlatKV OpenTelemetry Metrics
The FlatKV state store emits OpenTelemetry metrics through the process-wideMeterProvider (for example, a Prometheus exporter). These metrics let node operators observe commit throughput, catchup progress, snapshotting, rollbacks, and snapshot imports.
Available FlatKV Metrics
Labels
FlatKV metrics carry the following labels where applicable:Enabling Pebble Internal Metrics
Pebble’s internal (per-DB) metrics are governed by a single FlatKV-level knob:enable-pebble-metrics under [state-commit.flatkv] in app.toml (default true). The key is honored when present but is not part of the app.toml that seid init generates, so add it manually to change the default. Its value is propagated to every data DB (account, code, storage, legacy, and metadata) during initialization and overrides any per-DB EnableMetrics settings, so configure Pebble metrics through this knob rather than the individual per-DB settings.
LittDB OpenTelemetry Metrics
LittDB now emits its metrics through the process-wide OpenTelemetryMeterProvider instead of a private Prometheus client. When MetricsEnabled is set, LittDB configures a Prometheus exporter on the global provider and serves /metrics on MetricsPort (default 9101). The previous MetricsNamespace and MetricsRegistry config fields have been removed; all metric names now use a fixed litt_ prefix.
Available LittDB Metrics
Attributes
Migrating from Legacy LittDB Metrics
Metric names, units, and shape changed with the OpenTelemetry migration, so existing Prometheus and Grafana dashboards must be updated:- Latency metrics moved from millisecond summaries (for example
{namespace}_read_latency_ms) to second histograms (litt_read_latency_seconds). Adjust thresholds and panel formatting from milliseconds to seconds accordingly. - Counters and gauges gained a fixed
litt_prefix and explicit units, for examplebytes_readbecamelitt_bytes_readand the cache weight gauge becamelitt_chunk_cache_weight_bytes. - The per-cache series that were previously separate metric names (for example
chunk_read_cache_*andchunk_write_cache_*) are now the sharedlitt_chunk_cache_*metrics distinguished by thecacheattribute. - The
MetricsNamespaceandMetricsRegistryconfig fields no longer exist. Metric names are fixed, and metrics are always backed by the global OTel provider; supply the scrape port viaMetricsPort.
IBC OpenTelemetry Metrics
The IBC modules emit OpenTelemetry metrics through the process-wideMeterProvider (for example, a Prometheus exporter). These are emitted in parallel with the legacy ibc_* and tx_msg_* telemetry counters so you can migrate dashboards incrementally. The metrics are grouped by meter, one per IBC module.
Transfer Metrics (ibc_transfer_keeper meter)
The transfer gauges carry a
denom_class attribute. ibc_transfer_send carries destination_port, destination_channel, and a boolean source attribute indicating whether the sending chain is the token source. ibc_transfer_receive carries source_port, source_channel, and the boolean source attribute.
Core Client Metrics (ibc_core_client_keeper meter)
These metrics carry a
client_type attribute. ibc_client_update, ibc_client_upgrade, and ibc_client_misbehaviour also carry a client_id attribute. ibc_client_update additionally carries an update_type attribute (msg or proposal). ibc_client_misbehaviour also carries a msg_type attribute (value update) when the misbehaviour is detected during a client update.
Connection Metrics (ibc_connection meter)
Channel Metrics (ibc_channel meter)
Core Packet Metrics (ibc_core meter)
These packet metrics carry
source_port, source_channel, destination_port, and destination_channel attributes. ibc_core_timeout_packet additionally carries a timeout_type attribute (height or channel-closed).
Performance Testing
Example Benchmark Script using `eth_getLogs`
Example Benchmark Script using `eth_getLogs`