The Fundamental Problem: Three Siloed Realities #
In high-scale enterprise networks, each telemetry source provides only one dimension of an incident:
| Telemetry Stream | Standard Tools | Question It Answers | Example Signal |
|---|---|---|---|
| Metrics | OpenTelemetry, Prometheus, SNMP | WHAT is degrading? (Rates & Health) | P99 latency = 2.8s, interface drops = 12,450 pkts/s |
| Flows | Zeek (conn.log), IPFIX, NetFlow |
WHO is talking & affected? (Traffic & Behavior) | 10.140.2.12:54210 to 192.168.50.8:443, State = RSTO |
| Logs | Syslog, Traps, Control-Plane Logs | WHY did state change? (System Events) | %BGP-5-ADJCHANGE: neighbor 198.51.100.1 Down |
The Missing Link:
These systems have no shared primary key. A Prometheus metric knows an interface index (ifIndex: 14), a Zeek connection record knows an IP 5-tuple (10.140.2.12:54210 -> 192.168.50.8:443), and a Syslog daemon knows a network hostname (rtr-core-east-01).
Joining these disparate streams across millions of events per second requires a Dynamic Network Topology Graph.
The Glue: The Dynamic Topology Graph #
Static network configuration repositories (such as NetBox or CMDBs) are insufficient for runtime correlation because routing states mutate dynamically during link flaps, BGP recalculations, and failovers.
The platform maintains an in-memory graph (backed by RedisGraph or Neo4j) that tracks real-time state across Layer 2 (LLDP/CDP), Layer 3 (BGP/OSPF adjacencies), and interface IP bindings.
[Client Workload]
│
▼
[VPC Gateway: 10.140.2.1]
│
▼ (BGP Peer)
[rtr-edge-east-01] ──(Te1/0/1: Primary 10G)──► [Provider WAN: METRO-WAN-1042]
│
└──(Gi0/0/3: Backup 1G)──► [Provider WAN: BACKUP-WAN-0012]
│
▼
[Core Payment Gateway]Topology Entity Model in Redis #
Each network node and link is maintained as a queryable entity with short TTLs refreshed by polling workers and routing state listeners:
{
"node_id": "rtr-core-east-01",
"mgmt_ip": "198.51.100.24",
"interfaces": {
"14": {
"name": "GigabitEthernet0/0/3",
"speed_bps": 1000000000,
"circuit_id": "BACKUP-WAN-0012",
"egress_subnets": ["192.168.50.0/24"]
},
"2": {
"name": "TenGigabitEthernet1/0/1",
"speed_bps": 10000000000,
"circuit_id": "METRO-WAN-1042",
"egress_subnets": ["192.168.50.0/24"],
"status": "down"
}
}
}End-to-End Walkthrough: Resolving a Real Incident #
The Scenario #
The production Payment-Gateway-API experiences severe degradation. P99 latency jumps from 45ms to 2.8s, and 12% of customer requests terminate with HTTP 504 Gateway Timeout.
Here is how the automated platform correlates the three data streams to isolate the root cause:
[Step 1: Application / Prometheus Metric]
└─► Latency Spike Alert (P99 > 2.5s) on Service: "Payment-Gateway-API"
│
▼ (Query In-Memory Topology Graph for Active Path)
[Step 2: Flow Analysis (Zeek conn.log)]
└─► Identify 5-tuples traversing WAN via Circuit "METRO-WAN-1042"
└─► Anomaly Detected: TCP State = `RSTO` (Reset) & Severe Packet Retransmission
│
▼ (Map 5-Tuple to Active Egress Node & Interface)
[Step 3: Device Metrics (Streaming gNMI / SNMP)]
└─► Target: rtr-core-east-01, Interface "GigabitEthernet0/0/3"
└─► ifOutDiscards spiked to 12,450 pkts/sec; Capacity saturated at 98.5%
│
▼ (Query Log Engine for Target Node within [Event_Time ± 60s])
[Step 4: System Logs (Syslog RFC 5424)]
└─► 11:02:14 UTC: BGP-5-ADJCHANGE: neighbor 198.51.100.1 Down (Link Flap)
└─► 11:02:15 UTC: TRACKING-5-STATE: Route failover to Backup Circuit
│
▼
[Automated Root Cause Synthesis]
"BGP link flap on primary circuit (METRO-WAN-1042 / Te1/0/1) triggered failover
to backup 1G circuit (Gi0/0/3). Backup link saturated immediately, causing
12,450 drops/s and connection resets (RSTO) for Payment-Gateway-API."Step-by-Step Correlation Mechanics #
Step 1: Metric Ingestion Detects the Symptom (The “What”) #
The APM / Service Mesh metrics pipeline breaches an SLO alert:
{
"timestamp": "2026-09-14T11:02:30Z",
"metric": "http_request_duration_seconds",
"service": "Payment-Gateway-API",
"source_ip": "10.140.2.12",
"dest_ip": "192.168.50.8",
"p99_latency_ms": 2840
}Step 2: Flow Enrichment via Zeek (The “Who & Blast Radius”) #
The engine queries the normalized flow stream for traffic between 10.140.2.12 and 192.168.50.8 during that time slice:
{
"ts": "2026-09-14T11:02:32Z",
"uid": "CHz9qV2bJ49f8ZlK1a",
"id.orig_h": "10.140.2.12",
"id.orig_p": 54210,
"id.resp_h": "192.168.50.8",
"id.resp_p": 443,
"proto": "tcp",
"service": "ssl",
"duration": 4.12,
"orig_bytes": 1042,
"resp_bytes": 0,
"conn_state": "RSTO",
"history": "ShADadR",
"missed_bytes": 14200
}RSTO (connection reset by originator) paired with missed_bytes > 0 indicates severe transport-layer packet loss and TCP retransmissions along the transit path.
Step 3: Device & Interface Metrics via gNMI/SNMP (The “Where”) #
Using the destination subnet 192.168.50.0/24, the correlation engine evaluates the active egress path in the Topology Graph. It resolves the active physical path to router rtr-core-east-01 on egress interface GigabitEthernet0/0/3 (ifIndex: 14):
{
"timestamp": "2026-09-14T11:02:30Z",
"node": "rtr-core-east-01",
"ifIndex": 14,
"ifName": "GigabitEthernet0/0/3",
"ifOutOctets_rate_bps": 985000000,
"ifSpeed": 1000000000,
"ifOutDiscards_rate": 12450
}- Deduction: The backup interface is pinned at 98.5% capacity on a 1 Gbps link, discarding 12,450 packets/sec.
Step 4: Syslog Correlation Pinpoints the Cause (The “Why”) #
The engine executes a temporal slice query against Elasticsearch/OpenSearch for rtr-core-east-01 within a narrow 60s event window:
<189>2026-09-14T11:02:14.102Z rtr-core-east-01 BGP-5-ADJCHANGE: neighbor 198.51.100.1 (METRO-WAN-1042 10G) Down - Interface flap
<187>2026-09-14T11:02:15.010Z rtr-core-east-01 TRACKING-5-STATE: 10 ip route 0.0.0.0/0 Nexthop changed to 198.51.200.2 (Backup 1G)The sequence is established deterministically:
- Primary 10G link dropped.
- BGP withdrew the prefix and shifted traffic to the 1G backup circuit.
- The backup circuit overwhelmed its outbound buffers.
- Drops manifested as TCP timeouts and resets on the API tier.
Architectural Mechanics: How the Engine Runs in Real Time #
┌──────────────────────┐ ┌──────────────────────┐ ┌──────────────────────┐
│ Metrics (OTel/SNMP) │ │ Flows (Zeek/NetFlow)│ │ Logs (Syslog/Trap) │
└──────────┬───────────┘ └──────────┬───────────┘ └──────────┬───────────┘
│ │ │
▼ ▼ ▼
[Kafka Topic] [Kafka Topic] [Kafka Topic]
┌────────────────────────────────────────────────────────────────────────────┐
│ Stream Correlation Layer (Apache Flink / Rust Tokio) │
│ │
│ 1. Event-Time Alignment (Bounded Out-of-Order Watermarking) │
│ 2. Entity Resolution via In-Memory Topology Cache (Redis Graph) │
│ [IP 5-Tuple] ──► [Egress Interface] ──► [Node ID] ──► [Circuit ID] │
│ 3. Multi-Signal Coincidence Evaluation & Graph Traversal │
└─────────────────────────────────────┬──────────────────────────────────────┘
▼
[Consolidated Incident Object]
"Root Cause: Primary Circuit Flap Triggered Backup Link Drops"1. Watermarking and Event-Time Windowing #
Because logs, metrics, and flows travel through different ingestion paths, they arrive out of sequence. Correlating them requires an event-time sliding window (typically 30–60 seconds) with watermarking to handle bounded out-of-orderness:
$$ \text{Watermark} = \max(\text{EventTime}) - \Delta t_{\text{delay}} $$Stream processing engines (such as Apache Flink or dedicated async consumers) buffer incoming streams and evaluate joins only when the watermark passes the window boundary.
2. Path Traversal Algorithm #
When an alert fires on a service metric:
- Extract Endpoints: Parse source and destination IPs from the affected flow or APM trace.
- Trace Route Path: Query the topology cache for the active egress interface and transit hops between source and destination.
- Inspect Subgraph Telemetry: Traverse the resolved nodes and links within the incident window:
-
Inspect link egress utilization:
$$ \text{Utilization} = \frac{\Delta \text{ifHCOutOctets} \times 8}{\Delta t \times \text{ifSpeed}} > 0.90 $$ -
Check for queue discards during the window:
$$ \Delta \text{ifOutDiscards} > 0 \quad (\text{or rate} > 0\text{ pkts/sec}) $$ -
Search for state logs on traversed nodes:
event_type IN ('BGP_DOWN', 'INTERFACE_DOWN', 'OSPF_ADJCHANGE')
-
- Score & Synthesize: If an interface drop coincides with a link state change on the same transit hop, collapse all three signals into a single actionable incident record.
Production Realities & Edge Cases #
Handling NTP Drift across Autonomous Systems #
Routers managed by third-party NSPs often exhibit clock skew relative to internal infrastructure. If a router’s clock drifts by 15 seconds, strict timestamp-matching algorithms fail.
- Mitigation: Rely on
ingest_timestamp_ns(stamped by the edge collector upon packet arrival) for temporal windowing, usingdevice_timestamp_nsstrictly for relative delta computations.
Asymmetric Routing & Encapsulation #
In modern cloud architectures, the outbound path often differs from the return path (asymmetry), and packets may be wrapped in GENEVE, VXLAN, or IPsec tunnels.
- Mitigation: The flow normalizer must strip outer tunnel headers before generating flow hashes, ensuring that inner tenant 5-tuples match across physical underlay and virtual overlay devices.
What’s Next #
In Part 3, we will move from architecture to code: building a high-throughput Stream Normalizer in Rust using tokio and zero-copy binary parsing, benchmarking its throughput and memory footprint against a standard Python worker.