Architecting a Multi-NSP Telemetry Ingestion Pipeline at Enterprise Scale

Table of Contents

Network Observability Architecture - This article is part of a series.

Part 1: This Article

Building a unified observability plane across heterogeneous Network Service Providers (NSPs), on-prem WAN backbones, and hybrid cloud fabrics is one of the hardest data engineering challenges in enterprise infrastructure. This article provides an architect-level blueprint for designing a resilient, wire-speed telemetry ingestion pipeline.

The Core Challenge: The Heterogeneity Problem
#

Modern enterprise transport is fragmented across proprietary and open interfaces:

Cisco (IOS-XR / XE): Dial-out telemetry with vendor-specific YANG modules (Cisco-IOS-XR-*-oper) over gRPC, mixed with NetFlow v9.
Juniper (Junos): Junos Telemetry Interface (JTI) pushing OpenConfig YANG models via gNMI, alongside IPFIX.
Nokia (SR OS): Streaming telemetry via gRPC/dial-out combined with legacy SNMP traps for legacy interfaces.
Hyperscalers (AWS / Azure / Megaport): Cloud-native APIs, asynchronous event logs (AWS VPC Flow Logs via S3/SQS, Azure Network Watcher), and software-defined interconnection portals.

Without a strictly layered ingestion architecture, downstream analytics engines end up tightly coupled to vendor-specific transport semantics, resulting in pipeline fragility and runaway storage costs.

Target System Architecture
#

The blueprint decouples ingestion into four fault-isolated tiers: Collection & Protocol Adapters, Durable Buffering, Stream Normalization, and Polyglot Storage.

flowchart TB
    subgraph Sources["Heterogeneous Telemetry Sources"]
        direction LR
        S1["Cisco IOS-XR
gNMI / NetFlow v9"]
        S2["Juniper Junos
JTI / IPFIX"]
        S3["Nokia SR OS
gRPC / SNMP"]
        S4["Cloud / SD-WAN
VPC Flow / Equinix APIs"]
    end

    subgraph Tier1["Tier 1: Edge & Protocol Adapter Tier (Distributed POPs)"]
        direction TB
        A1["gNMI / gRPC Collectors
(Dial-out TLS, HTTP/2)"]
        A2["UDP Flow Receivers
(SO_REUSEPORT, Ring Buffers)"]
        A3["Async SNMP & API Workers
(PySNMP / SQS Consumers)"]
        ENV["Metadata Envelope Injection
(NSP_ID, Region, Ingest_TS, Device_ID)"]
        A1 --> ENV
        A2 --> ENV
        A3 --> ENV
    end

    subgraph Tier2["Tier 2: Event Spine (Kafka Cluster)"]
        direction TB
        K1["Topic: telemetry.raw.streaming
(Key: device_id)"]
        K2["Topic: telemetry.raw.flows
(Key: flow_hash)"]
        K3["Topic: telemetry.raw.snmp
(Key: device_id)"]
    end

    subgraph Tier3["Tier 3: Normalization & Context Enrichment (Flink / Rust)"]
        direction TB
        N1["Schema Normalizer
(YANG -> OTel, Flow -> Zeek)"]
        N2["Topology Enricher
(Redis Cache: Circuit ID, SLA)"]
        DLQ["Dead Letter Queue (DLQ)
(telemetry.dlq.malformed)"]
        N1 --> N2
        N1 -. Malformed .-> DLQ
    end

    subgraph Tier4["Tier 4: Polyglot Storage & Analytics"]
        direction LR
        D1[("Metrics TSDB
Timescale / VictoriaMetrics")]
        D2[("Flow & Log Store
ClickHouse / OpenSearch")]
        D3["AIOps / Anomaly Engine
Graph Root-Cause Agents"]
    end

    Sources --> Tier1
    ENV -->|mTLS / acks=all| Tier2
    Tier2 --> Tier3
    N2 --> D1
    N2 --> D2
    N2 --> D3

Detailed Architectural Walkthrough
#

1. Ingestion & Protocol Adapter Tier
#

Telemetry cannot follow a unified collection model because network protocols operate on fundamentally different transport mechanics:

A. Push-Based Streaming (gNMI / gRPC)
#

Devices act as clients initiating outbound connections (dial-out telemetry) to edge collectors behind layer-4 load balancers.
This eliminates the need to open inbound firewall ports into customer edge (CE) or provider edge (PE) devices.
Collectors are stateless daemons built on Rust (tokio) or Go (grpc-go) maintaining persistent HTTP/2 multiplexed streams.

B. UDP Flow Receivers (NetFlow v9 / IPFIX)
#

UDP drops packets silently during burst traffic when the OS socket buffer overflows. To guarantee zero loss during microbursts, apply Linux kernel tuning on the collector nodes:

# Increase maximum socket receive buffer size to 64MB
sysctl -w net.core.rmem_max=67108864
sysctl -w net.core.rmem_default=33554432

# Increase backpressure queue before socket dispatch
sysctl -w net.core.netdev_max_backlog=10000

Production Warning: Multi-threaded flow listeners must bind using SO_REUSEPORT across multiple CPU cores so the Linux kernel distributes UDP packet processing across RX rings, preventing a single thread from pinning a core.

C. Asynchronous Pull Schedulers (SNMP / Cloud APIs)
#

Legacy interfaces still require polling. Rather than running a monolithic poller, use an event-driven distributed scheduler.
Schedulers push polling tickets into an internal task queue. Stateless worker pools execute non-blocking SNMP queries using asynchronous engines.
Cloud NSP metrics (AWS VPC Flow from S3/SQS, Azure Event Hub) trigger serverless stream extractors pushing into Kafka.

2. The Universal Transport Envelope
#

To insulate the downstream stream processing layer from raw transport quirks, edge collectors wrap every raw payload in a deterministic Common Transport Envelope:

{
  "envelope_version": "1.1",
  "nsp_id": "equinix-fabric-eu",
  "source_protocol": "gnmi_openconfig",
  "source_ip": "198.51.100.24",
  "device_id": "zurich-cr01.prod.net",
  "region": "eu-central-1",
  "edge_ingest_ts_ns": 1726330800123456789,
  "payload_encoding": "protobuf",
  "schema_identifier": "openconfig-interfaces:2.0.0",
  "raw_payload": "Cg4yMDI2LTA5LTE0...<base64_or_binary>"
}

Why Separate Timestamps?
Never overwrite device time with ingestion time. Always preserve both device_ts (for sequence integrity and jitter analysis) and edge_ingest_ts_ns (for SLA pipeline latency tracking and watermarking).

3. Buffering & Partitioning Strategy (Kafka)
#

The message spine uses Apache Kafka or Apache Pulsar to decouple fast ingestion from compute-intensive normalization.

Topic Name	Partition Key	Guarantees
`telemetry.raw.streaming`	`device_id`	Preserves strict chronological state changes per router.
`telemetry.raw.flows`	`flow_hash` (`src_ip + dst_ip + port`)	Distributes high-throughput flow records evenly across partitions.
`telemetry.raw.snmp`	`device_id`	Prevents out-of-order counter wraparound evaluation.

4. Transformation & Normalization Tier
#

The stream normalization engine (Apache Flink or Rust-based stream workers) converts vendor dialects into industry standards:

[Vendor-Specific Dialect]           [Stream Engine]             [Standard Target]
Cisco native YANG / Protobuf ──┐
Juniper JTI OpenConfig       ──┼─► [Schema Transformer] ───► [OpenTelemetry Metrics]
Nokia SR OS Streaming        ──┘

Cisco NetFlow v9             ──┐
Juniper IPFIX                ──┼─► [Flow Normalizer]    ───► [Zeek JSON / conn.log]
AWS VPC Flow JSON            ──┘

Metrics Normalization (YANG to OpenTelemetry)
#

Problem: Cisco models interface discards as Cisco-IOS-XR-infra-statsd-oper:infra-statistics/interfaces/interface/latest/generic-counters/input-drops. Juniper models the same as openconfig-interfaces:interfaces/interface/state/counters/in-discards.
Solution: The normalization layer strips vendor semantics and translates metrics to OpenTelemetry Semantic Conventions:
- Metric: network.interface.discards.rx
- Attributes: { "device.name": "zurich-cr01", "interface.name": "TenGigE0/0/0/1" }

Flow Normalization (IPFIX / NetFlow to Zeek Format)
#

Raw binary flow records are normalized into Zeek connection schemas (conn.log), synthesizing missing connection state metrics:

Connection UID generation (uid = sha256(src_ip, dst_ip, src_port, dst_port, proto, epoch))
State flag evaluation (SF, RSTO, S0) to support security correlation downstream.

Dynamic Topology Enrichment (Redis In-Memory Fabric)
#

Raw device telemetry lacks business context. Stream processors query an in-memory Redis cluster populated by continuous network discovery:

Telemetry Record + Source IP + ifIndex 
              │
              ▼
   [Redis Topology Cache]
              │
              ├── Lookup: Interface Alias, Circuit ID, Peer ASN
              └── Lookup: Business SLA Tier ("Gold-Tier-WAN")
              │
              ▼
Enriched Stream Record (Forwarded to Storage)

Preventing ifIndex Drift: Network reboots can reorder SNMP ifIndex values. The enrichment tier maps lookups against unique tuples (device_id + ifName / ifPhysAddress) rather than ephemeral ifIndex numbers.

Concrete Transformation Pipeline
#

Here is how three distinct inputs converge into unified outputs:

Phase	Scenario A: Cisco Core (Push gNMI)	Scenario B: Nokia Edge (Pull SNMP)	Scenario C: AWS Transit Gateway (Cloud Log)
Ingress Format	Binary Protobuf over gRPC (Cisco YANG)	Polled OID `.1.3.6.1.2.1.2.2.1.14` (`ifInErrors`)	JSON flow logs delivered to S3 via SQS
Collector Tier	Ingress Daemon decrypts TLS, wraps in envelope	Async Worker unpacks PDU, wraps in envelope	Worker drains SQS event, fetches S3 batch
Kafka Topic	`telemetry.raw.streaming`	`telemetry.raw.snmp`	`telemetry.raw.flows`
Normalization	Protobuf decode $\to$ maps `in-octets` $\to$ OTel	Extract integer counter $\to$ delta rate calculation	Parse JSON 5-tuple $\to$ Zeek `conn.log` format
Final Payload	`metric: "network.interface.octets.rx", val: 4920194`	`metric: "network.interface.errors.rx", val: 0`	`{"uid": "C9xK...", "id.orig_h": "10.0.1.5", "conn_state": "SF"}`

Production Resiliency Patterns
#

1. Handling Malformed Telemetry via Dead Letter Queues (DLQ)
#

When an NSP applies a firmware upgrade, Protobuf schema definitions may change without upstream notice. Instead of letting consumers crash:

Deserialization errors are caught in an explicit try/catch block.
The complete raw envelope and stack trace are diverted to telemetry.dlq.deserialization_failures.
High DLQ ingress rates trigger PagerDuty alerts, giving engineers time to update the Schema Registry while raw data remains durable.

2. Edge Pre-Aggregation (WAN Cost Optimization)
#

Streaming full NetFlow/IPFIX payloads over international transit links creates prohibitive data-transfer bills:

Deploy regional Edge Aggregators at local Points of Presence (POPs).
Summarize raw packet counters into 10-second rolling window rollups locally.
Forward only the compressed, summarized records across long-haul WAN to the central Kafka cluster over mTLS.

3. Combating High-Cardinality TSDB Explosions
#

A single router with 500 sub-interfaces sending 20 counters every 10 seconds can overwhelm a time-series database.

Rule of Thumb: Never store raw ephemeral 5-tuples (src_ip:src_port -> dst_ip:dst_port) inside TSDB labels.
Route metrics (gauge, counter, rate) to columnar or time-series engines (TimescaleDB, VictoriaMetrics).
Route unbounded 5-tuple transaction records directly to columnar search engines (ClickHouse, OpenSearch).

What’s Next in this Series
#

In Part 2, we will implement the Edge Collection Layer in Rust, writing a zero-allocation UDP IPFIX receiver capable of processing 1,000,000 packets per second on a single node without packet loss.

Network Observability Architecture - This article is part of a series.

Part 1: This Article

The Core Challenge: The Heterogeneity Problem #

Target System Architecture #

Detailed Architectural Walkthrough #

1. Ingestion & Protocol Adapter Tier #

A. Push-Based Streaming (gNMI / gRPC) #

B. UDP Flow Receivers (NetFlow v9 / IPFIX) #

C. Asynchronous Pull Schedulers (SNMP / Cloud APIs) #

2. The Universal Transport Envelope #

3. Buffering & Partitioning Strategy (Kafka) #

4. Transformation & Normalization Tier #

Metrics Normalization (YANG to OpenTelemetry) #

Flow Normalization (IPFIX / NetFlow to Zeek Format) #

Dynamic Topology Enrichment (Redis In-Memory Fabric) #

Concrete Transformation Pipeline #

Production Resiliency Patterns #

1. Handling Malformed Telemetry via Dead Letter Queues (DLQ) #

2. Edge Pre-Aggregation (WAN Cost Optimization) #

3. Combating High-Cardinality TSDB Explosions #

What’s Next in this Series #