Pseudowire Drop on Customer EVC

Major

Resolved

Incident INC-000000693 • Created 2025-06-23 11:42 UTC

SLA DEADLINE

Met

ASSIGNED TO

Lisa Wong

CUSTOMERS AFFECTED

CORRELATED TICKETS

Network Topology

Operational

Degraded

Down

Fiber

Ethernet

MPLS

Affected Components

SYD PE

SYD

operational

BNE PE

BNE

operational

Primary Affected Component

Component Name

SYD-BNE Virtual Circuit

Type

L2VPN Service

Location

SYD-BNE Virtual Circuit

Current Status

operational

Incident Information

Description

Customer Ethernet Virtual Circuit (EVC) experienced pseudowire session drop between SYD and BNE endpoints, causing Layer 2 connectivity loss.

Root Cause Analysis

LDP session timeout due to control plane congestion during peak traffic hours. Pseudowire signaling packets were delayed beyond keepalive threshold.

Estimated Resolution

Resolved at 2025-06-23 12:30 UTC

Incident Analysis

System Logs

2025-06-19 13:46:12.345

ERROR

cor01-etp-454stpau-bne: BFD session removed on interface FourHundredGigE0/0/0/3

BFD neighbor 10.1.1.2 state changed from Up to Down

2025-06-19 13:46:15.123

WARN

monitoring-system: Interface FourHundredGigE0/0/0/3 operationally down

Link state change detected, triggering alarm

2025-06-19 13:46:18.567

ERROR

cor01-etp-47bourke-syd: Loss of Signal detected on optical interface

Optical power level dropped below threshold (-20.5 dBm)

2025-06-19 13:46:22.890

INFO

mpls-engine: TE tunnel Tunnel-te421 state changed to down

Tunnel rerouting initiated via backup path

2025-06-19 13:46:25.234

CRITICAL

service-monitor: Customer EVC services impacted

3 customer services affected by link failure

Resolution Steps

4/4 Complete

Verify pseudowire status and LDP session state

Assigned to: Lisa Wong

Completed: 2025-06-23 11:50 UTC

Analyze control plane traffic and congestion metrics

Assigned to: Lisa Wong

Completed: 2025-06-23 12:05 UTC

Adjust LDP keepalive timers and priority queuing

Assigned to: Lisa Wong

Completed: 2025-06-23 12:20 UTC

Re-establish pseudowire and verify EVC connectivity

Assigned to: Lisa Wong

Completed: 2025-06-23 12:30 UTC

Quick Actions

Emergency Contacts

NOC Manager

David Wilson

On-Call Engineer

Available 24/7

Impact Metrics

pseudowire Downtime48 minutes

ldp Session Flaps3 events

availability96.7%

error Rate0%