ACES
Systems Operational
Saturday, December 6, 2025 at 03:53:45

TE Tunnel Flap – SYD Core

Major
Resolved

Incident INC-000000691 • Created 2025-06-21 02:15 UTC

SLA DEADLINE
Met
ASSIGNED TO
Sarah Johnson
CUSTOMERS AFFECTED
2
CORRELATED TICKETS
1
Network Topology
10G10G100G40G40GSYD CoreSYD CoreSYD PE1SYDSYD PE2SYDBNE CoreBNE CoreMEL CoreMEL Core
Operational
Degraded
Down
Fiber
Ethernet
MPLS

Affected Components

SYD Core
SYD Core
operational
Primary Affected Component
Component Name
SYD Core Router
Type
Core Router
Location
SYD Core
Current Status
operational
Incident Information

Description

Traffic Engineering tunnel experiencing intermittent flapping on SYD core router, causing brief service interruptions and traffic rerouting events.

Root Cause Analysis

Memory leak in the MPLS forwarding engine causing periodic tunnel state changes. High memory utilization triggered automatic tunnel reconvergence.

Estimated Resolution

Resolved at 2025-06-21 03:05 UTC
Incident Analysis
System Logs
2025-06-19 13:46:12.345
ERROR
cor01-etp-454stpau-bne: BFD session removed on interface FourHundredGigE0/0/0/3
BFD neighbor 10.1.1.2 state changed from Up to Down
2025-06-19 13:46:15.123
WARN
monitoring-system: Interface FourHundredGigE0/0/0/3 operationally down
Link state change detected, triggering alarm
2025-06-19 13:46:18.567
ERROR
cor01-etp-47bourke-syd: Loss of Signal detected on optical interface
Optical power level dropped below threshold (-20.5 dBm)
2025-06-19 13:46:22.890
INFO
mpls-engine: TE tunnel Tunnel-te421 state changed to down
Tunnel rerouting initiated via backup path
2025-06-19 13:46:25.234
CRITICAL
service-monitor: Customer EVC services impacted
3 customer services affected by link failure
Resolution Steps
4/4 Complete

Identify tunnel flapping patterns and affected LSPs

Assigned to: Sarah Johnson

Completed: 2025-06-21 02:25 UTC

Analyze MPLS forwarding table and memory utilization

Assigned to: Sarah Johnson

Completed: 2025-06-21 02:40 UTC

Apply memory optimization patch to MPLS engine

Assigned to: Sarah Johnson

Completed: 2025-06-21 02:55 UTC

Monitor tunnel stability for 15 minutes

Assigned to: Sarah Johnson

Completed: 2025-06-21 03:05 UTC

Quick Actions
Emergency Contacts
NOC Manager
David Wilson
On-Call Engineer
Available 24/7
Impact Metrics
tunnel Flaps12 events
traffic Rerouting8 instances
availability94%
error Rate6.2%