ACES
Systems Operational
Saturday, December 6, 2025 at 03:52:35

400G DWDM Link Failure – SYD ⇌ BNE

Critical
Resolved

Incident INC-000000690 • Created 2025-06-19 13:40 UTC

SLA DEADLINE
Met
ASSIGNED TO
John Smith
CUSTOMERS AFFECTED
3
CORRELATED TICKETS
2
Network Topology
100G400G400G400G100GSYD CoreEquinix SY3SYD EdgeEquinix SY3DWDM SYDEquinix SY3DWDM BNENextDC B2BNE CoreNextDC B2BNE EdgeNextDC B2
Operational
Degraded
Down
Fiber
Ethernet
MPLS

Affected Components

DWDM SYD
Equinix SY3
operational
DWDM BNE
NextDC B2
operational
Primary Affected Component
Component Name
400G Wavelength (Unprotected)
Type
DWDM Transport
Location
Equinix SY3 – NextDC B2
Current Status
operational
Incident Information

Description

Two ETP devices were placed at SYD and BNE, connected via a 3rd-party (Telstra) 400G DWDM link. The link was affected due to an internal fiber failure in the 3rd-party wavelength provider's network, which caused statically mapped services (EVC) carried over 400G to be impacted. The link was unprotected, and the underlay logical connectivity (TE-tunnel) was also affected. Automated monitoring detected complete Loss of Signal (LOS) across all lanes on the 400G unprotected link between SYD (Equinix SY3) and BNE (NextDC B2). Statically mapped services and TE tunnels failed, impacting customer EVCs.

Root Cause Analysis

The incident was caused by an internal fiber failure in the Telstra (3rd-party) DWDM network. This resulted in a Loss of Signal (LOS) on the 400G link between BNE and SYD. BFD sessions and interfaces went down, followed by TE tunnel failures. The link was unprotected, so all services over it were affected. Because the link was unprotected, all services relying on it were immediately impacted. BFD sessions and IGP adjacencies also dropped, confirming a cascading logical failure.

Estimated Resolution

Resolved at 2025-06-20 12:29 UTC
Incident Analysis
System Logs
2025-06-19 01:46:53.000
ERROR
cor01-etp-454stpau-bne: BFD session to neighbor 172.21.6.48 on interface FourHundredGigE0/0/0/3 removed
BFD neighbor 172.21.6.48 state changed from Up to Down
2025-06-19 01:46:54.000
ERROR
bdr03-ipt-20wharfs-bne: ISIS adjacency to bdr03-ipt-47bourke-syd.au (Bundle-Ether41.421) Down, BFD session DOWN
BFD session to neighbor fe80::9e09:8bff:fe03:df10 on Bundle-Ether41.421 down (Control timer expired)
2025-06-19 01:46:58.000
CRITICAL
cor01-etp-454stpau-bne: Multiple OPTICS RX LOS (Loss of Signal) alarms on Optics0/0/0/3 (all lanes)
Optical power level dropped below threshold on all wavelength lanes
2025-06-19 01:47:58.000
ERROR
bdr01-etp-639garde-syd: Pseudowire 504973 (xc_504973:p2p_504973) changed: up → down
Customer EVC pseudowire session terminated due to tunnel failure
2025-06-19 01:47:59.000
ERROR
bdr02-etp-639garde-syd: Pseudowire 594950 (xc_594950:p2p_504950) changed: up → down
Customer EVC pseudowire session terminated due to tunnel failure
2025-06-19 01:51:43.000
ERROR
cor01-etp-454stpau-bne: Interface Down FourHundredGigE0/0/0/3 (Checks failed, FAIL)
Interface operationally down due to Loss of Signal
2025-06-19 01:52:07.000
ERROR
cor01-etp-47bourke-syd: Interface Down FourHundredGigE0/0/0/3 (Checks failed, FAIL)
Interface operationally down due to Loss of Signal
2025-06-19 01:52:10.000
WARN
cor01-etp-47bourke-syd: TE Tunnel down Tunnel-te421 (Checks failed but alert delayed, FAIL_DELAYED)
Traffic Engineering tunnel state changed to down due to interface failure
Resolution Steps
4/4 Complete

Detected LOS and BFD session removals at BNE device (cor01-etp-454stpau-bne)

Assigned to: John Smith

Completed: 2025-06-19 13:46 UTC

Observed Interface and Tunnel Down at SYD device (cor01-etp-47bourke-syd)

Assigned to: John Smith

Completed: 2025-06-19 13:51 UTC

Pseudowire and IGP adjacency drop confirmed on related paths

Assigned to: John Smith

Completed: 2025-06-19 13:52 UTC

Coordinated with Telstra to restore fiber; confirmed full recovery and cleared alerts

Assigned to: John Smith

Completed: 2025-06-20 12:29 UTC

Quick Actions
Emergency Contacts
NOC Manager
David Wilson
On-Call Engineer
Available 24/7
Impact Metrics
packet Loss100%
latency IncreaseN/A (link unreachable)
throughput0 Gbps
error RateVery High
affected Tunnels4 TE Tunnels
affected Pseudowires2 Customer EVCs
impact Duration22h 43m
service Availability0%