SDC VEG and SDC VEGN - Outage - 12/10/2025
Currently we are investigating a segment of the Las Vegas Metro networks that are causing a cascade outage. We know that it is impacting hypervisor related activity, no other segment has been impacted. SEIM, NETWORK, STORAGE and EDGE are all showing as up and functioning as normal. We will continue updates as additional information is made available.
1726 MST - Tech Sent advisement alert to Senior Techs and Management
1730 MST - Tech onsite requested additional support.
1750 MST - Secondary support has arrived.
1753 MST - Ticket moved to top priority and moved to 30m updates.
1741 MST - Escalation to Senior techs and Senior Management
1803 MST - Escalation was a success. All services have been restored. We are currently working on the root cause and failover features.
1807 MST - We are hard booting / replacing core switch 1 and core switch 2. It appears they were passing activity but stalling it to the uplink aggregate ports.
1809 MST - Additional support is working on PMX and VMW are performing checks now.
1922 MST - Discovery shows Core switches were originally believed at fault. This is not the case. The distribution switches had a cascading failure. This caused an unknown error to fail to the secondary. Upon fail over the distribution switch believed that there no errors and failed back. This cause a partial outage to the backbone between Veg and VegN. This was stabilize with manual intervention.
1934 MST - KVM/PMX is experiencing some minor pooling issues do to the load of restored VMs occuring. This is being worked on including manual pool management.
2002 MST - KVM/PMX Bridge restored. All capacity running normal.
2109 MST - Verification of data completed. SDC VEG
2109 MST - Verification of data completed. SDC VEGN
2313 MST - Verification follow up completed.
0419 MST - Verification follow up completed.
0624 MST - Ticket placed for observation.
0723 MST - RCA will be followed up.
0726 MST - Presently no RCA. The lite cause is a switch that behaved in a failed state that returned itself to production on the local edge side and required manual intervention for restoration of service.
0903 MST - Switch replacement plan for hot swap in are being prepared. This will not occur until afterhours and on the weekend. Target date will be the 14th.
1139 MST - Collapse switch will occur at Midnight and prep for tomorrow replacement. 2x Additional port groups will carry traffic during replacement to avoid a scheduled outage. The 2nd one will take place tomorrow evening at Midnight.