SDC PHIL - Power Related Outage - 02/17/2026 1310MST

1310 MST - Presently we are working with engineers on a power direct issue affecting the entire SDC PHIL site.  Both primary and secondary are impacted.

1316 MST ‐ Local DC has confirmed an entire floor power outage affecting primary and secondary services.  Working with local team for support.

1338 MST - Local engineers are working with power techs to restore services they caused drop in.  ETA less than 1h.

03092026 - We have confirmed that the underlying power was subject to an A+B+C Power configuration.  All systems were to placed in a round robin going to each load bank so that no one leg carried singular load.  We found that the tech staff that performed the wiring load did so incorrectly and that the operation by the datacenter was not communicated to us.  When they worked on the power causing the outage the physical hardware that was loaded incorrectly cause a power hard failure.  This cause the stack to begin to fail.  As the critical point was reached the remaining SAS infrastructure began to self protect and power off.  

Working with the techs we escalated this to management as the 2nd time of install occurred in the same manner, we had to place staff onsite to ensure that the proper wiring configuration was done correctly.  This extended the outage and the time to recover.  This has since been resolved.  We will no longer be using the Datacenter white gloves service to install any new work loads.

This concludes all finding on this matter.

1458MST - Local engineers are working on restoring VM / routing presently with power restoration.

1536 MST - Local power has been restored and local service is confirmed up/up

1540 MST - Power and routes are restoring, you may experience a change in routes temporarily dropping traffic and then automatically restoring.

1605 MST - Power has caused issues with storage on partial access to vm.  We are working on the restoring of that impeded service now. 

1617 MST - Techs working on impacted storage nodes.  ETA is less than 1h.

1706 MST - All services restored.  Please confirm all your VM.  Should additional support be required.  Open a ticket directly for the continued matter.