Microsoft Outage

Based on the latest information from Microsoft and our partners, ITS believes that this incident has been resolved.

If you are still experiencing problems related to Microsoft services, please contact the ITS Help Desk at https://support.clarku.edu, 508-793-7745, helpdesk@clarku.edu, or visit us on the first floor of the Academic Commons. Thank you.

Microsoft has reported there are still messages sent during the outage (11/12 @ 1pm - 11/13 @ 7am EST) being delivered, though it appears that at approximately 7am this morning a significant number of messages were delivered to Clark users, there still may be more to come. Messages sent after the outage are being delivered within an expected timeframe.

The status of this issue will remain as recovering until Microsoft has confirmed the issue is resolved. If you are still experiencing problems related to Microsoft services, please contact the ITS Help Desk at https://support.clarku.edu, 508-793-7745, helpdesk@clarku.edu, or visit us on the first floor of the Academic Commons.

We will provide another update by 8pm EST on November 14.

There are still a very limited number of integrations that are not fully functional, but the majority of impacted services have been restored. ITS is working to resolve any outstanding issues.

If you are still experiencing problems related to Microsoft services, please contact the ITS Help Desk at https://support.clarku.edu, 508-793-7745, helpdesk@clarku.edu, or visit us on the first floor of the Academic Commons. We will provide another update by 4pm EST.

The majority of access to Microsoft services has been restored and it appears that a significant backlog of mail has been delivered. There are still some limited integrations in specific pockets of campus related to Microsoft services that have not fully restored. ITS is currently working on these integrations.

If you are still experiencing problems related to Microsoft services, please contact the ITS Help Desk at https://support.clarku.edu, 508-793-7745, helpdesk@clarku.edu, or visit us on the first floor of the Academic Commons. We will provide another update by 12pm EST.

We are seeing increased signs of recovery. It appears that services are accessible again and a priority is being placed on new messages being sent/received as the backlog of queued email is being processed.

We expect that messages sent during the outage will be delivered.

If you are still experiencing problems accessing impacted services, please contact the ITS Help Desk at https://support.clarku.edu, 508-793-7745 after 8am, or visit them on the first floor of the Academic Commons. We will provide another update by 10am EST.

Microsoft is reporting progress overnight toward recovery, but we have seen no signs of recovery yet for Clark users.

Exchange Online, the service that supports email, calendaring, contacts, notes, and tasks, is the primary service impacted. Additional Microsoft services that integrate with Exchange Online, such as the calendaring parts of Teams, web access to OneDrive, certain features of Office applications, and others are impacted. The full list of impacted services is available at https://status.cloud.microsoft/.

Other services, such as Canvas, Banner, CUWeb, Insight, Zoom, SmartBuyPlus, and other non-Microsoft applications are available and have not been impacted by this outage.

ITS will continue to monitor this situation and provides updates here. The next update will be by 8:30am EST.

Microsoft continues to work on this significant, international outage. The most current information can be found directly from Microsoft at https://status.cloud.microsoft/.

Email remains inaccessible. Certain functions in Office, Copilot, SharePoint, and Teams may also be unavailable, but core messaging within Teams is functional. Other services, such as Canvas, Banner, CUWeb, Insight, Zoom, SmartBuyPlus, and other non-Microsoft applications are available and have not been impacted by this outage.

We will provide another update by 6:30am EST on November 13.

Microsoft is currently rolling back changes to try and restore impacted services. Email remains inaccessible. Certain functions in Office, Copilot, SharePoint, and Teams may also be unavailable, but core messaging within Teams is functional. A more detailed list of service impacts can be found at https://status.cloud.microsoft/. Other services, such as Canvas, Banner, CUWeb, Insight, SmartBuyPlus, and other non-Microsoft applications are available and have not been impacted by this outage.

This incident will be updated again by 10:30pm EST.

We have not seen evidence of service restoration for Clark users yet. We will continue to monitor this situation and expect our next update at 8:30pm EST.

Microsoft has provided the update below from https://status.cloud.microsoft/. We expect the next update before 6:30pm EST. ITS will continue to monitor the situation. Canvas, Banner, CUWeb, Insight, SmartyBuyPlus, and others are not impacted by this outage.

We’re continuing to pursue all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief. Our investigation has found that a core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding and contributes to the capacity loss.

To validate a potential recovery approach, we're fully isolating a portion of our service infrastructure, restarting the affected components, and gradually reintroducing traffic to confirm our services remain stable before applying this approach more broadly. We're also analyzing traffic patterns across our entire service infrastructure and evaluating whether isolating traffic within each region may allow affected regions to recover independently.

Microsoft has provided the update below from https://status.cloud.microsoft/. We expect the next update before 5:30pm EST. ITS will continue to monitor the situation. Canvas, Banner, CUWeb, Insight, SmartyBuyPlus, and others are not impacted by this outage.

We've validated an alternative health check method on a subset of affected infrastructure, which allowed the servers in that subset to successfully return to service, and we're now applying this change more broadly. We're also preparing an automated recovery process to continually detect and restart unresponsive servers, to help keep them in service while this change progresses. Additionally, our review of recent changes has identified a small number of configuration updates that we're investigating as potential contributors to the issue.

Microsoft has provided the update below from https://status.cloud.microsoft/. We expect the next update before 5pm EST. Canvas, Banner, CUWeb, Insight, SmartyBuyPlus, and others are not impacted by this outage.

We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've determined that the elevated load isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure. We're investigating what's causing these servers to become unavailable while we work to safely return them to service.

The latest update from Microsoft is shared below and is availble at https://status.cloud.microsoft/. Users may have intermittent access to email, Teams, OneDrive, and SharePoint. Services such at Canvas and CUWeb are still available. The next update from Microsoft is expected by 3:30pm EST.

Title: Users may be unable to connect to multiple Microsoft 365 services

User impact: Users may be unable to connect to multiple Microsoft 365 services.

More info: Potentially impacted services include:

• Exchange Online
• Microsoft Teams
• SharePoint Online
• Microsoft Copilot 365
• Microsoft Defender XDR

Current status: We’re investigating an issue impacting multiple Microsoft 365 services and we've determined the affected infrastructure is experiencing elevated load across multiple regions, which is likely contributing to the connectivity errors. We're restarting and repairing impacted portions of this infrastructure to restore capacity, and reviewing an unexpected increase in a specific type of service traffic to determine whether it's contributing to the elevated load. In parallel, we're evaluating options to redirect traffic to alternate infrastructure to expedite relief.

We have confirmed with Microsoft that there is an outage imapcting email services including mail delivery, Outlook, mobile devices, and outlook.com. We are monitoring the incident and will provide updates here as we have them.

We have received reports and can replicate problems access email and other Microsoft services. We are working to understand the scope and will update this notice as we learn more.