Issue Description:
Between February 26–March 1, 2026, Degreed experienced a significant service disruption in the US production environment related to Degreed Search Services, a core platform component that supports many application features.
During the early phase of the incident, the instability of the search infrastructure made the application search and other areas of the application (specifically Admin Catalog, Content (Add Content flows, Add to Featured Page, Add to Plans, Add to Pathways)) difficult to use. As mitigation efforts progressed, our engineering teams stabilized the system by operating the search infrastructure in a read-only mode, which allowed user experiences to function again while preventing further instability. on March 1, 2026 Degreed Search Services were fully re-enabled and pending queued data was processed, no data was lost. Daily and Weekly emails remain off to ensure stability while engineering team works to safely re-enable these emails. Other system emails are fully functional.
Customer Impact:
Affected Customers:
Organizations using the US Degreed production environment.
Impact Duration:
February 26–March 1, 2026 for application
February 26–continuing for daily and weekly emails. We expect to partially or fully restore all emails by Mar 20, 2026
Symptoms Experienced:
For roughly the first day of the incident, search and search related functions were difficult to use
Slow responses, timeouts, or errors across search and search related functions
After read-only mitigation steps were implemented, search and search related functions were stable again, though some functionality was impacted by read-only mode
Daily and weekly emails were not sent
Root Cause:
The issue originated from resource pressure and instability within Degreed Search Services, which handle indexing and retrieval of information used across the platform.
A background job responsible for generating daily and weekly email communications created higher-than-expected demand on the search infrastructure while the system was already operating near capacity. This combination caused the search services to become unstable under load.
Resolution:
Our engineering teams implemented several mitigation and stabilization actions:
Infrastructure stabilization: Upgraded and optimized the underlying search infrastructure to improve stability and capacity. We are continuing to invest in further improvements specifically for the search infrastructure in the coming days and weeks.
Read-only operating mode: Temporarily placed search services in read-only mode, allowing many core product experiences to continue functioning while preventing additional system strain.
Load reduction: Disabled the background processes responsible for sending daily and weekly email communications to reduce demand on the system while recovery actions were implemented.
These actions restored stability to the platform and allowed all other application functionality to resume.
Current Status:
Platform Stability: Degreed Search Services are currently stable and operating normally.
Application Functionality: All product experiences except daily and weekly emails are functioning as expected.
Email Communications: Daily and weekly email processing remains temporarily disabled while we complete additional system improvements. All other email functionality is operating normally. We expect to partially or fully restore daily and weekly emails by Mar 20, 2026
Our teams continue to monitor the system closely to ensure ongoing stability. We are also actively making further improvements to all mentioned systems.
What This Means:
Customers should now experience:
Restored access to all Degreed application functionality except daily and weekly emails
Stable platform performance and responsiveness
Continued availability of learning and content experiences
No data was lost
You may notice:
A temporary reduction in daily and weekly email communications while we complete restoration work
No action is required from customers.
Monitoring and Prevention:
To reduce the likelihood of similar incidents in the future, we have implemented and are continuing to implement the following improvements:
Expanded capacity and scaling thresholds for Degreed Search Services
Improved workload management for large background processing jobs. These are usually concurrent with sending daily and weekly emails
Additional monitoring and alerting for early detection of infrastructure stress
Updated operational procedures to allow faster mitigation if similar conditions arise
Many members of our engineering and platform teams are fully focused on restoring daily and weekly email processing as quickly and safely as possible. As restoration progresses, customers may see a temporarily lower volume of emails while we gradually return the system to full operation.