Summary
On 26 February from around 8 AM UTC, we received reports of the site being slow for users in US Production, mainly when searching and working with content.
Some areas of the product that may be impacted:
1. Discover Page
2. Admin Catalog
Content
3. Add Content Flows
Add to Featured Page
Add to Plans
Add to Pathways
4. Global search (Top navigation)
Current status
We’re pleased to share that Degreed Search services have been fully restored, and Search functionality is now operating at normal performance levels. Our engineering teams will continue to actively monitor service health to ensure ongoing stability.
Also, all remaining integration data that was queued in our database has now been successfully synced therefore all content, user, and learning information is fully up to date.
This will be our final message on this issue, and we sincerely appreciate your continued patience and understanding throughout the resolution process. For any related questions, please reach out to our Support team or your Account Director.
A full root cause analysis will be shared in the coming days. Thank you again for your patience and partnership.
Updates
March 1
02:25 UTC - Resolved - Search Performance and Integration Data Sync
We’re pleased to share that Degreed Search services have been fully restored, and Search functionality is now operating at normal performance levels. Our engineering teams will continue to actively monitor service health to ensure ongoing stability.
Also, all remaining integration data that was queued in our database has now been successfully synced therefore all content, user, and learning information is fully up to date.
This will be our final message on this issue, and we sincerely appreciate your continued patience and understanding throughout the resolution process. For any related questions, please reach out to our Support team or your Account Director.
A full root cause analysis will be shared in the coming days. Thank you again for your patience and partnership.
February 28
21:09 UTC - Update: Search Performance and Data Sync Progress
We continue restoring full performance of the Degreed Search services. Search functionality is currently operating, though performance is still not to the level we expect. Our engineering teams continue to actively monitor service health and address remaining elements that are performing below standard.
We’re also continuing to monitor and sync Search data from Wednesday, February 25. Once Search services are fully stable, we are still on track for the remaining integration data, in our database, to complete syncing by tomorrow evening (U.S. time).
We truly appreciate your continued patience and understanding as we work through these final steps. We’ll share another update later tonight (U.S. time) to keep you informed of our progress.
Thank you again for your understanding as we actively progress to full restoration.
00:13 UTC - Our teams continue to make progress addressing the underlying symptoms that impacted Degreed’s Search services. We’re targeting restoration of Search services (specifically Admin Catalog, Content (Add Content flows, Add to Featured Page, Add to Plans, Add to Pathways)), and updates to a user's Profile by tomorrow morning US time at which time performance for other parts of Degreed will improve as a result.
The integrations that update users, content, required learning, completions, and skills have successfully run, and all related data is in our database. Once Search services are fully operational, we’ll spend the weekend syncing this data so it's available to our Search services. We expect this sync to complete by Sunday evening US time.
We understand how important these capabilities are for your teams, and we truly appreciate your patience and partnership as we restore Search services. Once fully resolved, a full root cause analysis will be performed and shared with you. We’ll continue to keep you informed as we cross key milestones and share another update tomorrow morning US time.
February 27
11:31 UTC - Search results are now appearing more consistently. We are continuing to make infrastructure enhancements to ensure ongoing reliability.
12:13 AM UTC - Our teams continue to make progress addressing the underlying symptoms that impacted Degreed’s Search services. We’re targeting restoration of Search services (specifically Admin Catalog, Content (Add Content flows, Add to Featured Page, Add to Plans, Add to Pathways)), and updates to a user's Profile by tomorrow morning US time at which time performance for other parts of Degreed will improve as a result.
The integrations that update users, content, required learning, completions, and skills have successfully run, and all related data is in our database. Once Search services are fully operational, we’ll spend the weekend syncing this data so it's available to our Search services. We expect this sync to complete by Sunday evening US time.
We understand how important these capabilities are for your teams, and we truly appreciate your patience and partnership as we restore Search services. Once fully resolved, a full root cause analysis will be performed and shared with you. We’ll continue to keep you informed as we cross key milestones and share another update tomorrow morning US time.
February 26
11:09 AM UTC - The team is actively working to mitigate the ongoing issue, with investigation still in progress.
1:49 PM UTC - Our engineering team is actively working to remediate the issue across different parts of our search infrastructure. Some customers may continue to experience difficulties retrieving search results while this work is in progress.
2:28 PM UTC - We are seeing improvement in search performance. Our engineering team will continue mitigation efforts and monitor the systems closely to ensure full stability is restored.
4:42 PM UTC - Our engineering team continues to actively investigate and work toward resolving the intermittent search issues. Mitigation efforts remain underway and we are making progress. Thank you for your continued patience.
5:53 PM UTC - We are rolling out changes to address the message queue processing issue that has been contributing to the intermittent search instability. Our team remains fully focused on restoring search functionality as quickly as possible.
7:13 PM UTC - Update: We continue to investigate the intermittent search issues. Our engineering team remains fully engaged and is working diligently to identify a resolution. We understand the impact this has on our customers and we appreciate your continued patience.
10:48 PM UTC - Update: Our engineering team continues to actively investigate and work toward resolving the intermittent search issues. We do not have an ETA for full resolution at this time but remain fully committed to restoring search functionality as quickly as possible.
All other platform features and functionality remain fully operational.
11:44 PM UTC - We continue to experience intermittent instability within our search infrastructure. Our engineering team has isolated the affected segment and is actively shifting the workload to healthy systems to mitigate the impact. We will provide additional updates as they become available.