Summary
On March 27th, we experienced an issue impacting file processing where some files uploaded through FTP or the in-app upload workflow took longer than expected to complete, or remained in a “Processing” state without finalizing.
This issue has now been resolved.
What Happened
As part of ongoing infrastructure improvements, we've been enhancing the scalability of our integration jobs by migrating them from App Services to Kubernetes. During the migration of one of these background jobs, the service was scaled down more quickly than it was scaled up, which caused a backlog in some integration jobs and led to slower file ingestion.
Resolution
Once the backlog was identified through monitoring, the service was promptly scaled to the appropriate capacity. This allowed the system to process the queued jobs and restore normal ingestion performance.
Preventative Measures
To reduce the risk of recurrence, we have strengthened our scaling controls, improved migration validation processes, and enhanced monitoring to more quickly detect and respond to processing backlogs.