The 11.9.0 version upgrade included a new logic to remove jobs older than 6 months. While this function did not present an issue with product performance during QA testing, unfortunately, problems presented in the production environment when a large number of records were started to be pruned.
The pruning action caused high resource usage in the database, and resulted in database locks. This in turn caused a timeout or direct failure when creating new jobs on the Syrah platform.
The database clean-up logic has been disabled to resolve the issue for users on Syrah, and prevent the same issue from occurring on platforms where the 11.9.0 version upgrade was scheduled to be deployed in the following days.
Work on a subsequent code change to the clean-up logic has started immediately to apply the lessons learnt from the incident, and avoid the problem from resurfacing once the logic has been re-enabled on 11.9 platforms.
More rigorous risk assessment procedures for code changes have been introduced into the development and release review processes.