The agile room was quiet. Developers were focused on their tasks, while the testers introduced a new round of end-to-end tests. The scrum master was handling normal tasks for the sprint, often communicating with the product owner on outstanding questions. Things were normal. Things were good.
It Started So Simple
While building a new feature, one of the developers realized that the class name for one of the custom classes written in the Salesforce environment wasn't the best name that could have been chosen. Within the developers dedicated sandbox, the class was renamed to clear up the complication. Development continued in the sandbox and soon the code was ready for review.
To the entire team, the thought was that the original class name had not been deployed into production. So, knowing that renaming class files is kind of tricky within Salesforce, the developer updated each developer's sandbox with the renamed class. The full copy sandbox was updated in the same fashion. The updates were checked into Git without any issues and after a few more updates, the release was ready for production.
The Nightmare Begins
With the sun still hidden from the night time hours, the release was scheduled for deployment into production. No one checked to see if there was a full moon that night, but I am confident there was. Around 6:20 am, the build manager kicked off the production release process from Atlassian Bamboo. After a few seconds, the flurry of red text began to paint onto the continuous integration screen. The build was failing.
Much to everyone's surprise, the original class files were actually in the production instance. They were introduced, but left dormant in the prior sprint — which had escaped everyone's mind. As you might imagine, Salesforce does not allow class files to be updated or changed manually in a production org. For now, the release had to be put on hold. End-users expecting to see their latest features would have to wait until this problem could be resolved.
A number of attempts were made to use destructive changes to remove the classes and add the new classes, but the dependencies still existed in the production org's code version - which would not allow the delete of the classes to complete. In the end, a hot fix to the build had to be created, which re-introduced the classes as empty classes. These empty classes stayed in place until the next release, which removed them in the destructive changes process.
Our team certainly realizes the challenges with renaming objects within Salesforce. However, we didn't realize that the classes being renamed were already in the production org. Had that have been the case, I am confident the class would not have been renamed. It was certainly a lesson learned, which I believe the entire team will remember for quite a while.
Have a really great day!