A few years ago at a previous company, I had a legacy product built around an Orion app server and it was heavily integrated with the underlying Linux host system. Ops had been running this app in production for over a decade in its current configuration because it required a very specific custom setup in Linux in order to function. It made many calls out of specific tools installed on the host OS like Ghostscript and check-pass to name a few. The app also expected some of those tools to be installed in very specific (and non-standard) locations on the host system. It also had many expectations about users and directories on the host OS in order to compile, build and run the app. In the past in production, the OS of an existing machine was simply cloned in order to get a new host up and running, but this was no longer possible due to advances in hardware and OS kernals. Machines were no longer being upgraded to keep.
Due to the complexity of this app, all developers had to login to a specific development server that had the correct setup to do development for this app. There were complicated scripts that tried to help you setup your instance on this server and the specific ports were assigned to your instance when you ran it. It would frequently take hours to get your development setup working with correct branch of code to even begin develop on. To fix a bug or add a small feature to this system was measured in days or weeks, not hours. With everyone in development all competing for the same resources on this server, this host was frequently overloaded and the ability to run your instance was never reliable. You may have to wait 20-30 minutes for a build to complete just to test the latest changes you made to the code. I decide this needed to change, and set out to fix this problem.
My Requirements to Address
- Any developer should be able to run this app on their own laptop or desktop.
- Any developer should not have to be spend a day getting a specific branch of code running to begin development.
- All ports the app used need to be standardized to remove the 'port roulette' issues with everyone running on the same host.
- Simplify the setup and runtime scripts since some are incorrect and out-of-date.
- Speed up the build times by allowing the developer to build locally.
- This app can only run on very specific machines in the data center. IT needs more flexibility from this app.
- IT and OPS are concerned about the age of the hardware this app is running on in production, some of them have been EOL’ed years ago.
I started the process by just working on getting the app to run in a basic vanilla version on linux. During this process I discovered that the build required a custom and hacked version on Ant in order to build correctly. That version of Ant was so old and deprecated that it was no longer available from any public repo and had not been so some time. I was able to extract the custom version of Ant from off development server and package it up to be installed on my vanilla linux image. Many iterations followed. Once I got the app to build, getting it to run was the next challenge. Copying and/or installing the correct packages and custom linux utilities was time consuming but not hard. More iterations followed. Along the way, I standardize the ports with sensible defaults and simplified the setup and run scripts significantly.
It took a total of about 2 weeks to get this app to run on a Linux system built up from my vanilla install. Now I have a documented set of steps with all the file locations, dependencies and custom behavior that the app requires in order to run.
Creating a container from this documentation only took a day or so to complete. I am intentionally being vague here since you can use this technique for creating Docker Containers, Vagrant Machines, AWS AMIs, VMWare Virtual Machines, etc …
- Developers can develop and write code on their own laptops which resulted in an order of magnitude is development speed.
- Simple bugs can be fixed in a few hours and not days.
- Standing up a new code branch took a few minutes and not many hours.
- The ROI on this effort will be ongoing for the rest of the life of this apps life.
OPS / IT Benefits
- OPS was to be able to deploy this legacy app on newer hardware in production.
- Once the app was successfully moved off of the old servers, IT was able to decommission these machines, the oldest in the data center.
- Improved security since those OS versions had long been EOL’ed and had not received any security patches for quite some time.
This legacy app has been given a extension on life for both development and the company. Now it can run safely inside a container with other apps for at least a few more years while its replacement is being built. As of the this writing, this app is still used in production.