Perfecto is trusted by major enterprises to provide timely manner mobile, web and IOT cloud-based quality solutions. A major part of the R&D responsibility is to assure that its cloud-based solutions work seamlessly regardless of any new market event like new iOS or Android release, new browser or smartphone introduction and of course with any new feature or enhancements made to the cloud offering.
With that in mind, when it comes to DevOps, Perfecto is drinking its own champagne. Not only does Perfecto work with developers and testing teams across its global customer base to automate continuous testing throughout the DevOps cycle, the company has brought continuous integration, faster releases and practical DevOps processes/culture into its own engineering department.
Restructuring the Engineering Organization
In the past year, Perfecto’s engineering arm finalized a complete organizational restructure. Three years ago, development, QA, and operations teams worked separately, inhibiting collaboration and quick problem-solving. The whole organization ran with a single release train, with very high level of cross-team dependencies and responsibility handoffs. This impacted both velocities as well as the quality of the released product. Today, Dev, QA, and operations are organized in squads to create what Perfecto calls DevQAOps. Each squad is completely autonomous, has end-to-end responsibility for all SDLC aspects and phases starting from design, through implementation, building packaging the artifacts, handling the process of deployment, as well as monitoring and analyzing customer’s usage. Finally, the autonomous teams are also the ones who address any issues that are encountered in production.
By shifting responsibility, squads became closer to users. New feature design partnerships and controlled early access programs exposing new functionality are now commonplace resulting in shortened feedback loops. Gradual exposure mechanism which is controlled by automatic gates analysis was also added, to ensure that new features are being measured in the real world before they become mainstream and available to all customers.
The shift to squads with an end to end responsibility was complemented by other reinforcing changes. Perfecto adopted cross-functional guilds to define key standards such as UX, data persistence, messaging, deployment, monitoring, and test automation. To better align individual expectations, Perfecto created more defined roles for system architects and program manager. While each squad is fully responsible for the services they develop and operate as well as their own release train, Perfecto has implemented a central quality team view to ensure comprehensive quality spanning all aspects of the solution both pre and post-deployment of new services to production.
To keep the DevQAOps squads on track, Agile ceremonies such as daily stand-ups, weekly scrums, planning and retrospective meetings are followed. Additionally, squads track key performance indicators to allow continuously improve the team’s velocity and efficiency.
Organization Drives Culture
The result of these organizational changes has been greater autonomy and faster time to market, successfully simulating the DevOps-powered customer teams Perfecto supports.
Perfecto believes in great cross-team collaboration, rewarding and growing individual skills, learning by example, and continuous improvements. Thus, we continuously measure ourselves using metrics, like number of story points per developer, time spent on new features development, availability and performance of released features, defects solving metrics and more. Focusing on a team and individual accountability gathers everyone under the same objectives and ensures focused deliverables.
At Perfecto, it’s not all about the routine product roadmap development, but also about thinking outside the box. That’s why, we conduct at least two global hackathons a year, where we encourage the great talents in Perfecto to bring additional ideas and solutions to major problems either challenging our customers or faced by our squads. These hackathons resulted in many new features and product enhancements that are widely used by Perfecto’s top customers.
Automated Release Pipeline
Perfecto is living the everything as code mantra. If it matters, it must be in Git, it must be versioned and must be audited. This obviously applies to test code, but also to infrastructure setup, artifact packaging, deployment, configuration management, and finally also to service monitoring and usage tracking.
While overall responsibility was shifted to the DevQAOps team, we defined some gatekeepers to ensure all services meet the high standard Perfecto is committed to. Merging new functionality to the master requires test code be also committed. We defined production readiness criteria for new services before being exposed externally. These criteria cover all operational aspects such as gradual exposure and rollout plan, service monitoring, documentation and troubleshooting, scalability and usage collection.
From Monolithic to Microservices
Another crucial transition at Perfecto has been one that affects tools, people, and processes–the shift from a monolithic code base to microservices. Previously, Perfecto’s offering had a monolithic architecture spanning three locally hosted, single tenant servers. Perfecto, made a significant architectural change to further leverage cloud services to gain greater availability, scale, elasticity, cost-effectiveness from the operational standpoint and other long-term maintenance benefits.
Adopting a microservices strategy translates into more than 25 and growing, different microservices deployed as Docker containers. Each service is highly-available and load balanced, fully elastic enabling scalability and zero downtime upgrades. A sample of Perfecto’s Continuous Quality Lab services includes streaming video, reporting, image and text analysis, user and resource management.
From Own Hosting to the Public Cloud
To fully leverage the power of microservices based architecture, Perfecto has also moved many of the new services to run in the cloud. Perfecto now combines services running in its data centers with using AWS services like EC2, ECS, Lambda, S3, DynamoDB, and more.
Since switching to microservices, Perfecto has seen many productivity benefits. Overall solution performance has improved. Our ability to deliver innovation to market has also accelerated. Under a monolithic code base, Perfecto released software to the market about four times a year. Today, Perfecto operates under a hybrid release approach with new functionality being continuously released by the different squads on a cadence that varies from several times a day to once every 3 weeks, depending on the specific service.
The change in architecture has developers feeling the difference in their day-to-day activities. Previously, they spent about 60% of their time developing new features and 40% on maintenance and stability objectives. That ratio is now up to 80% innovating and only 20% stabilizing. The net benefit is a 7% increase in innovation throughput.In addition, since the beginning of the change, there’s been a 90% reduction in service outages and a 150% growth in system capacity
What’s more, the switch to a microservices framework has strengthened Perfecto’s ability to deliver same-day support for the latest mobile and web operating systems. Perfecto’s PCL™ - Perfecto Connectivity Layer. Upon release, the engineering team delivers an update daily to both internal stakeholders and customers about any issues occurring and how to address them. This cadence is a huge advantage for Perfecto’s customers, who must adjust their testing practices quickly as new devices and operating systems enter the market. For example, with an iOS release, the user base typically updates within 48 to 72 hours of its release. It’s critical to communicate fast and often.
Open Source @Perfecto
Perfecto embraces open-source. Not only its entire product offering spans around Dev and Test open-source frameworks, where Perfecto integrates into like Appium, Selenium, Espresso and more, but in addition and as part of the engineering strategy, Perfecto’s R&D squads own responsibility for their own technology stacks. Currently, teams use various open-source technologies, including platforms and frameworks like Spring Cloud, MongoDB, ReactJS, Redis, RabitMQ, Nginx, Docker, Prometheus, and much more.