Over a million developers have joined DZone.
Platinum Partner

The story of a massive system refactoring for the cloud: prologue

· DevOps Zone

The DevOps Zone is brought to you in partnership with New Relic.  Learn more about the common barriers to DevOps adoption so that you can come up with ways to win over the skeptics and kickstart DevOps.

A couple of months ago I started working at Buzzilla, a company developing "cutting edge technologies and revolutionary analysis and research methodologies that combine to create advanced solutions aimed at harnessing the vast opportunities presented by online conversation". Or in short, full-text search and analytics on BigData.

When I started there the existing system was based on a home-brewed solution for distributing a Lucene index across dozens of nodes. Since we are looking to expand way beyond the number of a couple of dozen servers, we really needed to recreate the system using tools better suited for the job, which we could also take to the cloud. And it is much more than just about the search engine.

Refactoring an operational distributed system is really a great challenge. While keeping it operational, you need to replace parts one by one, but you also get the opportunity to experiment with new tools and bleeding edge technologies. It is also a great subject for a series of blog posts, this one being the first of many to come. Among the items we had to tackle on which I'll blog are:

  • Building a distributed and highly available search engine
  • Various topics on full-text search: relevance, scoring, multi-lingual search, best practices for analysis and more
  • Choosing the right web framework for a web UI
  • Migrating from MySQL to NoSQL - and selecting the right NoSQL for the job
  • Caching
  • Keeping tabs of logs in a distributed environment
  • Planning for and recovering from failures and crashes
  • CI, deployment, versioning and backups
  • Identifying and fixing performance bottlenecks
  • Generating and displaying system stats and performance metrics

Such a refactoring process could easily turn out to be a disaster. This is the story of a careful planning and a great team working together, which made this a success. We are now in the final stages of migrating to the new system, and we have many more challenges pending after we have gone completely live. This series is going to span months and hopefully have some great content which will spark great discussions.

The DevOps Zone is brought to you in partnership with New Relic. Quickly learn how to use Docker and containers in general to create packaged images for easy management, testing, and deployment of software.


Published at DZone with permission of Itamar Syn-hershko , DZone MVB .

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}