DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

The software you build is only as secure as the code that powers it. Learn how malicious code creeps into your software supply chain.

Apache Cassandra combines the benefits of major NoSQL databases to support data management needs not covered by traditional RDBMS vendors.

Generative AI has transformed nearly every industry. How can you leverage GenAI to improve your productivity and efficiency?

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Related

  • How To Get Closer to Consistency in Microservice Architecture
  • Modeling Saga as a State Machine
  • Scaling Salesforce Apps Using Heroku Microservices - Part 2
  • In-house Tool for Performance Testing

Trending

  • Build a Simple REST API Using Python Flask and SQLite (With Tests)
  • Creating a Web Project: Caching for Performance Optimization
  • How to Build Real-Time BI Systems: Architecture, Code, and Best Practices
  • Code Reviews: Building an AI-Powered GitHub Integration
  1. DZone
  2. Data Engineering
  3. Databases
  4. Reporting in Microservices: How To Optimize Performance

Reporting in Microservices: How To Optimize Performance

In this article, learn about a new reporting microservice developed by Freschode to overcome reporting problems they faced.

By 
Karina Skrinnik user avatar
Karina Skrinnik
·
Nov. 15, 21 · Review
Likes (2)
Comment
Save
Tweet
Share
5.5K Views

Join the DZone community and get the full member experience.

Join For Free

Microservice architecture is becoming increasingly popular among app developers, thanks to its flexibility and scalability that allows for updates and new features to be added without overhauling the whole system. However, the use of separate user interfaces, databases, and servers for each microservice makes app optimization a challenge, considering the diversity of programming languages and API security issues. 

The Freshcode team faced the trouble of updating a legacy EdTech app running on MS SQL that used over 10,000 files created in ColdFusion and had dozens of microservices for disparate features. The app was intended for governments and education networks with hundreds of campuses and schools and had a full complement of features to appease both national universities and small schools. With conflicting microservice operations, reporting was a problem the FreshCode team had to tackle.

Monolithic vs Microservices

How To Improve Microservice Performance

Engaging different microservices caused a plethora of performance issues for the EdTech app, so we came up with an idea to develop a new reporting microservice. Instead of trying to reconcile conflicting technologies, the new microservice would get data directly from different databases, pass it through the messaging, and steam computation systems before storing it in a separate database. After that, the microservice could use the data from the database to create custom reports.

The task was further complicated by the need to develop two identical solutions (self-hosted and SaaS versions), so the team browsed several technology alternatives for every aspect of the reporting microservice to come up with the best stack for both deployment schemas.

Module Implementation in 6 Steps

Reporting Module Implementation in 6 Steps

The Freshcode team broke down the microservice development process into six stages. And as always, they started by analyzing requirements and selecting the best technologies and solutions.

1. Change Data Capturing

Oracle was the central reporting microservice database, so we went with StreamSets Data Collector as a customizable open-source solution with built-in Oracle CDC support. Matillion and Apache NiFi are equally user-friendly and beloved by developers, and they would have made the cut if they’d offered similar Oracle support.

2. Messaging System 

The development team chose Apache Kafka as a durable, scalable, and fault-tolerant messaging system. Though it requires expert DevOps, the solution boasts native in-stream computations and batch mode. Besides, unlike AWS Kinesis, Apache Kafka can be used in-house.

3. Streaming Computation Systems

Despite its complicated architecture, Apache Flink was the best choice for the EdTech app thanks to its fault-tolerance, scalability, low latency, and exactly-once state consistency. AWS Kinesis Services were also among the top choices, but they couldn’t be used on-premise, so Freshcode had to forego the AWS solution.

4. Data Lake

AWS S3 was chosen to create a depository of historical and current data. It’s a durable and cost-effective open-source solution that’s easy to integrate with other AWS products. The Freshcode team considered Apache Hadoop as a worthy alternative, though it’s a more challenging option when it comes to deployment and management.

5. Report Databases 

The reporting microservice had to work with other app modules, and the team suggested AWS Aurora as the best choice. It’s pretty fast for an SQL database while being highly scalable and durable. Though the minimally available instance is too big, it could be replaced with plain PostgreSQL. Other options the team considered included AWS Redshirt, Apache Druid, and Kinetica.

6. Report Microservice

The core features of the microservice included storing data objects and their relations, as well as generating custom reports based on the selected data objects.

Technological Stacks for SAAS and Self-Hosted Solutions

Freshcode developers settled on NodeJS for creating a microservice for the SaaS EdTech solution. It relied mostly on AWS products, like AWS S3, Aurora, and ElasticCache, complimented by StreamSets and Apache Kafka. This technology stack was ideal, as it could be easily replaced by self-hosted alternatives without disrupting the code and logic of the microservice for another deployment schema.

The in-house reporting microservice used Minio, PostgreSQL, and Redis instead as their APIs are fully compatible and provide seamless operation.

In-house reporting microservice used Minio, PostgreSQL, and Redis

The Bottom Line: Custom Reporting in Microservices

Successful implementation of the reporting microservice, along with other improvements achieved by Freshcode, enabled the EdTech company to update the design and architecture of the app and make it better through the addition of new and enhanced features. Besides, by removing reporting performance issues, the app became more flexible and scalable for in-house and SaaS deployment. 

If you want to learn more about the specifics of this project and the details of reporting microservice development with timeframe and budget estimates, check out the full EdTech case study. 

microservice app kafka AWS Database Open source Data (computing) teams hadoop

Opinions expressed by DZone contributors are their own.

Related

  • How To Get Closer to Consistency in Microservice Architecture
  • Modeling Saga as a State Machine
  • Scaling Salesforce Apps Using Heroku Microservices - Part 2
  • In-house Tool for Performance Testing

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!