DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workkloads.

Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • Exploring the Architecture of Amazon SQS
  • Unlocking the Benefits of a Private API in AWS API Gateway
  • Zero Trust for AWS NLBs: Why It Matters and How to Do It
  • Unlocking the Power of Serverless AI/ML on AWS: Expert Strategies for Scalable and Secure Applications

Trending

  • DZone's Article Submission Guidelines
  • Docker Base Images Demystified: A Practical Guide
  • How Large Tech Companies Architect Resilient Systems for Millions of Users
  • The Modern Data Stack Is Overrated — Here’s What Works
  1. DZone
  2. Software Design and Architecture
  3. Cloud Architecture
  4. AWS SQS in a Large Scale Application

AWS SQS in a Large Scale Application

What is and how to use AWS SQS to scale your application. This article shows readers how to design your internal infrastructure to decouple system components.

By 
Valerio Barbera user avatar
Valerio Barbera
·
Mar. 10, 23 · Tutorial
Likes (1)
Comment
Save
Tweet
Share
3.9K Views

Join the DZone community and get the full member experience.

Join For Free

In today’s article, I’m going to show you how we use AWS SQS in our Laravel application and how it helps us manage 1.6 billion operations each month.

In the image below, you can see our typical bill for a month of AWS SQS usage:AWS SQS

Before entering into the details of our system design, I would introduce the SQS technology, why you should consider a messaging system, and how to compare it with similar services.

What Is AWS SQS?

Amazon Simple Queue Service is a fully managed messages queueing system to enable communication between distributed software components or microservices.

Decoupling some parts of your system allows you to run and fail your software components independently. It also allows you to build easily scalable applications.

Message queues act as intermediaries:

Intermediaries

What Is a Message?

This is an important question I want to answer to help developers who are approaching this technology for the first time.

Every message queuing system accepts a text as the content of a message. The most common way to make different systems exchange structured data is to use JSON objects.

JSON can be easily serialized as text when pushing a new message onto the queue and deserialized when it is consumed on the other side of the system.

The Laravel Way

Laravel implements its own format when exchanging messages with queue systems. Here is an example:

The Laravel Way

The JSON is automatically created by the internal queue system of the framework when you dispatch a job. It is somewhat hidden from the eyes of the developer.

The JSON message contains all data needed by Laravel to understand what job class should be executed and under what conditions (attempts, maxTries, etc.).

If your application doesn’t have a prepackaged solution like the one provided by Laravel, you can define your own format. The goal is to provide the consumer of the queue with all information to process the desired task.

How To Consume Messages from the Queue

You have to implement a system that continuously polls the queue waiting for a message to arrive.

Laravel provides a built-in “worker” that you can run with a simple command:

Shell
 
php artisan queue:work


This command runs a background process that asks for new messages. When a message is retrieved, the worker can run the job class included in the message.

You can see this process in the \Illuminate\Queue\Worker.php class:

PHP
 
while (true) {
	// Before reserving any jobs, we will make sure this queue is not paused and
	// if it is we will just pause this worker for a given amount of time and
	// make sure we do not need to kill this worker process off completely.
	if (! $this->daemonShouldRun($options, $connectionName, $queue)) {
		$status = $this->pauseWorker($options, $lastRestart);

		if (! is_null($status)) {
			return $this->stop($status, $options);
		}

		continue;
	}
	
	...

	// If the daemon should run (not in maintenance mode, etc.), then we can run
	// fire off this job for processing. Otherwise, we will need to sleep the
	// worker so no more jobs are processed until they should be processed.
	if ($job) {
		$jobsProcessed++;

		$this->runJob($job, $connectionName, $options);

		if ($options->rest > 0) {
			$this->sleep($options->rest);
		}
	} else {
		$this->sleep($options->sleep);
	}
	
	...
}


Why Should You Use AWS SQS?

Since my product is a developer tool, I discuss with other fellow developers almost every week about the internal design of their software. 

In most of the applications I’ve had the opportunity to analyze, a messaging system was introduced to enable asynchronous task execution. 

The same is for Inspector.

I think this is the most common use case for messaging systems today. Microservices or large distributed software, in general, are very niche use cases. Probably belonging more to enterprise environments.

What Is an Asynchronous Task?

Imagine your product offers the ability to import account information by uploading large files. The user picks the file from the local PC and clicks “Import.”

This operation can take a while to be completed. You have to open the file, read each line, convert it into the appropriate format, and store this information in your system.

It could be impossible to handle this process at the time of an HTTP request.

Thanks to a message queueing system, we can store the file on our servers and push a message to the “import” queue with the path where the file is. 

The next step is to make a background system that waits for these messages to handle the import process. Users don’t need to wait until the import is done. In the meantime, they can continue to use the product or leave it waiting to be notified when the import is completed:

Import

How Inspector Uses AWS SQS

The most important challenge of Inspector is to be able to process large amounts of traffic, also guaranteeing good performance.

The endpoint where Inspector receives data from the monitored applications handles about 10 million HTTP requests per day. With this volume, the problem is that these data packets can’t be processed on the fly. 

At the beginning of our adventure, we used a self-managed Redis as a message queueing system. But, with increasing volumes, we have preferred to rely on a managed service.

The strategy is to use AWS SQS to be able to push these tasks to a queue, which will be consumed by another separate system in the background (the processing pipeline):

Processing Pipeline

Ingestion nodes are implemented in Node.js. They are built as a very simple and extremely scalable script. The worker servers that consume the queue are, instead, a system component built on top of the Laravel framework.

We could have chosen any other technology to manage the message queue. At this stage of the company, SQS guarantees scalability without any complexity for us since it is a fully managed service.

Other solutions could also save us costs. We prefer to guarantee our customers that the platform works perfectly and avoid the risk of introducing weaknesses that could cause unexpected downtimes.

Conclusion

At this point, you should have a better idea of what AWS SQS is, how to use AWS SQS to scale your application, why you should use AWS SQS, and more. I hope you have taken away some valuable information from this article. If so, feel free to like and share.

AWS Laravel Message queue

Published at DZone with permission of Valerio Barbera. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Exploring the Architecture of Amazon SQS
  • Unlocking the Benefits of a Private API in AWS API Gateway
  • Zero Trust for AWS NLBs: Why It Matters and How to Do It
  • Unlocking the Power of Serverless AI/ML on AWS: Expert Strategies for Scalable and Secure Applications

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!