Dedicated ETL pipelines are easy to set up but hard to scale, while common pipelines offer efficiency at the cost of complexity. Know which one to choose.
In this article, I will explain asynchronous retry mechanisms and demonstrate how to achieve retries by using the native functions of a broker using AWS SQS.
You can also do sharding between multiple queue sets (main + DL). Implementation classes will be shared so no additional coding would be necessary. The client can determine the queue set by a simple round robin at the initial publish.
Gabor, thanks a lot for pointing out this limitation. To me, "serverless" means, there should not be these kinds of limitations developers need to take care of. It seems that AWS is not following this approach. I just used AWS SQS as an example broker to explain the mechanism but this limitation can be a showstopper for some organizations using only AWS technologies.
CORE
Comments
Apr 06, 2018 · Murat Balkan
You can also do sharding between multiple queue sets (main + DL). Implementation classes will be shared so no additional coding would be necessary. The client can determine the queue set by a simple round robin at the initial publish.
Apr 04, 2018 · Murat Balkan
Gabor, thanks a lot for pointing out this limitation. To me, "serverless" means, there should not be these kinds of limitations developers need to take care of. It seems that AWS is not following this approach. I just used AWS SQS as an example broker to explain the mechanism but this limitation can be a showstopper for some organizations using only AWS technologies.