Are you using Celery to process python back-end tasks asynchronously? Have you wanted to get insight into their resource consumption and efficiency? Here’s a few useful ways to get insight into Celery performance when running tasks.
A simple celery task
For a quick review, Celery lets you turn any python method into an asynchronous task. Here’s a simple one:
Let’s trace Celery!
We’ll start with the good stuff. In the latest release of our Python instrumentation, oboeware-1.0, we have an updated API that makes it super-easy to gather performance data from any Python code, including Celery tasks. Here’s how we’d add tracing to our example task:
The key part is the @oboe.Context.trace decorator. Note that we’re assigning to the keys Controller and Action. This will be used by TraceView to segment the data. You can also optionally use the keys HTTP-Host and URL to indicate domain and URL to TraceView. Here’s what the data we’re gathering looks like in the dashboard (I added a few sqlalchemy queries to spice up the data):
The code at the top is configuration tunables: set the sample rate to a fractional value to trace only a fraction of your Celery tasks.
Event-based workers
You might be using event-based workers to save a few bytes of ram. TraceView also supports eventlet-based workers. However, you’ll need to install our TraceView-enabled greenlet module using pip or easy install:
|
1
|
$ pip install --extra-index-url=http://pypi.tracelytics.com greenlet-0.3.4-tly1.0
|
Slightly more interesting
Of course, you’re probably interested in more complex celery tasks than that. The good news is, all of our normal instrumentation works with your celery workers. Here’s a screenshot of one of our internal Celery workers in action:
Other resources
Looking for different tools? Good news–there’s a pretty healthy Celery ecosystem. Here’s a few useful packages and I’m sure there’s tons I’m forgetting–leave comments for anything else you find helpful with Celery!
- If you want to keep track of worker status, you might be interested in celerymon, a monitor for celery tasks. It keeps tabs of task execution and workers.
- Trying to track down a memory leak? Consider running dowser inside your celery worker.
Photo Credit: dottiemae
{{ parent.title || parent.header.title}}
{{ parent.tldr }}
{{ parent.linkDescription }}
{{ parent.urlSource.name }}