{{ !articles[0].partner.isSponsoringArticle ? "Platinum" : "Portal" }} Partner
python,devops,tool,performance,tips and tricks,tools & methods,apm,application performance management,celery,application performance,appneta,traceview

Tracing Celery Performance For Web Applications

Are you using Celery to process python back-end tasks asynchronously?  Have you wanted to get insight into their resource consumption and efficiency?  Here’s a few useful ways to get insight into Celery performance when running tasks.

A simple celery task

For a quick review, Celery lets you turn any python method into an asynchronous task.  Here’s a simple one:

Let’s trace Celery!

We’ll start with the good stuff. In the latest release of our Python instrumentation, oboeware-1.0, we have an updated API that makes it super-easy to gather performance data from any Python code, including Celery tasks. Here’s how we’d add tracing to our example task:


The key part is the @oboe.Context.trace decorator.  Note that we’re assigning to the keys Controller and Action.  This will be used by TraceView to segment the data.  You can also optionally use the keys HTTP-Host and URL to indicate domain and URL to TraceView.  Here’s what the data we’re gathering looks like in the dashboard (I added a few sqlalchemy queries to spice up the data):

celery performance in tracelytics

The code at the top is configuration tunables: set the sample rate to a fractional value to trace only a fraction of your Celery tasks.

Event-based workers

You might be using event-based workers to save a few bytes of ram.  TraceView also supports eventlet-based workers.  However, you’ll need to install our TraceView-enabled greenlet module using pip or easy install:

$ pip install --extra-index-url=http://pypi.tracelytics.com greenlet-0.3.4-tly1.0

Slightly more interesting

Of course, you’re probably interested in more complex celery tasks than that.  The good news is, all of our normal instrumentation works with your celery workers.  Here’s a screenshot of one of our internal Celery workers in action:

tracing complex celery tasks

Other resources

Looking for different tools?  Good news–there’s a pretty healthy Celery ecosystem.  Here’s a few useful packages and I’m sure there’s tons I’m forgetting–leave comments for anything else you find helpful with Celery!

celery performance

  • If you want to keep track of worker status, you might be interested in celerymon, a monitor for celery tasks.  It keeps tabs of task execution and workers.
  • Trying to track down a memory leak?  Consider running dowser inside your celery worker.

Photo Credit: dottiemae

Related Articles

Tracing Python — An API

Python and gevent

Advanced SQLAlchemy & Performance Concerns

{{ tag }}, {{tag}},

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}
{{ parent.authors[0].realName || parent.author}}

{{ parent.authors[0].tagline || parent.tagline }}

{{ parent.views }} ViewsClicks