Introducing Capacity Analysis for Python

editor's note: this article was original written by graham dumpleton.

last week, we released the latest version of our python agent . this release includes a few updates that were frequently requested by our users, including better support for tastypie which should be particularly useful for django users. one of the features i’m most excited about is the capacity analysis report and that’s what i’ll be discussing in this post.

the capacity analysis report helps you know if your app has enough instances deployed to keep up with request load. in other words, it helps you find out how busy your application is under varying loads, allowing you to tune your configuration for optimal performance. the reason i’m so interested in this report is, that as the author of apache/mod_wsgi, i am always being asked what is the best mix of processes and threads to use when configuring the wsgi server hosting your application.

unfortunately, there is no simple answer i can give as it depends on many factors. ultimately, the only way to ascertain what the optimal configuration is is by monitoring your specific web application when it has been deployed to production and use that information to iteratively tune the configuration. the capacity analysis report that comes with this version of the python agent provides you with key metrics you can use to answer this question.

what does the capacity analysis report show?

the capacity analysis report captures information in three separate charts:

the first chart, app instance busy , shows how busy your application instances are (i.e. what percent of time instances are processing requests). you can use this graph to determine if you have the right number of instances for your application. as app utilization approaches 100%, you application needs more instances to handle incoming requests.

to understand how this works, imagine your app constantly receives one request per minute. if your app could only serve one request per minute, your utilization would be 100%. if you halved the time it takes to serve a request, or doubled the number of app servers, your utilization would go down to 50%.  if multithread server processes are being used, the utilization figure is calculated across all threads which are actively handling requests in that process. a figure of 50% can therefore indicate that the equivalent of five out of ten available threads were being fully utilized during that period of time.

so essentially, the chart looks at how long your requests take to serve and how much capacity you have, and expresses the result as a percentage. when a multiprocess server configuration is being used, as well as the average being shown, the utilization of the least and most heavily loaded instances during that time are visible.

the next chart to look at is app instance analysis . this chart shows the total number of instances running along with the concurrent instance load. the concurrent instance load is the number of fully busy instances that would be needed to handle the load on your web application. it is computed as based on the average utilization from the app instance busy chart, multiplied by the number of instances you have running.

the final chart, app instance restarts by host , shows the number of instance restarts for each time interval. you can use this to figure out if your application instances are restarting too frequently.

how can i see the number of available threads?

the capacity analysis report isn’t a new feature of new relic. it’s actually been available as part of the ruby agent for some time. at this stage, we have simply taken the existing report as implemented for the ruby agent and updated the python agent so it can be used with it as well. because of this, the current report doesn’t break out any more detailed information about the number of threads which may be available and/or used within each process. we are working on adding this in some way, but in the interim it’s still possible to get this information by creating a custom dashboard which accesses and charts the underlying metrics which are used in creating the capacity analysis report for the python agent.

the names of the metrics available for the python agent which relate to capacity analysis and can be charted using a custom dashboard are:

when using these metrics in a custom dashboard, we can focus in on specific measures related to thread usage within application processes.

in some cases it can help to understand better the inner workings of specific wsgi servers. for example, there’s an interesting aspect of how apache/mod_wsgi daemon mode works that shows up in these particular charts. in this case, the apache/mod_wsgi daemon mode configuration was:

although mod_wsgi will create 25 threads, they are maintained in a lifo queue. this means that when handling a request, a preference is given to using a thread which has recently handled a request. if the number of threads is over-specified, any additional threads beyond what is required will sit there dormant and never be used. because they aren’t used, they won’t be counted against threads available by the agent.

this was the circumstance around this particular example. threads available averaged at less than four across all processes. even when the maximum accessed was considered, it only reached a maximum of 12 in any one process.

this meant that the configuration of ‘threads=25′ was actually a lot more than was needed to satisfy the load. a small amount of memory could have been saved by reducing the number of configured threads.

at the same time, overall capacity used reached up to 40%. some headroom is required for expansion and spikes in traffic, but the number of instances could also have been reduced.

what about co-routine based systems?

calculating how busy a process is for single or multithreaded systems is relatively straight forward. for co-routine based systems, such as gevent and eventlet, it gets a bit trickier. this is because there isn’t really an upper limit on the number of co-routines you could have so as to measure a utilization factor. instead, co-routines are created on demand for each request and destroyed at the end of the request.

as a result we are still tweaking how we can represent instance load for a co-routine based system and the capacity analysis report will currently not provide data when such a co-routine based system is used.

how much overhead does this introduce?

overhead of any monitoring is something we take very seriously. because of this, the code that performs the calculation of thread utilization which underlies these metrics is currently implemented as a c extension to the python agent. this allows us to implement the measure with no noticeable overhead. since it relies on a c extension, the system on which the agent package is installed on must have a compiler present. if it does not, the c extension can’t be compiled at the time of installation and the feature will be disabled. we are currently working on a pure python implementation, but we want to make sure that there will be no performance impacts due to the feature.

try it out for yourself

if you haven’t already, you should try new relic today to see how you can improve the performance of your python applications.