Nginx, Passenger and WSGI

It’s a little-known fact that Phusion Passenger, the awesome Rack webserver module, can also competently talk WSGI as well as fit into the Rack ecosystem. This means that not only can you run your Rack and Rails applications through it using Nginx or Apache (or in fact independently, using a cut-down version of Nginx) but also your WSGI-compliant Python applications.

This has been touched on in various levels of detail on Hongli’s own blog, the official Phusion blog, and the Dreamhost wiki, among other sites. You can piece together a working configuration between them but especially relying on the Dreamhost instructions you don’t really get a good idea of a generic configuration that would work outside of their tuned environment.

I’ve got an over-utilised HTPC at home that ends up not just serving as an XBMC frontend but a bunch of other things – IPv6 tunnel endpoint, VPN endpoint, IP traffic accounting system, monitoring station, wiki server, Mingle server and more. On 2GB of RAM this ends up being a significant amount, and most of the webapps are either being run as CGIs or proxied to via Apache. Since the wiki I’m running is MoinMoin, and I recently also found a way to run my CGIs through Passenger it seemed natural to do away with the separate webservers and run as much through Passenger as possible. Since all the cool kids are running Nginx these days on the basis of it having a much smaller memory footprint (and obviously other factors) I thought I would move to that from Apache at the same time and claim back some memory.

What I’ll describe below is a very quick run-through of setting up MoinMoin for operation through Passenger and Nginx. It should hopefully be somewhat reusable (the main point of writing this post, since I couldn’t find a decent generic guide through Google so far) but there are some details of the operation which I haven’t investigated entirely yet so don’t hold it against me!

Install Nginx (via Passenger)

If you are not familiar with Nginx, or if you are coming from the Apache world you may be dismayed, overjoyed, surprised or annoyed that Nginx doesn’t yet support dynamically-loaded modules like Apache does, no doubt for performance reasons. This means any additional modules like Passenger require a full recompile of the Nginx binary. For that reason I didn’t see a big point in installing the Ubuntu package for Nginx at all and just used the Passenger installer which has an option to download and install Nginx with Passenger already enabled.

Firstly, just install the Passenger gem:

root@oneiric:~# gem install passenger
Fetching: fastthread-1.0.7.gem (100%)
Building native extensions.  This could take a while...
Fetching: daemon_controller-1.0.0.gem (100%)
Fetching: rack-1.4.1.gem (100%)
Fetching: passenger-3.0.13.gem (100%)
Successfully installed fastthread-1.0.7
Successfully installed daemon_controller-1.0.0
Successfully installed rack-1.4.1
Successfully installed passenger-3.0.13
4 gems installed
Installing ri documentation for fastthread-1.0.7...
Installing ri documentation for daemon_controller-1.0.0...
Installing ri documentation for rack-1.4.1...
Installing ri documentation for passenger-3.0.13...
Installing RDoc documentation for fastthread-1.0.7...
Installing RDoc documentation for daemon_controller-1.0.0...
Installing RDoc documentation for rack-1.4.1...
Installing RDoc documentation for passenger-3.0.13...

Now use Passenger itself to download and install Nginx with its own recommended settings (option 1, when you have to choose), rather than rebuilding any available version for your distribution. Handily, the installer will give you pretty accurate commands to use if you don’t have some pre-requisites installed like a C++ compiler, so it is pretty idiot-proof.

root@oneiric:~# passenger-install-nginx-module
Welcome to the Phusion Passenger Nginx module installer, v3.0.13.

...LOTS of output...

I chose to install it in the suggested location, /opt/nginx. Assuming you got through the pre-requisites and Nginx compiled and installed successfully, it will also be somewhat configured for you. However, we are interested in running a WSGI application, so the standard Rack/Rails suggestions the installer provides are not entirely helpful. What is left is to set up the app, provide the bridge to WSGI and set up a location stanza in Nginx for it.

Install MoinMoin

At a former employer, we used MoinMoin wiki for internal (and later external) documentation and I’ve continued that by using it for personal documentation. It’s a very handy tool, especially being available from all our computers at home and remotely via VPN. It’s written in Python and works very well.

Installing is dead simple with setuptools:

root@oneiric:~# apt-get install -y python-setuptools
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  python-pkg-resources
Suggested packages:
  python-distribute python-distribute-doc
The following NEW packages will be installed:
  python-pkg-resources python-setuptools
0 upgraded, 2 newly installed, 0 to remove and 2 not upgraded.
Need to get 274 kB of archives.
After this operation, 1,274 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ oneiric-updates/main python-pkg-resources all 0.6.16-1ubuntu0.1 [62.7 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ oneiric-updates/main python-setuptools all 0.6.16-1ubuntu0.1 [212 kB]
Fetched 274 kB in 1s (180 kB/s)
Selecting previously deselected package python-pkg-resources.
(Reading database ... 54159 files and directories currently installed.)
Unpacking python-pkg-resources (from .../python-pkg-resources_0.6.16-1ubuntu0.1_all.deb) ...
Selecting previously deselected package python-setuptools.
Unpacking python-setuptools (from .../python-setuptools_0.6.16-1ubuntu0.1_all.deb) ...
Setting up python-pkg-resources (0.6.16-1ubuntu0.1) ...
Setting up python-setuptools (0.6.16-1ubuntu0.1) ...

root@oneiric:~# easy_install moin
Searching for moin
Reading http://pypi.python.org/simple/moin/
Reading http://moinmo.in/
Best match: moin 1.9.4
Downloading http://static.moinmo.in/files/moin-1.9.4.tar.gz
Processing moin-1.9.4.tar.gz
Running moin-1.9.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-Wp_XA9/moin-1.9.4/egg-dist-tmp-fEQzCQ
warning: no files found matching '*' under directory 'tests'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '*/CVS/*' found anywhere in distribution
warning: no previously-included files matching '*/.cvsignore' found anywhere in distribution
warning: no previously-included files matching 'underlay.tar' found anywhere in distribution
warning: no previously-included files matching 'README.underlay' found anywhere in distribution
zip_safe flag not set; analyzing archive contents...
MoinMoin.wikiutil: module references __file__
MoinMoin.packages: module references __file__
MoinMoin.events.__init__: module references __file__
MoinMoin.support.werkzeug.utils: module references __file__
MoinMoin.support.werkzeug.utils: module references __path__
MoinMoin.support.werkzeug.serving: module references __file__
MoinMoin.support.werkzeug.__init__: module references __file__
MoinMoin.support.werkzeug.__init__: module references __path__
MoinMoin.support.werkzeug.contrib.jsrouting: module MAY be using inspect.trace
MoinMoin.support.werkzeug.debug.tbtools: module MAY be using inspect.getsourcefile
MoinMoin.support.werkzeug.debug.__init__: module references __file__
MoinMoin.support.pygments.unistring: module references __file__
MoinMoin.support.pygments.formatters._mapping: module references __file__
MoinMoin.support.pygments.lexers._luabuiltins: module references __file__
MoinMoin.support.pygments.lexers._clbuiltins: module MAY be using inspect.trace
MoinMoin.support.pygments.lexers._phpbuiltins: module references __file__
MoinMoin.support.pygments.lexers._mapping: module references __file__
MoinMoin.converter.__init__: module references __file__
MoinMoin.script.account.__init__: module references __file__
MoinMoin.script.maint.__init__: module references __file__
MoinMoin.script.cli.__init__: module references __file__
MoinMoin.script.server.__init__: module references __file__
MoinMoin.script.import.__init__: module references __file__
MoinMoin.script.export.__init__: module references __file__
MoinMoin.script.index.__init__: module references __file__
MoinMoin.script.xmlrpc.__init__: module references __file__
MoinMoin.script.migration.__init__: module references __file__
MoinMoin.web.__init__: module references __file__
MoinMoin.web.static.__init__: module references __file__
MoinMoin.filter.__init__: module references __file__
MoinMoin.parser.__init__: module references __file__
MoinMoin.config.multiconfig: module references __file__
MoinMoin.macro.__init__: module references __file__
MoinMoin.userprefs.__init__: module references __file__
MoinMoin.formatter.__init__: module references __file__
MoinMoin.action.SpellCheck: module references __file__
MoinMoin.action.__init__: module references __file__
MoinMoin.theme.__init__: module references __file__
MoinMoin.xmlrpc.__init__: module references __file__
Adding moin 1.9.4 to easy-install.pth file
Installing moin script to /usr/local/bin

Installed /usr/local/lib/python2.7/dist-packages/moin-1.9.4-py2.7.egg
Processing dependencies for moin
Finished processing dependencies for moin

Now we have both MoinMoin and Nginx/Passenger installed as root, but don’t worry – they definitely won’t run as root.

Configure MoinMoin instance

Setting up MoinMoin to run as a WSGI app is actually very straightforward, but we’ll set it up as a standard server app explicitly here. By this I mean, not running the standalone python-based webserver included with it, using a separate user account for it and as a result keeping the data entirely separate from the installation base of the wiki, making it easier to upgrade later on.

root@oneiric:~# useradd -m moin -s /bin/bash
root@oneiric:~# su - moin
moin@oneiric:~$ mkdir wiki; cd wiki
moin@oneiric:~/wiki$ mkdir config public
moin@oneiric:~/wiki$ cp -a /usr/local/lib/python2.7/dist-packages/moin-1.9.4-py2.7.egg/share/moin/data/ .
moin@oneiric:~/wiki$ cp -a /usr/local/lib/python2.7/dist-packages/moin-1.9.4-py2.7.egg/share/moin/underlay/ .
moin@oneiric:~/wiki$ ln -s /usr/local/lib/python2.7/dist-packages/moin-1.9.4-py2.7.egg/MoinMoin/web/static/htdocs/

Here we are setting up the basic structure of the app:

• Config will contain our customised wiki config for this instance.
• Public is a requirement of Passenger. It can stay empty.
• Data will contain our wiki pages, from a starting point of what the installation gives you by default.
• Underlay contains built-in wiki page resources of MoinMoin. It shouldn’t need to change, but when I attempted to run it from a symlink to the installed files it failed on missing write permissions. I’m not sure why this happened but I’m more comfortable making a copy and referencing it than altering the original installation file permissions. Something to investigate later.
• Htdocs are static files used by MoinMoin (stylesheets, images etc). We just symlink to them.
  

Let’s create the configuration:

moin@oneiric:~/wiki$ cd config/
moin@oneiric:~/wiki/config$ cp /usr/local/lib/python2.7/dist-packages/moin-1.9.4-py2.7.egg/share/moin/config/wikiconfig.py .

We just customise the standard included configuration file. These are the entries I changed:

data_dir = os.path.join(instance_dir, '..', 'data', '') # path with trailing /
data_underlay_dir = os.path.join(instance_dir, '..', 'underlay', '') # path with trailing /
url_prefix_static = '/wiki/htdocs'
sitename = u'My Wiki'
page_front_page = u"FrontPage"

Passenger and WSGI

To run a WSGI app, Passenger needs a script in place which initialises the app and the WSGI interface. It is called passenger_wsgi.py and usually goes into the root of the app site. In my case, I wanted to serve the wiki from the /wiki/ URI path, so things were a little different but not much. Here’s what I dropped into the file:

moin@oneiric:~$ pwd
/home/moin
moin@oneiric:~$ cat passenger_wsgi.py
import sys, os

#The two following line ensure that you are using python2.7
#You can change it to another python version if you want
INTERP = "/usr/bin/python2.7"
if sys.executable != INTERP: os.execl(INTERP, INTERP, *sys.argv)

# Add to the search path so that we can do "from MoinMoin import ..."
sys.path.insert(0, '/usr/local/lib/python2.7/dist-packages/moin-1.9.4-py2.7.egg')

#Path to your $WIKI_CONFIG
sys.path.insert(0, '/home/moin/wiki/config')

from MoinMoin.web.serving import make_application

#Import to set shared to False, to serve the media files directly
application = make_application(shared=False)

This config file is largely based on what Dreamhost have provided in their wiki. The file does not need to be executable.

Now let’s set up Nginx and Passenger to start the app using this file. As mentioned, I want to serve the site from the path /wiki/ so I use a location stanza in /opt/nginx/conf/nginx.conf. I also set up a separate location stanza to serve the static site assets from htdocs directly.

        # Wiki
        location /wiki/ {
                alias /home/moin/;
                passenger_enabled on;
                passenger_user moin;
                passenger_base_uri /wiki;
        }
        location /wiki/htdocs/ {
                alias /home/moin/wiki/htdocs/;
        }

Here, the passenger_base_uri is the key to having the site served under a different path than from the root. This is one area that Nginx seems to make more complicated than Apache. Both have the ability to proxy to alternate webservers but only Apache can reverse proxypass and fix up links in the returned responses (although I believe the same can be done with Nginx scripting which I haven’t tried yet). Admittedly it’s a different tool than passing requests and responses through a loaded/compiled module but I haven’t yet found a good way to do it in Nginx (so undoubtedly a mechanism does exist).

Finally, we start up Nginx:

root@oneiric:~# /opt/nginx/sbin/nginx

Navigating to http://localhost/wiki/ now displays the front page of the so-far empty wiki. In my case, I just moved my data out of the previous path and into the location now served by Passenger (although with a symlink it is trivial to migrate from the standalone Python webserver to the Passenger-based WSGI system). Please do note that I haven’t dealt at all with the security system for MoinMoin which allows individual users to be identified and authentication/authorisation to be required.

Hopefully this has been useful for you, and I’ve run through the setup with a virtual machine from scratch to check that all the steps are correct. Please let me know though if I’ve made any mistakes, and of course any tips for a better Nginx configuration are always welcome!