Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

PyMongo's "use_greenlets" Followup

DZone's Guide to

PyMongo's "use_greenlets" Followup

· Java Zone
Free Resource

What every Java engineer should know about microservices: Reactive Microservices Architecture.  Brought to you in partnership with Lightbend.

Fern - (cc) Wingchi Poon

In December, I wrote that we are removing the idiosyncratic use_greenlets option from PyMongo when we release PyMongo 3.

In PyMongo 2 you have two options for using Gevent. First, you can do:

from gevent import monkey; monkey.patch_all()
from pymongo import MongoClient

client = MongoClient()

Or:

from gevent import monkey; monkey.patch_socket()
from pymongo import MongoClient

client = MongoClient(use_greenlets=True)

In the latter case, I wrote, "you could use PyMongo after calling Gevent's patch_socket without having to call patch_thread. But who would do that? What conceivable use case had I enabled?" So I removed use_greenlets in PyMongo 3; the first example code continues to work but the second will not.

In the comments, PyMongo user Peter Hansen replied,

I hope you're not saying that the only way this will work is if one uses monkey.patch_all, because, although this is a very common way to use Gevent, it's absolutely not the only way. (If it were, it would just be done automatically!) We have a large Gevent application here which cannot do that, because threads must be allowed to continue working as regular threads, but we monkey patch only what we need which happens to be everything else (with monkey.patch_all(thread=False)).

So Peter, Bernie, and I met online and he told us about his very interesting application. It needs to interface with some C code that talks an obscure network protocol; to get the best of both worlds his Python code uses asynchronous Gevent in the main thread, and it avoids blocking the event loop by launching Python threads to talk with the C extension. Peter had, in fact, perfectly understood PyMongo 2's design and was using it as intended. It was I who hadn't understood the feature's use case before I diked it out.

So what now? I would be sad to lose the great simplifications I achieved in PyMongo by removing its Gevent-specific code. Besides, occasional complaints from Eventlet and other communities motivated us to support all frameworks equally.

Luckily, Gevent 1.0 provides a workaround for the loss of use_greenlets in PyMongo. Beginning the same as the first example above:

from gevent import monkey; monkey.patch_all()
from pymongo import MongoClient

client = MongoClient()
def my_function():
    # Call some C code that drops the GIL and does
    # blocking I/O from C directly.
    pass

start_new_thread = monkey.saved['thread']['start_new_thread']
real_thread = start_new_thread(my_function, ())

I checked with Gevent's author Denis Bilenko whether monkey.saved was a stable API and he confirmed it is. If you use Gevent and PyMongo as Peter does, port your code to this technique when you upgrade to PyMongo 3.

Microservices for Java, explained. Revitalize your legacy systems (and your career) with Reactive Microservices Architecture, a free O'Reilly book. Brought to you in partnership with Lightbend.

Topics:

Published at DZone with permission of A. Jesse Jiryu Davis, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

THE DZONE NEWSLETTER

Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

X

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}