DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Last call! Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • A Guide to Maven 3 Beta

Trending

  • Enhancing Security With ZTNA in Hybrid and Multi-Cloud Deployments
  • Understanding and Mitigating IP Spoofing Attacks
  • Using Python Libraries in Java
  • Customer 360: Fraud Detection in Fintech With PySpark and ML

Motor 0.5 Beta: Simple Aggregation, Asyncio, Async, and Await

Today is a good day: I've published a beta of Motor, my async Python driver for MongoDB. This version is the biggest upgrade yet. Motor can now integrate with asyncio, as an alternative to Tornado.

By 
A. Jesse Jiryu Davis user avatar
A. Jesse Jiryu Davis
·
Nov. 16, 15 · Tutorial
Likes (1)
Comment
Save
Tweet
Share
4.9K Views

Join the DZone community and get the full member experience.

Join For Free

Today is a good day: I've published a beta of Motor, my async Python driver for MongoDB. This version is the biggest upgrade yet. Help me beta-test it! Install with:

python -m pip install --pre motor==0.5b0

Motor 0.5 still depends on PyMongo 2.8.0 exactly. That PyMongo version is outdated, I know, but I've decided not to tackle that issue right now.

You'll forgive me, because this Motor release is huge:

  • Asyncio
  • Aggregate
  • Python 3.5
  • Async and await

Asyncio

Motor can now integrate with asyncio, as an alternative to Tornado. My gratitude to Rémi Jolin, Andrew Svetlov, and Nikolay Novik for their huge contributions to Motor's asyncio integration.

The Tornado and asyncio APIs are kindred. Here is Motor with Tornado:

# Tornado API
from tornado import gen, ioloop
from motor.motor_tornado import MotorClient

@gen.coroutine
def f():
    result = yield client.db.collection.insert({'_id': 1})
    print(result)

client = MotorClient()
ioloop.IOLoop.current().run_sync(f)

And here's the new asyncio integration:

import asyncio
from motor.motor_asyncio import AsyncIOMotorClient

@asyncio.coroutine
def f():
    result = yield from client.db.collection.insert({'_id': 1})
    print(result)

client = AsyncIOMotorClient()
asyncio.get_event_loop().run_until_complete(f())

Unlike Tornado, asyncio does not include an HTTP implementation, much less a web framework. For those features, use Andrew Svetlov's aiohttp package. I wrote you a tiny example web application with Motor and aiohttp.

Aggregate

MotorCollection.aggregate now returns a cursor by default, and the cursor is returned immediately without a yield. The old syntax is no longer supported:

# Motor 0.4 and older, no longer supported.
cursor = yield collection.aggregate(pipeline, cursor={})
while (yield cursor.fetch_next):
    doc = cursor.next_object()
    print(doc)

In Motor 0.5, simply do:

# Motor 0.5: no "cursor={}", no "yield".
cursor = collection.aggregate(pipeline)
while (yield cursor.fetch_next):
    doc = cursor.next_object()
    print(doc)

In asyncio this uses yield from instead:

# Motor 0.5 with asyncio.
cursor = collection.aggregate(pipeline)
while (yield from cursor.fetch_next):
    doc = cursor.next_object()
    print(doc)

Python 3.5

Motor is now compatible with Python 3.5, which required some effort. It was hard because Motor doesn't just work with your coroutines, it uses coroutines internally to implement some of its own features, likeMotorClient.open and MotorGridFS.put. I had a method for writing coroutines that worked in Python 2.6 through 3.4, but 3.5 finally broke it. There is no single way to return a value from a Python 3.5 native coroutine or a Python 2 generator-based coroutine, so all Motor internal coroutines that return values were rewritten with callbacks. (See commit message dc19418c for an explanation.)

Async and Await

This is the payoff for my Python 3.5 effort. Motor works with native coroutines, written with the async and await syntax:

async def f():
    await collection.insert({'_id': 1})

Cursors from MotorCollection.find, MotorCollection.aggregate, orMotorGridFS.find can be iterated elegantly and very efficiently in native coroutines with async for:

async def f():
    async for doc in collection.find():
        print(doc)

How efficient is this? For a collection with 10,000 documents, this old-style code takes 0.14 seconds on my system:

# Motor 0.5 with Tornado.
@gen.coroutine
def f():
    cursor = collection.find()
    while (yield cursor.fetch_next):
        doc = cursor.next_object()
        print(doc)

The following code, which simply replaces gen.coroutine and yield withasync and await, performs about the same:

# Motor 0.5 with Tornado, using async and await.
async def f():
    cursor = collection.find()
    while (await cursor.fetch_next):
        doc = cursor.next_object()
        print(doc)

But with async for it takes 0.04 seconds, three times faster!

# Motor 0.5 with Tornado, using async for.
async def f():
    cursor = collection.find()
    async for doc in cursor:
        print(doc)

However, MotorCursor's to_list still reigns:

# Motor 0.5 with Tornado, using to_list.
async def f():
    cursor = collection.find()
    docs = await cursor.to_list(length=100)
    while docs:
        for doc in docs:
            print(doc)
        docs = await cursor.to_list(length=100)

The function with to_list is twice as fast as async for, but it's ungraceful and requires you to choose a chunk size. I think that async for is stylish, and fast enough for most uses.

Try Me!

I haven't always published betas before Motor releases, but this time is different. The asyncio integration is brand new. And since it required pervasive refactoring of Motor's core, the existing Tornado integration is rewritten as well. Python 3.5 support required yet another internal overhaul. I'm anxious to get early reports of all my new code in the wild.

Additionally, the change to aggregate is an API break. (There are also two subtler changes, see the changelog.) So I'm giving you a chance to opt in explicitly with pip install --pre before I make Motor 0.5 official.

So please: try it out! Install the beta:

python -m pip install --pre motor==0.5b0

Test your application with the new code. If you find issues, file a bug and I'll respond promptly. And if the beta goes smoothly, don't be silent!—tweet at me @jessejiryudavis and tell me! It's the only way I'll know the beta is working for you.

BETA (programming language)

Opinions expressed by DZone contributors are their own.

Related

  • A Guide to Maven 3 Beta

Partner Resources

×

Comments

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: