This week we welcome Jeff Forcier (@bitprophet) as our PyDev of the Week. Jeff is the current maintainer of the popular Fabric and Paramiko packages. He is also the creator of the Invoke package. You can check out other projects that Jeff contributes to on GitHub. He also has a blog that you might find interesting. Let’s take some time to get to know Jeff better!
Can you tell us a little about yourself (hobbies, education, etc):
My overall bio can be found at http://bitprophet.org/bio, but here are some basics!
- I have a bachelor’s degree in computer science (though I don’t hold with those who think one is required for a career in this field).
- Most of my hobbies involve a computer or TV screen – for example, I’m a devout video game addict, spend more time than I should just surfing the Web, and enjoy watching videos (anime, movies, TV; no Twitch habit yet!)
- When not glued to a display, I read a lot (mostly sci-fi and fantasy); take day trips around the beautiful, if disturbingly expensive, San Francisco Bay Area; and annoy my cats.
- Sometimes I combine these things – such as when my wife and I go on long walks playing Pokémon Go
Why did you start using Python?
My final semester of university included a course on mobile computing, which in 2004 was VERY different from what that term means today! The professor was a Python advocate and had us write our projects in it, to run on Sharp Zaurus PDAs. My project was a wifi-based advertising/bulletin-board server/client platform, written entirely in Python.
Prior to this I’d written Java for classwork and PHP for fun/jobs, and found I enjoyed Python’s expressiveness – it ‘clicked’ with how my mind works. Post-graduation, I wrote intranet apps in Plone and then Django, which further cemented Python as my language of choice.
What other programming languages do you know and which is your favorite?
Prior to Python, I wrote Visual Basic, PHP, and Java; haven’t used them since. After learning Python I also picked up Ruby and Clojure.
I enjoy Ruby; my ideal language would be Python with some Ruby syntax (blocks, trailing logic, etc.) or Ruby with a more explicit import system and Python’s package ecosystem (i.e. not limited to mostly web and ops.) At the moment everything I’m working with is Python, so my Ruby is gathering dust.
Ruby and Python are more similar than distinct – so I employ Clojure for some personal projects. Being LISP, it stretches my brain and encourages me to work with paradigms I don’t use much in Python, but is modern enough to avoid most of the practical frustrations found in other LISPs.
What projects are you working on now?
Too many! Which is my own fault. A full list is at http://bitprophet.org/projects/.
Fabric 2 and its underlying Invoke project are my primary foci right now. My time’s spent eyeballing the requirements for their 2.0 and 1.0 releases, respectively; and personal itch-scratching feature work. I’m privileged to use a lot of this OSS at my day job, so I allow myself tangents when something annoys me, or my users, enough.
Strongly related: Paramiko, which has high-priority feature areas needing a serious redo (like auth handling) on top of the usual ‘missing’ features and fixes.
Finally, I’ve been porting my test suites to a pytest-based test runner; the old one I used was based on nose and pytest has gotten so much neat stuff in the past few years, I made the jump. Plus the rewrite allowed for quality-of-life improvements on top of simply “you can use other pytest plugins with it!”
My other projects (including Alabaster, the Sphinx theme) get maintenance work and the occasional sprint.
How did you end up becoming the maintainer of these project(s)?
In ~2008 I needed to do config management in Python and there weren’t many options at the time. Fabric was created by another dev (Christian) who based it on Capistrano from the Rails world; it was a single .py and (like early Capistrano) wasn’t very UNIX-y in behavior or invocation, at version 0.1.x.
I submitted a large (pre-GitHub) PR that turned it into a multi-file package, made it more like a traditional POSIX app, and added some features; as it turns out, Christian was at a turning point on his end and asked if I just wanted the project wholesale. I said yes. 0.9.x was born; and we’re now up to 1.13, with a full-rewrite 2.0 around the corner.
Paramiko has powered Fabric since the beginning; about 5 years into my Fabric stewardship, I forked Paramiko (around 1.7.7) to publish critical bugfixes Paramiko’s author (Robey) hadn’t time to merge. He eventually got back to me and asked if I wanted to take over the original. I said yes and merged my fork back in. Since then, I’ve published versions 1.7.7.x through 1.18, then a (non-rewrite) 2.0, 2.1, etc.
Alabaster started life as a (non-hostile) fork of Kenneth Reitz’s Sphinx theme, which in turn was a gentle rework of Armin Ronacher’s theme for his projects. Alabaster’s intent was to add a lot of configurability to the theme package, for example making it easier to use across multiple sites (as I do with most of my projects’ split www/docs sites.) It’s now the default Sphinx theme!
Invoke (née Fabric 2) is 100% original work, as is my barely-used-by-anyone-else test runner, spec (nose) / pytest-relaxed (pytest).
What challenges have you faced while developing/maintaining these popular packages?
The biggest challenge is simply scaling with the amount of activity the projects get. I’m able to work on them 1-2 days a week due to the arrangement with my day job, but as with most OSS, these projects really want more like 1-2 full-time developers.
Making matters worse are my tendencies towards micromanagement and pickiness – they’ve prevented me from giving out full commit/release privileges to others, which would help get more patches out more quickly.
That said, I have a handful of awesome folks who help with ticket triage/patch review; I still feel compelled to at least skim nearly every ticket, but seeing a trusted name in the comments makes it much easier to defer, close, or just-merge sometimes!
Backwards compatibility (something I take very, perhaps too, seriously – see “pickiness” above) introduces many challenges, not only around how to make certain changes but also how to handle them in changelogs, release branches, etc.
Finally, I’m a poster child for impostor syndrome; it’s amazing how you can have millions of PyPI downloads and still beat yourself up for being “bad at computers”!
What sorts of things can we look forward to in Fabric 2?
I have a nice handy list at http://docs.fabfile.org/en/v2/upgrading.html#why-upgrade, some of my personal favorites so far:
- Python 3 compatibility, finally.
- Explicit, single-responsibility-principle driven classes instead of functions referencing global module state, which brings a ton of other good stuff like “you can use threads instead of multiprocessing!”
- Massively expanded configuration system.
- POSIX/GNU style command-line flags for task arguments; no more weird custom stuff.
- Command/subprocess execution now way more powerful, more consistent between local/remote, and usable totally standalone (easy-to-use replacement for manual, stdlib subprocess based hacks!)
- Elegant nested SSH gateway support.
Which Python libraries are your favorite (core or 3rd party)?
In no particular order:
- requests and sphinx are near-ubiquitous, for good reason
- cryptography has been great to work with, and its split high/low level API approach is increasingly valuable as the programming world gets more security-savvy
- flake8 and coverage are invaluable for code insight
- humanfriendly (or tqdm), tabulate, and blessings all make for attractive user interfaces (spinners, progress bars, tables, colors) – a feature area I’m increasingly utilizing
- pathlib, which I should use more than I do
- httpie is amazing for introspecting anything web/http based on the CLI
Where do you see Python going as a programming language?
- Execution speed: Golang is hard to beat, but PyPy’s doing good work, and optional typing in Python 3 can only help (also just for those who like typing, period).
- Deploying code: Python justifiably gets a bad rap compared to “copy a static binary,” but the Packaging Authority is doing awesome work and there’s more coming. Wheels are already a huge boon over sdists or eggs.
- Managing dependencies: npm has the benefit of being a next-gen system, with the bells and whistles that entails; but again, the PyPA does great work and I’m excited for upcoming changes (e.g. Pipfile).
- Async: an increasingly common need in our multi-core age; Golang and JS have strong async paradigms, but Python 3.5+ has made solid gains in native async syntax and hopefully will keep improving (plus the built-in syntax is only one option!).
Not a comparison to another language, but: I think helping programmers write secure programs by default is rapidly increasing in importance. Python has an opportunity to be a leader here, given our community is blessed with no few security domain experts, and tends to encourage APIs which do the right thing by default while still being flexible.
What is your take on the current market for Python programmers?
It’s hard for me to be objective here, my resumé and overall visibility (presumably) means I get more recruiters contacting me than average – but as far as I can tell the market is doing quite well, at least as well as the overall tech sector and possibly more so.
I think Pythonistas who have, or can get, a leg up on the interfaces between Python and the current hot topics (containers, functions-as-a-service, machine learning, etc.) will get an edge when it comes to job opportunities. Take it from someone who lags behind on such topics!
Thanks for doing the interview!