If you follow-along on the tech blog you’ve seen the news that we opened our Python driver this week. We’re pretty excited. From the response so far, you are too! If you’ve actually gone and pulled the source, however, you may have noticed that there’s more than just SQL support in there. There are also APIs for domain management.
The SQL driver builds on core code that we started developing over a year ago to automate tests and tools in-house. What you can see today is more a prototype than what we plan to support for the product, so expect the APIs to change in the next few months. With that in mind, I don’t want to do an exhaustive guide to programming with these modules. I do, however, want to give y’all a sense of what you can do today and where we’re going..
Part of what NuoDB provides out of the box is a simple provisioning and management model. The agents running on each host provide basic monitoring, process management and automation capabilities. This is what the web console and command line manager tool builds on.
Management commands in the current version of the product are simple XML messages sent over a mutually-authenticated and encrypted TCP session. Yes, yes, we’re working on REST interfaces and we’ll definitely post when they’re available. In the meantime, what we use internally to drive all our testing frameworks are these python interfaces.
Note that the examples here can all be run interactively from a python shell or scripted. As long as you run the install step from the python getting started entry and have NuoDB installed locally you can follow-along at home.
Get started: view the domain
What can these modules do? Well, for one, they give you the ability to see what’s happing across all of your hosts (what we call a Domain). To monitor and manage a domain, the first step is to connect to any of the available brokers. Let’s say you have a broker running on the localhost with the default properties. To connect as the user “domain” with the password “bird” you say:
from pynuodb.entity import Domain d = Domain('localhost', 'domain', 'bird')
That’s it. You’ve established a management connection. The first thing you might want to do is see all of the available hosts (referred to as “peers” in this version of the APIs):
for peer in d.getPeers(): print str(peer)
That will print out all of the hosts in a format like this:
The Domain class also has a getDatabases() routine you can use to see all of the running databases. Both getDatabases() and getPeers() return class instances. You can see the definition for these in the entity module. Both of these classes let you get the associated processes. So, to print out all the running processes by database you’d say:
for db in d.getDatabases(): print str(db) for p in db.getProcesses(): print str(p)
Pretty simple, huh? Of course, processes (Transaction Engines or Storage Managers) here are also represented by a class. If you want to get more detail about that process, or manage it in any way, that’s the starting-point.
Next step: start some processes
The above examples give you the ability to write simple view and monitoring tools. What if you want to actually automate the creation of a database? You start in the same way, by connecting to a broker. Then you issue commands to specific peers.
Let’s take a simple example. Suppose you’re on your laptop and you’ve just got a local broker running. Connect to it with the Domain() class, and then ask for the “entry peer”:
broker = d.getEntryPeer()
What you got is a reference to the peer that you used to “enter” the domain. In other words, the local broker. To start the simplest database you need a single Storage Manager and Transaction Engine:
# start the SM, initializing a new archive directory broker.startStorageManager('teas', '/tmp/teas-data', True) # start the TE with an initial user 'dba' (password is 'dba') dbaOptions = [('--dba-user', 'dba'), ('--dba-password', 'dba')] broker.startTransactionEngine('teas', options=dbaOptions)
You’ve now scripted the startup of a fully ACID database on your localhost named “teas”. Yes, I like tea. A lot. Building on the above example to print out the processes what you’ll see is something like this:
teas localhost:50450 [pid=48802] (TE) localhost:50439 [pid=48801] (SM)
Note that you didn’t have to start these on the localhost. These interfaces let you connect to any broker, and then issue commands to any host in the domain, all from your local system. This makes it really easy to setup tests, automation and simple tools to help you be more productive with our database!
Starting databases and seeing that they’ve started is pretty useful, but you may want to have something always monitoring the state of your domain. To do this, you’ll want to start by dropping your current connection:
We’re going to re-connect, but this time we’re going to supply a listener class to get notified about key events. For instance, let’s say we want to know every time a process starts or stops. Here’s the class we’d write:
class ProcessListener: def processJoined(self, p): print str(p) def processLeft(self, p): print str(p)
Now that you’ve got that class defined, re-connect to the domain as before but provide an instance of your new listener:
d = Domain('localhost', 'domain', 'bird', ProcessListener())
This time when you connected you should have immediately seen your TE and SM from above printed out. That’s because on connection you got event notifications for what’s already running. If you added another TE or SM, from any management client attached to any broker, you’ll get notified here. See the header comment in the entity module for details about all the events that are supported in the current code.
These kinds of simple events make automation pretty easy. For instance, you now know enough to write a script that reacts to process failure by picking an available host and re-starting a process there. You also know how to expand the transactional layer of a database on-demand.
This is obviously a pretty simple introduction into what we can do in the management layer of our system. Like I said at the start, this is also just a first-version of management interfaces for python. There’s a lot of functionality that we haven’t exposed at this layer yet but that we’ll be adding as the summer moves on.
We’re also in the process now of expanding our management features to make things like database startup a lot easier. Look for more details on this blog in the coming weeks … for now I’ll just say that it’s going to give you a simple way to describe what you want and then start databases of known classes instead of process-by-process. If that seems interesting to you then I hope you’ll take what we have today, play around a little and give us feedback about where you’d like us to take this!