Matt Butcher

The JSON/HTTP model may not be the best fit for IoT technologies.

Let's ditch Go for a little and see what we can conjure with a little Rust.

You've been writing Go. But you're feeling an urge to test the waters with Rust. This is a guide to make this switch easy.

It is possible to disable HTTP access on S3 bucket, limiting S3 traffic to only HTTPS requests. The documentation is scattered around the Amazon AWS documentation, but the solution is actually straightforward. All you need to do to block HTTP traffic on an S3 bucket is add a Condition in your bucket's policy. AWS supports a global condition for verifying SSL. So you can add a condition like this: "Condition": { "Bool": { "aws:SecureTransport": "true" } } Here's a complete example: { "Version": "2008-10-17", "Id": "some_policy", "Statement": [ { "Sid": "AddPerm", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "s3:GetObject", "Resource": "arn:aws:s3:::my_bucket/*", "Condition": { "Bool": { "aws:SecureTransport": "true" } } } ] } Now accessing the contents of my_bucket over HTTP will produce a 403 error, while using HTTPS will work fine.

I've been hunting for good database tools to perform that class of tasks that we all need, but that we end up re-implementing over and over again. One such task is database migrations. I've been experimenting with Goose to provide general-purpose database migration support. What Is Goose? Goose is a general purpose database migration manager. The idea is simple: You provide SQL schema files that follow a particular naming convention You provide a simple dbconf.yml file that tells Goose how to connect to your various databases Goose provides you simple tools to upgrade (goose up), check on (goose status), and even revert (goose down) schema changes. Goose does this by adding one more table inside your database. This table tracks which schema changes it has made. Based on its history, it can tell which scheme updates need to be run and which have already been run. While Goose is written in Go (golang), it is agnostic about what language your app is written in. Getting Started I got Goose up and running in less than 30 minutes, and you can probably do it faster. I already have an empty Postgres database called foo. But it has no tables. I have an existing codebase, too (MyProject). Here is the process for configuring Goose to manage the database schema management. First, I create the db/ directory, which will house all of the Goose-specific files, including my schema. $ cd MyProject $ mkdir db $ cd db $ vim dbconf.yml # Open with the editor of your choice. The dbconf.yml file contains a list of databases along with the relevant information for connecting to each. Mine looks something like this: test: driver: postgres open: user=foo dbname=foo_test sslmode=disable development: driver: postgres open: user=foo dbname=foo_dev sslmode=disable (Important: use spaces, not tabs, in YAML.) Now I have two databases configured. One for testing and one for development. By default, Goose assumes the target database is development. The above is just configured to connect to the PostgreSQL instance locally running. If I need support for a remote host, I can add host=... password=... (and remove sslmode=disable). At this point, I can generate a new migration. $ cd .. # Back to MyProject/, not in db/ $ goose create NewSchema sql goose: created db/migrations/20140311133014_NewSchema.sql $ vim db/migrations/20140311133014_NewSchema.sql # Use whatever editor you like Notice that the goose create command will create a new SQL file that follows Goose's naming convention. (That trailing sql on the command is important. goose create can also generate go migration files) My new schema file has two sections: a section for goose up and a section to rollback with goose down: -- +goose Up CREATE TABLE foo ( -- ... ); -- +goose Down DROP TABLE foo; With that done, I can now very easily create by development database: $ goose up If I want to setup test instead, I use the -env flag: $ goose -env=test up And that's it! In subsequent schema files, I may ALTER existing tables or CREATE new ones, and so on. Just about anything that your SQL engine can execute can be passed through Goose. (Though there are some formatting annotations you need to use for things like stored procedures.) Goose Pros In addition to the general ease of use of Goose, here are some additional features that I really like: You do not need your entire codebase to execute Goose. Our deployment box, for example, only has the Goose db/ directory, not the rest of the code. It is largely language neutral if you're just migrating SQL. It works with PostgreSQL, MySQL, and SQLite. The history table that it creates is human-readable, which makes it easy for me to see what's been going on. It supports environment variable interpolation. Don't want your password inside the dbconf.yml file? Just do something like this: development: driver: postgres open: user=foo dbname=foo_dev sslmode=disable password=$MY_DB_PASSWORD This will cause Goose to check the environment for a variable named $MY_DB_PASSWORD. Goose Cons Honestly, I have very few. Right now, you need the Go runtime to install and build Goose. Of course, you can compile Goose once, and then use it wherever. While it has support for Go language migrations, it would be nice to be able to write migration scripts that are executed via the shell. That way, one could use Bash, Python, Perl, or whatever else to trigger migrations. But, hey... this is a pretty minor complaint. Overall, though, Goose is a fantastic tool for handling migrations with ease.

[Matt Butcher is a topic expert featured in the DZone 2014 Cloud Platform Research Report, which you can download for free.] Ask a cloud-savvy developer what PaaS is, and you will get an answer like this: A PaaS is a cloud service that lets developers deploy applications into the cloud without having to manage the underlying infrastructure layer. A year or two ago, PaaS systems were monolithic. A single vendor or solution, like Heroku, would provide one system that handled all aspects of PaaS. But things are changing. With a plethora of Open Source tools like Docker, Packer, Serf, CoreOS, Dokku, and Flynn, it is now possible to build your own PaaS. But what exactly makes up a PaaS? I will take a functional approach to defining PaaS by asking what are the things that a PaaS does? PaaS can be viewed as a workflow with several functional phases. Each phase accomplishes a specific goal in the process of moving an application onto a production platform. The phases are not necessarily serial steps. They may run in parallel, and not in the order listed below. The five functional phases of a PaaS are: Deployment Provisioning Lifecycle management Service management Reporting 1. Deployment The deployment phase is responsible for moving an application from its source (typically a developer's machine) to the PaaS. Some of the common ways of doing this include: Running a git remote on the PaaS and handling git push events from clients. (Heroku, OpenShift, Flynn and Dokku all use this method. Elastic Beanstalk uses a variation on this.) Sending the code as a bundle (often a gzipped tar). Cloud Foundry uses this method, as does Stackato. Compiling the code locally and copying the resulting executable to the PaaS. When a PaaS receives a deployment, it kicks off processes to move that app into a running state. The exact order of those processes varies, so I will keep them in the order in which they appeared above. 2. Provisioning In the provisioning phase, the PaaS sets up the infrastructure necessary for running the app. "Infrastructure" is a broad and sometimes nebulous term, but here are some common provisioning targets: Setting up containers and/or compute instances Configuring networking Installing or configuring operating system services (e.g. Apache) Installing or configuring libraries (e.g. Ruby Gems) Many PaaS systems spread provisioning responsibilities across multiple tools. One tool may create a compute instance, while another tool may install libraries. But all are sharing the same responsibility: create the environment in which the application will run. 3. Lifecycle Management Once the PaaS has a copy of the app as well as an environment capable of running the app, it needs to manage the execution of the app. This is lifecycle management. Common tasks of lifecycle management include: Starting the app Monitoring the app's running state Monitoring or reporting on the app's resource consumption Restarting an app upon failure Stopping or restarting the app on command Some minimal PaaS systems offer only basic lifecycle management (e.g. start and stop), while highly sophisticated ones may include autoscaling, auto-throttling, and hot (zero-downtime) deployments. 4. Service Management This phase is not one that all PaaS layers perform. In fact, I would go so far as to say that it is not a mandatory piece of PaaS, But it certainly is useful when present. All PaaS systems run applications (that is, after all, what they're for). But some go a step beyond and provide services that may be attached to an application. These services run outside of the application container or compute instance. Services might include: Databases Networked file systems Message queues Caches Aggregated logging "Old guard" systems (like Cloud Foundry) share a service (e.g. MySQL) across multiple applications. Some of the newer container-based approaches like CoreOS may supplant this model by making it simpler to run services in specially-designated containers. (Check out the Serf project for a similar approach.) Why don't all PaaS systems need this layer? One reason is that many cloud providers already have comparable services in the form of DBaaS, MQaaS, and so on. 5. Reporting and Monitoring This final phase is the most banal. Most of the application's lifecycle is not spent on deployment or provisioning or service management. It's spent running. During an applications life, there are many interesting things that can occur. There are lifecycle events that we'd like to know about, like restarts. There are environmental conditions of interest, like resource utilization and system performance. And, of course, there is application data that we would like to monitor, like log files and application metrics. Many, but by no means all, PaaS platforms provide at least some level of reporting. Here are some examples: Amazon Elastic Beanstalk integrates with AWS Cloud Watch, and also aggregates system log files per application. ActiveState Stackato provides a web console with copious logs, and can show real-time statistics about an application and its surrounding environment. Heroku can optionally send events to a Loggly backend (which is a service). Conclusion: PaaS and Mini-PaaS As we've seen, each functional phase of PaaS can be done to greater or lesser degrees of complexity. Old guard PaaS systems often come feature-packed. But with PaaS building blocks like Docker, Flynn, and CoreOS, building a special purpose tailored mini-PaaS is not out of the question. Just take a look at Deis and Dokku for solutions with varying degrees of complexity.