A Deep Discussion of the Joyent Cloud API
Join the DZone community and get the full member experience.
Join For FreeThe following article was written by Grig Gheorghiu on his blog, Agile Testing.
Here's some notes I took while doing some initial experiments with provisioning machines in the Joyent Cloud. I used their CloudAPI directly, although in the future I also want to try the libcloud
Joyent driver. The promise of the Joyent Cloud 'SmartMachines' is that
they are really Solaris zones running on a SmartOS host, and that gives
you more performance (especially I/O performance) than regular virtual
machines such as the ones offered by most cloud vendors. I have yet to
fully verify this performance increase, but it's next on my TODO list.
Installing the Joyent CloudAPI tools
I did the following on an Ubuntu 10.04 server:
- installed node.js -- I downloaded it in tar.gz format from http://nodejs.org/dist/v0.6.19/node-v0.6.19.tar.gz then I ran the usual './configure; make; make install'
- installed the Joyent smartdc node package by runing 'npm install smartdc -g'
- created new ssh RSA keypair: id_rsa_joyentapi (private key) and id_rsa_joyentapi.pub (public key)
- ran the sdc-setup utility, pointing it to the US-EAST-1 region:
# sdc-setup https://us-east-1.api.joyentcloud.com Username (login): (root) myjoyentusername Password: The following keys exist in SmartDataCenter: [1] grig Would you like to use an existing key? (yes) no SSH public key: (/root/.ssh/id_rsa.pub) /root/.ssh/id_rsa_joyentapi.pub If you set these environment variables, your life will be easier: export SDC_CLI_URL=https://us-east-1.api.joyentcloud.com export SDC_CLI_ACCOUNT=myjoyentusername export SDC_CLI_KEY_ID=id_rsa_joyentapi export SDC_CLI_IDENTITY=/root/.ssh/id_rsa_joyentapi
- added recommended environment variables (above) to .bash_profile, sourced the file
# sdc-listdatacenters { "us-east-1": "https://us-east-1.api.joyentcloud.com", "us-west-1": "https://us-west-1.api.joyentcloud.com", "us-sw-1": "https://us-sw-1.api.joyentcloud.com" }
# sdc-listdatasets [ { "id": "988c2f4e-4314-11e1-8dc3-2bc6d58f4be2", "urn": "sdc:sdc:centos-5.7:1.2.1", "name": "centos-5.7", "os": "linux", "type": "virtualmachine", "description": "Centos 5.7 VM 1.2.1", "default": false, "requirements": {}, "version": "1.2.1", "created": "2012-02-14T05:53:49+00:00" }, { "id": "e4cd7b9e-4330-11e1-81cf-3bb50a972bda", "urn": "sdc:sdc:centos-6:1.0.1", "name": "centos-6", "os": "linux", "type": "virtualmachine", "description": "Centos 6 VM 1.0.1", "default": false, "requirements": {}, "version": "1.0.1", "created": "2012-02-15T20:04:18+00:00" }, { "id": "a9380908-ea0e-11e0-aeee-4ba794c83c33", "urn": "sdc:sdc:percona:1.0.7", "name": "percona", "os": "smartos", "type": "smartmachine", "description": "Percona SmartMachine", "default": false, "requirements": {}, "version": "1.0.7", "created": "2012-02-13T19:24:17+00:00" }, etc
# sdc-listpackages [ { "name": "Large 16GB", "memory": 16384, "disk": 491520, "vcpus": 3, "swap": 32768,Cloud Analytics API "default": false }, { "name": "XL 32GB", "memory": 32768, "disk": 778240, "vcpus": 4, "swap": 65536, "default": false }, { "name": "XXL 48GB", "memory": 49152, "disk": 1048576, "vcpus": 8, "swap": 98304, "default": false }, { "name": "Small 1GB", "memory": 1024, "disk": 30720, "vcpus": 1, "swap": 2048, "default": true }, etc
# sdc-createmachine --dataset sdc:sdc:percona:1.3.9 --package "Large 16GB" { "id": "7ccc739e-c323-497a-88df-898dc358ea40", "name": "a0e7314", "type": "smartmachine", "state": "provisioning", "dataset": "sdc:sdc:percona:1.3.9", "ips": [ "A.B.C.D", "X.Y.Z.W" ], "memory": 16384, "disk": 491520, "metadata": { "credentials": { "root": "", "admin": "", "mysql": "" } }, "created": "2012-06-07T17:55:29+00:00", "updated": "2012-06-07T17:55:30+00:00" }
# sdc-createmachine --dataset sdc:sdc:ubuntu-10.04:1.0.1 --package "Small 1GB" --name ggtest { "id": "dc856044-7895-4a52-bfee-35b404061920", "name": "ggtest", "type": "virtualmachine", "state": "provisioning", "dataset": "sdc:sdc:ubuntu-10.04:1.0.1", "ips": [ "A1.B1.C1.D1", "X1.Y1.Z1.W1" ], "memory": 1024, "disk": 30720, "metadata": { "root_authorized_keys": "" }, "created": "2012-06-07T19:28:19+00:00", "updated": "2012-06-07T19:28:19+00:00" }
# sdc-listmachines [ { "id": "36b50e4c-88d2-4588-a974-11195fac000b", "name": "db01", "type": "smartmachine", "state": "running", "dataset": "sdc:sdc:percona:1.3.9", "ips": [ "A.B.C.D", "X.Y.Z.W" ], "memory": 16384, "disk": 491520, "metadata": {}, "created": "2012-06-04T18:03:18+00:00", "updated": "2012-06-07T00:39:20+00:00" }, { "id": "dc856044-7895-4a52-bfee-35b404061920", "name": "ggtest", "type": "virtualmachine", "state": "running", "dataset": "sdc:sdc:ubuntu-10.04:1.0.1", "ips": [ "A1.B1.C1.D1", "X1.Y1.Z1.W1" ], "memory": 1024, "disk": 30720, "metadata": { "root_authorized_keys": "" }, "created": "2012-06-07T19:30:29+00:00", "updated": "2012-06-07T19:30:38+00:00" }, ]
# sdc-createmachine --dataset sdc:sdc:ubuntu-10.04:1.0.1 --package "Small 1GB" --name ggtest2 --metadata user-script='hostname ggtest2; echo ggtest2 > /etc/hostname' { "id": "379c0cad-35bf-462a-b680-fc091c74061f", "name": "ggtest2", "type": "virtualmachine", "state": "provisioning", "dataset": "sdc:sdc:ubuntu-10.04:1.0.1", "ips": [ "A2.B2.C2.D2", "X2.Y2.Z2.W2" ], "memory": 1024, "disk": 30720, "metadata": { "user-script": "hostname ggtest2; echo ggtest2 > /etc/hostname", "root_authorized_keys": "" }, "created": "2012-06-08T23:17:44+00:00", "updated": "2012-06-08T23:17:44+00:00" }
- Disk I/O operations
- Kernel thread executions
- TCP connections established
- MySQL queries
- HTTP server operations
- System load average
- which metric to collect
- an optional predicate based on the metric's fields (e.g., only collect data from certain hosts, or data for certain operations)
- an optional decomposition based on the metric's fields (e.g., break down the results by server hostname)
- how frequently to aggregate data (e.g., every second, every hour, etc.)
- how much data to keep (e.g., 10 minutes' worth, 6 months' worth, etc.)
- other configuration options
# sdc-describeanalytics "metrics": [ { "module": "cpu", "stat": "thread_samples", "label": "thread samples", "interval": "interval", "fields": [ "zonename", "pid", "execname", "psargs", "ppid", "pexecname", "ppsargs", "subsecond" ], "unit": "samples" }, { "module": "cpu", "stat": "thread_executions", "label": "thread executions", "interval": "interval", "fields": [ "zonename", "pid", "execname", "psargs", "ppid", "pexecname", "ppsargs", "leavereason", "runtime", "subsecond" ], etc
# sdc-createinstrumentation -m fs -s logical_ops { "module": "fs", "stat": "logical_ops", "predicate": {}, "decomposition": [], "value-dimension": 1, "value-arity": "scalar", "enabled": true, "retention-time": 600, "idle-max": 3600, "transformations": {}, "nsources": 0, "granularity": 1, "persist-data": false, "crtime": 1340228876662, "value-scope": "interval", "id": "17", "uris": [ { "uri": "/myjoyentusername/analytics/instrumentations/17/value/raw", "name": "value_raw" } ] }
To list the instrumentations you have created so far, you use sdc-listinstrumentations:
# sdc-listinstrumentations [ { "module": "fs", "stat": "logical_ops", "predicate": {}, "decomposition": [], "value-dimension": 1, "value-arity": "scalar", "enabled": true, "retention-time": 600, "idle-max": 3600, "transformations": {}, "nsources": 2,/ "granularity": 1, "persist-data": false, "crtime": 1340228876662, "value-scope": "interval", "id": "17", "uris": [ { "uri": "/myjoyentusername/analytics/instrumentations/17/value/raw", "name": "value_raw" } ] } ]
# sdc-getinstrumentation -v 17 { "value": 1248, "transformations": {}, "start_time": 1340229361, "duration": 1, "end_time": 1340229362, "nsources": 2, "minreporting": 2, "requested_start_time": 1340229361, "requested_duration": 1, "requested_end_time": 1340229362 }
You can dig deeper into a specific metric by decomposing it by different fields, such as the application name. For example, to see filesystem logical operation by application name, you would call:
# sdc-createinstrumentation -m fs -s logical_ops --decomposition execname { "module": "fs", "stat": "logical_ops", "predicate": {}, "decomposition": [ "execname" ], "value-dimension": 2, "value-arity": "discrete-decomposition", "enabled": true, "retention-time": 600, "idle-max": 3600, "transformations": {}, "nsources": 0, "granularity": 1, "persist-data": false, "crtime": 1340231734049, "value-scope": "interval", "id": "18", "uris": [ { "uri": "/myjoyentusername/analytics/instrumentations/18/value/raw", "name": "value_raw" } ] }
# sdc-getinstrumentation -v 18 { "value": { "grep": 4, "ksh93": 5, "cron": 7, "gawk": 15, "svc.startd": 2, "mysqld": 163, "nscd": 27, "top": 159 }, "transformations": {}, "start_time": 1340231762, "duration": 1, "end_time": 1340231763, "nsources": 2, "minreporting": 2, "requested_start_time": 1340231762, "requested_duration": 1, "requested_end_time": 1340231763 }
Another useful technique is to isolate metrics pertaining to a specific host (or 'zonename' in Joyent parlance). For this, you need to specify a predicate that will filter only the host with a specific id (you can see the id of a host when you call sdc-listmachines). Here's an example that captures the CPU wait time for a Percona SmartMachine which I provisioned earlier:
# sdc-createinstrumentation -m cpu -s waittime -p '{"eq": ["zonename","36b50e4c-88d2-4588-a974-11195fac000b"]}' { "module": "cpu", "stat": "waittime", "predicate": { "eq": [ "zonename", "36b50e4c-88d2-4588-a974-11195fac000b" ] }, "decomposition": [], "value-dimension": 1, "value-arity": "scalar", "enabled": true, "retention-time": 600, "idle-max": 3600, "transformations": {}, "nsources": 0, "granularity": 1, "persist-data": false, "crtime": 1340232271092, "value-scope": "interval", "id": "19", "uris": [ { "uri": "/myjoyentusername/analytics/instrumentations/19/value/raw", "name": "value_raw" } ] }
You can combine decomposition with predicates. For example, here's how to create an instrumentation for CPU usage time decomposed by CPU mode (user, kernel):
# sdc-createinstrumentation -m cpu -s usage -n cpumode -p '{"eq": ["zonename","36b50e4c-88d2-4588-a974-11195fac000b"]}' { "module": "cpu", "stat": "usage", "predicate": { "eq": [ "zonename", "36b50e4c-88d2-4588-a974-11195fac000b" ] }, "decomposition": [ "cpumode" ], "value-dimension": 2, "value-arity": "discrete-decomposition", "enabled": true, "retention-time": 600, "idle-max": 3600, "transformations": {}, "nsources": 0, "granularity": 1, "persist-data": false, "crtime": 1340232361944, "value-scope": "point", "id": "20", "uris": [ { "uri": "/myjoyentusername/analytics/instrumentations/20/value/raw", "name": "value_raw" } ] }
# sdc-getinstrumentation -v 20 { "value": { "kernel": 24, "user": 28 }, "transformations": {}, "start_time": 1340232390, "duration": 1, "end_time": 1340232391, "nsources": 2, "minreporting": 2, "requested_start_time": 1340232390, "requested_duration": 1, "requested_end_time": 1340232391 }
# sdc-createinstrumentation -m mysql -s queries -p '{"eq": ["zonename","36b50e4c-88d2-4588-a974-11195fac000b"]}' { "module": "mysql", "stat": "queries", "predicate": { "eq": [ "zonename", "36b50e4c-88d2-4588-a974-11195fac000b" ] }, "decomposition": [], "value-dimension": 1, "value-arity": "scalar", "enabled": true, "retention-time": 600, "idle-max": 3600, "transformations": {}, "nsources": 0, "granularity": 1, "persist-data": false, "crtime": 1340232562361, "value-scope": "interval", "id": "22", "uris": [ { "uri": "/myjoyentusername/analytics/instrumentations/22/value/raw", "name": "value_raw" } ] }Overall, I found the Joyent Cloud API and its associated Analytics API fairly easy to use, once I got past some nomenclature quirks. I also want to mention that the support I got from Joyent was very, very good. Replies to questions regarding some of the topics I discussed here were given promptly and knowledgeably. My next step is gauging the performance of MySQL on a SmartMachine, when compared to a similar-sized instance running in the Amazon EC2 cloud. Stay tuned.
Opinions expressed by DZone contributors are their own.
Comments