Michael Maximilien ("Max") kicked off this month’s Cloud Foundry Community Advisory Board meeting with reflection of the 3 releases of Cloud Foundry that have occurred since last month's meeting - v178, v179 and v180.
One of the features of the recent upgrades is a move to the Ubuntu 14.04 "Trusty" stemcell.
Max pointed out that there have been some security issues with Ubuntu 10.04 "Lucid" stemcells, which this solves. This was a kernel exploit that would be an issue in multi-tenant environments where users are allowed to run untrusted code. Therefore, Max recommends that people should consider moving over to it.
James said that when they added container support to the Lucid stemcells they had to backport support for this into Lucid. This made for a very customized environment which has made things difficult.
It was attempted for several weeks to patch the security issue, said James. Since Trusty resolves this issues, they put work into moving to Trusty instead. James also strongly recommended people move to Trusty, because then the standard Canonical supported releases can be used. Lucid LTS support actually expires early next year, whereas Trusty LTS support will expire in late 2019.
The new Trusty stemcells have been certified to work with cf-release v178 and higher.
Max said that 3 weeks ago 3 IBMers went to Pivotal's offices, in addition to himself, Michael and the others that usually go there. This was to work on Docker integration with Cloud Foundry.
James said there is a video coming from Onsi Fakhouri (Product Manager of Diego) in the next few days, which demonstrates pushing an application into Cloud Foundry, which uses a Docker image. This is also tied into Loggregator and the Router. This Docker image based application is then scaled up to 300 instances in 40 seconds across 10 Cells. A Cell is essentially the new "DEA" (Droplet Execution Agent) in Diego, which each runs on a separate VM.
The video also shows 2 Cell VMs being killed and Diego re-balancing the lost containers across the remaining 8 Cells.
James said that Diego is still in its early stages and there are still gaps, but he is impressed with the progress that has been made - especially with integration with the Router and Loggregator.
For those not familiar with BOSH's Cloud Provider Interface (CPI) layer, here is an explanation from Matt Stine...
"BOSH was initially developed as the means of deploying and managing the Cloud Foundry PaaS platform, but it’s much more generally applicable. BOSH can today deploy any distributed system, unchanged, to any of several popular IaaS providers: VMware vSphere, VMware vCloud Director, Amazon Web Services, and OpenStack." - BOSH and Cloud API Compatibility by Matt Stine.
Max mentioned that Dmitriy Kalinin and the BOSH team have built a Diego CPI and he thinks the performance is very good. James said they did 40+ containers in a minute, but there was some lock contention that could be removed to further improve this.
This is similar to how BOSH-lite can spin up the components of Cloud Foundry, via cf-release, in Warden containers on a single VM. The difference is that Diego is a distributed system, so it can spread the components across many machines on an existing Diego installation. This also means that the running instances of components should be ephemeral and have no persistent state.
James said that many of the components of Cloud Foundry do not require persistent state. He also said that BOSH is compossible, so you can combine different CPIs into a single BOSH release. Therefore, you can use the Diego CPI for ephemeral components and another CPI for those that require persistent state.
James stressed that this is a working protocol. It is not ready for production, but has great potential.
Docker Service Broker
Ferran Rodenas ("Ferdy") from Pivotal explained about the new service broker, cf-containers-broker, that he has created.
He said that users of Cloud Foundry generally find it hard to deploy and use Cloud Foundry services. This new service broker allows for easily deploying services as Docker containers.
In the chat, Dr. Nic from Stark & Wayne said, "It took about 40 minutes last night to add etcd to Ferdy's broker. Most of the time was spent creating a Dockerfile using Ferdy's common base image and figuring out how to encode a picture."
You can read more about this in Ferdy's blog post, "Docker Service Broker for Cloud Foundry".
Regarding moving this to the Cloud Foundry Incubator, James said they plan to give it a month to gather feedback from the community. The benefit of it being in the Incubator, said James, is that anyone can sign the CLA and contribute. Otherwise contributing is a lot more complicated and may involve legal departments.
James asked if anyone had objections to provisionally putting multiple of these Docker service brokers into incubation. There were no objections.
Colin Humphreys from CloudCredo has created a new project that he calls Cloud Focker. It is a lightweight Cloud Foundry utilizing Docker containers that are designed to make it easier for developers to develop applications destined for Cloud Foundry.
Colin said that he liked the Micro-CF which came with Cloud Foundry version 1 and this is an even lighter weight Cloud Foundry that can run on your laptop. "It's all about fast feedback. It's about how quickly can a developer get feedback on their application. How do we give them production-like environments as quickly and easily as possible?" said Colin.
Buildpack development is also easier with Cloud Focker, Colin said.
"Cloud Flocker" was proposed as alternative name.
Static File Buildpack
Logging And Metrics
Alex Jackson from Pivotal spoke of the Loggregator single-point-of-failure being removed. When they did this, they added a local agent called Metron that collects all the logs from local application instances and is now starting to also collect metrics.
Initially they are supporting counter and gauge metrics. This was so Diego could begin using these metrics, which is the first component to do so. Alex said that they are also using Metron internally with the Loggregator component.
Metron has a /varz end-point, so it can act as a bridge to the existing varz and the Collector system. Therefore, said Alex, your components no longer have to deal with this, since Metron will deal with this for you.
There is a Go library called dropsonde to emit metrics to Metron.
The syslog aggregator has been adjusted for the VM Ops logs. Alex said they used to only forward vcap.*, but now you will get everything that is sent to syslog.
Additional debugging has been added to the CLI. When you set CF_TRACE=true you will now see additional information related to the Loggregator connection.
Future updates will include a fix to a bug related to syslog drains. After a period of time it ceases sending data to the syslog drain. This bug was introduced fairly recently, said Alex.
Alex said that they are also working on the metrics consumers (subscription endpoint), so that metrics can be streamed out of Cloud Foundry in the same way you can currently stream logs out of Loggregator.
Alex clarified, in response to a question from Dr.Nic, that this will be a different stream to the Loggregator output.
Greg Oehman from Pivotal gave the update on the CLI.
Since the last meeting there have been two releases.
Coming up are feature flags and environment variable groups (see below).
Greg said that they are working on a CI migration to use Go-CD.
CLI plugins are on the horizon. Greg said that he has not been involved in all of the conversations, but he has looked back through the history of conversations related to this. The new direction is being documented and it will be put out for comments. Work on this will begin quite soon and Greg said that they are hoping to see this in the next couple of months.
Greg said that in some areas of the CLI response messages are not clear or actionable. They are hoping to improve this to to be more consistently actionable for end-users.
Mark Kropf from Pivotal gave the Runtime updates and said that support is planned for vSphere in cf-release by adding vSphere templates.
Environment Variable Groups
Mark said that cluster administrators will be able to set environment variable groups. This is to set staging and runtime environment variables globally across all applications, such as a HTTP proxy, rather than relying on developers to always setting these every time.
A design document can be found here.
Guillaume Berche from Orange suggested that possibly the design document for Environment Variable Groups, hosted on Google Documents, was not properly sending emails to Pivotal authors when comments were posted. He had not seen responses on comments made since August 19th. Mark said he would review the document and make sure those comments were addressed.
Buildpack Dependency Mapping
Buildpack dependency mapping is a way for buildpacks to describe their dependencies, so that an operator can query the Cloud Controller API and get a list of what the buildpacks are providing. They will also be able to get a list of dependencies that the applications have consumed directly from the buildpacks. The goal is to allow Cloud Foundry cluster administrators to react to CVE (Common Vulnerabilities and Exposures) reports and quickly determine whether they are affected.
It was clarified that this is only for dependencies explicitly provided by buildpacks and will not include additional dependencies that the application defines itself.
Many people have commented on the proposal document for Process Types, said Mark. This is for formal support of process types in a Procfile when you push your application. Multiple types of application process types can be deployed with a single "cf push", rather than having to push multiple times.
By default the "web" type is used if no type is specified.
Other types will be used for background workers or cron jobs.
There is a Google Document which describes this and can be commented on.
Chris Ferris from IBM noted in the chat that all design documents can be found on GitHub.
With an upcoming feature Org managers will be able to install and set the detection order of buildpacks specific to their Org. This is in the backlog and Pivotal are working on a proposal retroactively, since this was submitted from someone in the ecosystem, said Mark.
ActiveDirectory Group Mapping
Mark said that the Runtime team will be supporting the Identity (UAA) team to get tight integration with ActiveDirectory so that membership of an ActiveDirectory group can be mapped to a specific membership within a Org and Space or to a role in an Org and Space.
CLI Spending Prompts
James Bayer from Pivotal said that feedback from Pivotal Web Services and IBM's BlueMix is that it is easy for end-users of public PaaS to consume resources via the CLI without realizing the costs associated with this.
It has been proposed that prompts and warnings be added to the CLI workflow, so that there are no surprises when end-users receive their bill.
The API already exposes whether a service is paid or free, so it would involve exposing this is the CLI.
James said that the Notifications team have made good progress. They are now integrating with the UAA team for switching over to using the Notifications API for sending things like "forgotten password" messages to users.
If you are a Service Broker author and you want to send messages to consumers of your services, then the Notifications API is the standard way to do that, said James.
CF Summit Europe
Lukas from Swisscom said in the chat that Thursday August 28th will be a first round-table-call about a potential "CloudFoundry Summit in Europe". Hopefully that went well :)
The next Cloud Foundry Community Advisory Board Meeting will be Wednesday, September 25th at 8am PST for 90 minutes.
Chat room: http://webconf.soaphub.org/conf/room/CF-CAB-OpenForum
US Toll Free: 1-888-426-6840
Participant Code: 87517362
All other countries can find dialin numbers here: http://goo.gl/RnNfc1