This week's CAB call started with Dr. Max introducing Chip Childers. Chip started by saying "Register for the Summit! Register for the Summit! Register for the Summit!". He was of course talking about the CF Summit, which is occurring 11th to 12th May in Santa Clara. He said if anyone has any difficulties with coming, such as financial or logistic, they are happy to try to find ways to help.
Chip said they are working getting more structured governance around code contribution and getting the road-map planning in place. He said it will take time and they are working the commitments that were made between the members to each other for how the projects are structured, governed and how new contributions can be managed. This is through the Dojo program, as well as just good coordination in situations where people develop new features and submit pull requests.
First PMC Council Meeting
The first official PMC (Project Management Committee) Council meeting, which will include the chair of each project management committee (e.g. BOSH, Runtime, Servers, Utilities), will take place on April 9th. There are a lot items on the agenda for Foundation-wide topics that need to be addressed. This includes...
- Intellectual property management
- Legal documentation
- Policies that have to be in-place across all the projects
- How will security be handled more formally as a Foundation? (accepting security vulnerabilities, updates for upstream projects)
Core Services - MySQL and Riak CS
Marco Nicosia recently joined Pivotal to take over the core servers from Shannon Coen. This includes the MySQL and Riak CS services.
In the most recent release of cf-mysql Marco says they released a new proxy layer called switchboard which is written at Pivotal as open-source software, in Go. This helps achieve the critical balancing features that allow the applications to adjust to server downtime without interruption.
The Riak CS service includes some updated pieces from Basho which they hope will address a couple of significant customer bugs.
Shannon Coen from Pivotal gave the update on platform features around services. He said the most significant work has been around continued work for asynchronous service operations. Shannon conceded that this has taken a long time, but said that they are now close to the finish line for an MVP. They plan to publish documentation soon. They hope people will try it out and give feedback.
Shannon said it is worth noting that IBM has been working on support for service keys. This will allow developers to create credentials for service instances in a standalone manner (without requiring an application). They are leveraging the bind endpoints in the broker API, but there are new endpoints in the Cloud Controller API. The difference between service keys and bindings is that service keys do not require an application.
I dropped off the call here and missed 30 seconds or so. Please let me know if I missed anything significant
James Bayer from Pivotal gave an update on Loggregator, saying that a lot of work has been related to scalability and robustness. He said Mike Youngstrom of the LDS Church told him that his experience using the latest Loggregator code from cf-release since v200 has been successful and has resolved the occasional crash they were seeing.
statsd support has been added for the Loggregator components themselves to communicate their metrics via statsd. This is currently going through acceptance testing and they would like get this into cf-release this week. They would like the Cloud Controller to start using this mechanism instead of the collector mechanism.
A question was asked as to where the statsd work is implemented and where is it exposed. James replied that the way it works currently is that the components expose a HTTP interface called /varz and the collector would poll /varz and retrieve the metrics. statsd is more of a push mechanism, rather than a pull. The components can be emitting the metrics themselves via the statsd protocol. The system is then able to propagate those metrics downstream.
A follow-up question was "Is Loggregator involved in shipping it, or is Loggregator just using statsd now?". James stressed that this is not exposed to developers and it is "inside the blackbox of Cloud Foundry".
Dr. Max from IBM gave the update on the command-line client from the report Greg Oehman from Pivotal had sent to him.
- Work has been done on stacks to support both the Lucid and Trusty filesystems. Now you can do cf stacks to show the stacks available. The cf app command also shows the stack being used for an application.
- There is new plugin cf-app-stack-changer that automates the migration from the Lucid to the Trusty filesystem.
- A series of plugins have been added to cli-plugin-repo. These include Brooklyn, for interacting with the Service Broker for Apache Brooklyn and kibana-me-logs, for launching the Kibana UI for an application..
- The CLI is now using the NOAA API for Loggregator.
- It is now possible to set multiple domains within an application manifest file.
- A pull request was merged for viewing Security Group Rules.
- Another pull request was merged for viewing Shared Private Domains.
Sree Tummidi, the product manager for UAA at Pivotal, gave an update on the UAA.
She said two weeks prior they released the multi-tenancy version of UAA. This included the documentation for the updated endpoints. They hope to do another release this week that includes further documentation and dynamic configuration of LDAP endpoints. They plan to continue to add support for SAML. Currently they do not support trusting of SAML groups or claims about the user. This will be their next focus.
Dmitry Kalinin from Pivotal gave the BOSH update. He said they have recently been working on a few internal improvements and a few UX improvements. They have started several tracks on trying to support dynamic provisioning via BOSH. This means that when the end-user requests services, the service brokers will call to BOSH and BOSH will provision new VMs on-the-fly. To support this work they needed to extract the IaaS configuration as a separate configuration file. They have called this "cloud-config" and it has several stories in the BOSH backlog (search label:"cloud-config").
There will be a separate CLI command bosh cloud-config. For the deployment manifest this means that all IaaS integration is extracted out into a separate file. This includes resource pools and networking. They will be sharing more information about this through the bosh-notes repository. This repository was created recently to enable clearer explanation and easier-to-share BOSH proposals.
Multi-region AWS support was recently completed for stemcells.
vSphere error messaging has been improved. This was to help debug why VMs would not start up in certain areas. While they were doing this they also changed the agent networking to be more generic. This means the agent no longer knows which infrastructure is associated with which networking type. Dmitry said you might see that on AWS the agent switches to using manual or static networking on the VMs even though AWS always provides DHCP. Although, this does make the networking configuration more consistent throughout all the CPIs.
They have externalized the vSphere API as part of the external CPIs work, which is ongoing. This was the last CPI that had to be changed. That was done after removing the database from the vSphere API. There were a couple of regressions after that work was completed, but they have recently fixed those bugs.
The BOSH-lite pipeline has been updated. They have switched to Concourse CI. This has made it a lot easier to produce new builds for BOSH-lite. There is a link in the BOSH-lite README to the CI pipeline.
Dr. Max mentioned that Concourse is a project that Alex Surachi at Pivotal created. "It's a really cool open-source project for CI".
On the chat, Mike Youngstrom from the LDS Church asked "What are the possibilities of upgrading vsphere stemcells to hardware version 8?". Dmitry said he will get back to Mike on the mailing-list once he has more information.
v205 included all of the new buildpacks. This brought the size of cf-release to about 5.1Gb. Michael Dalessio from Pivotal posted some proposals on the mailing-list as to how this could be reduced.
v206 will be released very soon (if not before this post goes live). This addresses an issue that developers experience when switching stacks. This relates to compiling native gems and the buildpack cache.
Work is continuing on process types. More information will be shared on the patterns being used on vcap-dev in the next couple of weeks.
A spike has just been finished on Route Services to prove the concept of forwarding traffic to an application as a reverse-proxy before going to your actual application. On Pivotal's production environment this resulted in a 20ms overhead, which they deemed acceptable. They will now proceed with the actual implementation.
Mike Dalessio from Pivotal gave the Buildpacks update. He said the Buildpacks team have released buildpacks which support both root filesystems; lucid64 and cflinuxfs2.
Work was done to bring the PHP buildpack inline with the tool-chain they are using for building other buildpacks. Now, all the buildpacks are using a manifest file to declare what they package. There are a couple of common tools to make sure the buildpacks are behaving the same way. This is part of the work towards unifying the common behaviors of buildpacks.
There is new repository to do buildpack runtime acceptance tests. Mike thinks that these will also be useful for anyone downstream who is packaging their own buildpacks. The acceptance tests work off of the buildpack's manifest file, so anything that you are packaging in there can now be acceptance tested against a live instance of Cloud Foundry.
There was one remaining license issue with the buildpacks, which was for the Go buildpack. The upstream maintainers of this buildpack were convinced to add a new license.
Mike has sent an email to vcap-dev proposing some changes to what is being packaged in the offline buildpacks and to reduce the number of binary dependencies. He is looking for feedback over the next few days. He is expecting the reduce the size of buildpacks by a factor of 3 or 4.
James Bayer from Pivotal gave an update on Diego. They have done a lot of work to improve stability and reliability to handle a production workload.
SAP has been working on private Docker registry that would sit inside the Cloud Foundry network if you are using Docker images for your applications. This would help with lifecycle events, such as losing a copy of your application. You would be able to always create a new copy of your application from the Docker registry. This work is in the process of being pulled in and they have been working with the Cloud Controller team to make the interconnect between Diego and the Cloud Controller possible.
"Which version of Diego will be compatible with cf-release version v206?", was asked. James said that Onsi had posted this information for v205 and plans to do a better job of communicating this information each time there is a release. Currently, v206 has not been released.
We'll see you at the Cloud Foundry Summit on the 11th-12th May! The next CAB call is scheduled for 13th May at 8am PST, but keep an eye on the vcap-dev mailing list for updates.