API Developer Outreach Research for the Department of Veterans Affairs: Part 9
Check out the final post in this API developer outreach research series and explore a set of interview questions and answers.
Join the DZone community and get the full member experience.
Join For FreeThis is is final part (you can find Part 8 here) of a series on a write-up for research I conducted with my partner Skylight Digital. The team conducted a series of interviews with leading public and private sector API platforms regarding how they approached developer outreach, and then I wrote it up as a formal report, which the Skylight Digital team then edited and polished. We are looking to provide as much information as possible regarding how the VA and other federal agencies should consider crafting their API outreach efforts.
Appendix A: Interview Questions
For each interview, we asked the following questions:
- What role do you play within the organization?
- What is the purpose and scope of your API developer outreach program?
- What does success look like for you?
- What are the key elements or practices (e.g, documentation, demos, webinars, conferences, blog posts) that you’re using to drive and sustain effective adoption of your API(s)?
- Do you make use of an API developer sandbox to drive and sustain adoption? If so, please describe how you’ve designed that environment to be useful and usable to developers.
- What types of metrics do you use to measure adoption effectiveness and to inform future decisions about how you evolve your program?
- How is your outreach program structured and staffed? How did it start out and evolve over time?
- Imagine that you’re charged with ensuring the complete failure of an API developer outreach program. What things would you do to ensure that failure?
- What big ideas do you have for evolving your outreach program to make it even more effective?
- Any other thoughts you’d like to share? If so, please feel free!
Appendix B: CMS Blue Button API interview notes
1. What role do you play within the organization?
Product Manager.
2. What is the purpose and scope of your API developer outreach program?
CMS is offering several APIs. One approach is put it out there and hope they come. The second approach is leveraging APIs on the Socrato platform and hoping there’s adoption. The third approach is partnership agreements with various private partners.
Purpose of the Blue Button outreach program is to drive adoption of the API.
3. What does success look like for you?
Worked with various stakeholders to define the metrics in terms of value. For example, the number of unique organizations who are experimenting with the API. Originally set a target of 500 organizations. Up to 450 currently. Use it as a proxy measurement of value. Not sure how beneficiaries are benefiting yet, but adoption is a proxy indicator.
Also measuring the number of beneficiaries who have linked their data to Blue Button API — for example, Google Verily.
4. What are the key elements or practices (e.g, documentation, demos, webinars, conferences, blog posts) that you’re using to drive and sustain effective adoption of your API(s)?
Documentation is the first key element. Originally hosted documentation on GitHub pages, but now using bluebutton.cms.gov.
Had challenges over the year making clear what Blue Button is, value, etc.
Relaunched during recent HIMSS conference. Resulted in a number of sign-ups from the press.
Set up a blog to provide a getting started guide, how sign-up works, etc. Working pretty well.
Assigned a designated Developer Evangelist who pushes content, participates in forums, and hits the niche conference circuit.
5. Do you make use of an API developer sandbox to drive and sustain adoption? If so, please describe how you’ve designed that environment to be useful and usable to developers.
Sandbox has been a big part. Have a synthetic dataset. In healthcare, this is essential from a privacy standpoint.
Have had challenges with synthetic data in terms of making them realistic.
Have a streamlined process for accessing and onboarding into the environment. Experience is not currently great, and working on improving.
Developer portal is only for sandbox environment. No portal for production, so disjointed at this point.
6. What types of metrics do you use to measure adoption effectiveness and to inform future decisions about how you evolve your program?
Look at the adoption as a funnel. That has helped to drive a culture shift around how folks think about metrics and the role they play.
Top of the funnel is evangelism/traffic/etc., portal sign-ups, registered an app, and how many of those who have registered made API calls.
7. How is your outreach program structured and staffed? How did it start out and evolve over time?
Have a developer evangelist.
Periodically, the team gets on the phone with developer evangelist and talks about ideas for building awareness and driving adoption.
Using email + CRM tools for outreach. Trying to improve marketing automation.
8. Imagine that you’re charged with ensuring the complete failure of an API developer outreach program. What things would you do to ensure that failure?
Don’t take a “build it and they will come” approach.
Not acting quickly enough on insights gained from the funnel process, including reaching out to individuals to help them through the process (i.e., proactive outreach).
9. What big ideas do you have for evolving your outreach program to make it even more effective?
Double down on what’s working to incrementally grow — conferences, webinars, etc.
Serving as a blueprint for other organizations, particularly at a state & local level.
N/A
Appendix C: Census interview notes
1. What role do you play within the organization?
Developer engagement lead, serving as the comms arm for Census’ API efforts. First-line of defense for any developer engagement. Participate in hackathons, etc.
Also working on CitySDK. This was in response to the observation that developers were having trouble working with Census APIs. SDK currently not being maintained because of lost lead developer and funding for it. Personally learning how to develop in order to maintain myself.
2. What is the purpose and scope of your API developer outreach program?
Save developers time and help them understand the nuances of Census data and how to use the data. Engage in communications and help inform the development of API products.
3. What does success look like for you?
Happy developers.
4. What are the key elements or practices (e.g, documentation, demos, webinars, conferences, blog posts) that you’re using to drive and sustain effective adoption of your API(s)?
Proactive engagement: work on driving the government’s engagement in National Day of Civic Hacking in collaboration with NASA and Code for America. Hackathons are great for user research and testing new features.
Reactive: have a Slack and Gitter channel.
Haven’t tried blog posts yet. Thinking about interviewing developers who are using APIs and turning their stories into blog posts. Haven’t been able to get legal approval yet for that.
Being able to figure out what data Census has and how that data can be made available to developers is key.
Webinars are good, especially those focused on showing how to use Census data and build something simple using the API.
Use a data search tool — American FactFinder; developers can use this to find out what variables they are interested in and then trace that data back to an API that they can use to access programmatically.
5. Do you make use of an API developer sandbox to drive and sustain adoption? If so, please describe how you’ve designed that environment to be useful and usable to developers.
No, haven’t done anything like yet. Have a discovery tool that is part of the API.
6. What types of metrics do you use to measure adoption effectiveness and to inform future decisions about how you evolve your program?
Initially focused on the number of API keys registered. Starting focusing on metrics of use. A small number of users drive 80% of the API use. Developers tend to move data into their environment for performance purposes (e.g., caching). Plus, they worry about the government shutting down and data not being accessible.
7. How is your outreach program structured and staffed? How did it start out and evolve over time?
One person focused on outreach.
The entire team dedicated to building the APIs and making the data available and accessible. Each product gets its own endpoint; there are about 30 now and planning to do 70 more.
8. Imagine that you’re charged with ensuring the complete failure of an API developer outreach program. What things would you do to ensure that failure?
Create terrible, hand-rolled documentation with a lot of pictures; don’t give developers a channel to communicate; don’t listen to them; don’t give them a way to test out data before granting a key; and don’t have metrics for measuring developer happiness and just focus on usage metrics.
9. What big ideas do you have for evolving your outreach program to make it even more effective?
Focus more on product excellence and marketing. A lot of developers don’t like being pushed to. Most developers go to hackathons for social interaction and career progression, not because they love working with your data and using your APIs.
Instead of trying to focus too much on attracting random developers, focus on the developers who are engaged now and reach out to them. Would like to introduce a CRM system to help manage these relationships better.
Learned it’s important to grab as much data during the registration process in order to gain better insight into developer intent, behavior, and characteristics. Found it’s very hard to get information after developers have gained access; they don’t typically respond to email requests for information.
Hackathons, etc. are huge investments of time and resources.
Biggest lesson learned is to always focus on the principle of how to make it as easy for the developers as possible.
Make heavy use of GraphQL because it’s introspective and helps developers understand the API in greater detail. Like GraphQL because it can evolve without breaking things.
Appendix D: OpenFEC Interview Notes
1. What role do you play within the organization?
Tech Lead at FEC, mostly focused on OpenFEC. Involved in FEC campaign finance data for ten years.
2. What is the purpose and scope of your API developer outreach program?
Don’t have an overarching formal approach in place yet, but do provide support to developers through a variety of channels when they reach out.
3. What does success look like for you?
Number one measure of success is providing developers accurate data and making sure that is available at all times. Also making sure that developers can find the data that they need.
4. What are the key elements or practices (e.g, documentation, demos, webinars, conferences, blog posts) that you’re using to drive and sustain effective adoption of your API(s)?
Don’t have a formal developer outreach program. Have a small team of developers, designers, content designers, and product managers.
Would like to be more proactive about engaging.
Do have some mechanisms for getting feedback. For example, the project is open source, which encourages developers to add issues, contribute, etc. Developers can send emails to existing team tries to get back as quickly as possible. Would like to have a ticket management system to help facilitate workflow around support. Do have a google group in which community of developers can ask and answer questions.
Do have a developer page. And work hard to keep those up-to-date.
Use GSA’s API umbrella to manage users, which handles rate limiting and caching.
Have a try-it-out feature on developer page powered by Swagger API, and can just put params in to see how the API works.
5. Do you make use of an API developer sandbox to drive and sustain adoption? If so, please describe how you’ve designed that environment to be useful and usable to developers.
Don’t have a sandbox, but do provide instructions for setting up locally. And also provide a sample database that can be copied. Since all data is public, don’t have to worry about PII. Amazon provides a free sandbox environment to anyone with .gov domain that can be used to set-up a test environment.
6. What types of metrics do you use to measure adoption effectiveness and to inform future decisions about how you evolve your program?
Don’t have formal metrics. Using the GSA API Umbrella, there is some data available, not currently using that data to drive decisions.
7. How is your outreach program structured and staffed? How did it start out and evolve over time?
The existing team of developers, etc. are providing outreach support.
8. Imagine that you’re charged with ensuring the complete failure of an API developer outreach program. What things would you do to ensure that failure?
Making breaking changes and don’t tell anyone; not responding to and fixing data quality issues, which would undermine the credibility of the API; couldn’t serve the data in an efficient manner.
9. What big ideas do you have for evolving your outreach program to make it even more effective?
Have discussed holding a hackathon or conference, similar to what Blue Button is doing.
Holding a quarterly developer conference call to starting answering questions and discuss ideas.
Regardless of activity, starting to build a community.
Pushing our code to code.gov to increase awareness.
There are commercial restrictions around the use of open FEC data.
Appendix E: Salesforce interview notes
1. What role do you play within the organization?
Vice President of Developer Relations. My team doesn’t create APIs but we drive awareness and adoption through producing:
- Technical content
- Demos & sessions at events
- Tools like SDKs and interactive docs
- Marcom (social, email, etc)
- Community Building
2. What is the purpose and scope of your API developer outreach program?
Unlike companies who solely monetize services, our primary purpose is to enable developers to create custom integrations with Salesforce.
As an advanced enterprise software platform, its critical that Salesforce works well with all the other enterprise apps you might have. In addition, many of our customers are building custom web or mobile apps that need access to customer data.
Because we are a platform, our purpose is not limited to just “API outreach” but is inclusive both of our APIs and also our app development platform.
Some of the primary use-cases for our APIs are to extract data for archival purposes, surface customer data in third-party or bespoke apps, or to enable business systems to work together in other ways. We also have a family of machine learning APIs known as Einstein Vision and Einstein Language that can be used by any application to create predictions from unstructured images and text.
3. What does success look like for you?
Ultimately, we know that our customers are demanding more skilled Salesforce Developers every day to support their custom implementations so our primary metrics are monthly signups and active monthly usage.
Our goal is to keep the growth of our developer population in line with our overall customer growth to ensure there are enough developers in the ecosystem to service our customers' needs.
We also measure job postings for Salesforce Developers, traffic to our website, engagement with our events like Dreamforce and TrailheaDX, and usage of our online learning portal trailhead.salesforce.com.
4. What are the key elements or practices (e.g, documentation, demos, webinars, conferences, blog posts) that you’re using to drive and sustain effective adoption of your API(s)?
Blogging, webinars, documentation (including an API Explorer), training classes, sample code, first-party events (we do regional workshops called DevTrails, large regional events called Salesforce World Tours, and big global events called Dreamforce and TrailheaDX)
In addition, we have built a substantial community program with over 200 developers groups around the world and an MVP program to recognize top contributors in the community. We’re in the process of expanding our open source footprint (sample code, apps, and SDKs).
5. Do you make use of an API developer sandbox to drive and sustain adoption? If so, please describe how you’ve designed that environment to be useful and usable to developers.
We have a free environment called the Salesforce Developer Edition that anyone can sign up for. There is no expiration date so you can use it as long as you’d like, and you’re allowed to sign up for as many as you’d like for the times that you need a “clean sandbox.”
We’ve also tightly integrated these developer editions into the Trailhead learning platform so that as a developer is going through an online course (called a module) or a tutorial (called a project), they can actually complete hands-on challenges in their developer environment and get validation that they have done it correctly.
To protect our business interests, these developer environments have limits in terms of API usage, storage, user licenses, etc. With these limits, we do our best to find a balance between empowering the developers and protecting the business. And we have a process where developers can request increased limits.
6. What types of metrics do you use to measure adoption effectiveness and to inform future decisions about how you evolve your program?
We use our CRM platform to measure all the ways we touch our developers and how that impacts success.
7. How is your outreach program structured and staffed? How did it start out and evolve over time?
When we launched the program, we started small. 1-2 evangelists, a program manager for community/events, and a web developer. It grew from there.
We don’t share details about our current team size publicly. However, the key components of our program are as follows with approximate percentages:
- Marketers (12%)
- Evangelists (35%)
- Community Program Managers (12%)
- Event Producers/Program Managers (13%)
- Marketing Operations: Email, Development, Creative, etc (25%)
8. Imagine that you’re charged with ensuring the complete failure of an API developer outreach program. What things would you do to ensure that failure?
Gate all the content.
Don’t provide a developer edition.
Don’t share use-cases or sample code.
Don’t find any partners.
Don’t engage with your users or help them be successful.
9. What big ideas do you have for evolving your outreach program to make it even more effective?
We have a ton of content but it’s not all discoverable.
Working on making our pages more data-driven and dynamic so we can expose all our resources.
We’re also moving a lot of our content to GitHub as well as creating a more organized process to store our code samples.
Many of them are orphaned in blog posts and other places.
N/A
Opinions expressed by DZone contributors are their own.
Comments