Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Understanding the Words We Use to Describe Machine Learning APIs

DZone's Guide to

Understanding the Words We Use to Describe Machine Learning APIs

The API Evangelist has profiled the machine learning engines of major cloud providers: Amazon, Google, and Azure. Come check it out!

· AI Zone
Free Resource

Find out how AI-Fueled APIs from Neura can make interesting products more exciting and engaging. 

I spend a lot of time trying out new APIs, working to understand what it is they do or do not do. I have a pretty robust way of looking at APIs, profiling the company and the APIs they offer, but when I’m wading through the marketing content, API documentation, and other resources, I am regularly stumped by the language that is used to describe what an API does. Honestly, this problem isn’t exclusive to machine learning APIs, but with the recent explosion in artificial intelligence, machine learning, deep learning, cognitive, and other types of algorithmic voodoo, the words being used seem to have gone to entirely new levels.

I am interested in understanding what it is an API does. I want to go from zero to understanding in 2.34 seconds. I don’t want to wade through marketing and documentation to understand what an API does. I want to find simple, concise language that properly describes an API. In the world of artificial intelligence, this can be difficult to do and is something that varies from provider to provider. Some machine learning API providers are better at describing what they do, while others seem to prefer hype and fluff when it comes to explaining what is actually possible.

As I continue my work profiling Amazon, Microsoft, and Google APIs, I want to develop an approach to helping me separate what an API does and what a provider says it does. I am going to continue profiling each API using OpenAPI and labeling them with a common set of tags I’m using to quantify what machine learning APIs actually do. As I’m doing this, I also decided to add an extra tag field called x-hype-tags, which gives me a way to track each of the additional words I found in the marketing and documentation, that I may not be actually using to describe what the API does — maintaining much of the API providers intent.

One thing that contributes to the confusion of what is machine learning is that, very similar to APIs, it can be almost anything. While packaged in many different ways, most of the major machine learning providers have a machine learning engine for training and executing models, as well as some ready to go models that accomplish some common ML tasks for developers. Here are the primary machine learning engines I’ve been profiling out of the major cloud providers:

  • Amazon Machine Learning: Amazon Machine Learning is a service that makes it easy for developers of all skill levels to use machine learning technology.
  • Google Machine Learning: Google Cloud Machine Learning Engine is a managed service that enables you to easily build machine learning models that work on any type of data, of any size
  • Azure Machine Learning: A fully-managed cloud service that enables you to easily build, deploy, and share predictive analytics solutions.

I am going to be profiling these engines in their own way, but before I understand the infinite possibilities, I want to better understand the finite possibilities with each of these machine learning platforms. To help me develop a way of tagging and quantifying what APIs do and the hype around them, I took a single type of machine learning API from each of these providers and augmented their OpenAPI with more descriptive tagging.

To begin, I started with Amazon’s Rekognition API, crafting a basic OpenAPI that outlines the technical details of each possible API request and response, with the added tagging describing what the API does:

Swagger: "2.0"
info:
  title: Amazon Rekognition
  description: Amazon Rekognition is a service that makes it easy to add image analysis to your applications.
  version: 1.0.0
host: rekognition.us-west-2.amazonaws.com
basePath: /
schemes:
  - https
produces:
  - application/json
consumes:
  - application/json
paths:
  /:
    post:
      summary: Detects faces
      description: Detects faces within an image (JPEG or PNG) that is provided as input.
      parameters:
        - in: header
          name: X-Amz-Target
          type: string
          required: true
          default: RekognitionService.DetectFaces
        - in: body
          type: string
          required: true
          schema:
            $ref: "#/definitions/definitions"
      responses:
        200:
          description: Successful Response
          schema:
            type: array
            items:
              $ref: '#/definitions/ImageProcess'
      tags:
        - Machine Learning
        - Facial Recognition
        - Image Recognition
      x-hype-tags:
        - analysis
        - detect objects
        - detect scenes
        - detect faces
        - recognize celebrities
        - sophisticated
        - deep learning
        - image classification
        - proven
        - highly scalable
        - scientists
        - deep neural network models
        - facial recognition
definitions:
  Image:
    properties:
      view_sort:
        description: The image to be processed.
        type: object
        schema:
          $ref: "#/definitions/S3Object"
  S3Object:
    properties:
      Bucket:
        description: The bucket.
        type: string
      Name:
        description: The name of image.
        type: string
  ImageProcess:
    properties:
      FaceDetails:
        description: The bucket.
        type: string
        type: array
        schema:
          $ref: "#/definitions/FaceDetails"
      OrientationCorrection:
        description: The bucket.
        type: string
  FaceDetails:
    properties:
      BoundingBox:
        description: The bounding box.
        type: object
        schema:
          $ref: "#/definitions/BoundingBox"
      Confidence:
        description: Confidence score.
        type: integer
      Landmarks:
        description: The bucket.
        type: object
        schema:
          $ref: "#/definitions/Landmarks"
      Pose:
        description: The pose.
        type: object
        schema:
          $ref: "#/definitions/Pose"
      Quality:
        description: The quality.
        type: object
        schema:
          $ref: "#/definitions/Quality"
  BoundingBox:
    properties:
      Height:
        description: The height.
        type: integer
      Left:
        description: The left.
        type: integer
      Top:
        description: The top.
        type: integer
      Width:
        description: The width.
        type: integer
  Landmarks:
    properties:
      Type:
        description: The bucket.
        type: string
      X:
        description: The bucket.
        type: integer
      Y:
        description: The bucket.
        type: integer
  Pose:
    properties:
      Pitch:
        description: The pitch.
        type: integer
      Roll:
        description: The pitch.
        type: integer
      Yaw:
        description: The pitch.
        type: integer
  Quality:
    properties:
      Brightness:
        description: The brightness.
        type: string
      Sharpness:
        description: The sharpness.
        type: string

Then I went through Azure’s Computer Vision, crafting a basic OpenAPI that outlines the technical details of each possible API request and response, with the added tagging describing what the API does:

Swagger: "2.0"
info:
  title: Azure Computer Vision
  description: The cloud-based Computer Vision API provides developers with access to advanced algorithms for processing images and returning information.
  version: 1.0.0
host: westcentralus.api.cognitive.microsoft.com
basePath: /vision/v1.0
schemes:
  - https
produces:
  - application/json
consumes:
  - application/json
paths:
  /analyze:
    post:
      summary: Run Image Detection
      description: Run image detection and annotation for a batch of images.
      parameters:
        - in: query
          type: string
          name: visualFeatures
        - in: query
          type: string
          name: details
        - in: query
          type: string
          name: language
        - in: body
          type: string
          required: true
          schema:
            $ref: "#/definitions/image"
      responses:
        200:
          description: Successful Response
          schema:
            type: array
            items:
              $ref: '#/definitions/image'
      tags:
        - Machine Learning
        - Facial Recognition
        - Image Cropping
        - Image Moderation
        - Optical Character Recognition
        - Image Tagging
        - Image Categorization
      x-hype-tags:
        - Crop
        - Flag
        - Adult Content
        - Color Schemes
        - Optical Character Recognition
        - Handwritten Text
        - Text
        - Domain-Specific content
        - Descriptions
        - Facial Recognition
        - Categorize
        - Tag
        - Advanced Algorithms
        - Type
        - Quality
        - Analyze
definitions:
  image:
    properties:
      url:
        description: A letter to sort in the UI.
        type: string
  annotation:
    properties:
      tags:
        description: The tags.
        type: array
        schema:
          $ref: "#/definitions/tag"
      description:
        description: The
        type: object
        schema:
          $ref: "#/definitions/description"
  tag:
    properties:
      name:
        description: Name of tag.
        type: string
      name:
        description: Score of tag.
        score: string
  description:
    properties:
      tags:
        description: Tags
        type: array
      captions:
        description: The captions
        type: array
        schema:
          $ref: "#/definitions/caption"
  caption:
    properties:
      text:
        description: The tet of the captions
        type: string
      confidence:
        description: The tet of the captions
        type: integer

Then last I look at Google’s Cloud Vision API, crafting a basic OpenAPI that outlines the technical details of each possible API request and response, with the added tagging describing what the API does:

Swagger: "2.0"
info:
  title: Google Cloud Vision API
  description: Derive insight from images with our powerful Cloud Vision API.
  version: v1
host: vision.googleapis.com
basePath: /v1
schemes:
  - https
produces:
  - application/json
consumes:
  - application/json
paths:
  /images:annotate:
    post:
      summary: Derive Insight
      description: Derive insight from images with our powerful Cloud Vision API.
      parameters:
        - in: body
          type: string
          required: true
          schema:
            $ref: "#/definitions/ImageRequest"
      responses:
        200:
          description: Successful Response
          schema:
            type: array
            items:
              $ref: '#/definitions/ImageResponse'
      tags:
        - Machine Learning
        - Facial Recognition
        - Object Recongition
        - Optical Character Recognition
        - Sentiment Analysis
        - Similar Content
        - Landmark Detection
        - Language Identification
        - Image Moderation
      x-hype-tags:
        - Vision
        - Understand
        - Powerful
        - Categorization
        - Object Recongition
        - Facial Recongition
        - Optical Character Recognition
        - Bulk Metadata
        - Moderation
        - Offensive Content
        - Sentiment Analysis
        - Analysis
        - Insights
        - Improves Over Time
        - Celebrities
        - Similar Content
        - Language Identification
        - Logo Detection
        - Landmark Detection
        - Image Attributes
definitions:
  ImageRequest:
    properties:
      requests:
        description: The image request envelope.
        type: object
        required: true
        schema:
          $ref: "#/definitions/AnnotateImageRequest"
  AnnotateImageRequest:
    properties:
      image:
        description: Image object.
        type: object
        required: true
        schema:
          $ref: "#/definitions/Image"
      features:
        description: Image object.
        type: object
        required: true
        schema:
          $ref: "#/definitions/Feature"
      imageContext:
        description: Image object.
        type: object
        required: true
        schema:
          $ref: "#/definitions/ImageContext"
  ImageResponse:
    properties:
      responses:
        description: The image response envelope.
        type: object
        required: true
        schema:
          $ref: "#/definitions/AnnotateImageResponse"
  AnnotateImageResponse:
    properties:
      faceAnnotations:
        description: Face Annotations.
        type: object
        required: true
        schema:
          $ref: "#/definitions/FaceAnnotation"
      landmarkAnnotations:
        description: Landmark Annotations.
        type: object
        required: true
        schema:
          $ref: "#/definitions/EntityAnnotation"
      logoAnnotations:
        description: Logo Annotations.
        type: object
        required: true
        schema:
          $ref: "#/definitions/EntityAnnotation"
      labelAnnotations:
        description: Label Annotations.
        type: object
        required: true
        schema:
          $ref: "#/definitions/EntityAnnotation"
      textAnnotations:
        description: Text Annotations.
        type: object
        required: true
        schema:
          $ref: "#/definitions/EntityAnnotation"
      fullTextAnnotation:
        description: Full Text Annotations.
        type: object
        required: true
        schema:
          $ref: "#/definitions/fullTextAnnotation"
      safeSearchAnnotation:
        description: Safe Search Annotations.
        type: object
        required: true
        schema:
          $ref: "#/definitions/safeSearchAnnotation"
      imagePropertiesAnnotation:
        description: Image Properties Annotations.
        type: object
        required: true
        schema:
          $ref: "#/definitions/ImageProperties"
      cropHintsAnnotation:
        description: Crop Hints Annotations.
        type: object
        required: true
        schema:
          $ref: "#/definitions/CropHintsAnnotation"
      webDetection:
        description: Web Detection.
        type: object
        required: true
        schema:
          $ref: "#/definitions/WebDetection"
      error:
        description: Error.
        type: object
        required: true
        schema:
          $ref: "#/definitions/Status"

These OpenAPIs aren’t 100% complete. I’m more concerned with developing the tagging schema for use across many APIs than I am with the perfect OpenAPI. I’ll publish them to GitHub and continue to evolve, polish, and work to make each OpenAPI as complete as possible. I’m looking to settle on a schema before I got to work adding the extra x-hype-tags to additional OpenAPIs, pushing beyond what OpenAPI 2.0 dictates when it comes to API tagging. I’ll use the primary tags for each OpenAPI as the distilled down categorization of what each ML API does, while the x-hype-tags will give me a summary of what is used in the marketing copy, and the API documentation for each ML API provider.

It is important to me that I can quickly pull up APIs that accomplish specific business tasks. This requires each API to be tagged well. I also want to be able to apply some sort of hype score to each API, which reflects the number of adjectives and buzzwords in use in marketing and documentation. I feel these x-hype-tags reflect the hype in ML API space, and the tags reflect the realities of what ML APIs are actually capable of doing. When I profile new ML APIs, I will be adding any word that catches my eye to the x-hype-tags, but I will be thinking about more deeply and more carefully evolving the core set of more meaningful APIs I am using to categorize the ML APIs I track on.

Once I profile Amazon, Microsoft’s, and Google’s ML APIs in this way, I am going to go through Algorithmia and other leading ML API providers and profiling their APIs, adding to both my core machine learning tagging vocabulary as well as my machine learning hype tagging dictionary. Then, I’ll step back a little and think about the bigger picture of the words we are choosing to describe our ML APIs. I am looking to develop a better way to be cataloging the types of ML APIs out there and understand the language API providers are choosing to use when they describe what it is that they deliver. We need to be able to make sense of the ML APIs we already have, as well as each of the upcoming waves of AI and ML voodoo that is being developed right now and due to be unleashed on the world in coming years.

To find out how AI-Fueled APIs can increase engagement and retention, download Six Ways to Boost Engagement for Your IoT Device or App with AI today.

Topics:
api ,machine learning ,ai

Published at DZone with permission of Kin Lane, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}