API Crafting Secrets: Into Primal API

Our API is one of the evolutionary results of over 8 years of R&D in our field, culminating in the web-facing DaaS platform that we currently promote as Primal Data API.

Interview with Justin Kominar, Senior Software Developer at Primal. 

Tell us a bit about Primal and its Data API
Primal let’s you treat your users like individuals. Our interest graph powers personalized mobile and Web experiences by modelling your user’s interests, but not in a big-data way. When you understand users better, your applications can do and find great things for them.

Primal’s Data API takes your interest inputs as simple keywords (composed as semi-structured URIs) and returns an interest graph of concepts related to your input. The concepts are organized in a graph structure of narrower, broader and related concepts.

Primal also returns a payload of example content from various popular content sources (or you can pick your own set of content sources) that have been filtered and organized using the interest graph.

This one-two punch of data (interest graph + example content) allows you to generate all kinds of personalized solutions with one simple API call.

Tell us more about the development process of your API (how long, methodology, framework, etc)
Our API is one of the evolutionary results of over 8 years of R&D in our field, culminating in the web-facing DaaS platform that we currently promote as Primal Data API. Although it borrows a lot from REST’s fundamentals, it is not a pure RESTful API as we rely on the URL to express structured input. The result of our approach allows you to encapsulate quite a bit of complexity in a simple call to our service.

Primal is a dyed-in-the-wool follower of Clayton Christensen’s school of disruptive innovation. Company-wide, we used Agile development processes, preferring quick development cycles that allow us to measure, test, rinse and repeat regularly.

What are the key components of your core API infrastructure?
It will come as no surprise to you that there are many bits and pieces of infrastructure that help keep the entire Primal ecosystem running smoothly. We leverage the Java Virtual Machine (JVM) to power the core intellectual property that drives the Primal engine. From there, a Python stack runs the DaaS layer which manifests itself as the actual API that developers talk with. Our content wrangling infrastructure is powered by the fantastic ElasticSearch project, in concert with some custom Java work coordinating the ingestion of content from all over the web. Finally, all of our front-end web services are built on a NodeJS stack, using the Express web framework. This entire Primal infrastructure is hosted in the Amazon Cloud for all of the elastic-y goodness that Amazon provides.

We are currently developing a new autonomous agent-based infrastructure using Scala to handle massive load/scalability demands, under the direction of Derek Wyatt, our Director of Architect, who literally “wrote a book on Scala”. We’re very excited about the opportunities this new system will enable, so stay tuned!

Last, but certainly not least, we leverage 3Scale’s API management infrastructure for controlling the access to the data API. The biggest win we get from 3Scale is the ease with which we were able to tie into their services, freeing our developer resources up to focus on the tasks that really matter to help make Primal a roaring success.

What drove your choices in terms of technology and infrastructure?
Much of our existing infrastructure was based on some pretty classical decisions over the course of our R&D years. Java and Python were obvious choices for their maturity and speed – not to mention there was always a steady pool of developer resources in the Waterloo community to draw from.

As Primal grows, we’re starting to feel the need for greater concurrency at all levels of our systems. The move to NodeJS for our front-end systems (away from a legacy system built on PHP) has helped a lot in that respect. However, far more importantly, our back-end infrastructure is in the process of being migrated to a more message-based architecture utilizing the JVM. We chose this path because of the fabulous Hotspot compiler, the modern garbage collection, an extensive community of open libraries and support, and the concurrent robustness that a message-based system can deliver. Getting away from stateless calls by moving to a less RPC-based design will allow Primal to scale to meet all of our future needs.

What proxy are you using for your API and why?
Our environment utilizes the Varnish proxy primarily because it was the recommended way to tie in with 3Scale. Varnish has been a reliable component, and true to their word, it was very simple to integrate to route all of our API calls through, enabling a pretty quick API throttling/management solution for us.

What are the specificities of your API?
As we briefly discussed earlier, Primal’s Data API has two main data outputs: an interest graph and exemplary content fresh from the Web. To get this data, you input interests (the semi-structured input I mentioned earlier) and tell Primal about the Web content sources you care about. These data Primal generates can be used to power new Web and mobile experiences – ones that seem more personal and inline with a user’s interests. There are a number of applications for Primal, but they all boil down to understanding your users and finding interesting content for them.

One of the key differentiators of how Primal works is that we perform our work in real-time, on data that is not predetermined. This allows you to leverage our technology on virtually any topic or field of interest, with content that’s as up-to-date as the Web itself.

Primal is not a search engine! With search engines, more keywords often means a narrower set of relevant results since you are giving it more and more words that need to be present in the document to return a “hit”. Primal is unlike a search engine in this regard — the more context you can provide in your input to Primal, the richer the results will be. More keywords means more context, and more context means better results.

Primal thrives in the “long tail”, where other services often choke. As a simple example, you could ask Primal for a result about Java with a query like this:

However, this result returns a mixed bag of concepts related to coffee, the island, and the programming language.

Now give Primal a bit more context about what you really care about:

… and now Primal returns you concepts about constructors, garbage collection, arguments, stack, etc.

Primal’s Data API can presently return you results in JSON or RSS format.  JSON is great for the API-slingers out there, and RSS makes it really simple to generate feeds of custom-filtered content that can be integrated into many pre-existing publication systems.

The structure of the JSON response is annotated using the widely-accepted standard skos (Simple Knowledge Organization System – for concepts) and dc (Dublin Core – for content) namespaces, to help you easily identify and interoperate with specific elements in the response.

Further, Primal’s Data API has a set of calls that allow you to define collections of Web-accessible content sources. You can request content from our default content categories (which include thousands of the Web’s most popular content sources), or create your own custom content sources by with your own website URLs or RSS feeds.

How can developers get a better taste of what your API does?

We’ve made it pretty easy to kick Primal’s tires if you feel that there might be some synergy between Primal and your own development projects.

The first stop would be to try our API Explorer.  It’s a quick interactive tool on our website that allows you to make easy no-strings-attached sample calls to the API to get a sense of what the JSON response would look like for your given input.

The next step is to register for a free API key and give the Data API a try.  Our freemium model allows you to sign up for Primal and make 5,000 calls per month, on the house, for as long as you need.  Perform a quick integration in the programming environment of your choice, or make the POST and GET calls in your favourite REST client.

Finally, there are plenty of use cases, documentation and examples available on the Developer section of our website for you to read over to get a better taste of what Primal has to offer.

Treat your users like individuals – they’ll love you for it. Primal’s interest graph powers truly personalized mobile and Web experiences. Our cloud-based data service has a simple API, generates interest graphs in JSON format, and is free to evaluate – for as long as you need.