DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Data Engineering
  3. Data
  4. Searching OpenAPI Parameters For Signs Of Archival Data

Searching OpenAPI Parameters For Signs Of Archival Data

How do we look for historical data in OpenAPI parameters? There is a wealth of data available within the growing number of APIs available online today. As we...

Kin Lane user avatar by
Kin Lane
·
Aug. 13, 18 · Analysis
Like (2)
Save
Tweet
Share
2.43K Views

Join the DZone community and get the full member experience.

Join For Free

How do we look for historical data in OpenAPI parameters?

There is a wealth of data available within the growing number of APIs available online today. As we work to profile new APIs for the Streamdata.io API Gallery and optimize our approach to filtering, searching, and understand what is relevant across a large number of APIs, we are investing heavily in the ability to search API providers who provide archival data. Many of the customers we are talking with, especially those who operate in the financial sector, are looking for data with 2-5 year history — something that doesn't always exist in the API world.

We find a lot of interesting APIs, but we need to be able to also identify APIs that potentially offer historical data via their API. We can't look at every API manually, to see if they offer archive, it is something we are needing to do at scale, and we find that using OpenAPI parameters is the best way to automate the discovery of historical data. Since we profile all of the APIs in the Streamdata.io API Gallery using OpenAPI, documenting the details of each API, we can easily use the data gathered as part of the discovery of historical data.

There are a number of ways to search for signs of historical data at scale across hundreds or thousands of OpenAPI definitions. One good way to find APIs that might have historical components is to look for the occurrence of "date", but of course, you need to filter out some common occurrences like "validate", "update", and other variances to help keep the results focused and clean. If an API has a parameter for one of its path that has a date, there is the potential that the API can be searched, filtered, and worked with by date. Of course, it will depend on how far back data goes, but an API endpoint possessing a date parameter provides a very good start.

Next, we are looking at other related words that might produce historical results, parameters like "history", "historical", "archive", "past", "year", and other parameter names that might be historically relevant. Providing us with a growing vocabulary when it comes to flagging, and then manually checking how far back an APIs data store actually goes. Hopefully, we can automate other aspects of the historical data discovery process, and optimize how we find archival data. For now, just refining our search, and then manually reviewing each individual will give us more insight into the best ways to prioritize which APIs we should be looking at as part of our search.

This effort doesn't tell us 100% which APIs have historical data. It does give us a head start when it comes to finding them. First, we are discovering and indexing new APIs each day, and the more OpenAPI definitions we find and produce, the wider our search index becomes for finding valuable data. The wider and more precise our vocabulary becomes, the more we'll learn about identifying APIs that have historical data. Currently, we are refining our historical API search vocabulary, and once we identify a logical set of APIs, we'll publish the list to the Streamdata.io API Gallery. Providing an easier way to identify the APIs that have the highest Streamrank, as well as possessing historical data that can be accessed as well.

Data (computing) API

Published at DZone with permission of Kin Lane, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Learning by Doing: An HTTP API With Rust
  • Connecting Your Devs' Work to the Business
  • Key Considerations When Implementing Virtual Kubernetes Clusters
  • Pipes And Filters Pattern

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: