Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

SoChain BitCoin, DogeCoin and LiteCoin Data REST API With Apache NiFi

DZone's Guide to

SoChain BitCoin, DogeCoin and LiteCoin Data REST API With Apache NiFi

Using data science techniques to process cryptocurrency in streams with the open source platform, Apache NiFi.

· Big Data Zone ·
Free Resource

The Architect’s Guide to Big Data Application Performance. Get the Guide.

Introduction

SoChain provides a fast set of public, freely available APIs (don't abuse them) to access information on various networks.

If you need this for critical work, please donate.

One of the things you will see in this simple flow is that NiFi excels in ingesting REST and working with JSON. As you'll see, NiFi is great for splitting it up, shredding it, filtering it, manipulating it, and extracting from it. With the resulting usable objects, we can build a schema that will also us to do record processing. 

I just hosted a Future of Data Princeton Meetup in Woodbridge New Jersey with some amazing speakers sponsored by ChainNinja. While this was all about Blockchain for Enterprise and no cryptocurrency was involved it made we want to investigate some cryptocurrency data. As you can see, manipulating complex JSON data, filtering, modifying, routing, and scripting with its values is trivial in Apache NiFi.

In my next article, I am investigating Hyperledger and Ethereum for enterprise solutions integration with Apache NiFi, Impala, Hive, Kudu, HBase, Spark, Kafka, and other enterprise technologies.

Steps

We read from the URL.

I send the original file to immutable HDFS storage.

In another branch, I will use EvaluateJSONPath to pull out one attribute to use to get detail records.

$.data.blocks 

I use that attribute to build a deeper REST call to get the details for the latest block.

https://chain.so/api/v2/block/BTC/${block_no} 

This is in invokeHTTP, which is a scriptable HTTP(S) call. This often comes in handy.

In the next EvaluateJSONPath, I pull out all the high-level attributes of the JSON file. I want these for all the records as master fields. These are repeated.

After that, I split out the two arrays of data beneath that into two separate branches. I will breach these down into individual records for parsing. I could also apply a schema and handle these are groups of records.

This is an example of reading a REST API and creating a unique name per call. Also, notice it's easy to handle HTTPS as well as HTTP.

Example of a Unique File Name We Can Script:

${filename:append('btc.'):append(${now():format('yyyymmddHHMMSS'):append(${md5}):append('.json')})}

REST URL Example:  https://chain.so/api/v2/get_info/LTC

Example of the Value of the Apache NiFi Provenance. (These are the attributes acquired for one flowfile).

Attribute Values
Access-Control-Allow-Headers
Origin,Accept,Content-Type,X-Requested-With,X-CSRF-Token
Access-Control-Allow-Methods
GET,POST
Access-Control-Allow-Origin
*
CF-RAY
49e564b17e23923c-EWR
Cache-Control
no-cache, no-store, max-age=0, must-revalidate
Connection
keep-alive
Content-Type
application/json; charset=utf-8
Date
Thu, 24 Jan 2019 20:54:07 GMT
Expect-CT
max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Expires
Fri, 01 Jan 1990 00:00:00 GMT
Pragma
no-cache
Server
cloudflare
Set-Cookie
__cfduid=d6f52ee1552c73223442296ff7230e9fd1548363246; expires=Fri, 24-Jan-20 20:54:06 GMT; path=/; domain=.chain.so; HttpOnly, _mkra_ctxt=1a7dafd219c4972a7562f232dc63f524--200; path=/; max-age=5
Status
200 OK
Strict-Transport-Security
max-age=31536000;includeSubDomains
Transfer-Encoding
chunked
X-Content-Type-Options
nosniff
X-Download-Options
noopen
X-Frame-Options
SAMEORIGIN
X-Request-Id
20d3f592-50b6-40cf-a496-a6f915eb463b
X-Runtime
1.018401
X-XSS-Protection
1; mode=block
bits
172fd633
block_no
559950
blockhash
0000000000000000001c68f61ddcc30568536a583c843a7d0c9606b9582fd7e5
fee
0.05142179
filename
btc.201949241501759.json
fragment.count
1
fragment.identifier
cec10691-82e9-402b-84a9-7901b084f10a
fragment.index
0
gethttp.remote.source
chain.so
invokehttp.remote.dn
CN=ssl371663.cloudflaressl.com,OU=PositiveSSL Multi-Domain,OU=Domain Control Validated
invokehttp.request.url
https://chain.so/api/v2/block/BTC/559950
invokehttp.status.code
200
invokehttp.status.message
OK
invokehttp.tx.id
bc8a0a18-0685-4a2c-97fa-34541b9ea929
merkleroot
41eb6f68477e96c9239ae1bbe4e5d4d02529c6f7faebc4ad801730d09609a0ef
mime.type
application/json; charset=utf-8
mining_difficulty
5883988430955.408
network
BTC
next_blockhash
Empty string set
nonce
1358814296
path
./
previous_blockhash
0000000000000000001b2b3d3b5741462fe31981a6c0ae9335ed8851e936664b
schema
chainsotxinputinfo
schema.name
chainsotxinputinfo
segment.original.filename
btc.201949241501759.json
sent_value
3977.10078351
size
470242
time
1548362873
uuid
3c1d72b4-e993-4b32-a679-0741a44aeefb

An example input record:

{
  "input_no" : 0,
  "address" : "3N7Vid17hE1ofGcWR6bWEmtQBQ8kKQ7iKW",
  "value" : "0.20993260",
  "received_from" : {
    "txid" : "4e0f00cddb8e3d98de7f645684dc7526468d1dc33efbbf0bc173ed19c6556896",
    "output_no" : 4
  }
}

An Example LiteCoin Record 

{
  "status" : "success",
    "data" : {
      "name" : "Litecoin",
      "acronym" : "LTC",
      "network" : "LTC",
      "symbol_htmlcode" : "Ł",
      "url" : "http://www.litecoin.com/",
      "mining_difficulty" : "6399667.35869154",
      "unconfirmed_txs" : 8,
      "blocks" : 1567929,
      "price" : "0.00000000",
      "price_base" : "BTC",
      "price_update_time" : 1548451214,
      "hashrate" : "178582229079753"
    }
  }

Learn how taking a DataOps approach will help you speed up processes and increase data quality by providing streamlined analytics pipelines via automation and testing. Learn More.

Topics:
cryptocurrency ,big data ,rest api ,apache nifi ,apache nifi tutorial

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}