Writing an API Wrapper in Golang

I had a really time-limited effort to do to prove how to write a command line wrapper for an open API a customer is developing.

The target REST API is the jquants-api, as presented in a previous article.

I chose to implement the wrapper in Golang, which proved to be extremely fast and pleasant to do. The task was eventually done in a short evening, and the resulting Golang wrapper with core features has been uploaded on GitHub.

This is the short story on the process to write the API and the few different programming steps to get there.

Goals

So first, let’s list the programming tasks that we will have to deal with:

• Create a test, and supporting code, checking we can save the username and password in an edn file compatible with the jquants-api-jvm format
• Write another test and supporting code to retrieve the refresh token
• Write another test and supporting code to retrieve the ID token
• Write another test and supporting code using the ID token to retrieve daily values
• Publish our wrapper to GitHub
• Use our Go library in another program

Start by Writing a Test Case, Preparing and Saving the Login Struct to Access the API

We always talk about writing code using TDD — now’s the day to do it. Check that we have code to enter and save the username and password in an edn file compatible with the jquants-api-jvm format.

In a helper_test.go file, let’s write the skeleton test for a PrepareLogin function.

package jquants_api_go

import (
	"fmt"
	"os"
	"testing"
)

func TestPrepareLogin(t *testing.T) {
	PrepareLogin(os.Getenv("USERNAME"), os.Getenv("PASSWORD"))
}

Here, we pick up the USERNAME and PASSWORD from the environment, using os.GetEnv.

We will write the prepare function in a helper.go file. It will:

• Get the username and password as parameters
• Instantiate a Login struct
• Marshal this as an EDN file content

func PrepareLogin(username string, password string) {
    var user = Login{username, password}
    encoded, _ := edn.Marshal(&user)
    writeConfigFile("login.edn", encoded)
}

Our Login struct will first simply be:

type Login struct {
    UserName string `edn:"mailaddress"`
    Password string `edn:"password"`
}

And the call to edn.Marshal will create a byte[] array content that we can write to file, and so writeConfigFile will simply call os.WriteFile with the array returned from the EDN marshaling. 

func writeConfigFile(file string, content []byte) {
    os.WriteFile(getConfigFile(file), content, 0664)
}

To be able to use the EDN library, we will need to add it to the go.mod file with:

require olympos.io/encoding/edn v0.0.0-20201019073823-d3554ca0b0a3

Before running the test, be sure to enter your jquants API’s credential:

export USERNAME="[email protected]"
export PASSWORD="yourpassword"

And at this stage, you should be able to run go test in the project folder, and see the following output:

PASS
ok      github.com/hellonico/jquants-api-go    1.012s

You should also see that the content of the login.edn file is properly filled:

cat ~/.config/jquants/login.edn

{:mailaddress "[email protected]" :password "yourpassword"}

Use the Login to Send an HTTP Request to the jQuants API and Retrieve the RefreshToken

The second function to be tested is TestRefreshToken, which sends a HTTP post request with the username and password and retrieve the refresh token as an answer of the API call. We update the helper_test.go file with a new test case:

func TestRefreshToken(t *testing.T) {
    token, _ := GetRefreshToken()
    fmt.Printf("%s\n", token)
}

The GetRefreshToken func will:

• Load user stored in file previously and prepare it as JSON data
• Prepare the HTTP request with the URL and the JSON formatted user as body content
• Send the HTTP request
• The API will returns data that will store in a RefreshToken struct
• And let’s store that refresh token as an EDN file

The supporting GetUser will now load the file content that was written in the step before. We already have the Login struct, and will then just use edn.Unmarshall()  with the content from the file.

func GetUser() Login {
    s, _ := os.ReadFile(getConfigFile("login.edn"))
    var user Login
    edn.Unmarshal(s, &user)
    return user
}

Note, that, while we want to read/write our Login struct to a file in EDN format, we also want to marshal the struct to JSON when sending the HTTP request.

So the metadata on our Login struct needs to be slightly updated:

type Login struct {
    UserName string `edn:"mailaddress" json:"mailaddress"`
    Password string `edn:"password" json:"password"`
}

We also need a new struct to read the token returned by the API, and we also want to store it as EDN, just like we are doing for the Login struct:

type RefreshToken struct {
    RefreshToken string `edn:"refreshToken" json:"refreshToken"`
}

And now, we have all the bricks to write the GetRefreshToken function:

func GetRefreshToken() (RefreshToken, error) {
    // load user stored in file previously and prepare it as json data
    var user = GetUser()
    data, err := json.Marshal(user)

    // prepare the http request, with the url, and the json formatted user as body content
    url := fmt.Sprintf("%s/token/auth_user", BASE_URL)
    req, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(data))
    
    // send the request
    client := http.Client{}
    res, err := client.Do(req)

    // the API will returns data that will store in a RefreshToken struct
    var rt RefreshToken
    json.NewDecoder(res.Body).Decode(&rt)

    // and let's store that refresh token as an EDN file
    encoded, err := edn.Marshal(&rt)
    writeConfigFile(REFRESH_TOKEN_FILE, encoded)

    return rt, err
}

Running go test is a little bit more verbose, because we print the refreshToken to the standard output, but the tests should be passing!

{eyJjdHkiOiJKV1QiLC...}

PASS
ok      github.com/hellonico/jquants-api-go    3.231s

Get the ID Token

From the Refresh Token, you can retrieve the IdToken which is the token then used to send requests to the jquants API. This is has almost the same flow as GetRefreshToken, and to support it we mostly introduce a new struct IdToken with the necessary metadata to marshal to/from edn/json.

type IdToken struct {
    IdToken string `edn:"idToken" json:"idToken"`
}

And the rest of the code this time is:

func GetIdToken() (IdToken, error) {
    var token = ReadRefreshToken()

    url := fmt.Sprintf("%s/token/auth_refresh?refreshtoken=%s", BASE_URL, token.RefreshToken)

    req, err := http.NewRequest(http.MethodPost, url, nil)
    client := http.Client{}
    res, err := client.Do(req)

    var rt IdToken
    json.NewDecoder(res.Body).Decode(&rt)

    encoded, err := edn.Marshal(&rt)
    writeConfigFile(ID_TOKEN_FILE, encoded)

    return rt, err
}

Get Daily Quotes

We come to the core of the wrapper code, where we use the IdToken, and request daily quote out of the jquants HTTP API via a HTTP GET request.

The code flow to retrieve the daily quotes is:

• As before, read ID token from the EDN file
• Prepare the target URL with parameters code and dates parameters
• Send the HTTP request using the idToken as a HTTP header
• Parse the result as a daily quotes struct, which is a slice of Quote structs

The test case simply checks on non-nul value returned and prints the quotes for now.

func TestDaily(t *testing.T) {
    var quotes = Daily("86970", "", "20220929", "20221003")
    
    if quotes.DailyQuotes == nil {
        t.Failed()
    }

    for _, quote := range quotes.DailyQuotes {
        fmt.Printf("%s,%f\n", quote.Date, quote.Close)
    }
}

Supporting code for the func Daily is shown below:

func Daily(code string, date string, from string, to string) DailyQuotes {
    // read id token
    idtoken := ReadIdToken()

    // prepare url with parameters
    baseUrl := fmt.Sprintf("%s/prices/daily_quotes?code=%s", BASE_URL, code)
    var url string
    if from != "" && to != "" {
        url = fmt.Sprintf("%s&from=%s&to=%s", baseUrl, from, to)
    } else {
        url = fmt.Sprintf("%s&date=%s", baseUrl, date)
    }
    // send the HTTP request using the idToken
    res := sendRequest(url, idtoken.IdToken)

    // parse the result as daily quotes
    var quotes DailyQuotes
    err_ := json.NewDecoder(res.Body).Decode(&quotes)
    Check(err_)
    return quotes
}

Now we need to fill in a few blanks:

• The sendRequest needs a bit more details
• The parsing of DailyQuotes is actually not so straightforward

So, first let’s get the sendRequest func out of the way. It sets a header using http.Header, and note that you can add as many headers as you want there. Then it sends the HTTP GET request and returns the response as-is.

func sendRequest(url string, idToken string) *http.Response {

    req, _ := http.NewRequest(http.MethodGet, url, nil)
    req.Header = http.Header{
        "Authorization": {"Bearer " + idToken},
    }
    client := http.Client{}

    res, _ := client.Do(req)
    return res
}

Now to the parsing of the daily quotes. If you use Goland as your editor, you’ll notice that if you copy-paste a JSON content into your Go file, the editor will ask to convert the JSON to go code directly!

Pretty neat.

type Quote struct {
    Code             string   `json:"Code"`
    Close            float64  `json:"Close"`
    Date             JSONTime `json:"Date"`
    AdjustmentHigh   float64  `json:"AdjustmentHigh"`
    Volume           float64  `json:"Volume"`
    TurnoverValue    float64  `json:"TurnoverValue"`
    AdjustmentClose  float64  `json:"AdjustmentClose"`
    AdjustmentLow    float64  `json:"AdjustmentLow"`
    Low              float64  `json:"Low"`
    High             float64  `json:"High"`
    Open             float64  `json:"Open"`
    AdjustmentOpen   float64  `json:"AdjustmentOpen"`
    AdjustmentFactor float64  `json:"AdjustmentFactor"`
    AdjustmentVolume float64  `json:"AdjustmentVolume"`
}

type DailyQuotes struct {
    DailyQuotes []Quote `json:"daily_quotes"`
}

While the defaults are very good, we need to do a bit more tweaking to unmarshal Dates properly. What follows comes from the following post on how to marshal/unmarshal JSON dates.

The JSONTime type will store its internal date as a 64bits integer, and we add the functions to JSONTime to marshall/unmarshall JSONTime. As shown, the time value coming from the JSON content can be either a string or an integer.

type JSONTime int64

// String converts the unix timestamp into a string
func (t JSONTime) String() string {
    tm := t.Time()
    return fmt.Sprintf("\"%s\"", tm.Format("2006-01-02"))
}

// Time returns a `time.Time` representation of this value.
func (t JSONTime) Time() time.Time {
    return time.Unix(int64(t), 0)
}

// UnmarshalJSON will unmarshal both string and int JSON values
func (t *JSONTime) UnmarshalJSON(buf []byte) error {
    s := bytes.Trim(buf, `"`)
    aa, _ := time.Parse("20060102", string(s))
    *t = JSONTime(aa.Unix())
    return nil
}

The test case written at first now should pass with go test.

"2022-09-29",1952.000000
"2022-09-30",1952.500000
"2022-10-03",1946.000000
PASS
ok      github.com/hellonico/jquants-api-go    1.883s

Our helper is now ready and we can adding some CI to it.

CircleCI Configuration

The configuration is character to character close to the official CircleCI doc on testing with Golang.

We will just update the Docker image to 1.17.

version: 2.1
jobs:
  build:
    working_directory: ~/repo
    docker:
      - image: cimg/go:1.17.9
    steps:
      - checkout
      - restore_cache:
          keys:
            - go-mod-v4-{{ checksum "go.sum" }}
      - run:
          name: Install Dependencies
          command: go get ./...
      - save_cache:
          key: go-mod-v4-{{ checksum "go.sum" }}
          paths:
            - "/go/pkg/mod"
      - run: go test -v

Now we are ready to set up the project on CircleCI:

The required parameters USERNAME and PASSWORD in our helper_test.go can be set up directly from the Environment Variables settings of the CircleCI project:

Any commit on the main branch will trigger the CircleCI build (or you can manually trigger it of course) and if you’re all good, you should see the success steps:

Our wrapper is well-tested. Let’s start publishing it.

Publishing the Library on GitHub

Providing our go.mod file has the content below:

module github.com/hellonico/jquants-api-go

go 1.17

require olympos.io/encoding/edn v0.0.0-20201019073823-d3554ca0b0a3

The best way to publish the code is to use git tags. So let’s create a git tag and push it to GitHub with:

git tag v0.6.0
git push --tags

Now, a separate project can depend on our library by using it in their go.mod.

require github.com/hellonico/jquants-api-go v0.6.0

Using the Library from an External Program

Our simplistic program will parse parameters using the flag module, and then call the different functions just like it was done in the test cases for our wrapper.

package main

import (
    "flag"
    "fmt"
    jquants "github.com/hellonico/jquants-api-go"
)

func main() {

    code := flag.String("code", "86970", "Company Code")
    date := flag.String("date", "20220930", "Date of the quote")
    from := flag.String("from", "", "Start Date for date range")
    to := flag.String("to", "", "End Date for date range")
    refreshToken := flag.Bool("refresh", false, "refresh RefreshToken")
    refreshId := flag.Bool("id", false, "refresh IdToken")

    flag.Parse()

    if *refreshToken {
        jquants.GetRefreshToken()
    }
    if *refreshId {
        jquants.GetIdToken()
    }

    var quotes = jquants.Daily(*code, *date, *from, *to)

    fmt.Printf("[%d] Daily Quotes for %s \n", len(quotes.DailyQuotes), *code)
    for _, quote := range quotes.DailyQuotes {
        fmt.Printf("%s,%f\n", quote.Date, quote.Close)
    }

}

We can create our CLI using go build.

go build

And the run it with the wanted parameters here:

• Refreshing the ID token
• Refreshing the refresh token
• Getting daily values for entity with code 86970 between 20221005 and 20221010

./jquants-example --id --refresh --from=20221005 --to=20221010 --code=86970

Code: 86970 and Date: 20220930 [From: 20221005 To: 20221010]
[3] Daily Quotes for 86970 
"2022-10-05",2016.500000
"2022-10-06",2029.000000
"2022-10-07",1992.500000

Nice work. We will leave it to the user to write the remaining statements and listedInfo that are part of the JQuants API but not yet implemented in this wrapper.