Writing an API Wrapper in GoLang

2022-10-16 2027 words 10 minutes

Contents

I had a really time-limited effort to do to prove how to write a command line wrapper for an open API a customer is developping.

The target Rest API is the jquants-api, as presented in a previous post.

I chose to implement the wrapper in GoLang, which proved to be extremely fast and pleasant to do. The task was eventually done in a short evening, and the resulting Golang wrapper with core features has been uploaded on github.

This is the short story on the process to write the API and the few different programming steps to get there.

Goals

So first, let’s list the programming tasks that we will have to deal with:

create a test, and supporting code, checking we can save the username and password in an edn file compatible with the jquants-api-jvm format.
write another test and supporting code to retrieve the refresh token
write another test and supporting code to retrieve the id token
write another test, and supporting code using the id token to retrieve daily values
publish our wrapper to github
use our go library in another program

We always talk about writing code using TDD, now’s the day to do it. Checking we have code to enter and save the username and password in an edn file compatible with the jquants-api-jvm format.

In a helper_test.go file, let’s write the skeleton test for a PrepareLogin function.


package jquants_api_go

import (
	"fmt"
	"os"
	"testing"
)

func TestPrepareLogin(t *testing.T) {
	PrepareLogin(os.Getenv("USERNAME"), os.Getenv("PASSWORD"))
}

Here, we pick up the USERNAME and PASSWORD from the environment, using os.GetEnv.

The prepare function, we will write in a helper.go file. It will:

get the username and password as parameters,
instanciate a Login struct
marshall this as an EDN file content.

func PrepareLogin(username string, password string) {
	var user = Login{username, password}
	encoded, _ := edn.Marshal(&user)
	writeConfigFile("login.edn", encoded)
}

Our Login struct will first simply be:

type Login struct {
	UserName string `edn:"mailaddress"`
	Password string `edn:"password"`
}

And the call to edn.Marshal will create a byte[] array content that we can write to file, and so writeConfigFile will simply call os.WriteFile with the array returned from the EDN marshalling.

func writeConfigFile(file string, content []byte) {
	os.WriteFile(getConfigFile(file), content, 0664)
}

To be able to use the EDN library, we will need to add it to the go.mod file, with:

require olympos.io/encoding/edn v0.0.0-20201019073823-d3554ca0b0a3

Before running the test, be sure to enter your jquants API’s credential:

export USERNAME="youremail@you.com"
export PASSWORD="yourpassword"

And at this stage, you should be able to run go test in the project folder, and see the following output:

PASS
ok  	github.com/hellonico/jquants-api-go	1.012s

You should also see that the content of the login.edn file is properly filled:

cat ~/.config/jquants/login.edn

{:mailaddress "youremail@you.com" :password "yourpassword"}

The second function to be tested, is TestRefreshToken which sends a HTTP post request, with the username and password and retrieve the refresh token as an answer of the API call. We update the helper_test.go file with a new test case:

func TestRefreshToken(t *testing.T) {
	token, _ := GetRefreshToken()
	fmt.Printf("%s\n", token)
}

The GetRefreshToken func will:

load user stored in file previously and prepare it as json data
prepare the http request, with the url, and the json formatted user as body content
send the http request
the API will returns data that will store in a RefreshToken struct
and let’s store that refresh token as an EDN file

The supporting GetUser will now load the file content that was written in the step before. We already have the Login struct, and will then just use edn.Unmarshall() with the content from the file.


func GetUser() Login {
	s, _ := os.ReadFile(getConfigFile("login.edn"))
	var user Login
	edn.Unmarshal(s, &user)
	return user
}

Note, that, while we want to read/write our Login struct to a file in EDN format, we also want to marshall the struct to JSON when sending the http request.

So the metadata on our Login struct needs to be slightly updated:

type Login struct {
	UserName string `edn:"mailaddress" json:"mailaddress"`
	Password string `edn:"password" json:"password"`
}

We also need a new struct to read the token returned by the API, and we also want to store it as EDN, just like we are doing for the Login struct:

type RefreshToken struct {
	RefreshToken string `edn:"refreshToken" json:"refreshToken"`
}

And now, we have all the bricks to write the GetRefreshToken function:

func GetRefreshToken() (RefreshToken, error) {
	// load user stored in file previously and prepare it as json data
	var user = GetUser()
	data, err := json.Marshal(user)

	// prepare the http request, with the url, and the json formatted user as body content
	url := fmt.Sprintf("%s/token/auth_user", BASE_URL)
	req, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(data))
	
	// send the request
	client := http.Client{}
	res, err := client.Do(req)

    // the API will returns data that will store in a RefreshToken struct
	var rt RefreshToken
	json.NewDecoder(res.Body).Decode(&rt)

	// and let's store that refresh token as an EDN file
	encoded, err := edn.Marshal(&rt)
	writeConfigFile(REFRESH_TOKEN_FILE, encoded)

	return rt, err
}

Running go test is a little bit more verbose, because we print the refreshToken to the standard output, but the tests should be passing!

{eyJjdHkiOiJKV1QiLC...}

PASS
ok  	github.com/hellonico/jquants-api-go	3.231s

Get the Id Token

From the Refresh Token, you can retrieve the IdToken which is the token then used to send requests to the jquants API. This is has almost the same flow as GetRefreshToken, and to support it we mostly introduce a new struct IdToken with the necessary metadata to marshall to/from edn/json.

type IdToken struct {
	IdToken string `edn:"idToken" json:"idToken"`
}

And the rest of the code this time is:

func GetIdToken() (IdToken, error) {
	var token = ReadRefreshToken()

	url := fmt.Sprintf("%s/token/auth_refresh?refreshtoken=%s", BASE_URL, token.RefreshToken)

	req, err := http.NewRequest(http.MethodPost, url, nil)
	client := http.Client{}
	res, err := client.Do(req)

	var rt IdToken
	json.NewDecoder(res.Body).Decode(&rt)

	encoded, err := edn.Marshal(&rt)
	writeConfigFile(ID_TOKEN_FILE, encoded)

	return rt, err
}

Get Daily Quotes

We come to the core of the wrapper code, where we use the IdToken, and request daily quote out of the jquants http api, via a HTTP GET request.

The code flow to retrieve the daily quotes is:

as before, read id token from the EDN file
prepare the target url with parameters code and dates parameters
send the HTTP request using the idToken as a HTTP header
parse the result as a daily quotes struct, which is a slice of Quote structs

The test case simply checks on non-nul value returned and prints the quotes for now.

func TestDaily(t *testing.T) {
	var quotes = Daily("86970", "", "20220929", "20221003")
	
	if quotes.DailyQuotes == nil {
		t.Failed()
	}

	for _, quote := range quotes.DailyQuotes {
		fmt.Printf("%s,%f\n", quote.Date, quote.Close)
	}
}

Supporting code for the func Daily is shown below:

func Daily(code string, date string, from string, to string) DailyQuotes {
	// read id token
	idtoken := ReadIdToken()

	// prepare url with parameters
	baseUrl := fmt.Sprintf("%s/prices/daily_quotes?code=%s", BASE_URL, code)
	var url string
	if from != "" && to != "" {
		url = fmt.Sprintf("%s&from=%s&to=%s", baseUrl, from, to)
	} else {
		url = fmt.Sprintf("%s&date=%s", baseUrl, date)
	}
	// send the HTTP request using the idToken
	res := sendRequest(url, idtoken.IdToken)

	// parse the result as daily quotes
	var quotes DailyQuotes
	err_ := json.NewDecoder(res.Body).Decode(&quotes)
	Check(err_)
	return quotes
}

Now we need to fill in a few blanks.

the sendRequest needs a bit more details
the parsing of DailyQuotes is actually not so straight forward.

So, first let’s get out of the way the sendRequest func. It sets a header using http.Header, and note that you can add as many headers as you want there. Then sends the Http Get request and returns the response as is.


func sendRequest(url string, idToken string) *http.Response {

	req, _ := http.NewRequest(http.MethodGet, url, nil)
	req.Header = http.Header{
		"Authorization": {"Bearer " + idToken},
	}
	client := http.Client{}

	res, _ := client.Do(req)
	return res
}

Now to the parsing of the daily quotes. If you use Goland as your editor, you’ll notice that if you copy paste a json content into your go file, the editor will ask to convert the json to go code directly!

Pretty neat.


type Quote struct {
	Code             string   `json:"Code"`
	Close            float64  `json:"Close"`
	Date             JSONTime `json:"Date"`
	AdjustmentHigh   float64  `json:"AdjustmentHigh"`
	Volume           float64  `json:"Volume"`
	TurnoverValue    float64  `json:"TurnoverValue"`
	AdjustmentClose  float64  `json:"AdjustmentClose"`
	AdjustmentLow    float64  `json:"AdjustmentLow"`
	Low              float64  `json:"Low"`
	High             float64  `json:"High"`
	Open             float64  `json:"Open"`
	AdjustmentOpen   float64  `json:"AdjustmentOpen"`
	AdjustmentFactor float64  `json:"AdjustmentFactor"`
	AdjustmentVolume float64  `json:"AdjustmentVolume"`
}

type DailyQuotes struct {
	DailyQuotes []Quote `json:"daily_quotes"`
}

While the defaults are very good, we need to do a bit more tweaking to unmarshall Dates properly. What follows comes from the following post on how to marshall/unmarshall json dates.

The JSONTime type will store its internal date as a 64bits integer, and we add the functions to JSONTime to marshall/unmarshall JSONTime. As shown, the time value coming from the json content can be either a string or an integer.

type JSONTime int64

// String converts the unix timestamp into a string
func (t JSONTime) String() string {
	tm := t.Time()
	return fmt.Sprintf("\"%s\"", tm.Format("2006-01-02"))
}

// Time returns a `time.Time` representation of this value.
func (t JSONTime) Time() time.Time {
	return time.Unix(int64(t), 0)
}

// UnmarshalJSON will unmarshal both string and int JSON values
func (t *JSONTime) UnmarshalJSON(buf []byte) error {
	s := bytes.Trim(buf, `"`)
	aa, _ := time.Parse("20060102", string(s))
	*t = JSONTime(aa.Unix())
	return nil
}

The test case written at first, now should pass with go test

"2022-09-29",1952.000000
"2022-09-30",1952.500000
"2022-10-03",1946.000000
PASS
ok  	github.com/hellonico/jquants-api-go	1.883s

Our helper is now ready and we can adding some CI to it.

Circle CI Configuration

The configuration is character to character close to the official CircleCI doc on testing with golang.

We will just update the docker image to 1.17

version: 2.1
jobs:
  build:
    working_directory: ~/repo
    docker:
      - image: cimg/go:1.17.9
    steps:
      - checkout
      - restore_cache:
          keys:
            - go-mod-v4-{{ checksum "go.sum" }}
      - run:
          name: Install Dependencies
          command: go get ./...
      - save_cache:
          key: go-mod-v4-{{ checksum "go.sum" }}
          paths:
            - "/go/pkg/mod"
      - run: go test -v

Now we are ready to setup the project on circleci:

The required parameters USERNAME and PASSWORD in our helper_test.go can be setup directly from the Environment Variables settings of the circleci’s project:

Any commit on the main branch will trigger the circleci build (or you can manually trigger it of course) and if you’re all good, you should see the success steps:

Our wrapper is well tested, let’s start publishing it.

Publishing the library on github

Providing our go.mod file has the content below:

module github.com/hellonico/jquants-api-go

go 1.17

require olympos.io/encoding/edn v0.0.0-20201019073823-d3554ca0b0a3

The best way to publish the code is to use git tags. So let’s create a git tag and push it to github with:

git tag v0.6.0
git push --tags

Now, a separate project can depend on our library by using in their go.mod

require github.com/hellonico/jquants-api-go v0.6.0

Using the library from an external program.

Our simplistic program will parse parameters using the flag module, and then call the different functions just like it was done in the test cases for our wrapper.

package main

import (
	"flag"
	"fmt"
	jquants "github.com/hellonico/jquants-api-go"
)

func main() {

	code := flag.String("code", "86970", "Company Code")
	date := flag.String("date", "20220930", "Date of the quote")
	from := flag.String("from", "", "Start Date for date range")
	to := flag.String("to", "", "End Date for date range")
	refreshToken := flag.Bool("refresh", false, "refresh RefreshToken")
	refreshId := flag.Bool("id", false, "refresh IdToken")

	flag.Parse()

	if *refreshToken {
		jquants.GetRefreshToken()
	}
	if *refreshId {
		jquants.GetIdToken()
	}

	var quotes = jquants.Daily(*code, *date, *from, *to)

	fmt.Printf("[%d] Daily Quotes for %s \n", len(quotes.DailyQuotes), *code)
	for _, quote := range quotes.DailyQuotes {
		fmt.Printf("%s,%f\n", quote.Date, quote.Close)
	}

}

We can create our CLI using go build

go build

And the run it with the wanted parameters, here:

refreshing the id token
refreshing the refresh token
getting daily values for entity with code 86970 between 20221005 and 20221010

./jquants-example --id --refresh --from=20221005 --to=20221010 --code=86970

Code: 86970 and Date: 20220930 [From: 20221005 To: 20221010]
[3] Daily Quotes for 86970 
"2022-10-05",2016.500000
"2022-10-06",2029.000000
"2022-10-07",1992.500000

Nice work. We will leave it to the user to write the remaining statements and listedInfo that are part of the JQuants API but not yet implemented in this wrapper.