Over a million developers have joined DZone.

Building a Real-Time GitHub Issue Tracker Using Java and Deepstream

See how you can incorporate Java into into deepstream, an open-source, real-time data streaming server, so you can build an issue tracker connected to GitHub.

· Integration Zone

Learn how API management supports better integration in Achieving Enterprise Agility with Microservices and API Management, brought to you in partnership with 3scale

In this post, we'll be building a simple GitHub issue tracker similar to waffle.io that shows active issues on a project with specific workflow labels and updates in real time as issues are created and updated on GitHub. 

Image title

Why Use deepstream?

deepstream.io is an open-source, real-time data streaming server designed for speed and reliability. It's also very simple to get up and running, and there are clients available in a number of programming languages, as well as for JavaScript in the browser. That means we could use deepstream as an out-of-the-box dynamic backend for a static hosted webpage. 

deepstream will take care of managing our data and sharing it to the browser client, so we only need to setup deepstream and build a connector that keeps it up-to-date with our GitHub issues. 

This post will focus on building a connector using deepstream's Java API and Kohsuke Kawaguchi's GitHub API for Java. I will assume a basic understanding of Java. For brevity, we'll be using a frontend that I built earlier. If you'd like to read more about integrating deepstream with React.js, read the deepstream react tutorial

Setting Up

If you'd like to follow along, you can clone the completed project on GitHub. 

  • Within the client directory, run npm install to install the client dependencies, then run python -m SimpleHTTPServer to start a test server. You should then be able to view the client at localhost:8000.
  • Install and start a local deepstream server. For more information see the Quickstart Tutorial.
  • Create a GitHub personal access token with repository access. Then create a file ~/.github containing:
    oauth={your personal access token}
    It's also possible to put your GitHub credentials in this file to avoid this step — see here.
  • I recommend setting up a test GitHub repository with some issues, and labeling some of them with 'roadmap', 'ready', 'in progress', 'awaiting review', or 'in review'. It's also a good idea to give those labels some nice colors — we'll be using those later on.
  • Install ngrok which will allow us to listen for external connections from the webhook we'll use later on. Start it using ngrok http 8080, and make note of the forwarding address.
  • We're using Gradle to manage dependencies.

Building the GitHub connector

Project Setup

In our main method, we set the details of our local deepstream server, the webhook URI given when you start ngrok, and the GitHub repository that we're getting our issues from.

You'll need to update these details for the connector to work.

public static void main(String[] args) throws IOException {
    String deepstreamURI = "0.0.0.0:6021";
    String webhookURI = "http://024680ab.ngrok.io";
    String repo = "username/GithubBoard-Tutorial-Test";
    new GithubConnector(deepstreamURI, webhookURI, repo);
}

Setting up API Objects

We first setup a connection to our deepstream client.

private DeepstreamClient deepstreamClient = null;
...
try {
  deepstreamClient = new DeepstreamClient(deepstreamURI);
  deepstreamClient.login(new JsonObject());
} catch //...

... then setup a connection to GitHub...

private GitHub gitHub;
...
gitHub = GitHub.connect();
GHRepository repository = gitHub.getRepository(repo);

Initial State

Our connector will need to fetch any open issues from GitHub's API and create corresponding records in deepstream for rendering in the client.

Interesting Labels

We only care about certain labels, so we'll create a list of these and add them to a deepstream list.

List<String> interestingLabels = Arrays.asList(
        "roadmap", "ready", "in progress", "awaiting review", "in review");

// add label list to deepstream
deepstreamClient.record.getList("github-board-labels")
        .setEntries(interestingLabels);

Issue Lists

Now we can start inserting issues into lists based on which labels they have.

In our initializeIssueLists() method, we make deepstream lists for each of the labels and ensure that they're empty.

for (String labelName: interestingLabels) {
    deepstreamClient.record.getList(labelName).setEntries(new ArrayList<String>());
}

We get a list of open issues from GitHub.

List<GHIssue> issues = repository.getIssues(GHIssueState.OPEN);

Then we add each issue to all the lists for each label it has.

// put the issues into lists by label
for (GHIssue issue: issues){
    String issueId = Integer.toString(issue.getId());
    for (GHLabel label : issue.getLabels()) {
        // add the issue to the label list
    }
}

However, we're only interested in some of the labels. We also want to create a record with our issue id to store the issue data.

String labelName = label.getName();
if (interestingLabelSet.contains(labelName)){
    // add the issue id to the label list
    deepstreamClient.record.getList(labelName)
            .addEntry(issueId);

    // add a record for the issue
    deepstreamClient.record.getRecord(issueId)
            .set("title", issue.getTitle())
            .set("url", issue.getHtmlUrl())
            .discard();
}

We call this method from the constructor.

initializeIssueLists(repository, interestingLabels);

Coloring Labels

GitHub allows each label to have a color, so we get that from the API and pass it to the client for rendering.

Inside our setupIssueColors method, we simply create a deepstream record that maps label names to hex colors.

// set the label colors
List<GHLabel> repoLabels = repository.listLabels().asList();
Record labelColorsRecord = deepstreamClient.record.getRecord("github-board-label-colors");

for (String labelName: interestingLabels) {
    GHLabel label = getLabelWithName(repoLabels, labelName);
    if (label != null)
        // add records for each label's color
        labelColorsRecord.set(labelName, label.getColor());
    else {
        System.out.printf("Label '%s' does not exist on the repository", labelName);
    }
}

Live Updates

We use the GitHub webhook API to recieve updates when issues are modified.

Event Listener

First, we start a server to listen for webhook events.

Our startServer() method instantiates the RequestHandler which we'll describe shortly.

Subscribe to Issue Events

So that we are notified when the issues are modified, we subscribe through the GitHub webhook API, specifying GHEvent.ISSUES, as this is the only event we're interested in.

List<GHEvent> events = Arrays.asList(GHEvent.ISSUES);
repository.createWebHook(new URL(webhookURI), events);

Handling Events

Events are handled in the RequestHandler.handle() method. GitHub IssuesEvents carry a JSON payload that gives details of the issue that triggered the event.

The 'action' field shows what caused the event. The ones we're interested in are label modifications and edits.

When an issue is edited, we update the corresponding record.

if (action.equals("edited")){
    //...
    deepstreamClient.record.getRecord(issueId)
            .set("title", issueTitle)
            .set("url", issueUrl);

When an issue is labeled, we add the issue to the corresponding list.

} else if (action.equals("labeled")) {
    deepstreamClient.record
            .getList(issueEvent.get("label").getAsJsonObject().get("name").getAsString())
            .addEntry(issueId);

Likewise, we remove issues from lists when they are unlabeled.

} else if (action.equals("unlabeled")) {
    deepstreamClient.record
            .getList(issueEvent.get("label").getAsJsonObject().get("name").getAsString())
            .removeEntry(issueId);
}

Feel free to look through the rest of the code on GitHub and add any features you'd like. 

Unleash the power of your APIs with future-proof API management - Create your account and start your free trial today, brought to you in partnership with 3scale.

Topics:
java ,github ,webhooks ,api ,realtime ,data streaming ,react.js ,tutorial

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

SEE AN EXAMPLE
Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.
Subscribe

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}