Building Dynamic D3.js Web Apps With Database Data

d3.js is a javascript library for producing dynamic, interactive data visualizations in web browsers, using the widely implemented svg, html5, and css standards. the cdata api server enables you to generate rest apis for 80+ data sources, including both on-premises and cloud-based databases. this article walks through setting up the cdata api server to create a rest api for a sqlite database and creating a simple d3.js web application that has live access to the database data. the d3.js app dynamically builds a simple bar chart based on the database data. while the article steps through most of the code, you can download the sample project and sqlite database to see the full source code and test the functionality for yourself.

setting up the api server

if you have not already done so, download the cdata api server . once you have installed the api server, follow the steps below to run the application, configure the application to connect to your data (the instructions in this article are for the included sample database), and then configure the application to create a rest api for any tables you wish to access in your d3 app.

enable cors

if the d3 web app and api server are on different domains, the d3 library will generate cross-domain requests. this means that cors (cross-origin resource sharing) must be enabled on any servers queried by d3 web apps. you can enable cors for the api server on the server tab in the settings page:

1. click the checkbox to enable cross-origin resource sharing (cors).
2. either click the checkbox to allow all domains without '*' or specify the domains that are allowed to connect in access-control-allow-origin .
3. set access-control-allow-methods to "get,put,post,options".
4. set access-control-allow-headers to "authorization".
5. click save changes.

configure your database connection

follow the steps below to configure the api server to connect to your database:

1. navigate to the connections tab on the settings page.
2. click add connection.
3. configure the connection in the resulting dialog: name your connection, select sqlite as the database, and fill the database field with the full path to the sqlite database (the included database is chinook.db from the sqlite tutorial ).

configure a user

next, create a user to access your database data through the api server. you can add and configure users on the users tab of the settings page. in this simple d3 app for viewing data, create a user that has read-only access: click add, give the user a name, select get for the privileges, and click save changes.

an authtoken is then generated for the user. you can find authtokens and other information for each user on the users tab:

accessing tables

having created a user, you are ready to enable access to the database tables:

1. click the add resources button on the resources tab of the settings page.
2. select the data connection you wish to access and click next.
3. with the connection selected, enable resources by selecting each table name and clicking next.

sample urls for the rest api

having configured a connection to the database, created a user, and added resources to the api server, you now have an easily accessible rest api based on the odata protocol for those resources. below, you will see a list of tables and the urls to access them. for more information on accessing the tables, you can open the api page from the navigation bar. to work with javascript, you can append the @json parameter to the end of urls that do not return json data by default.

as with standard odata feeds, if you wish to limit the fields returned, you can add a $select parameter to the query, along with other standard url parameters, such as $filter , $orderby , $skip , and $top .

building a d3.js application

with the api server setup completed, you are ready to build the sample d3.js app. the following steps walk through the source code for the web app contained in the .zip file, making note of any relevant sections.

index.html

this file contains all of the sources for the d3.js web app, including script references, styles, basic html to create a general layout for the web app, and javascript code using the d3.js library to dynamically populate the app and generate a bar chart based on data from the api server.

<body>

this html element contains the basic html code for the web app: a drop down menu to select the table, drop down menus for the domain and range of the bar chart, and an svg to contain the bar chart itself. the specific attributes and values of each element are defined dynamically using the d3 javascript library.

the body element also contains the script element that encompasses the javascript used to dynamically build the web app.

<script>

the first lines of javascript set up the svg for the bar chart and initialize the other global variables. after the initial setup, the functions to populate the drop down menus for the table and columns are called.

var svg = d3.select("svg"),
    margin = {top: 20, right: 20, bottom: 120, left: 60},
    width = +svg.attr("width") - margin.left - margin.right,
    height = +svg.attr("height") - margin.top - margin.bottom;

var x = d3.scaleband().rangeround([0, width]).padding(0.2),
    y = d3.scalelinear().rangeround([height, 0]);

var g = svg.append("g")
    .attr("transform", "translate(" + margin.left + "," + margin.top + ")");

var username = "read_only";
var authtoken = "********************";

var selectedtable = "";
var selecteddomain = "";
var selectedrange = "";

populatetableoptions();

populatedomainandrangeoptions();

populatetableoptions

this function uses the d3.js library to perform the http get against the api server to retrieve the list of tables, populate the options for the drop down menu used to choose a table, set the behavior for when the selection changes, set the selected table for the web app, and set up the button for retrieving specific table data.

populatetableoptions() {
  d3.json("http://localhost:8153/api.rsc/")
    .header("authorization", "basic " + btoa(username + ":" + authtoken))
    .get(function(error, data) {
      if (error) throw error;

      var values = data.value;

      d3.select("select.tableselect")
        .on('change', function() {
          clearchart();
          selectedtable = d3.select("select.tableselect").property("value");
          populatedomainandrangeoptions(selectedtable);
          d3.select("button.databutton")
          .text(function(d) {
            return "get [" + selectedtable + "] data"; 
          });
        })
        .selectall('option')
        .data(values)
        .enter().append("option")
        .text(function(d) {
          return d.name; 
        });

      selectedtable = d3.select("select.tableselect").property("value");

      d3.select("button.databutton")
        .on('click', function() {
          clearchart();
          buildchart();
        })
        .text(function(d) {
          return "get [" + selectedtable + "] data";
        });
    });
}

populatedomainandrangeoptions

this function retrieves a list of available columns from the api server based on the selected table, calls the populatecolumnoptions function to populate the domain and range drop down menus with those columns, and sets the selected domain and range.

populatedomainandrangeoptions() {
  d3.json("http://localhost:8153/api.rsc/" + selectedtable + "/$metadata?@json")
    .header("authorization", "basic " + btoa(username+":"+authtoken))
    .get(function(error, data) {
      if (error) throw error;
      populatecolumnoptions("domain", data);
      populatecolumnoptions("range", data);
      selecteddomain = d3.select("select.domainselect").property("value");
      selectedrange = d3.select("select.rangeselect").property("value");
    });
}

populatecolumnoptions

this function uses the raw data from the api server column request to populate the domain and range drop down menus and define the behavior for both drop down menus when the selection changes.

populatecolumnoptions(data) {
  var values = data.items[0]["odata:cname"];
  var axes = ["domain", "range"];

  axes.foreach(function(axis) {

    d3.select("select." + axis + "select")
      .selectall("*")
      .remove();

    d3.select("select." + axis + "select")
      .on('change', function() {
        clearchart();
        if (axis == "domain")
          selecteddomain = d3.select("select." + axis + "select").property("value");
        else if (axis == "range")
          selectedrange = d3.select("select." + axis + "select").property("value");
      })
      .selectall('option')
      .data(values)
      .enter().append("option")
      .text(function(d) {
        return d; 
      });
  });
}

buildchart

this function does all of the work to build the bar chart, using the values from the selected domain column to fill out the x-axis and the values from the selected range column to set the height of the bars and fill out the y-axis. the function retrieves all of these values by performing an http get request against the api server, requesting only the selected columns from the selected table.

buildchart() {
  d3.json("http://localhost:8153/api.rsc/" + selectedtable + "/?$select=" + selecteddomain + "," + selectedrange)
      .header("authorization", "basic " + btoa(username + ":" + authtoken))
      .get(function(error, data) {
        if (error) throw error;

        var values = data.value;

        x.domain(values.map(function(d) { return d[selecteddomain].tostring(); }));
        y.domain([0, d3.max(values, function(d) { return d[selectedrange]; })]);

        g.append("g")
            .attr("class", "axis axis--x")
            .attr("transform", "translate(0," + height + ")")
            .call(d3.axisbottom(x));

        g.selectall("g.tick")
            .attr("text-anchor", "end")
            .selectall("text")
            .attr("transform", "rotate(-45)");

        g.append("g")
            .attr("class", "axis axis--y")
            .call(d3.axisleft(y))
            .append("text")
            .attr("transform", "rotate(-90)")
            .attr("y", 6)
            .attr("dy", "0.71em")
            .attr("text-anchor", "end")
            .text("value");

        g.selectall(".bar")
          .data(values)
          .enter().append("rect")
            .attr("class", "bar")
            .attr("x", function(d) { return x(d[selecteddomain].tostring()); })
            .attr("y", function(d) { return y(d[selectedrange]); })
            .attr("width", x.bandwidth())
            .attr("height", function(d) { return height - y(d[selectedrange]); })
            .attr("title", function(d) { return d[selectedrange]; })
            .text(function(d) { return d[selectedrange]; });
      });
}

clearchart

this function simply removes all of the bar chart elements from the svg, allowing you to build new charts whenever you need.

clearchart() {
  d3.select("svg")
    .select("g")
    .selectall("*")
    .remove();
}

configuring the d3.js web app

with the connection to the data configured and the source code for the web app reviewed, you are ready to start the web app. you need to have node.js installed on your machine and you will need to install or run a web server in order to properly manage the requests and responses between the web app and the api server.

setting up the web app

in the next few steps, you will set up your web app, and create and populate the package.json file.

1. in the command line, change to the directory with the source files: cd ./apiserver-d3
2. once in the directory, interactively create the package.json file: npm init
3. follow the prompts to set the name, version, description, and more for the package. since this is a sample project, you can safely use the default values or blanks for the test command, git repository, keywords, author, and license. with package.json created, simply edit the file and replace "test": "echo \"error: no test specified\" && exit 1" with "start": "node node_modules/http-server/bin/http-server -p 3000 -o" in the "scripts" element. this will start a light-weight http server when you use npm start to run the web app.

installing a lightweight http server

with the package.json file created, you can now install a lightweight http server to run the web app if needed:

running the web app

now that you created the package.json file and installed the necessary modules, you are ready to run the web app. to do so, simply navigate to the directory for the web app in a command-line interface and execute the following command:

when the web app launches, you will be able to see the drop down menus for the table, domain, and range, along with the button to click to build the chart. the list of tables and columns are retrieved from the api server and include all of the tables you added as resources when configuring the api server.

once you select the table and columns, you can click the get [ table ] data button to retrieve data from your database via the api server. the bar chart will be built based on the table you selected and the columns you selected for the domain and range.

below you can see a sample bar chart based on the number of bytes in a given song track, listed by the title of the track.