We all know that first-class UX is about delighting users. As designers and developers, we usually think about what we can ADD to our interfaces to wow our customers: responsive layouts, spinning widgets, and lightning-fast load times probably top the list of 'must haves' for a UI to be considered 'modern.' What we don't usually consider is what we should SUBTRACT from our applications to make the user experience a great one. This article is a quick run through of how we improved an unpleasant user experience by removing cumbersome steps and unnecessary clicks followed by a slightly deeper dive on the technical steps we took to do it.
At DataPortfolio, we build tools that help DevOps do more with less. One module of our product monitors the progress of batch workloads; series of actions with inputs and outputs that combine to accomplish a larger task. We call each of those actions a 'stage' of a 'pipeline,' and stages within a pipeline can be arranged in a Directed Acyclic Graph.
The First Approach
We tackled the DAG problem back-to-front by first creating the relational storage structures and the APIs to support interacting with the server-side graph. We operate on an MVP model of development, so our first UI modeled the way the database worked. Users would create a pipeline, then create stages within that pipeline. Once the user created all of the stages within a pipeline they would then use a drop-down to select the stages that they were related to. In other words, for each stage, the user would manually select which other stages were to happen first, and which stages were to happen after this stage were completed.
This approach worked, but users hated it. The general comment was "Wait, what is this again?" They got the bigger concept of what they were doing, but the minutia of all the button clicking got them off track. They eventually internalized the concept but no one was happy with all that clicking. This workflow was NOT delighting them.
A Better Way
We knew that there had to be a better way to accomplish this task. We whiteboarded a few approaches but were all excited by the concept of letting users drag-and-drop stages and their dependencies. No one on our team had experience with this kind of thing, so we were a little apprehensive when we started looking for a library to make this work. We checked out a number of 'graphing' tools but found that they often lacked the drag-and-drop composition functionality we needed. We then evaluated 'diagramming' tools that had drag-and-drop composition, were attractive, and rendered quickly. We ended up seriously evaluating six products (open and closed source) and doing test integrations for three of them.
After our evaluation and POC process, we decided to perform a full integration using the team's favorite library: JointJS/Rappid. We began demoing our POC to our users and it was clear they 'got it' right away. They could now intuitively compose full DAG pipelines in one UI without lots of unnecessary clicking. We were able to ADD to the UX by SUBTRACTING a whole bunch of nonsense and with a lot less effort than we expected.
When we demo our solution to technical crowds we often get asked about the diagramming and rendering functionality and how we do it. Before this implementation we also didn't have much idea about how it might be done, so we wrote this article to show you just how easy it can be to build first-class user interfaces. We hope that by pulling back the curtain you can get some ideas on how you might be able to ADD value by SUBTRACTING obnoxious user interface flows with less effort than you thought.
We'll save our graphing and diagramming evaluation process for another article and just show you what we did and how we did it.
We've discussed that the goal is to build a DAG that represents the stages in our data pipeline. Our initial approach involved selecting predecessors for each stage of a pipeline using a drop-down selector box populated with all the other stages in a pipeline. We decided that a better approach would be to assemble this pipeline graphically; we wanted to allow users to drag-and-drop the connections/dependencies between stages.
The Rappid trial package came with a handful of fully operational demo projects, one of which was a 'kitchen sink' project that had the core of what we needed: a 'stencil' that contained a visual grid of selectable objects and a grid-canvas that the objects could be dragged onto. Users could easily drag connector lines from 'ports' on the side of one object to visually connect them to another object on the canvas. That was all well and good, but what next?
Rappid 'Kitchen Sink' Demo App
We dug into the source code and began to understand the key components without too much effort. We liked a few things right away:
- All of the drag-and-drop icons were image files (e.g. PNG), which meant we had complete control over the size and shape of the icons to be used. This wasn't the case with some of the other tools we evaluated; they locked you down to a limited set of shapes (unless you were ready to cut deep into the source code).
- The data models that you could attach to an object were easy to define and link up using plain JSON. There was a pre-made panel that allowed users to input arbitrary data as well as sliders for scalars and drop-downs that could be populated with server-side data.
- Icons could be hooked into double-click handlers, allowing users to double-click a 'stage' icon and be routed to a drill-down page with more information about that stage.
- The styling of the canvas and stencil area was straightforward CSS, leading to easy branding.
We were confident that the drag-and-drop capabilities were in place, but we still needed to address the DAG layout concerns.
Automagic Layout of Diagrams
Many diagramming/graphing tools can lay out the nodes of a graph in a visually attractive way based on a display algorithm. Most tools support a pluggable model for layouts, allowing you to leverage an open-sourced layout model to arrange the nodes in a way that makes sense for your use case (e.g. hierarchical for an org chart, fixed positioning for a floor plan, etc.). However, not all layout algorithm implementations are compatible with all diagramming libraries. We had no interest in getting into the layout-algorithm-creation game, so support for a DAG layout algorithm was a must-have for us.
We confirmed that the open-sourced Dagre library supports Rappid/JointJS and came pre-packaged with the toolkit. Selecting the DAG layout algorithm was straightforward and allowed us to snap hand-positioned elements into a clean-looking DAG layout with the click of a button (erm, call of a function).
We loved the basic functionality of the Dagre algorithm, but it does have its limits. Our first implementation attempt allowed users to create a pipeline, all of its stages and map of the inputs and outputs of each stage in one graph. The Dagre algorithm wouldn't allow us to cluster the input/output files together, which lead to a jumbled layout. We solved this by stripping down the pipeline designer UI to only concern itself with stage connections and having the users configure stages within their own interface. This ended up being fortuitous; as stage configuration complexity grew, the 'all-in-one' UI approach would have broken down anyway.
Saving to the Back-End/API
Once we were confident that Rappid could satisfy the drag-and-drop composition and DAG layout requirements, we had to tackle saving our graph layout to the back-end API/Database structure. Serializing the Rappid graph to JSON is done in a single line of code and the format is easy to understand and parse.
The data is organized as a series of nodes and links, so the save routine is simply a matter of converting the node and link objects to the back-end format expected and persisting them, plus a little pre-persistence object-walking to do some client-side validation
Note the "id" field of the "basic.stageDef" objects and how they are linked together by the "source.id" and "target.id" fields of the "app.Link" object.
Loading From the Back-End
Loading data from the back-end into the drag-and-drop canvas is the inverse of saving it - make a few API calls to get the data in back-end form, de-serialize it into the graph form and then render it - easy-peasy. The Rappid framework allows you to render only the display canvas and easily hide grid lines, which allowed us to make a read-only real-time graph that showed the status of a currently executing data-pipeline with a few lines of code.
The 'read-only' version of the drag-and-drop canvas is fed similar data as the 'interactive' version with a few additional pieces of status information. Only the layout canvas is rendered this time (no stencils, data-model controls, and users can't change the links between objects), and the additional status information determines which stage image is to be loaded - one with a black background for 'initializing,' a blue background for 'running,' a green background for 'complete,' and a red background for 'failure.' As new/updated status information arrives in the front-end, we re-draw our graph and the colors change before the user's eyes.
DataPortfolio uses this same approach to visually display the inputs, outputs, and log files associated with a given stage, too, except this graph is composed of information entirely created by the system, NOT by the user. These graphs update in real-time and the graph layout is adjusted automatically thanks to the Dagre layout algorithm.
Addition Through Subtraction
We hope you've gotten something you can apply to your day-to-day from this article or at least enjoyed seeing how we added to our user's experience by subtracting a cumbersome UI process. We're by no means done improving the UX of DataPortfolio and haven't even begun to reach the limits of what Rappid can do for us. Reach out to us if you've got tips or tricks about Addition Through Subtraction or advice on leveraging Rappid to it's fullest - we'd love to hear from you!