Data Visualization on the Mainframe
Data Visualization on the Mainframe
After attending a seminar about visualizing the genealogy of pop/rock music, I was inspired to think even more seriously about the value of mainframe data visualization.
Join the DZone community and get the full member experience.Join For Free
Hortonworks Sandbox for HDP and HDF is your chance to get started on learning, developing, testing and trying out new features. Each download comes preconfigured with interactive tutorials, sample data and developments from the Apache community.
When we compare past methods of mainframe data analysis to the capabilities of modern data visualization software, it’s amazing how much more information a well-designed visualization can provide.
I recently attended a seminar of Edward Tufte, a statistician, artist, and Professor Emeritus of Political Science, Statistics, and Computer Science at Yale University. I was mesmerized by the wealth and power of information displayed in an amazing visualization image Tufte showed, entitled “Genealogy of Pop/Rock Music,” designed by the artist Reebee Garofalo and found in Tufte’s book, Visual Explanations.
Figure 1: Genealogy of Pop/Rock Music. (Concept and design: Reebee Garofalo. Graphics services: Damon Rarey and Jean Nicolazzo. Tufte, Edward. Visual Explanations: Images and Quantities, Evidence and Narrative. Graphics Press; Cheshire, CT, 1997.)
If you zoom in on Garofalo’s image, it’s amazing to see the impact of the British Invasion, starting with the Beatles and morphing into many of the bands I grew up with. I also found myself sad to see the demise of Rockabilly, which Elvis pioneered and Buddy Holly and Roy Orbison built upon. Finally, I had to laugh as I recalled listening to Bubblegum (though I had never called it that), music made popular by Sonny and Cher and the Partridge family. I’m actually relieved that one faded away quietly.
With that, I was in the mood to reminisce. I thought back to my own discovery that the visualization of data relationships on the mainframe provides a similar wealth and power of information as Garofalo’s image.
Determining Data Relationships Before Data Visualization on the Mainframe
I started thinking seriously about the value of mainframe data visualization a few years ago while working with a group of people tuning and debugging large data extracts for a customer in South America. The customer needed to fit into a much shorter window large, complex data extracts taking far too long to run. The customer needed agility, but what they had was a bottleneck. Our mutual frustrations included very complex data relationships, limitations of the operating system and difficulty communicating due to a language barrier. We needed one solution to clarify all of these issues.
But as a new product owner coming from a mainframe background, I was really only familiar with the erstwhile wonders and marvels of the historical 3270 green screen. Because we were using a mainframe product to manage our customer’s extracts, we were relegated to painfully analyzing lines upon lines of formatted green screen text, and a whole lot of hitting “PF8” and “PF7” to scroll up and down pages of data, trying to make sense of it all.
As is often the case in the mainframe world, we seasoned engineers defaulted to drawing large rectangles on a whiteboard and connecting them with straight lines, representing our data objects and the relationships between them. It was mainframe work and we were “mature,” so the rectangles were necessarily large. We soon ran out of whiteboard, and it became too difficult to adequately gather the information required to improve the extracts.
Life After Mainframe Data Visualization of Data Relationships
It took us weeks of white-boarding and head-scratching before we discovered a better solution: exporting the mainframe text data into a mainframe software package — an Eclipse-based GUI — that painted an easily understood, concise rendering of the data with which we were working.
Seen in this new light, our subject matter experts were able to notice inconsistencies in regards to how data relationships were defined. Several of the objects from which we were extracting had multiple parent objects, and in some cases, recursive looping occurred. Our visualization of the data relationships made this easier to identify and fix, after which we established best practices to ensure issues wouldn’t recur. Visualization of the data relationships became a cornerstone in these improved practices.
Leaving Behind the Green Screen (and Bubblegum)
Today, data visualization of data relationships is essentially at the heart of what I do as a seasoned product owner of two years. Along with my product manager and development Scrum team, I’m now involved in a much larger effort to mainstream the mainframe, bringing the platform into Agile and DevOps, putting a delightful twist on those wonders and marvels I referred to as being found on the green screen.
While I value my history doing data analysis with green screens and whiteboards — arduous as it was — I’m relieved this method is fading away like the Bubblegum pop stars of the 70s. Not that I disdain the old way of doing data analysis on the mainframe, or Bubblegum music for that matter, but clear data visualization of data relationships provides much greater value to mainframe shops looking for ways to innovate for the fast-paced digital economy.
Published at DZone with permission of Bill Mackey , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.