DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Popular
  3. Open Source
  4. This Will Surprise You

This Will Surprise You

Here's an interesting way to visualize the repositories in your infrastructure to combat inconsistency.

Rod Johnson user avatar by
Rod Johnson
·
May. 17, 19 · Tutorial
Like (1)
Save
Tweet
Share
3.87K Views

Join the DZone community and get the full member experience.

Join For Free

How many different Docker base images is your organization running? Which ports do your Docker containers expose? How many versions of core technologies like TypeScript or Spring Boot are you running? How many versions of any particular library?

Chances are, the answers are going to surprise you. Inconsistency brings risk and slows progress. Different technologies and versions have different bugs, capabilities, security vulnerabilities, and implications for your developers. Inconsistency makes it hard to move forward quickly.

The move toward cloud-native applications has brought fragmentation that can approach chaos. We no longer have a small number of very large repos, but a very large number of small repos. In previous posts I've discussed the risks of fragmentation in CI files; today I'll discuss several other kinds of fragmentation, and show how Atomist can help you better understand your organization today.

The Problem: Inconsistency and Drift

Consider TypeScript version. One of the awesome things about TypeScript is how fast it moves. It's good to be able to use those cool new features consistently. But occasionally we do encounter breaking changes, and the longer repos remain un-updated, the harder they become to fix.

Atomist's client framework and supporting libraries are written in TypeScript. Here's an interactive visualization of Atomist's open source code on GitHub, breaking down the TypeScript versions:

Image title


This sunburst is rendered using d3. The innermost ring is the TypeScript version specified in package-json. The outermost ring shows individual repos. Clicking on the intermediate rings makes their names visible and provides clickable links.

Most repos are fairly up-to-date, but some need attention. Those with the oldest versions may be dead — that itself is useful, actionable information.

The story is similar for versions of any library. Consider Atomist's microgrammar parsing library, used by a subset of projects. The innermost ring is the organization, as this screenshot shows data from private repositories at atomisthq as well as open source at atomist. The next is the version of the library specified in package.json. The next is the version that was resolved, according to package-lock.json, which may not be the same. (For example, ^0.6.0 resolved to 0.6.1 in one repo.) Again, the outermost ring shows individual repos.

It would be good to get up to date, and especially to move off milestone versions. Again, it seems that there may be dead repos that should be archived.

Let's dig deeper down the stack. Consider Docker base images, again across both Atomist organizations:

This visualization is surprising and demands further investigation. We need multiple base images for different types of projects, but why multiple versions of any one base image? Why 3.3, 3.4 and latest of alpine? Why 2 versions of java and 7 of node?

Let's consider the spread of exposed Docker ports, which may also have security implications:

No great surprises or obvious action items here. But as well as surfacing potential risks, this visualization helps understand what each repository does, and how many there are of each type. For example, those exposing 2866 are Atomist SDMs.

These visualizations provide useful, actionable information about Atomist's own code. What could you learn about your organization?

Analyzing Your Own Organization

We'll be building visualization functionality into the Atomist service. However, you can get insights into your own code right now using open source by following these steps:

  1. If you don't already have the Atomist CLI, install it via npm i --global @atomist/cli.
  2. Clone https://github.com/atomist-blogs/org-visualizer.
  3. In its root directory, run npm i
  4. Run npm run build
  5. Run npm link to install the spider binary that can analyze GitHub organizations.
  6. (Optional) If you need to work with private repos, ensure that your GitHub token is available to the Node processes via a GITHUB_TOKEN environment variable.
  7. Run spider <org> to analyze a GitHub organization. To see the open source part of the data shown in this post, type spider atomist.
  8. Start the server via atomist start --local and navigate to http://localhost:2866 to see the visualizations.
Spidering currently only supports GitHub.com. We'll soon add support for GitHub Enterprise and BitBucket. .

Surface What You Most Care About

These specific reports are valuable, but the true power of this approach is in its extensibility. While there are existing point solutions for handling certain aspects of projects, the open source Atomist analysis framework allows you to analyze and expose aspects of your choice, enabling you to better understand what matters in your organization, even if it's unique to your needs.

There are three layers in org-visualizer, all extensible and customizable:

  1. The analysis framework (from @atomist/sdm-pack-analysis) enables scanners that extract relevant data from projects. Here's a real scanner that extracts Node data.
  2. A query layer that can create JSON structures from persisted analyses and expose them through Express routes. This layer provides support for grouping, splitting and rendering data, to any number of levels. Here's the TypeScript version query.
  3. A simple UI using Handlebars and d3 for client-side rendering. JSON data is also exposed over HTTP.

Anything in any of your repos is accessible to Atomist analysis. Suppose you want to check whether all your repos have a code of conduct — an important consideration in open source organizations. Add the following scanner. Its implementation benefits from the simple, testable, Project abstraction that Atomist provides:

After analyzing with this scanner enabled, you can visualize the new path within the analysis model by visiting here.

Whoops, action required here. We still have a few repos within the Atomist open source org that don't have a code of conduct — something we'll fix quickly, now we're aware of it.

What data could you extract and query to deliver insights concerning your unique needs? What generic data could you extract and query to help the community? Pull requests welcome!

Atomist: An Enabler for the Cloud Native Era

Gathering and exposing this information is possible because, unique among software delivery technologies, Atomist understands your repositories. This understanding allows you to specify consistent delivery policy across your entire organization, and also makes Atomist a great basis for reporting.

Try the org-visualizer today on your own to learn more about your own repositories. Join the Atomist community Slack and tell us if there are particular insights you'd like. Learn more about Atomist to see how it can help with development and delivery at scale.

In my next post, I'll look at how Atomist can also help to start rectifying problems identified this way.

Open source

Published at DZone with permission of Rod Johnson, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • The Importance of Delegation in Management Teams
  • Java Development Trends 2023
  • Streamlining Your Workflow With the Jenkins HTTP Request Plugin: A Guide to Replacing CURL in Scripts
  • 9 Ways You Can Improve Security Posture

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: