OS Framework Selection: How to Spot Immature Frameworks During Selection

New frameworks appear all the time in the open-source space. While a new and cutting edge framework is exciting, there is quite a risk attached to immaturity, including incompleteness, bugs, incompatible upgrades, and unpredictable direction.

You may also like: Open Source and Its Tools Are Critical to Modernization

In most projects, you have no time to battle the shortcomings of a new open-source framework, and you rather be productive with proven technology. This article is all about assessing the maturity of a framework during selection.

Why Framework Maturity Matters

A framework is supposed to help you be more productive. You don't want the time saved by productivity to go to waste again, chasing after framework bugs or fixing damage caused by incompatible upgrades, which is typically seen with newbie frameworks. Or even worse, the framework's vendor started with an incomplete vision and pivots in a direction that goes completely against your needs.

Once a framework isn't beta anymore, it's not always obvious how finished and market-ready it really is. Of course, frameworks say "we've been using it ourselves first," but that means little. Internal developers have a perk you don't have: They can rely on help from the original creators, immediately available within office-chair-rolling distance and able to fix bugs on the spot when needed.

First question: How many years has it been around?

For durable production software, you should prefer mature frameworks; those that are a few years old, used by multiple companies on serious projects, and a considerable size and visible community.

Peek Through the Back Door

With frameworks that look mature enough, you should sneak in through the back door and see what is going on in the kitchen. These are the things to look out for:

Not-Implemented API Functions

Click through the API to find specific functions that are not implemented yet. You'll see this often with young frameworks.

If not-implemented functions are about more advanced or very specific niche logic that you're not going to need now, that's fine: They know where they want to go, they are going there, and they described it to let you know it's coming. So, you can take that into account.

But if you find not-implemented functions that cover the really basic, low-level stuff: bad news. Dismiss the list — at least for now.

Scan the Release Notes for Incompatible Upgrades

A good indicator for a framework that might change direction later is looking for announcements of incompatible upgrades in the release notes. Phrases like "minor changes needed when upgrading" are understatements and bad signs, especially if it's frequent and unannounced.

"Minor" in this context often is a euphemism for "it's just one change, but to be applied in a thousand places." If data storage is broken by an upgrade, that's a big red flag — dismiss.

Why is it an indicator? If the framework developers knew where they were going already, they'd have thought a lot about how to design the API so that it lasts as long as possible without incompatible changes ever. They know such events are hugely annoying for developers, and it's a community traction risk that can be your indicator to start looking for another framework.

Signs of well-designed breaking changes: they built an entirely new API next to the old one and announced six months to a year in advance; a long cooldown period so that you can make the changes at your own pace; and awaited a few patch releases to be sure it's stable before following the early movers.

Bottom line: Features dropped when the framework is young: bad sign. Features dropped when the framework is mature and because no one uses the feature anymore: good sign.

Browse the Bug Database

Even the best-written software has bugs, and having an accessible bug database in itself is a good sign. It's the content that matters.

• A small bug database is bad news: not many people use it, or the users have very little faith in their problems getting fixed, big red flag, to the dismiss list.

• A big bug database actually is a good sign, it means a lot of people are using it seriously; of course, only if the majority of bugs are solved.

It's the kind of bugs you find there that tell a lot about how far they are and what they struggle with.

Sample your way through the bug database randomly and seek the highest-rated bugs or the longest open bugs. And from a handful of recently solved bugs, scroll to see the tone of voice in the discussion before it. It is surprising how much you can learn from superficially browsing and scrolling through that information.

If they are struggling with basics and fundamentals, or they have big problems they don't manage to solve despite stacking complaints, here you find your reasons to move it to the dismiss list.

Also look at who fixed the bugs, and how long ago; it could be an abandoned open-source community. Find where the community flocked next, maybe it's a good candidate for you.

Scan Success Stories for Omissions

If there are famous logos on a framework's site with success stories behind them, it's fairly reliable info. These big brand's lawyer-hawks and employees involved will have scrutinized and approved these success stories. They don't want to see lies with their name under it.

But the framework's vendor wants to make the success story as shiny as possible. You should look for what the success story doesn't say — how it's used, at what scale, money amounts saved, numbers of users, etc.

If you were the framework's marketer, what would you want to say on its web site? Is it there? And reversely, be aware that only a logo with no success story could mean that the founders just did a freelance gig for them in the distant past.

Suppose you come across an authentic, detailed, and non-anonymous disaster testimonial, take it seriously. Why? For one customer who dares to speak up publicly about a fail, there are at least ten more who feel ashamed and rather keep their skeletons in the closet. There's a fair chance you'll become one of those ten. But investigate, it might be FUD.

Check Open Job Positions

Open job positions at a framework's home company can give a brutally honest insight in what the company is struggling with, and where they indirectly admit they fall short. Together, it gives some idea of what is going on in there, how mature the organization is, and whether you should be worried about a possible decline or growing way too fast.

Follow the Money

Always know who's behind it. If they have funding, it will be in the "about" section. For a young framework, a big-money source is a good sign, but not a reliable predictor of a long life per se. Google search press releases to find out if it's one-time donation or strategic support.

There are very good frameworks that are 100 percent unpaid open source because the makers are scientists or idealists, or just don't like managers and investors breathing down their necks. Usually, you get superior technical quality but infrequent updates and long bug fix wait time.

Often, the money source explains why they do what they do. It tells you development direction, revenue/pricing model, and what you can expect in the future.

Over the last few years, tech giants have released hundreds of open-source initiatives and invested billions of dollars in them. But giants can afford to terminate projects that stalled (for example, heavily marketed VR platform Google Daydream just got killed). Giants can freely reassign focus to frameworks that got more traction. Not fun if you depend on it. Just as with any party, wait until it's a few years old before depending on it.

Be on the lookout for mixed interests: if a giant offers free hosting if you use their framework, you can feel the reason for this generosity.

Another example: If they encourage to use their browser framework by including their URL to the .js, .css, or font file on their servers, the real reason is getting free user data from the millions of sites that <link rel>   to it, so they can sell access to this data. Maybe you and your current customers don't care about that, but a future customer might be held to strict compliance rules or fears of leaking usage figures to their competitor.

Such a catch will not always be a direct problem for you. But go look for it so that you know what you're getting yourself into, and possibly might need to get rid of later.

What If it Has Bugs?

It is often underestimated how stunningly low the quality of a framework can be. It's new, it's buggy, it's incomplete, they move fast, and they break things. Your things.

About Fixing Bugs Yourself

If it's an open-source framework and not minified, it often is easy to pinpoint a root cause in a framework bug and propose a fix. The problem is what happens next.

If you fix it in your own copy, you create a fork. You are responsible for merging with the official upgrades and keeping it bug-free, each time, requiring a structural understanding of the framework.

You really don't want to fix bugs in the framework yourself.

About Joining an Open-Source Community

If it's open-source, you could consider becoming part of the development community. But that can be the Road of No Return — you still have to wait and see if a proposed bug fix is accepted into an official release, and when — if ever.

How long does it take for a fix to be released? Can you afford to wait that long with your project?

Mind that the founding fathers may already have a completely different solution in mind and it will be a while before they pick that part up, let alone how long until it's done.

Or, it is possible that they do all of this in their free time and don't feel rushed by anyone or anything at all.

If, and only if, a framework is the absolute #1 part of your core business, it is worthwhile to join a framework's development community. If you are able to be a big player by donating a lot of developer capacity, it can work for you. Internet giants fund and buy open-source projects all of the time. 

Being part of an open-source project is great; it's motivating, brings personal growth, and expands your career perspectives. But do not hope to steer an immature open-source project in your direction by sponsoring it with your developer time.

To the framework's organization, you are a friendly neighbor offering to help paint their house, but in return, you demand to pick the colors. That does not work. I saw several such cases fail, and none that succeeded. I'm glad I always declined to partake.

Dealing With Framework Bugs, Realistically

Nothing is bug-free. Let's be realistic about that.

The only tidy solution for a framework bug is using a workaround as follows: Do not call the framework directly, replace all calls by calling a wrapper function that you wrote that calls the framework for you. Let's name it "isolator."

Inside the isolator, add a workaround, a commented-out real solution, and a comment-line with the bug ID and URL from their bug tracker (if you had time to report the bug with them). If they release a version with that bug fixed, uncomment the desired use and comment-out the workaround. Make unit tests for the isolator so that you can verify it's broken in the current version and fixed in the new one.

If it's a bug you can't work around and can't live without — bad news. There's nothing you can do, except replace the entire framework with another one.

Conclusion

When selecting frameworks, watch for unimplemented API calls and investigate their bugs database, job vacancies, and money source. You will know if the framework struggles with immaturity and bugs in the basic, or very exotic, bugs you can live with. Avoid becoming a framework's free guinea pig department.

All in all, it is better to select frameworks that are a few years in their maturity so that they take less of a risk.

Further Reading

Open Source and Its Tools Are Critical to Modernization

16 Free and Open-Source Business Intelligence Tools