OS Framework Selection: Reversible Real-World Trial

When selecting an open-source framework, the proof of the pudding is in the eating. You did your research, you did a thorough review process, and one candidate stood out among the rest and offered the best solutions for your use case.

So, the framework looks great, but does it really live up to all of its promises? You must test it thoroughly before adopting it for good. You know that a "try-me" tutorial is not enough. However, performing a real test is a lot of work.

You may also like: OS Framework Selection: How to Spot Immature Frameworks During Selection

Here are some realistic and efficient tips for testing a framework.

Reversible Production Trial Concept

The idea is as follows: test it all the way through the Develop-Test-Acceptance-Production (DTAP) pipeline. Yes, use it in production.

Of course, you take all the usual precautions. DTAP procedures, automated unit test, and deploy, in short, everything you would do with a "normal" release cycle.

One other thing that is vitally important: This trial must be fully reversible. If it was satisfactory, the code stays. If it failed, you remove it.

Trial code must be written in such a way that you can toggle it on and off and organized so that you can remove it out quickly if it fails at any point in the journey from development to production.

A framework must be used in your existing codebase before deciding to use it permanently. Not in a test or mockup, not as a tutorial, but in real, actual production software that real end-users are going to use.

You need to learn how the beast behaves in your real-world and whether it works for your environment, your developers, and your end-users. Others may have been super successful with it, but they have different situations and requirements.

During the trial, you embed the framework in your codebase, and you should add precautions for quick and easy removal later. This too is part of the test. If it is difficult to do this, the framework failed.

Frameworks that can't be embedded for easy removal do not belong in a sustainable codebase and will become a legacy problem later. They must not be selected to begin with!

Why in Production? Are You Nuts?

Only in real production use, you will learn the true practical value of a framework.

Demos and tutorials tell very little, even if they worked fine. When you use it for real, there always comes a string of catch-22s, gotchas, effects of real data on a real scale, performance issues, and unexpected side-effects. These will never show up in demos. Vendors make sure the demos work.

Half of the time, problems you get are not even the framework's fault, but something in the environment you use it in. But it needs to be discovered and dealt with nonetheless.

If you don't test all the way into production, you skip the question the entire trial was about: Does it behave in the real world? Fundamentally, putting something in production is a test too.

But what's the alternative? Test the new framework no further than acceptance? Then what? You still need to go into production someday, officially, without the toggles and precautions of a reversible trial. If that fails, it fails out in the open, and you have to release an official emergency-patch. Not cool.

A nice upside about such a real-life trial is if it was successful, you wrote useful code and you didn't waste time on throwaway stuff; nor do you risk that impatient commercial people force you to base serious software on a throwaway prototype, as often happens after hackathons (click here for more on hackathon abuse).

What to Watch During the Trial Period

Obviously, you want to see "if it works," but a trial offers more opportunities to learn.

Unpredictable Humans

Users are the number one reason why you should test all the way through production.

Real users aren't scripts; they react unpredictably. For example, if a super slow feature was made very fast by the new framework, users might start to click on it so often that it causes an overload situation elsewhere in the system.

If something turns out slow, real users might get impatient and click repeatedly, causing longer event queues to process, worsening the overload and causing multiple executions of the same action.

You won't see such things with acceptance users obediently clicking through a test script and taking time to make notes in the test report in between each action.

Opportunities in Acceptance Trialing

When it comes to your trial, everyone hopes it's a success, and if it fails, everyone hopes it buckles in test or acceptance, not in production.

Acceptance is the ideal environment to create overload situations and excessive data quantities, and see how it holds. Normally, acceptance will run a copy of real production data on the same size hardware as production, and you'd run automated tests that behave similarly to real-world users.

Benchmark Performance and Scale Times Ten

You should test for tomorrow's circumstances. You should make a random data generator and conduct tests with ten times the data size and ten times the load you expect in real use.

If it can handle ten times, you can be confident about production.

If it fails, no worries. Keep lowering the load and data size until it runs fine. Now you know the breaking point. This gives a rough idea of how much time you have until real user growth and data growth reach that breaking point. If that breaking point is too close, a completely different approach is needed, and now you know it before going live.

Did the Framework Make You More Productive?

A framework's goal is supposed to make you and your team more productive, directly or indirectly.

You should monitor how your fellow developers feel about coding for this platform. At first, there are always complaints, because it's new, you are learning and everything has quirks. If the complaints remain or increase, you should worry about how productive you will be with the framework, and perhaps dismiss it early.

After all, you and your team are the real users of a framework. If it doesn't work for you, it doesn't work. If an API is hard to use, ask around and reach a consensus on whether the hardships outweigh the benefits. While everyone is busy using the platform, the question "will it make us more productive?" is often overlooked, but it's by far the most important question the trial should answer.

Pay Attention to Error Message Quality

Error message quality is vital for productivity. It's only shown during the trial when you are actually writing code to use the framework, use real data, make mistakes, and things start spewing out errors.

What you want to see:

• Error messages that tell precisely what, why, and where you did something wrong in a language that you understand

• Error messages with specific IDs or names that are easy to Google

• API functions that neatly validate input parameters and not crash halfway inside

• Stack traces that show the call hierarchy of a crash

• Specific errors, rarely see generic fallback-errors that could mean anything

If error messages are poor, you're chasing ghosts for hours, hanging on the phone with the service desk, project delays due to waiting for days, weeks, or months for answers posted in community forums and StackOverflow.

Quality error messages are a massive time saver and productivity gain, which you can safely multiply by the number of developers using the framework.

What to Do if the Trial Disappoints

Suppose it performs like a snail and crashes when you just look at it; there was a big elephant in the room happily concealed by marketing, all mistakes say "general error id:1", or developing on it deserves a special place in hell, in short, it disappoints? Feel blessed!

If it didn't work, you have learned something. (Apple R&D proverb)

Though you wish it had fallen off the list earlier, it is not a waste of time. You will now see the other candidate frameworks with new eyes. Remarks in discussion forums that didn't land first, suddenly get meaning. The knowledge you won makes integrating another framework a lot faster and easier. You should reevaluate not just your top-two or top-three but more alternatives with the new knowledge, so dig in your dismiss list too.

What to Do if the Trial Was a Success

When the framework is introduced, and it is a success, it stays in your code and is most likely to be used extensively. There are three side-effects to watch out for now:

1: Happy Flow Overfocus

If early results are above expectations, you can get too optimistic about the amount of "dull" work needed for a complete implementation. Optimistic estimates put stakeholders up with expectations that aren't met, after which the framework is blamed, and not the unrealistic happy flow mindset.

This is another reason why the framework should be trialed all the way into production, so the amount of dull work is felt too and taken into account in the development planning accordingly.

2: Freed of Performance Considerations

If the framework was needed for performance, and the results were impressive, it could backfire: there is a risk to get sloppy and don't think about efficiency, because it will be fast enough anyway. Then, all gains go to waste.

Everybody knows it sucks to work on performance bugs, so it's nice to be "freed" from that chore for a while: but this is a treacherous and short-lived situation.

While premature optimization is wrong, many performance problems are foreseeable. Even with the fastest framework, performance should remain a design consideration for any developer.

3: Three Months Off the Schedule

If the situation calls for drastic measures to get dramatic time savings immediately, new frameworks are not the way to do it.

Often a "saves three months" expectation is communicated as an argument to get a framework selection/introduction approved. The truth is that the time savings of a framework can be much bigger, but they come much later. For the first project, planners should expect no time savings or a small loss.

People need to learn using a framework, embedding it in the codebase. The actual time savings will happen a few months later and can be truly dramatic. I know a case that took multiple weeks per customer before, that turned into hours, then into a simple mouse click; but this was achieved step by step, over the course of two years.

Conclusion

If you have selected a framework, you should test it all the way through the DTAP pipeline into real production. Only then can you get honest and reliable metrics that are useful, performant, and productive. Do it reversibly so that you can go back if it disappoints and keep if it's a success.

Further Reading

OS Framework Selection: How to Spot Immature Frameworks During Selection

Happy Overflow Focus

Resolving Performance Bugs in 10 Steps