In this tutorial, learn how to integrate Grafana with an internal app using the AuthProxy module of Grafana.

What is an Operator and why did we create one for IBM App Connect?

Learn about a new framework that allows processing methods asynchronously with retries in case of failure and the support of load-balancing and failover.

You may be familiar with using DevTools to modify a website's HTML. In this article and video, discover two more tools: contenteditable and designMode.

This article will import a standard GitLab library and the underlying data into two databases to see if GitLab could be started then do a comparison test.

This article compares how well these two databases support GitLab to a certain extent and reflect the compatibility with standard PostgreSQL.

Jira is feature-rich and exceptionally flexible, bending to meet your needs. Here are the add-ons and integrations to improve your software delivery processes.

This tutorial explains how to install the IBM App Connect Operator on an Amazon EKS cluster and deploy an integration using it.

Can we take microservice architecture patterns and apply them to the frontend?

As the head of DevOps, my frustration was approaching Will Smith slapping Chris Rock levels. Here’s what happened.

Ever wonder how Kubernetes would handle a 1000-nodes Cassandra cluster?

KEDA is an event-driven autoscaler that horizontally scales a container by adding additional pods based on the number of events needing to be processed.

The article discusses the architectures currently popular for achieving this Serverless Architecture for Scale use cases and how and when we can use them.

Getting Docker working as a local network host on Windows requires quite a lot of kludgy hodgepodge of hacks to work, but it can be done.

In this second part, learn how to centralize IAM for multiple AWS accounts, create and use EC2 instance profiles, and implement just-in-time access with Vault.

Covering the basics of managing AWS IAM with Terraform and learn how to delete the Root/User Access key, enforce MFA, customize password policy, and more.

This article will show how to expose an application using the service type load balancer.

A few weeks ago I wrote a blog on synchronizing multithreaded integration tests, which was republished on DZone Javalobby from where it received a comment from Robert Saulnier who quite rightly pointed out that you can also use join() to synchronize a worker thread and its unit tests. This got me thinking, just how many ways can you synchronise multi-threaded integration tests? So, I started counting... and came up with: Using a random delay. Adding a CountDownLatch Thread.join() Acquiring a Semaphore With a Future and ExecutorService Now, I’m not going to explain all the following in great detail, I’ll let the code speak for itself, except to say that all the code samples do roughly the same thing: the unit test creates a ThreadWrapper instance and then calls its doWork() method (or call() in the case of the Future). The unit test’s main thread then waits for the worker thread to complete before asserting that the test has passed. For the sample code demonstrating points 1 and 2 take a look at my original blog on Synchronizing Multithreaded Integration Tests, though I wouldn’t recommend point 1: using a random delay. Thread.join() public class ThreadWrapper { private Thread thread; /** * Start the thread running so that it does some work. */ public void doWork() { thread = new Thread() { /** * Run method adding data to a fictitious database */ @Override public void run() { System.out.println("Start of the thread"); addDataToDB(); System.out.println("End of the thread method"); } private void addDataToDB() { try { Thread.sleep(4000); } catch (InterruptedException e) { e.printStackTrace(); } } }; thread.start(); System.out.println("Off and running..."); } /** * Synchronization method. */ public void join() { try { thread.join(); } catch (InterruptedException ex) { Thread.currentThread().interrupt(); } } } public class ThreadWrapperTest { @Test public void testDoWork() throws InterruptedException { ThreadWrapper instance = new ThreadWrapper(); instance.doWork(); instance.join(); boolean result = getResultFromDatabase(); assertTrue(result); } /** * Dummy database method - just return true */ private boolean getResultFromDatabase() { return true; } } Acquiring a Semaphore public class ThreadWrapper { /** * Start the thread running so that it does some work. */ public void doWork() { doWork(null); } @VisibleForTesting void doWork(final Semaphore semaphore) { Thread thread = new Thread() { /** * Run method adding data to a fictitious database */ @Override public void run() { System.out.println("Start of the thread"); addDataToDB(); System.out.println("End of the thread method"); semaphore.release(); } private void addDataToDB() { try { Thread.sleep(4000); } catch (InterruptedException e) { e.printStackTrace(); } } }; aquire(semaphore); thread.start(); System.out.println("Off and running..."); } private void aquire(Semaphore semaphore) { try { semaphore.acquire(); } catch (InterruptedException e) { e.printStackTrace(); } } } public class ThreadWrapperTest { @Test public void testDoWork() throws InterruptedException { ThreadWrapper instance = new ThreadWrapper(); Semaphore semaphore = new Semaphore(1); instance.doWork(semaphore); semaphore.acquire(); boolean result = getResultFromDatabase(); assertTrue(result); } /** * Dummy database method - just return true */ private boolean getResultFromDatabase() { return true; } } With a Future public class ThreadWrapper implements Callable { @Override public Boolean call() throws Exception { System.out.println("Start of the thread"); Boolean added = addDataToDB(); System.out.println("End of the thread method"); return added; } /** * Add to the DB and return true if added okay */ private Boolean addDataToDB() { try { Thread.sleep(4000); } catch (InterruptedException e) { e.printStackTrace(); } return Boolean.valueOf(true); } } public class ThreadWrapperTest { @Test public void testCall() throws ExecutionException, InterruptedException { ThreadWrapper instance = new ThreadWrapper(); ExecutorService executorService = Executors.newFixedThreadPool(1); Future future = executorService.submit(instance); Boolean result = future.get(); assertTrue(result); } } Having listed all these methods, the next thing to consider is which one is the best? In asking that question you have to define the word “best” in terms of best for what? Best for simplicity? Maintainability? Speed or code size? After all Programming the Art of Making the Right Decision. You may have guessed that I don’t like the random delay idea and prefer the use of a CountDownLatch. Thread.join() is a bit old school; remember that the likes of Semaphore and CountDownLatch were written to improve on the original Java threading techniques. ExecutorService seems a little heavy weight for what we need here. At the end of the day the choice of technique really comes down to personal preference. Finally, I’ve listed five ways of synchronizing multi-threaded integration tests; however, if you can think of any others please let me know... The source code for this blog is available on Github.

There is a lot of talk in the devops community about the importance of sharing principles and values, and about silo busting: breaking down the “wall of confusion” between developers and operations to create agile, cross-functional teams. Radical improvement through fundamental organizational changes and building an entirely new culture. But it doesn’t have to be that hard. All it took us was 3 simple, but important, steps. Reliability First When we first launched our online platform, things were pretty crazy. Sales was busy with customer feedback and onboarding more customers. Development was still finishing the backlog of features that were supposed to already be done, responding to changes from sales and partners, and helping to support the system. Ops was trying to stabilize everything, help onboard more customers and address performance issues as more customers came on. We were all rushing forwards, but not always in the same direction. Our CEO recognized this and made an important decision. He made reliability the #1 priority – the reliability and integrity of our systems and of our customers’ data, and the services we provided. For everyone: not just ops, but development, sales, marketing, compliance, admin. Above everything else. It was more important not to mess up the customers that we had than to get new customers or hit deadlines or cut costs. Reliability, resilience, integrity have remained our #1 driver for the company over several years as we continued to grow. This meant that everyone was working towards the same goals – and the goals were easy to understand and measure: improving MTTF, MTTD and MTTR windows; reducing bug counts and variability in response time, improving results of audits and pen tests. It gave people more reasons to work together at more levels.. It reduced politics and conflicts to a minimum. Development’s first priority changed from pushing features out ASAP to making sure that the system was running optimally and that any changes wouldn't negatively impact customers. This meant more time spent with ops on understanding the run-time, more time troubleshooting operational issues, more reviews and regression testing and stress testing, anticipating compatibility issues, planning for roll-back and roll-forward recovery. Smaller, more frequent releases Spending some more time on testing and reviews and working with ops meant that it took longer to complete a feature. But we still had to keep up with customer demands – we still had to deliver. We did this by shortening the software release cycle, from 2-3 months to 2-3 weeks or sometimes shorter. Delivering less in each release, sometimes only 1 new feature or some fixes, but delivering much more often. If a change or feature had to be delayed, if developers or testers needed more time to make sure that it was ready, it wasn't a big deal – if something wasn't ready this week, it would be ready next week or soon after, still fast enough for the business and for customers. Planning and working in shorter horizons meant that development could respond faster to changes in direction and changing priorities, so developers were always focused on delivering what was most important at the time. Shorter releases drove development to be more agile, to think and work faster. To automate more testing. To pay more attention to deployment, make the steps simpler and more efficient – and safer. Fewer changes batched together made it easier to review and test. Less chances to make mistakes. Easier to understand what went wrong when we did make a mistake, and less time to fix it. RCA – Learn from Mistakes We still made mistakes, shit still happened. When something went seriously wrong, it was my job to explain it to our customers and owners. What went wrong, why, and what we were going to do to make sure that it didn’t happen again. We didn't know about blameless post mortems, but this is the way we did it anyway. We got developers and testers and ops and managers together in Root Cause Analysis sessions to carefully examine what happened, what went wrong, understand why, and fix it. We made sure that people focused on the facts and on problem solving: what happened, what happened next, what did we see, what didn’t we see, why? What could we do to fix it or to prevent it from happening again or to recognize and respond to problems like this more effectively in the future? Better training, better tools, better procedures, better documentation, better error handling, better testing and reviews, better configuration checks and run-time checking, better information and better ways of communicating it. Focusing on details and problems, not people. Proving that it was ok to make mistakes, but not ok to hide them. We got much better: at operations, testing, design, deployment, monitoring, incident handling. And better as an organization. We built transparency and trust within and across teams. We learned how to move forward from failure, and to be more resilient and confident in our ability to deal with serious problems. Delivering Better and Faster Together We didn't restructure or change who we were as an organization. Dev and ops still work in separate organizations for different managers in different countries. They have their own projects and their own ways of working, and they don’t always speak the same language or agree on everything. We have lots of checks and balances and handoffs and paperwork between dev and ops to make sure that things are done properly and to make the regulators happy. There are still more steps that we could automate or simplify, more we can do to build out our Continuous Delivery pipelines, more things we can get out of Puppet and Vagrant and other cool tools. But if devops is about developers and operations sharing responsibility for the system, trusting each other and helping each other to make sure that the system is always working correctly and optimally, looking for better solutions together, delivering better and faster – then we’ve been doing devops for a while now.

Separate out analysis to preserve commit stage processing time The entry point of a Continuous Delivery pipeline is its Commit Stage, and as such manages the compilation, unit testing, analysis, and packaging of source code whenever a change is committed to version control. As the commit stage is responsible for identifying defective code it represents a vital feedback loop for developers, and for that reason Dave Farley and Jez Humble recommend a commit stage that is “ideally less than five minutes and no more than ten” – if the build process is too slow or non-deterministic, the pace of development can soon grind to a halt. Both compilation and unit testing tasks can be optimized for performance, particularly when the commit stage is hosted on a multi-processor Continuous Integration server. Modern compilers require only a few seconds for compilation, and a unit test suite that follows the Michael Feathers strategy of no database/filesystem/network/user interface access should run in parallel in seconds. However, it is more difficult to optimize analysis tasks as they tend to involve third-party tooling reliant upon byte code manipulation. When a significant percentage of commit stage time is consumed by static analysis tooling, it may become necessary to trade-off unit test feedback against static analysis feedback and move the static analysis tooling into a separate Analysis Stage. The analysis stage is triggered by a successful run of the commit stage, and analyses the uploaded artifact(s) and source code in parallel to the acceptance testing stage. If a failure is detected the relevant pipeline metadata is updated and Stop The Line applies. That binary cannot be used elsewhere in the pipeline and further development efforts should cease until the issue is resolved. For example, consider an organisation that has implemented a standard Continuous Delivery pipeline. The commit stage has an average processing time of 5 minutes, of which 1 minute is spent upon static analysis. Over time the codebase grows to the extent that commit stage time increases to 6 minutes, of which 1 minute 30 seconds is spent upon static analysis. With static analysis time growing from 20% to 25% the decision is made to create a separate Analysis stage, which reduces commit time to 4 minutes 30 seconds and improves the developer feedback loop. Static analysis is the definitive example of an automated task that periodically needs human intervention. Regardless of tool choice there will always be a percentage of false positives and false negatives, and therefore a pipeline that implements an Analysis Stage must also offer a capability for an authenticated human user to override prior results for one or more application versions.