In this tutorial, we will look at an example of creating a chatbot using Microsoft's Azure Bot Service.

Can we take microservice architecture patterns and apply them to the frontend?

As the head of DevOps, my frustration was approaching Will Smith slapping Chris Rock levels. Here’s what happened.

Secure data today while building and implementing the overall data protection plan for the future.

I'll share some of the most interesting questions, comments from the public, and, of course, the main highlights of the session from the TSQA 2022 Conference.

Ever wonder how Kubernetes would handle a 1000-nodes Cassandra cluster?

KEDA is an event-driven autoscaler that horizontally scales a container by adding additional pods based on the number of events needing to be processed.

Get started with testing in Rust, including a look at Cargo, cfg macros, and defining features.

Platform Engineering takes care of the common shared services that other development teams rely on to build their products and services.

The article discusses the architectures currently popular for achieving this Serverless Architecture for Scale use cases and how and when we can use them.

Getting Docker working as a local network host on Windows requires quite a lot of kludgy hodgepodge of hacks to work, but it can be done.

In this second part, learn how to centralize IAM for multiple AWS accounts, create and use EC2 instance profiles, and implement just-in-time access with Vault.

Covering the basics of managing AWS IAM with Terraform and learn how to delete the Root/User Access key, enforce MFA, customize password policy, and more.

SRE is a modern approach to managing the risks inherent in running complex, dynamic software deployments – risks like downtime, slowdowns, and the like.

This article will show how to expose an application using the service type load balancer.

I have talked about human filters and my plan for digital curation. These items are the fruits of those ideas, the items I deemed worthy from my Google Reader feeds. These items are a combination of tech business news, development news and programming tools and techniques. Making accessible icon buttons (NCZOnline) Double Shot #1097 (A Fresh Cup) Life Beyond Rete – R.I.P Rete 2013 (Java Code Geeks) My Passover Project: Introducing Rattlesnake.CLR (Ayende @ Rahien) Super useful jQuery plugins for responsive web design (HTML5 Zone) Android Development – Your First Steps (Javalobby – The heart of the Java developer community) Never Ever Rewrite Your System (Javalobby – The heart of the Java developer community) Telecommuting, Hoteling, and Managing Product Development (Javalobby – The heart of the Java developer community) The Daily Six Pack: April 1, 2013 (Dirk Strauss) Optimizing Proto-Geeks for Business (DaedTech) Learning Bootstrap Part 2: Working with Buttons (debug mode……) Rumination on Time (Rob Williams' Blog) Unit-Testing Multi-Threaded Code Timers (Architects Zone – Architectural Design Patterns & Best Practices) Metrics for Agile (Javalobby – The heart of the Java developer community) Detecting Java Threads in Deadlock with Groovy and JMX (Inspired by Actual Events) Entrepreneurs: Stop participating in hackathons just to win them (VentureBeat) How to hack the recruitment process to find the best developers for your startup or agency (The Next Web) Hardware Hacks: MongoLab + Arduino (Architects Zone – Architectural Design Patterns & Best Practices) The Daily Six Pack: March 30, 2013 (Dirk Strauss) I hope you enjoy today’s items, and please participate in the discussions on those sites.

A few weeks ago I wrote a blog on synchronizing multithreaded integration tests, which was republished on DZone Javalobby from where it received a comment from Robert Saulnier who quite rightly pointed out that you can also use join() to synchronize a worker thread and its unit tests. This got me thinking, just how many ways can you synchronise multi-threaded integration tests? So, I started counting... and came up with: Using a random delay. Adding a CountDownLatch Thread.join() Acquiring a Semaphore With a Future and ExecutorService Now, I’m not going to explain all the following in great detail, I’ll let the code speak for itself, except to say that all the code samples do roughly the same thing: the unit test creates a ThreadWrapper instance and then calls its doWork() method (or call() in the case of the Future). The unit test’s main thread then waits for the worker thread to complete before asserting that the test has passed. For the sample code demonstrating points 1 and 2 take a look at my original blog on Synchronizing Multithreaded Integration Tests, though I wouldn’t recommend point 1: using a random delay. Thread.join() public class ThreadWrapper { private Thread thread; /** * Start the thread running so that it does some work. */ public void doWork() { thread = new Thread() { /** * Run method adding data to a fictitious database */ @Override public void run() { System.out.println("Start of the thread"); addDataToDB(); System.out.println("End of the thread method"); } private void addDataToDB() { try { Thread.sleep(4000); } catch (InterruptedException e) { e.printStackTrace(); } } }; thread.start(); System.out.println("Off and running..."); } /** * Synchronization method. */ public void join() { try { thread.join(); } catch (InterruptedException ex) { Thread.currentThread().interrupt(); } } } public class ThreadWrapperTest { @Test public void testDoWork() throws InterruptedException { ThreadWrapper instance = new ThreadWrapper(); instance.doWork(); instance.join(); boolean result = getResultFromDatabase(); assertTrue(result); } /** * Dummy database method - just return true */ private boolean getResultFromDatabase() { return true; } } Acquiring a Semaphore public class ThreadWrapper { /** * Start the thread running so that it does some work. */ public void doWork() { doWork(null); } @VisibleForTesting void doWork(final Semaphore semaphore) { Thread thread = new Thread() { /** * Run method adding data to a fictitious database */ @Override public void run() { System.out.println("Start of the thread"); addDataToDB(); System.out.println("End of the thread method"); semaphore.release(); } private void addDataToDB() { try { Thread.sleep(4000); } catch (InterruptedException e) { e.printStackTrace(); } } }; aquire(semaphore); thread.start(); System.out.println("Off and running..."); } private void aquire(Semaphore semaphore) { try { semaphore.acquire(); } catch (InterruptedException e) { e.printStackTrace(); } } } public class ThreadWrapperTest { @Test public void testDoWork() throws InterruptedException { ThreadWrapper instance = new ThreadWrapper(); Semaphore semaphore = new Semaphore(1); instance.doWork(semaphore); semaphore.acquire(); boolean result = getResultFromDatabase(); assertTrue(result); } /** * Dummy database method - just return true */ private boolean getResultFromDatabase() { return true; } } With a Future public class ThreadWrapper implements Callable { @Override public Boolean call() throws Exception { System.out.println("Start of the thread"); Boolean added = addDataToDB(); System.out.println("End of the thread method"); return added; } /** * Add to the DB and return true if added okay */ private Boolean addDataToDB() { try { Thread.sleep(4000); } catch (InterruptedException e) { e.printStackTrace(); } return Boolean.valueOf(true); } } public class ThreadWrapperTest { @Test public void testCall() throws ExecutionException, InterruptedException { ThreadWrapper instance = new ThreadWrapper(); ExecutorService executorService = Executors.newFixedThreadPool(1); Future future = executorService.submit(instance); Boolean result = future.get(); assertTrue(result); } } Having listed all these methods, the next thing to consider is which one is the best? In asking that question you have to define the word “best” in terms of best for what? Best for simplicity? Maintainability? Speed or code size? After all Programming the Art of Making the Right Decision. You may have guessed that I don’t like the random delay idea and prefer the use of a CountDownLatch. Thread.join() is a bit old school; remember that the likes of Semaphore and CountDownLatch were written to improve on the original Java threading techniques. ExecutorService seems a little heavy weight for what we need here. At the end of the day the choice of technique really comes down to personal preference. Finally, I’ve listed five ways of synchronizing multi-threaded integration tests; however, if you can think of any others please let me know... The source code for this blog is available on Github.

There is a lot of talk in the devops community about the importance of sharing principles and values, and about silo busting: breaking down the “wall of confusion” between developers and operations to create agile, cross-functional teams. Radical improvement through fundamental organizational changes and building an entirely new culture. But it doesn’t have to be that hard. All it took us was 3 simple, but important, steps. Reliability First When we first launched our online platform, things were pretty crazy. Sales was busy with customer feedback and onboarding more customers. Development was still finishing the backlog of features that were supposed to already be done, responding to changes from sales and partners, and helping to support the system. Ops was trying to stabilize everything, help onboard more customers and address performance issues as more customers came on. We were all rushing forwards, but not always in the same direction. Our CEO recognized this and made an important decision. He made reliability the #1 priority – the reliability and integrity of our systems and of our customers’ data, and the services we provided. For everyone: not just ops, but development, sales, marketing, compliance, admin. Above everything else. It was more important not to mess up the customers that we had than to get new customers or hit deadlines or cut costs. Reliability, resilience, integrity have remained our #1 driver for the company over several years as we continued to grow. This meant that everyone was working towards the same goals – and the goals were easy to understand and measure: improving MTTF, MTTD and MTTR windows; reducing bug counts and variability in response time, improving results of audits and pen tests. It gave people more reasons to work together at more levels.. It reduced politics and conflicts to a minimum. Development’s first priority changed from pushing features out ASAP to making sure that the system was running optimally and that any changes wouldn't negatively impact customers. This meant more time spent with ops on understanding the run-time, more time troubleshooting operational issues, more reviews and regression testing and stress testing, anticipating compatibility issues, planning for roll-back and roll-forward recovery. Smaller, more frequent releases Spending some more time on testing and reviews and working with ops meant that it took longer to complete a feature. But we still had to keep up with customer demands – we still had to deliver. We did this by shortening the software release cycle, from 2-3 months to 2-3 weeks or sometimes shorter. Delivering less in each release, sometimes only 1 new feature or some fixes, but delivering much more often. If a change or feature had to be delayed, if developers or testers needed more time to make sure that it was ready, it wasn't a big deal – if something wasn't ready this week, it would be ready next week or soon after, still fast enough for the business and for customers. Planning and working in shorter horizons meant that development could respond faster to changes in direction and changing priorities, so developers were always focused on delivering what was most important at the time. Shorter releases drove development to be more agile, to think and work faster. To automate more testing. To pay more attention to deployment, make the steps simpler and more efficient – and safer. Fewer changes batched together made it easier to review and test. Less chances to make mistakes. Easier to understand what went wrong when we did make a mistake, and less time to fix it. RCA – Learn from Mistakes We still made mistakes, shit still happened. When something went seriously wrong, it was my job to explain it to our customers and owners. What went wrong, why, and what we were going to do to make sure that it didn’t happen again. We didn't know about blameless post mortems, but this is the way we did it anyway. We got developers and testers and ops and managers together in Root Cause Analysis sessions to carefully examine what happened, what went wrong, understand why, and fix it. We made sure that people focused on the facts and on problem solving: what happened, what happened next, what did we see, what didn’t we see, why? What could we do to fix it or to prevent it from happening again or to recognize and respond to problems like this more effectively in the future? Better training, better tools, better procedures, better documentation, better error handling, better testing and reviews, better configuration checks and run-time checking, better information and better ways of communicating it. Focusing on details and problems, not people. Proving that it was ok to make mistakes, but not ok to hide them. We got much better: at operations, testing, design, deployment, monitoring, incident handling. And better as an organization. We built transparency and trust within and across teams. We learned how to move forward from failure, and to be more resilient and confident in our ability to deal with serious problems. Delivering Better and Faster Together We didn't restructure or change who we were as an organization. Dev and ops still work in separate organizations for different managers in different countries. They have their own projects and their own ways of working, and they don’t always speak the same language or agree on everything. We have lots of checks and balances and handoffs and paperwork between dev and ops to make sure that things are done properly and to make the regulators happy. There are still more steps that we could automate or simplify, more we can do to build out our Continuous Delivery pipelines, more things we can get out of Puppet and Vagrant and other cool tools. But if devops is about developers and operations sharing responsibility for the system, trusting each other and helping each other to make sure that the system is always working correctly and optimally, looking for better solutions together, delivering better and faster – then we’ve been doing devops for a while now.

Separate out analysis to preserve commit stage processing time The entry point of a Continuous Delivery pipeline is its Commit Stage, and as such manages the compilation, unit testing, analysis, and packaging of source code whenever a change is committed to version control. As the commit stage is responsible for identifying defective code it represents a vital feedback loop for developers, and for that reason Dave Farley and Jez Humble recommend a commit stage that is “ideally less than five minutes and no more than ten” – if the build process is too slow or non-deterministic, the pace of development can soon grind to a halt. Both compilation and unit testing tasks can be optimized for performance, particularly when the commit stage is hosted on a multi-processor Continuous Integration server. Modern compilers require only a few seconds for compilation, and a unit test suite that follows the Michael Feathers strategy of no database/filesystem/network/user interface access should run in parallel in seconds. However, it is more difficult to optimize analysis tasks as they tend to involve third-party tooling reliant upon byte code manipulation. When a significant percentage of commit stage time is consumed by static analysis tooling, it may become necessary to trade-off unit test feedback against static analysis feedback and move the static analysis tooling into a separate Analysis Stage. The analysis stage is triggered by a successful run of the commit stage, and analyses the uploaded artifact(s) and source code in parallel to the acceptance testing stage. If a failure is detected the relevant pipeline metadata is updated and Stop The Line applies. That binary cannot be used elsewhere in the pipeline and further development efforts should cease until the issue is resolved. For example, consider an organisation that has implemented a standard Continuous Delivery pipeline. The commit stage has an average processing time of 5 minutes, of which 1 minute is spent upon static analysis. Over time the codebase grows to the extent that commit stage time increases to 6 minutes, of which 1 minute 30 seconds is spent upon static analysis. With static analysis time growing from 20% to 25% the decision is made to create a separate Analysis stage, which reduces commit time to 4 minutes 30 seconds and improves the developer feedback loop. Static analysis is the definitive example of an automated task that periodically needs human intervention. Regardless of tool choice there will always be a percentage of false positives and false negatives, and therefore a pipeline that implements an Analysis Stage must also offer a capability for an authenticated human user to override prior results for one or more application versions.

Continuous Delivery and DevOps are interdependent, not equivalent Since the publication of Dave Farley and Jez Humble’s seminal book on Continuous Delivery in 2010, its rise within the IT industry has been paralleled by the growth of the DevOps movement. While Continuous Delivery has an explicit goal of optimising for cycle time and an established set of principles and practices, DevOps is a more organic philosophy that is defined as “aligning development and operations roles and processes in the context of shared business objectives“, and gradually codifying into principles and practices. Continuous Delivery and DevOps possess a shared background in agile methods and Lean Thinking, and a shared desire to eliminate Waterscrumfall silos – but what is the nature of their relationship? In Continuous Delivery, practitioners such as Jez Humble have warned that organisations require “a culture that enables collaboration and understanding between the functional groups that deliver IT services“, which refers to the culture-centric principles – Continuous Improvement, Done Means Released, and Everybody Is Responsible – that reduce handover delays between siloed teams. DevOps provides an implementation strategy for these principles – its emphasis upon “the integration of Agile principles with Operations practices” aligns Development and Operations working practices and encourages cooperation. However, these principles can be also implemented independently of DevOps – for example, an organisation might forego a QA team in favour of mandatory Development support for production releases, as at Facebook. In DevOps, one of the four key areas described by Patrick Debois is Extend Delivery To Production. The intention is for the delivery mechanism to act as a focal point for collaboration between Development and Operations, resulting in improved speed/reliability of releases and a sense of shared responsibility for production systems. Continuous Delivery offers an implementation strategy for this key area – a deployment pipeline provides a shared one-button workflow, encourages the emergence of a shared codebase and toolchain, and facilitates a release cadence that minimises change sets and the risk of failure. However, it should be noted that Extend Delivery To Production could be accomplished without Continuous Delivery – for example, a push-based Continuous Deployment mechanism might underpin the value stream instead of a pull-based pipeline, as at IMVU. From the above we can surmise that Continuous Delivery and DevOps are interdependent, but the inherent fuzziness of the DevOps philosophy allows different interpretations of the relationship. For example, Jeff Sussna recently contended that “delivering software as service makes operations an explicit part of the customer value proposition… customers view functionality and operability as inseparable aspects of service” and that by defining DevOps “not in terms of how IT structures itself, but rather in terms of what customers expect” we can say “DevOps IS Continuous Delivery“. While it is an interesting approach to couple DevOps to customer expectations, the commonly accepted definitions focus upon internal organisational change in order to meet business objectives, which may or may not include operability as a first-class concept. It is evident that SaaScustomers will have explicit operability requirements, but for many organisations the reality is that customers explicitly expect functionality and timeliness while implicitly expecting operability. For example, Jeff uses a restaurant review metaphor to describe the combined value of functionality and operability (“the food was great but the service was terrible“), but restaurant customers cannot observe back-of-house operability and will likely only comment upon front-of-house operability if it impacts upon functionality and/or timeliness. Jeff also makes a comparison of nomenclature, suggesting that for agile development and Continuous Delivery the name describes the value… in the case of DevOps, the name describes the implementation, not the desired outcome“. Surely the desired outcome of DevOps is expressed in the portmanteau – Development and Operations teams seamlessly working together to deliver value-adding features to the customer.