I presented a new talk at GOTO Chicago 2015 about how to change a team’s attitude towards writing automated tests. The talk covers the same case study as Groovy vs Java for Testing, adopting Spock in MongoDB, but this is a more process/agile/people perspective, not a technical look at the merits of one language over another. Slides available below. As always, the slides are not super-useful out of context, but they do contain my conclusions (also note that due to a technology fail, my hand-drawn style is even more hand-drawn than usual). Questions I sadly did not have a lot of time for questions during the presentation, but thanks to the wonders of modern technology, I have a list of unanswered questions which I will attempt to address here. Is testing to find out your system works? Or is it so you know when your system is broken? Excellent question. I would expect that if you have a system that’s in production (which is probably the large majority of the projects we work on), we can assume the system is working, for some definition of working. Automated testing is particularly good at catching when your system stops doing the things you thought it was doing when you wrote the tests (which may, or may not, mean the system is genuinely “broken”). Regression testing is to find out when your system is no longer doing what you expect, and automated tests are really good for this. But testing can also make sure you implement code that behaves the way you expect, especially if you write the tests first. Automated tests can be used to determine that your code is complete, according to some pre-agreed specification (in this case, the automated tests you wrote up front). So I guess what I’m trying to say is, when you first write the tests you have tests that, when they pass, proves the system works (assumingyour tests are testing the right things and/or not giving you false positives). Subsequent passes show that you haven’t broken anything. At what level do “tests documenting code” actually become useful? And who is/should the documentation be targeted to? In the presentation, my case study is the MongoDB Java Driver. Our users were Java programmers, who were going to be coding using our driver. So in this example, it makes a lot of sense to document the code using a language that our users understood. We started with Java, and ended up using Groovy because it was also understandable for our users and a bit more succinct. On a previous project we had different types of tests. The unit and system tests documented what the expected behaviour was at the class or module level, and was aimed at developers in the team. The acceptance tests were written in Java, but in a friendly DSL-style way. These were usually written by a triad of tester, business analyst and developer, and documented to all these guys and girls what the top-level behaviour should be. Our audience here was fairly technical though, so there was no need to go to the extent of trying to write English-language-style tests, they were readable enough for a reasonably techy (but non-programmer) audience. These were not designed to be read by “the business” - us developers might use them to answer questions about the behaviour of the system, but they didn’t document it in a way that just anyone could understand. These are two different approaches for two different-sized team/organisations, with different users. So I guess in summary the answer is “it depends”. But at the very least, developers on your own team should be able to read your tests and understand what the expected behaviour of the code is. How do you become a team champion? I.e. get authority and acceptance that people listen to you? In my case, it was just by accident - I happened to care about the tests being green and also being useful, so I moaned at people until it happened. But it’s not just about nagging, you get more buy-in if other people see you doing the right things the right way, and it’s not too painful for them to follow your example. There are going to be things that you care about that you’ll never get other people to care about, and this will be different from team to team. You have two choices here - if you care that much, and it bothers you that much, you have to do it yourself (often on your own time, especially if your boss doesn’t buy into it). Or, you have to let it go - when it comes to quality, there are so many things you could care about that it might be more beneficial to drop one cause and pick another that you can get people to care about. For example, I wanted us to use assertThat instead of assertFalse (or true, or equals, or whatever). I tried to demo the advantages (as I saw them) of my approach to the team, and tried to push this in code reviews, but in the end the other developers weren’t sold on the benefits, and from my point of view the benefits weren’t big enough to force the issue. Those of us who cared, used assertThat. For the rest, I was just happy people were writing and maintaining tests. So, pick your battles. You’ll be surprised at how many people do get on board with things. I thought implementing checkstyle and setting draconian formatting standards was going to be a tough battle, but in the end people were just happy to have any standards, especially when they were enforced by the build. Do you report test, style, coverage, etc failures separately? Why? We didn’t fail on coverage. Enforcing a coverage percentage is a really good way to end up with crappy tests, like for getters/setters and constructors (by the way, if there’s enough logic in your constructor that it needs a test, You’re Doing It Wrong). Generally different types of failures are found by different tools, so for this reason alone they will be reported separately - for example, checkstyle will fail the build if it doesn’t conform to our style standards, codenarc fails it for Groovy style failures, and Gradle will run the tests in a different task to these two. What’s actually important, though, is time-to-failure. For checkstyle, for example, it will fail on something silly like curly braces in the wrong place. You want this to fail within seconds, so you can fix the silly mistake quickly. Ideally you’d have IntelliJ (perhaps) run your checks before it even makes it into your CI environment. Compiler errors should, of course, fail things before you run a test, short-running tests should fail before long-running tests. Basically, the easier it is to fix the problem, the sooner you want to know, I guess. Our build was relatively small and not too complex, so actually we ran all our types of tests (integration and unit, both Groovy and Java) in a single task, because this turned out to be much quicker in Gradle (in our case) than splitting things up into a simple pipeline. You might have a reason to report stuff separately, but for me it’s much more important to understand how fast I need to be aware of a particular type of failure. Sometimes I find myself modifying code design and architecture to enable testing. How can I avoid damaging design? This is a great question, and a common one too. The short answer is: in general writing code that’s easier to test leads to a cleaner design anyway (for example, dependency injection at that appropriate places). If you find you need to rip your design apart to test it, there’s a smell there somewhere - either your design isn’t following SOLID principals, or you’re trying to test the wrong things. Of course, the common example here is testing private methods - how do you test these without exposing secrets? I think for me, if it’s important enough to be tested it’s important enough to be exposed in some way - it might belong in some sort of util or helper (right now I’m not going to go into whether utils or helpers are, in themselves a smell), in a smaller class that only provides this sort of functionality, or simply a protected method. Or, if you’re testing with Groovy, you can access private methods anyway so this becomes a moot point (i.e. your testing framework may be limiting you). In another story from LMAX, we found we had created methods just for testing. It seemed a bit wrong to have these methods only available for testing, but later on down the line, we needed access to many of these methods In Real Life (well, from our Admin app), so our testing had “found” a missing feature. When we came to implement it, it was pretty easy as we’d already done most of it for testing. My co-workers often point to a lack of end-to-end testing as the reason why a lot of bugs get out to production even though they don’t have much unit tests nor integration tests. What, in your experience, is a good balance between unit tests, integration tests and end-to-end testing? Hmm, sounds to me like “lack of tests” is your problem! This is a big (and contentious!) topic. Martin Fowler has written about it, Google wrote something I completely disagree with (so I’m not even going to link to it, but you’ll find references in the links in this paragraph), and my ex-colleague Adrian talks about what we, at LMAX, meant by end-to-end tests. I hope that’s enough to get you started, there’s plenty more out there too. How did you go about getting buy in from the team to use Spock? I cover this in my other presentation on the topic - the short version is, I did a week-long spike to investigate whether Spock would make testing easier for us, showed the pros and cons to the whole team, and then led by example writing tests that (I thought) were more readable than what we had before and, probably most importantly, much easier to write than what we were previously doing. I basically got buy-in by showing how much easier it was for us to use the tool than even JUnit (which we were all familiar with). It did help that we were already using Gradle, so we already had a development dependency on Groovy. It also helped that adding Spock made no changes to the dependencies of the final Jar, which was very important. Over time, further buy-in (certainly from management) came when the new tests started catching more errors - usually regressions in our code or regressions in the server’s overnight builds. I don’t think it was Spock specifically that caught more problems - I think it was writing more tests, and better tests, that caught the issues. Can we do data driven style tests in frameworks like junit or cucumber? I don’t think you can in JUnit (although maybe there’s something out there). I believe someone told me you can do it in TestNG. Are there drawbacks to having tests that only run in ci? I.e I have Java 8 on my machine, but the test requires Java 7 Yes, definitely - the drawback is Time. You have to commit your code to a branch that is being checked by CI and wait for CI to finish before you find the error. In practice, we found very little that was different between Java 7 and 8, for example, but this is a valid concern (otherwise you wouldn’t be testing a complex matrix of dependencies at all). In our case, our Java 6 driver used Netty for async capabilities, as the stuff we were using from Java 7 wasn’t available. This was clearly a different code path that wasn’t tested by us locally as we were all running Java 8. Probably more importantly for us is we were testing against at least 3 different major versions of the server, which all supported different features (and had different APIs). I would often find I’d broken the tests for version 2.2 as I’d only been running it on 2.6, and had forgotten to either turn off the new tests for the old server versions, or didn’t realise the new functionality wouldn’t work there. So the main drawback is time - it takes a lot longer to find out about these errors. There are a few ways to get around this: Commit often!! And to a branch that’s actually going to be run by CI Make your build as fast as possible, so you get failures fast (you should be doing this anyway) You could set up virtual machines locally or somewhere cloudy to run these configurations before committing, but that sounds kinda painful (and to my mind defeats a lot of the point of CI). I set up Travis on my fork of the project, so I could have that running a different version of Java and MongoDB when I committed to my own fork - I’d be able to see some errors before they made it into the “real” project. If you can, you probably want these specific tests run first so they can fail fast. E.g. if you’re running a Java 6 & MongoDB 2.2 configuration on CI, run those tests that only work in that environment first. Would probably need some Gradle magic, and/or might need you to separate these into a different set of folders. The advantage of this approach though is if you set up some aliases on your local machine you could sanity check just these special cases before checking in. For example, I had aliases to start MongoDB versions/configurations from a single command, and to set JAVA_HOME to whichever version I wanted. Do you have any tips for unit tests that pass on dev machines but not on Jenkins because it’s not as powerful as our own machines? E.g. Synchronous calls timeout on the Jenkins builds intermittently. Erk! Yes, not uncommon. No, not really. We had our timeouts set longer than I would have liked to prevent these sorts of errors, and they still intermittently failed. You can also set some sort of retry on the test, and get your build system to re-run those that fail to see if they pass later. It’s kinda nasty though. At LMAX they were able to take testing seriously enough to really invest in their testing architecture, and, of course, this is The Correct Answer. Just often very difficult to sell. If you ask where are tests and dev asks if code is correct? And you say yes. Then dev asks why you’re delaying shipping value, how do you manage that? These are my opinions: Your code is not complete without tests that show me it’s complete. Your code might do what you think it’s supposed to do right now, but given Shared Code Ownership, anyone can come in and change it at any time, you want tests in place to make sure they don’t change it to break what you thought it did The tests are not so much to show it works right now, the tests are to show it continues to work in future Having automated tests will speed you up in future. You can refactor more safely, you can fix bugs and know almost immediately if you broke something, you can read from the test what the author of the code thought the code should do, getting you up to speed faster. You don’t know you’re shipping value without tests - you’re only shipping code (to be honest, you never know if you’re shipping value until much later on when you also analyse if people are even using the feature). Testing almost never slows you down in the long run. Show me the bits of your code base which are poorly tested, and I bet I can show you the bits of your code base that frequently have bugs (either because the code is not really doing what the author thinks, or because subsequent changes break things in subtle ways). If you say code is hard to understand and dev asks if you seriously don’t understand the code, how do you explain you mean easy to understand without thinking rather than ‘can I compile this in my head’? I have zero problem with saying “I’m too stupid to understand this code, and I expect you’re much smarter than me for writing it. Can you please write it in a way so that a less smart person like myself won’t trample all over your beautiful code at a later date through lack of understanding?” By definition, code should be easy to understand by someone who’s not the author. If someone who is not the author says the code is hard to understand, then the code is hard to understand. This is not negotiable. This is what code reviews or pair programming should address. What is effective nagging like? (Whether or not you get what you want) Mmm, good question. Off the top of my head: Don’t make the people who are the target of the nagging feel stupid - they’ll get defensive. If necessary, take the burden of “stupidity” on yourself. E.g. “I’m just not smart enough to be able to tell if this test is failing because the test is bad or because the code is bad. Can you walk me through it and help me fix it?” Do at least your fair share of the work, if not more. When I wanted to get the code to a state where we could fail style errors, I fixed 99% of the problems, and delegated the handful of remaining ones that I just didn’t have the context to fix. In the face of three errors to fix each, the team could hardly say “no” after I’d fixed over 6000. Explain why things need to be done. Developers are adults and don’t want to be treated like children. Give them a good reason and they’ll follow the rules. The few times I didn’t have good reasons, I could not get the team to do what I wanted. Find carrots and sticks that work. At LMAX, a short e-mail at the start of the day summarising the errors that had happened overnight, who seemed to be responsible, and whether they looked like real errors or intermittencies, was enough to get people to fix their problems2 - they didn’t like to look bad, but they also had enough information to get right on it, they didn’t have to wade through all the build info. On occasion, when people were ignoring this, I’d turn up to work with bags of chocolate that I’d bought with my own money, offering chocolate bars to anyone who fixed up the tests. I was random with my carrot offerings so people didn’t game the system. Give up if it’s not working. If you’ve tried to phrase the “why” in a number of ways, if you’ve tried to show examples of the benefits, if you’ve tried to work the results you want into a process, but it’s still not getting done, just accept the fact that this isn’t working for the team. Move on to something else, or find a new angle. 1 I had a colleague at LMAX who was working with a hypothesis that All Private Methods Were Evil - they were clearly only sharable within single class, so provided no reuse elsewhere, and if you have the same bit of code being called multiple times from within the same class (but it’s not valuable elsewhere) then maybe your design is wrong. I’m still pondering this specific hypothesis 4 years on, and I admit I see its pros and cons. 2 This worked so well that this process was automated by one of the guys and turned into a tool called AutoTrish, which as far as I know is still used at LMAX. Dave Farley talks about it in some of hisContinuous Delivery talks. Resources My talk that specifically looks at the advantages of Spock over JUnit, plus some Spock-specific resources. I love Jay Fields book Working Effectively With Unit Tests - if I could have made the whole team read this before moving to Spock, we might have stuck with JUnit. Go read everything Adrian Sutton has written about testing at LMAX. If not everything, definitely Abstraction by DSL and Making End-to-End Tests Work If you can’t make it all the way through Dave Farley and Jez Humble’s excellent Continuous Delivery book, do take a look at one of Dave’s presentations on the subject, for example The Rationale for Continuous Delivery or The Process, Technology and Practice of Continuous Delivery - my own talk was around testing, but I’m working off the assumption that you’re at least running some sort of Continuous Integration, if not Continuous Delivery. Martin Fowler has loads of interesting and useful articles on testing. Abstract What can you do to help developers a) write tests b) write meaningful tests and c) write readable tests? Trisha will talk about her experiences of working in a team that wanted to build quality into their new software version without a painful overhead - without a QA / Testing team, without putting in place any formal processes, without slavishly improving the coverage percentage. The team had been writing automated tests and running them in a continuous integration environment, but they were simply writing tests as another tick box to check, there to verify the developer had done what the developer had aimed to do. The team needed to move to a model where tests provided more than this. The tests needed to: Demonstrate that the library code was meeting the requirements Document in a readable fashion what those requirements were, and what should happen under non-happy-path situations Provide enough coverage so a developer could confidently refactor the code This talk will cover how the team selected a new testing framework (Spock, a framework written in Groovy that can be used to test JVM code) to aid with this effort, and how they evaluated whether this tool would meet the team’s needs. And now, two years after starting to use Spock, Trisha can talk about how both the tool and the shift in the focus of the purpose of tests has affected the quality of the code. And, interestingly, the happiness of the developers.

That Supplier is Better For You Since releasing the 2015 State of the Software Supply Chain Report, there has been a lot of great discussion across the industry on best practices for managing the complexity introduced by the volume and velocity of the components used across your software supply chain. Today I want to focus on the huge ecosystem of open source projects (“suppliers”) that feed a steady stream of innovative components into our software supply chains. In the Java ecosystem alone, there are now over 108,000 suppliers of open source components. Across all component types available to developers (e.g., RubyGems, NuGet, npm, Bower, PyPI, etc.), estimates now reach over 650,000 suppliers of open source projects. Source: ModuleCounts.com However, like in traditional manufacturing, not all suppliers deliver parts of comparable quality and integrity. My latest research, the 2015 State of the Software Supply Chain Report, shows that some open source projects use restrictive licenses and vulnerable sub-components, while other projects are far more diligent at updating the overall quality of their components. Choosing the best and fewest suppliers can improve the quality and integrity of the applications we deliver to our customers. While I am hosting a webinar next week to share many of the detailed report findings, I wanted to share a few of the more meaningful stats here. Your 7,600 Suppliers My research for the report revealed many new perspectives on “suppliers” across the software supply chains. First of all, I saw that the average large development organization consumed over 240,000 open source components last year — sourced from over 7,600 open source projects. Average number of open source component downloads per organization in calendar 2014 On the surface, the huge reliance on open source projects is a great thing. Development teams have chosen to not write those pieces themselves, but have sourced the needed components from outside suppliers. This practice speeds development, enables more innovation, and ensures time-to-release goals are achieved. The use of open source is so prolific today, few of us could ever imagine reducing the use of those components and their suppliers in the future. At the same time that we benefit from open source, our high paced, high volume consumption practices don’t allow us the time needed to do the due diligence on the suppliers or open source projects where we source our component parts from. Average number of known vulnerabilities downloaded by organization in calendar 2014. For example, of the 240,000 average component downloads in 2014, the same businesses sourced an average of 15,000 components that included known security vulnerabilities. In many cases, developers were downloading vulnerable component versions, when safer versions of those same components were available from the open source projects. While no one intends to download components with known vulnerabilities, the problem is exacerbated due to the lack the visibility into a better recommended version. Fewer Suppliers, Less Context Switching Choosing an open source project supplier should be considered an important strategic decision in organizations because changing a supplier (“open source project”) used is far more effort than swapping out a specific component. Like traditional suppliers, open source projects have good and bad practices impacting the overall quality of their component parts. Traditional manufacturing supply chains intentionally select specific parts from approved suppliers. They also rely on formalized sourcing and procurement practices. This practice also focuses the organization on using the best and fewest suppliers — an effort that improves quality, reduces context switching, and also accelerates mean time to repair when defects are discovered. One industry example from the report describes how Toyota manages 125 suppliers for their Prius to help sustain competitive advantages over GM who manages over 800 suppliers for the Chevy Volt. By contrast, development teams working with software supply chains often rely on an unchecked variety of supply, where each developer or development team can make their own sourcing and procurement decisions. The effort of managing over 7,600 suppliers introduces a drag on development and is contrary to their need to develop faster as part of agile, continuous delivery and devops practices. Coming to Terms When you come to terms with the volume of consumption and the massive ecosystem of suppliers you can source your components from, you quickly realize it is impossible to address this issue with a manual review process. Any organizations clutching to these outdated manual practices are will continue to be outgunned by the velocity by their software supply chains. Just as traditional manufacturing supply chains have turned to automation, software development teams need to take the same approach by further automating their software supply chains. Information about suppliers and the quality of their projects needs to be made available to developers at the time they are selecting components. Information about the latest versions, features, licenses, known vulnerabilities, popularity of versions being used, and the cadence of new releases should be made available to developers in an automated way. Automating the availability of this information about suppliers can lead to better and fewer suppliers being used. Be sure to read the full 2015 State of the Software Supply Chain Report for more information about open source suppliers and organizations sourcing practices. The report also highlights current and best practices being used in organizations that are managing their use of suppliers that feed their software supply chains.

A huge amount has happened recently. Following the official launch of CUBA on 1st of June, we have rolled out a new release, published our first article on a few Java sites and presented the platform at the Devoxx UK сonference in London. But before the rush continues, about it is an apt time to articulate the philosophy behind CUBA. The first words associated with enterprise software development will probably be: slow, routine, complex and convoluted - nothing exciting at all! A common approach to combat these challenges is raising the level of abstraction - so that developers can operate with interfaces and tools encapsulating internal mechanisms. This enables the focus on high-level business requirements without the need to reinvent common processes for every project. Such a concept is typically implemented in frameworks, or platforms. The previous CUBA article explained why CUBA is more than just a bunch of well-known open-source frameworks comprehensively integrated together. In brief, it brings declarative UI with data aware visual components, out-of-the-box features starting from sophisticated security model to BPM, and awesome development tools to complement your chosen IDE. You can easily find more details on our Learn page, so instead of listing all of them I'll try to "raise the abstraction level" and explain the fundamental principles of CUBA. Practical The platform is a living organism, and its evolution is mostly driven by specific requests from developers. Of course, we constantly keep track of emerging technologies, but we are rather conservative and employ them only when we see that they can bring tangible value to the enterprise software development. As a result, CUBA is extremely practical; every part of it has been created to solve some real problem. Integral Apart from the obvious material features, the visual development environment provided by CUBA Studio greatly reduces the learning curve for newcomers and juniors. It is even more important that the platform brings a unified structure to your applications. When you open a CUBA-based project, you will always know where to find a screen, or a component inside of it; where the business logic is located and how is it invoked. Such an ability to quickly understand and change the code written by other developers cannot be underestimated as a significant benefit to continual enterprise development. An enterprise application lifecycle may last tens of years and your solution must constantly evolve with the business environment, regardless of any changes in your team. For this reason, the flexibility to rotate, or scale up or down the team when needed, is one of the major concerns for the companies, especially those who outsource development or have distributed teams. Open One of the key principles of CUBA is openness. This starts with the full platform source code, which you have at hand when you work on a CUBA-based project. In addition, the platform is also open in the sense that you can change almost any part of it to suit your needs. You don't need to fork it to customize some parts of the platform - it is possible to extend and modify the platform functionality right in your project. To achieve this, we usually follow the open inheritance pattern, providing access to the platform internals. We understand that this can cause issues when the project is upgraded to a newer platform version. However, from our experience, this is far less evil than maintaining a fork, or accepting the inability to adapt the tool for a particular task. We could also make a number of specific extension points, but in such case we would have to anticipate how application developers will use the platform. Such predictions always fail, sooner or later. So instead we have made the whole platform extension-friendly: you can inherit and override platform Java code including the object model, XML screens layout and configuration parameters. Transitively, this remains true for CUBA-based projects. If you follow a few simple conventions, your application becomes open for extension, allowing you to adapt the single product for many customers. Symbiotic CUBA is not positioned as a “thing-in-itself”. When a suitable and well-supported instrument already exists and we can integrate with it without sacrificing platform usability, we will integrate with it. An illustration of such integrations is full-text search and BPM engines, JavaScript charts and Google Maps API. At the same time, we have had to implement our ownreport generator from scratch, because we could not find a suitable tool (technology and license wise). The CUBA Studio follows this principle too. It is a standalone web application and it doesn't replace your preferred IDE. You can use Studio and the IDE in parallel, switching between them to accomplish different tasks. WYSIWYG approach, implemented in Studio, is great for designing the data model and screens layout, while the classic Java IDE is the best for writing code. You can change any part of your project right in the IDE, even things created by Studio. When you return to Studio, it will instantly parse all changes, allowing you to keep on developing visually. As you see, instead of competing with the power of Java IDEs, we follow a symbiotic approach. Moreover, to raise coding efficiency, we’ve developed plugins for the most popular IDEs. When we integrate with a third-party framework, we always wrap it in a higher level API. This enables replacing the underlying implementation if needed and makes the whole platform API more stable long term and less dependent on the constant changes in the integrated third-party frameworks. However, we don't restrict the direct use of underlying frameworks and libraries. It makes sense if CUBA API does not fit a particular use case. For example, if you can't do something via Generic UI, you can unwrap a visual component and get direct access to Vaadin (or Swing). The same applies for data access; if some operation is slow or not supported by ORM, just write SQL and run it via JDBC or MyBatis. Of course, such “hacks” lead to more complex and less portable application code, but they are typically very rare compared to the use of standard platform API. This knowledge of inherent flexibility and a sense of “Yes you can” adds a lot of confidence to developers. Wide use area We recommend using CUBA if you need to create an application with anything starting from 5-10 screens, as long as they consist of standard components like fields, forms, and tables. The effect from using CUBA grows exponentially with the complexity of your application, independent of the domain. We have delivered complex projects in financial, manufacturing, logistics and other areas. As an example, a non-obvious, but popular use case is using CUBA as the backend and admin UI, while creating the end-user interface with another, lighter or more customizable web technology. I hope you will see some use cases of the platform for yourself, so in the next articles we’ll focus on “what's under the hood” – as we provide a detailed overview of the different CUBA parts.

As device-enabled subscribers demand more, Calix introduces multiple wavelength NG-PON2 solutions primed for next generation applications ANAHEIM, CA - June 29, 2015 - Calix, Inc. (NYSE: CALX), the world leader in gigabit fiber deployments, today announced new cards for its E-Series portfolio that introduce both increased systems capacity and ITU/FSAN NG-PON2 support. By adding 10 gigabit per second (10 Gbps) time and wavelength division multiplexed (TWDM) PON NG-PON2 with both fixed and tunable wavelengths, Calix is paving the way for service providers to leverage next generation fiber solutions that redefine the broadband experience. These solutions, when combined with Calix Compass software applications and GigaCenter platforms, extend far beyond gigabit "speeds and feeds" to encompass superior Wi-Fi performance and compelling cloud-based applications, and lay the foundation for a superior gigabit and multi-gigabit experience for residential and business subscribers. Service providers globally are seeing an explosion of cloud-connected devices. In North America alone, the average person is projected to have over 11 of these devices by 2019. As these devices proliferate and are used to stream and share rich multimedia content, they will place enormous pressure on the access infrastructure. By 2019, 80 percent of all internet traffic will be IP video, while global cloud traffic will nearly quadruple over the same period. (Statistics from Cisco VNI and Cisco GCI) Service providers need solutions that can meet these emerging demands today while seamlessly adding new technologies for even higher scale and capacity in the future. Extending from the award-winning GigaCenter at the subscriber premises all the way to the cloud-based Compass software, Calix next generation fiber solutions arm service providers with technologies to address challenges and opportunities in subscriber support, analytics, and service enhancement. Potential issues impacting the subscriber broadband experience, such as device bandwidth contention in the ultra HD video enabled home, management of the device-rich smart home, and sub-par Wi-Fi performance, can be remedied before they manifest themselves to subscribers. Calix has a long history of serving fiber access customers with an architectural philosophy of a unified access infrastructure, supporting both business and residential services. Calix service provider customers have found the financial benefits of this architecture to be compelling, resulting in industry-leading efficiency in installed equipment costs and on-going operational savings. The Calix NG-PON2 strategy extends this leadership, expanding the capacity of fiber networks and allowing service providers to keep pace with the needs of business and residential customers alike over a common infrastructure. "From being the vendor with the first commercial deployments of GPON over a decade ago, to our unique auto-detect technology and pay-as-you-grow modular fiber architecture, Calix has consistently led the way in fiber access innovation," said Michel Langlois, Calix senior vice president of systems products. "This leadership in innovation is why over 1000 service providers across the globe rely on Calix for fiber access solutions, including nearly 100 delivering a gigabit service experience to their residential subscribers. Next generation PON provides fertile ground for a new wave of innovations, and Calix is again leading the way with significant contributions to development of the NG-PON2 standard, including key submissions that will reduce deployment costs and technical complexity and assure 2.4 GPON coexistence. Demonstrations of both NG-PON2 with tunable TWDM wavelengths and fixed wavelength 10G PON will take place this fall." Calix next generation PON solutions will extend across the entire access infrastructure, from the device-enabled subscriber premises to the data center or central office: Subscriber Premises: The industry leading Calix premises portfolio will be expanded to include NG-PON2 technology, including both fixed and tunable optics, across the full range of subscriber applications - from business services to MDU and SFU residential applications with Carrier Class Wi-Fi. Compass software-as-a-service applications will also be expanded to support these technologies and bring a true multi-gigabit experience to subscribers. Access Infrastructure: The award winning E7 portfolio will be enhanced by NG-PON2 cards that support both 10 Gbps fixed and TWDM wavelengths with pluggable optics. Optimized for areas of high bandwidth demand and congestion, these cards will support the delivery of multiple wavelengths of symmetrical 10 Gbps services. Data Center / Central Office: The E7 portfolio will be augmented by new high capacity cards that tap into new levels of performance in both system switching and uplink capacity. "Calix has a long history of new innovations in the fiber access industry, making key contributions to fiber access standards and making the fiber access business model work for service providers," said Teresa Mastrangelo, principal and founder of Broadbandtrends LLC. "Now as we move into the emerging era of next generation PON, it's no surprise to see Calix announcing a comprehensive portfolio addressing a range of applications. It is clear that NG-PON2 is going to be the PON technology that enables the multi-gigabit experience of the future, and Calix looks to be ready as this market emerges with a robust solutions oriented strategy." The new Calix innovations, including its new next generation PON and recently announced G.fast solutions for MDU applications, will be highlighted at the Calix booth #413 this week at the FTTH Conference & Expo in Anaheim, California.

Persistence of model objects is a part of many Java projects and a part which deserves, and often gets, high test coverage as one of the key layer integration points in the code. However, I've often felt the testing paradigms for this can be cumbersome, often involving a large amount of setup with an equivalent amount of validation. This can be tedious to both create and maintain. As a solution to this I've been testing persistence with a different pattern; by combining both the exparity-stub and the hamcrest-bean library you can thoroughly test model persistence in a few lines of test code as per the snippet below; .. User user = aRandomInstanceOf(User.class); User saved = dao.save(user); assertThat(dao.getUserById(saved.getId()), theSameBeanAs(saved)); The test snippet above is small but in those few lines will thoroughly test that all fields in a graph can be persisted and retrieved without loss, that any JPA or other mapping is valid, and that your queries are valid. For a complete example we'll work through testing a simple DAO for storing and retrieving User objects using the in-memory H2 database for simplicity. The same example will work for any persistence mechanism. Before we get started with an example lets briefly outline what the libraries are and what they do. The Exparity-Stub Library The exparity-stub libraries provides a set of static methods for creating stubs of model objects, object graphs, collections, types, and primitive types. For our example we'll be creating random stubs because we want to completely fill the graph with junk data and check it can be written down. exparity-stub offers two approaches to this, the RandomBuilder or the BeanBuilder. The RandomBuilder provides a terser notation to create random objects with less code. For example: User user = RandomBuilder.aRandomInstanceOf(User.class); List users = RandomBuilder.aRandomListOf(User.class); String anyString = RandomBuilder.aRandomString(); Whereas the BeanBuilder provides a fluent interface with finer control for building individual objects and graphs, for example; User user = BeanBuilder.aRandomInstanceOf(User.class) .excludeProperty("Id").build(); For this example i'm going to use the BeanBuilder so I can exclude the User.Id property from being populated by the random builder. The Hamcrest-Bean Library The hamcrest-bean library is an extension library to the Java Hamcrest library. The hamcrest-bean library provides a set of matchers specifically for testing Java objects and object graphs and performs deep inspections of those objects. It supports exclusions and overrides to allow fine control, if required, of how matching of any property, path, or type is handled, for example: User expected = new User("Jane", "Doe"); assertThat(new User("John", "Doe"), BeanMatchers.theSameAs(expected).excludeProperty("FirstName")); A Sample Project The sample project I'll work through is persistence of a simple User object with a child list of UserComment objects. This simple graph will be persisted to a H2 database with hibernate handling the Object-Relational Mapping (ORM) mapping, and Java Persistence Annotation (JPA) used to mark-up the model. The Model Below are the two model classes; first the User class. package org.exparity.hamcrest.bean.sample.dao; import java.util.*; import javax.persistence.*; @Entity @Table public class User { @Id @GeneratedValue(strategy = GenerationType.SEQUENCE) private Long id; private Date createTs; private String username, firstName, surname; @OneToMany(cascade = CascadeType.ALL, fetch = FetchType.EAGER) private List comments = new ArrayList<>(); public Long getId() { return id; } public void setId(Long id) { this.id = id; } public Date getCreateTs() { return createTs; } public void setCreateTs(Date createTs) { this.createTs = createTs; } public String getUsername() { return username; } public void setUsername(String username) { this.username = username; } public String getFirstName() { return firstName; } public void setFirstName(String firstName) { this.firstName = firstName; } public String getSurname() { return surname; } public void setSurname(String surname) { this.surname = surname; } public List getComments() { return comments; } public void setComments(List comments) { this.comments = comments; } } Followed by the UserComment class. package org.exparity.hamcrest.bean.sample.dao; import java.util.Date; import javax.persistence.*; @Table @Entity public class UserComment { private Long id; private Date timestamp; @Transient private String text; private String title; public Date getTimestamp() { return timestamp; } public void setTimestamp(Date timestamp) { this.timestamp = timestamp; } public String getText() { return text; } public void setText(String text) { this.text = text; } public String getTitle() { return title; } public void setTitle(String title) { this.title = title; } } Followed by the UserComment class. package org.exparity.hamcrest.bean.sample.dao; import java.util.Date; import javax.persistence.*; @Table @Entity public class UserComment { private Long id; private Date timestamp; @Transient private String text; private String title; public Date getTimestamp() { return timestamp; } public void setTimestamp(Date timestamp) { this.timestamp = timestamp; } public String getText() { return text; } public void setText(String text) { this.text = text; } public String getTitle() { return title; } public void setTitle(String title) { this.title = title; } } The Data Access Object (DAO) Next up we write our DAO layer. I've excluded the UserDAO interface from this post but it is available in the sample project ongithub .The full, if somewhat crude, implementation of the UserDAO is below. package org.exparity.hamcrest.bean.sample.dao; import org.hibernate.boot.registry.StandardServiceRegistryBuilder; import org.hibernate.cfg.Configuration; import org.hibernate.*; public class UserDAOHibernateImpl implements UserDAO { private final SessionFactory factory; public UserDAOHibernateImpl(final String resourceFile) { this.factory = new Configuration() .addAnnotatedClass(User.class) .addAnnotatedClass(UserComment.class) .buildSessionFactory( new StandardServiceRegistryBuilder() .loadProperties(resourceFile) .build()); } @Override public User save(final User user) { Session session = factory.getCurrentSession(); Transaction txn = session.beginTransaction(); try { session.save(user); txn.commit(); } catch (final Exception e) { txn.rollback(); } return user; } @Override public User getUserById(Long userId) { Session session = factory.getCurrentSession(); Transaction txn = session.beginTransaction(); try { return (User) session.get(User.class, userId); } finally { txn.rollback(); } } } Integration Test And finally, onto our integration test. The hibernate.properties will create an instance of an in-memory database and create the necessary tables on instantiation of the DAO. hibernate.dialect=org.hibernate.dialect.H2Dialect hibernate.connection.username=sa hibernate.connection.password= hibernate.connection.driver_class=org.h2.Driver hibernate.connection.url=jdbc:h2:mem:test hibernate.current_session_context_class=thread hibernate.cache.provider_class=org.hibernate.cache.internal.NoCacheProvider hibernate.show_sql=true hibernate.hbm2ddl.auto=update The integration test is below. package org.exparity.hamcrest.bean.sample.dao; import static org.exparity.hamcrest.BeanMatchers.theSameBeanAs; import static org.exparity.stub.bean.BeanBuilder.aRandomInstanceOf; import static org.hamcrest.MatcherAssert.assertThat; import static org.hamcrest.Matchers.*; import org.junit.Test; public class UserDAOHibernateImplTest { @Test public void canSaveAUser() { User user = aRandomInstanceOf(User.class).excludeProperty("Id").build(); UserDAOHibernateImpl dao = new UserDAOHibernateImpl("hibernate.properties"); User saved = dao.save(user); User loaded = dao.getUserById(saved.getId()); assertThat(loaded, not(sameInstance(user))); assertThat(loaded, theSameBeanAs(user)); } } Let's break the test down step by step to see what each step is doing and why the test is put together this way. 1) Model Setup User user = aRandomInstanceOf(User.class).excludeProperty("Id").build(); Create a random instance of the User class and it's associates using exparity-stub. The instance will be populated with random data with the exception of the Id property. I've excluded the Id property so that is left null to test that the id is being generated in the database. 2) DAO Setup UserDAOHibernateImpl dao = new UserDAOHibernateImpl("hibernate.properties") Instantiate the DAO ready to be tested, passing in the property file to use for the test. The hibernate properties used will configure an in-memory instance of H2 and create the schema automatically. 3) Exercise the DAO User saved = dao.save(user); User loaded = dao.getUserById(saved.getId()); Save the random instance of the model set up in step (1) and then query the object back out again. 4) Verify the results assertThat(loaded, not(sameInstance(user))); assertThat(loaded, theSameBeanAs(user)); The first line verifies that the loaded User instance is not the same instance as the originally saved User. This prevents false positive results when the loaded instance is returned directly from a cache. The second line uses hamcrest-bean to perform a deep comparison of the loaded User instance against the original user instance. Running the Test The first run of the test yields an error; specifically a hibernate warning because a @Id annotation has been missed on UserComment. org.hibernate.AnnotationException: No identifier specified for entity: org.exparity.hamcrest.bean.sample.dao.UserComment at org.hibernate.cfg.InheritanceState.determineDefaultAccessType(InheritanceState.java:277) at org.hibernate.cfg.InheritanceState.getElementsToProcess(InheritanceState.java:224) at org.hibernate.cfg.AnnotationBinder.bindClass(AnnotationBinder.java:775) at org.hibernate.cfg.Configuration$MetadataSourceQueue.processAnnotatedClassesQueue(Configuration.java:3845) at org.hibernate.cfg.Configuration$MetadataSourceQueue.processMetadata(Configuration.java:3799) at org.hibernate.cfg.Configuration.secondPassCompile(Configuration.java:1412) at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1846) at org.exparity.hamcrest.bean.sample.dao.UserDAOHibernateImpl.(UserDAOHibernateImpl.java:15) at org.exparity.hamcrest.bean.sample.dao.UserDAOHibernateImplTest.canSaveAUser(UserDAOHibernateImplTest.java:18) A fix to the UserComment object and we can run the test again. @Table @Entity public class UserComment { @Id @GeneratedValue(strategy = GenerationType.SEQUENCE) private Long id; private Date timestamp; @Transient private String text; private String title; ... After running the test again we get another failure. The presence of the @Transient annotation on the UserComment.text property is preventing the value being persisted java.lang.AssertionError: Expected: the same as but: User.Comments[0].Text is null instead of "mDAWDJXbheIHbbHLR1NNVJqAki49RvaVwQtKD38r79u0y3MTDD" at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8) at org.exparity.hamcrest.bean.sample.dao.UserDAOHibernateImplTest.canSaveAUser(UserDAOHibernateImplTest.java:19) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) Another change to the UserComment object to remove the @Transient annotation and we can run the test again. @Table @Entity public class UserComment { @Id @GeneratedValue(strategy = GenerationType.SEQUENCE) private Long id; private Date timestamp; private String text; private String title; ... After running the test again it all passes. Try It Out To try hamcrest-bean and exparity-stub out for yourself include the dependency in your maven pom or other dependency manager. org.exparity hamcrest-bean 1.0.10 test org.exparity exparity-stub 1.1.5 test

Value-Driven SON® provides the right mobile network experience for the right customers at the right time Cellwize, the innovative Self-Organizing Networks provider, has revealed how its Value-Driven SON® solution can provide mobile operators with the means to ensure better customer experience for selected audiences, thereby driving new business for operators. Cellwize’ Value-Driven SON shifts the value generated by the network to the end customer – moving from a Network-Centric SON to a Customer-Centric activity. The latest whitepaper from Cellwize “Value-Driven SON – Putting the Customer at the Network’s Center” is a must read for forward thinking operators who are introducing customer-focused metrics. As operators globally look to maximize user satisfaction, several are using metrics to measure quality of experience (QoE), net promoter scores (NPS) and data service revenues. In fact some are starting to use the term Key Quality Indicator (KQI) to describe these metrics and distinguish them from the more operational KPIs. And, unlike KPIs, KQI’s are generally at a customer level – either for selected customers or a segment of customers. “We believe that SON should be driven by the business objectives of the operator and not by the immediate need to reduce complexity and costs in the network itself,” said Ofir Zemer, CEO of Cellwize. “And since operator’s business objectives aim to deliver superior value to various groups of end-users; Value-Driven SON allows mobile operators to turn this value to enhanced revenue. Value-Driven SON connects customer insight generated in and around the network with network optimization technology, and is driven by the operators’ business goals. Ultimately, the network is there to serve a purpose and the shift to the user will enable a quicker path for operators to deliver better performance and better user experience.” The Whitepaper illustrates the benefits of a customer-centric approach with Value-Driven SON through two types of use cases: commuters and enterprise customers. The first depicts Value-Driven SON for Space & Time use cases - prioritizing network resources to meet the demand of commuters along a specific route experiencing poor service quality, such as dropped calls, session termination, latency and data throughput. The second, explains how Value-Driven SON can prioritize and maximize network performance for a defined group of customers e.g. enterprise customers. Download the whitepaper to find out more about “Value-Driven SON®: Putting the Customer at the Network’s Center”. Additional information Cellwize elastic-SON® and Value Driven-SON®

Manufacturing has been undergoing a significant number of changes in recent years, and I’ve touched on a number of them on this blog. Whether it’s the rise of TechShop style co-working spaces, or the opening up of intellectual property via communities such as Quirky. Industrial giant GE have been at the heart of these things, so it is perhaps no surprise that they are leading an effort by the Digital Manufacturing Design and Innovation Institute (DMDII) to create a Digital Manufacturing Commons. Opening up manufacturing This will provide a digital marketplace to more effectively connect up supply chains. It will consist of an open source platform along the lines of the platform developed by GE in partnership with DARPA and MIT a few years ago. “The Digital Manufacturing Commons will open up innovation and collaboration in ways that create a whole new renaissance in manufacturing,” GE say. “The open source platform we are building with our DMDII partners truly will democratize access to the tools of manufacturing innovation for companies, universities, institutes and entrepreneurs big and small.” The aim is for innovation within manufacturing to be supported, and tremendous potential unlocked for manufacturers within the US. The project hopes to connect manufacturers, supply chains and entrepreneurs to accelerate new products to market. With agility increasingly important, GE hope that the platform will bring the various facets of the supply chain together in a productive way, but especially between the physical and digital worlds that are so heavily influencing manufacturing today. Creating the 21st Century assembly line This is especially evident in the amount of data that’s generated throughout the lifecycle of a product, whether it’s in the design stage, the sourcing, production, distribution, point of sale or even when it’s being used by customers. Maintaining a seamless flow of data across each of these stages is likely to be hugely valuable. Whilst this is very far from the case at the moment, it’s hoped that the Digital Manufacturing Commons will provide a platform to support such sharing. “Manufacturing operations will achieve the agility and speed that we have seen in other digital industries,” GE say. With the plan being to recruit over 100,000 users within the next few years, this is certainly a project to watch with interest. Original post

Many people believe that marketing is just advertising or selling. But, in reality, marketing is actually a conversation that you the entrepreneur start by meeting your customers’ needs. One of the biggest challenges in marketing we see is executives who resist it by asking, “Yes, but how will this help me sell more?” Many businesses struggle with just such a sales- or product-focused culture, forgetting that they were formed to solve a customer problem and that the solution to that problem was born of insights gleaned from talking to potential customers. It’s this forgetfulness that makes marketing look less like it’s about serving customers and more like it’s about (only) sales and advertising. And that’s not good. In fact, for many business-to-business brands, marketing originally emerged from the sales team’s need for more leads in the field. In the consumer space, marketers started on the traditional advertising side, where the game was all about reach and frequency, and where the goal was to get the brand message out to the target audience. The legacy of such thinking leads to the biggest mistake marketers make: They make the message all about themselves. When I started my own content-marketing journey, I remember, I was always hearing, “How much more stuff will this help us sell?” Yet the worst way to try to reach your target audience in today’s digital, consumer-led world is to try to sell yourself directly. Businesses that succeed in reaching their customers have stopped trying to interrupt the content their consumers are interested in and instead have started creating, publishing and sharing the content their customers enjoy. Related: A Guide For Creating Consistently Great Content Answer customers’ questions, and you may earn the right to tell them more about yourself. But building a company that focuses on helping people versus selling stuff? That’s a question of culture! And culture flows straight from the boss. So, if you’re a CEO, your job is to build a customer-focused culture of content. And here are three questions to pose that help you do just that. 1. What is a culture of content? Effective marketing is the art of providing the best answers to your buyers’ questions, and that’s a content problem — which in turn is a job that flows to just about every employee. Everyone in your business produces content. Everyone has an email address and a few social accounts. Content drives real business value when it connects with your potential customers. Late last year, Altimeter published a report on how to foster a culture of content, to which I was honored to contribute. The research pointed to education, executive buy-in and employee advocacy as key components of a culture of content. 2. How do you shift away from a culture of selling? Most businesses believe that the best way to drive new sales is to talk about themselves, thinking that if they’re not outright asking for new business, they won’t get it. Today, however, customers tune out such promotional messages. They can tell which content is trying to sell. Businesses need to make the customer the hero of their stories, exhibiting empathy in a real and emotional way. Many businesses forget that one of the most effective ways to use content to drive traffic is to simply answer your customers’ most basic questions. If you sell widgets, the first question your content should answer is, “What are widgets?” And then: “How can widgets help a business like mine?” Once you’ve done that on a regular basis, you can answer why your widgets are best. There is no magic pill for effective content that drives traffic. The best businesses have a documented strategy for publishing helpful, high-quality content; they drive that strategy by publishing on a consistent basis. Publishing audience-focused content more than once per day is much more effective than publishing less frequently. And frequency requires a culture of content. 3. How can brands establish a culture of content? The best way to build a “culture of content” is to help your employees understand what problem you set out to solve. The mission must be bigger than the service or product you sell. The brand is about more than what you sell, it’s about people. Companies that do this well understand the larger world they operate in and how they fit into it; they activate their employees to tell authentic and personal stories about how they contribute. As a content marketer, I have often found myself teaching others how to write, share on social media and build their personal brands. To succeed takes executives who embody this spirit — living and breathing the notion that your brand is bigger than what you sell. Creating and defining a culture of content, then, starts with the CEO. However, it’s also the daily job of everyone else. What content have you produced today that will help a potential customer? Have you coached any executives on how to turn their presentations into slideshares and blog posts? Have you encouraged any of your thought leaders to start contributing more often? It’s time to get started to create this bright new customer-winning culture Original post

In this post, I demonstrate one approach for generating JSON Schema from an XML Schema (XSD). While providing an overview of an approach for creating JSON Schema from XML Schema, this post also demonstrates use of a JAXB implementation (xjc version 2.2.12-b150331.1824 bundled with JDK 9 [build 1.9.0-ea-b68]) and of a JSON/Java binding implementation (Jackson 2.5.4). The steps of this approach for generating JSON Schema from an XSD can be summarized as: Apply JAXB's xjc compiler to generate Java classes from XML Schema (XSD). Apply Jackson to generate JSON schema from JAXB-generated Java classes. Generating Java Classes from XSD with JAXB's xjc For purposes of this discussion, I'll be using the simple Food.xsd used in my previous blog post A JAXB Nuance: String Versus Enum from Enumerated Restricted XSD String. For convenience, I have reproduced that simple schema here without the XML comments specific to that earlier blog post: Food.xsd It is easy to use the xjc command line tool provided by the JDK-provided JAXB implementation to generate Java classes corresponding to this XSD. The next screen snapshot shows this process using the command: xjc -d jaxb .\Food.xsd This simple command generates Java classes corresponding to the provided Food.xsd and places those classes in the specified "jaxb" subdirectory. Generating JSON from JAXB-Generated Classes with Jackson With the JAXB-generated classes now available, Jackson can be applied to these classes to generate JSON from the Java classes. Jackson is described on its main portal page as "a multi-purpose Java library for processing" that is "inspired by the quality and variety of XML tooling available for the Java platform." The existence of Jackson and similar frameworks and libraries appears to be one of the reasons that Oracle hasdropped the JEP 198 ("Light-Weight JSON API") from Java SE 9. [It's worth noting that Java EE 7 already hasbuilt-in JSON support with its implementation of JSR 353 ("Java API for JSON Processing"), which is not associated with JEP 198).] One of the first steps of applying Jackson to generating JSON from our JAXB-generated Java classes is to acquire and configure an instance of Jackson's ObjectMapper class. One approach for accomplishing this is shown in the next code listing. Acquiring and Configuring Jackson ObjectMapper for JAXB Serialization/Deserialization /** * Create instance of ObjectMapper with JAXB introspector * and default type factory. * * @return Instance of ObjectMapper with JAXB introspector * and default type factory. */ private ObjectMapper createJaxbObjectMapper() { final ObjectMapper mapper = new ObjectMapper(); final TypeFactory typeFactory = TypeFactory.defaultInstance(); final AnnotationIntrospector introspector = new JaxbAnnotationIntrospector(typeFactory); // make deserializer use JAXB annotations (only) mapper.getDeserializationConfig().with(introspector); // make serializer use JAXB annotations (only) mapper.getSerializationConfig().with(introspector); return mapper; } The above code listing demonstrates acquiring the Jackson ObjectMapper instance and configuring it to use a default type factory and a JAXB-oriented annotation introspector. With the Jackson ObjectMapper instantiated and appropriately configured, it's easy to use thatObjectMapper instance to generate JSON from the generated JAXB classes. One way to accomplish this using the deprecated Jackson class JsonSchema is demonstrated in the next code listing. Generating JSON from Java Classes with Deprecated com.fasterxml.jackson.databind.jsonschema.JsonSchema Class /** * Write JSON Schema to standard output based upon Java source * code in class whose fully qualified package and class name * have been provided. * * @param mapper Instance of ObjectMapper from which to * invoke JSON schema generation. * @param fullyQualifiedClassName Name of Java class upon * which JSON Schema will be extracted. */ private void writeToStandardOutputWithDeprecatedJsonSchema( final ObjectMapper mapper, final String fullyQualifiedClassName) { try { final JsonSchema jsonSchema = mapper.generateJsonSchema(Class.forName(fullyQualifiedClassName)); out.println(jsonSchema); } catch (ClassNotFoundException cnfEx) { err.println("Unable to find class " + fullyQualifiedClassName); } catch (JsonMappingException jsonEx) { err.println("Unable to map JSON: " + jsonEx); } } The code in the above listing instantiates acquires the class definition of the provided Java class (the highest level Food class generated by the JAXB xjc compiler in my example) and passes that reference to the JAXB-generated class to ObjectMapper's generateJsonSchema(Class) method. The deprecated JsonSchemaclass's toString() implementation is very useful and makes it easy to write out the JSON generated from the JAXB-generated classes. For purposes of this demonstration, I provide the demonstration driver as a main(String[]) function. That function and the entire class to this point (including methods shown above) is provided in the next code listing. JsonGenerationFromJaxbClasses.java, Version 1 package dustin.examples.jackson; import com.fasterxml.jackson.databind.AnnotationIntrospector; import com.fasterxml.jackson.databind.JsonMappingException; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.databind.type.TypeFactory; import com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector; import com.fasterxml.jackson.databind.jsonschema.JsonSchema; import static java.lang.System.out; import static java.lang.System.err; /** * Generates JavaScript Object Notation (JSON) from Java classes * with Java API for XML Binding (JAXB) annotations. */ public class JsonGenerationFromJaxbClasses { /** * Create instance of ObjectMapper with JAXB introspector * and default type factory. * * @return Instance of ObjectMapper with JAXB introspector * and default type factory. */ private ObjectMapper createJaxbObjectMapper() { final ObjectMapper mapper = new ObjectMapper(); final TypeFactory typeFactory = TypeFactory.defaultInstance(); final AnnotationIntrospector introspector = new JaxbAnnotationIntrospector(typeFactory); // make deserializer use JAXB annotations (only) mapper.getDeserializationConfig().with(introspector); // make serializer use JAXB annotations (only) mapper.getSerializationConfig().with(introspector); return mapper; } /** * Write out JSON Schema based upon Java source code in * class whose fully qualified package and class name have * been provided. * * @param mapper Instance of ObjectMapper from which to * invoke JSON schema generation. * @param fullyQualifiedClassName Name of Java class upon * which JSON Schema will be extracted. */ private void writeToStandardOutputWithDeprecatedJsonSchema( final ObjectMapper mapper, final String fullyQualifiedClassName) { try { final JsonSchema jsonSchema = mapper.generateJsonSchema(Class.forName(fullyQualifiedClassName)); out.println(jsonSchema); } catch (ClassNotFoundException cnfEx) { err.println("Unable to find class " + fullyQualifiedClassName); } catch (JsonMappingException jsonEx) { err.println("Unable to map JSON: " + jsonEx); } } /** * Accepts the fully qualified (full package) name of a * Java class with JAXB annotations that will be used to * generate a JSON schema. * * @param arguments One argument expected: fully qualified * package and class name of Java class with JAXB * annotations. */ public static void main(final String[] arguments) { if (arguments.length < 1) { err.println("Need to provide the fully qualified name of the highest-level Java class with JAXB annotations."); System.exit(-1); } final JsonGenerationFromJaxbClasses instance = new JsonGenerationFromJaxbClasses(); final String fullyQualifiedClassName = arguments[0]; final ObjectMapper objectMapper = instance.createJaxbObjectMapper(); instance.writeToStandardOutputWithDeprecatedJsonSchema(objectMapper, fullyQualifiedClassName); } } To run this relatively generic code against the Java classes generated by JAXB's xjc based upon Food.xsd, I need to provide the fully qualified package name and class name of the highest-level generated class. In this case, that's com.blogspot.marxsoftware.foodxml.Food (package name is based on the XSD's namespace because I did not explicitly override that when running xjc). When I run the above code with that fully qualified class name and with the JAXB classes and Jackson libraries on the classpath, I see the following JSON written to standard output. Generated JSON {"type":"object","properties":{"vegetable":{"type":"string","enum":["CARROT","SQUASH","SPINACH","CELERY"]},"fruit":{"type":"string"},"dessert":{"type":"string","enum":["PIE","CAKE","ICE_CREAM"]}} Humans (which includes many developers) prefer prettier print than what was just shown for the generated JSON. We can tweak the implementation of the demonstration class's methodwriteToStandardOutputWithDeprecatedJsonSchema(ObjectMapper, String) as shown below to write out indented JSON that better reflects its hierarchical nature. This modified method is shown next. Modified writeToStandardOutputWithDeprecatedJsonSchema(ObjectMapper, String) to Write Indented JSON /** * Write out indented JSON Schema based upon Java source * code in class whose fully qualified package and class * name have been provided. * * @param mapper Instance of ObjectMapper from which to * invoke JSON schema generation. * @param fullyQualifiedClassName Name of Java class upon * which JSON Schema will be extracted. */ private void writeToStandardOutputWithDeprecatedJsonSchema( final ObjectMapper mapper, final String fullyQualifiedClassName) { try { final JsonSchema jsonSchema = mapper.generateJsonSchema(Class.forName(fullyQualifiedClassName)); out.println(mapper.writerWithDefaultPrettyPrinter().writeValueAsString(jsonSchema)); } catch (ClassNotFoundException cnfEx) { err.println("Unable to find class " + fullyQualifiedClassName); } catch (JsonMappingException jsonEx) { err.println("Unable to map JSON: " + jsonEx); } catch (JsonProcessingException jsonEx) { err.println("Unable to process JSON: " + jsonEx); } } When I run the demonstration class again with this modified method, the JSON output is more aesthetically pleasing: Generated JSON with Indentation Communicating Hierarchy { "type" : "object", "properties" : { "vegetable" : { "type" : "string", "enum" : [ "CARROT", "SQUASH", "SPINACH", "CELERY" ] }, "fruit" : { "type" : "string" }, "dessert" : { "type" : "string", "enum" : [ "PIE", "CAKE", "ICE_CREAM" ] } } } I have been using Jackson 2.5.4 in this post. The classcom.fasterxml.jackson.databind.jsonschema.JsonSchema is deprecated in that version with the comment, "Since 2.2, we recommend use of external JSON Schema generator module." Given that, I now look at using the new preferred approach (Jackson JSON Schema Module approach). The most significant change is to use the JsonSchema class in the com.fasterxml.jackson.module.jsonSchemapackage rather than using the JsonSchema class in the com.fasterxml.jackson.databind.jsonschema package. The approaches for obtaining instances of these different versions of JsonSchema classes are also different. The next code listing demonstrates using the newer, preferred approach for generating JSON from Java classes. Using Jackson's Newer and Preferred com.fasterxml.jackson.module.jsonSchema.JsonSchema /** * Write out JSON Schema based upon Java source code in * class whose fully qualified package and class name have * been provided. This method uses the newer module JsonSchema * class that replaces the deprecated databind JsonSchema. * * @param fullyQualifiedClassName Name of Java class upon * which JSON Schema will be extracted. */ private void writeToStandardOutputWithModuleJsonSchema( final String fullyQualifiedClassName) { final SchemaFactoryWrapper visitor = new SchemaFactoryWrapper(); final ObjectMapper mapper = new ObjectMapper(); try { mapper.acceptJsonFormatVisitor(mapper.constructType(Class.forName(fullyQualifiedClassName)), visitor); final com.fasterxml.jackson.module.jsonSchema.JsonSchema jsonSchema = visitor.finalSchema(); out.println(mapper.writerWithDefaultPrettyPrinter().writeValueAsString(jsonSchema)); } catch (ClassNotFoundException cnfEx) { err.println("Unable to find class " + fullyQualifiedClassName); } catch (JsonMappingException jsonEx) { err.println("Unable to map JSON: " + jsonEx); } catch (JsonProcessingException jsonEx) { err.println("Unable to process JSON: " + jsonEx); } } The following table compares usage of the two Jackson JsonSchema classes side-by-side with the deprecated approach shown earlier on the left (adapted a bit for this comparison) and the recommended newer approach on the right. Both generate the same output for the same given Java class from which JSON is to be written. /** * Write out JSON Schema based upon Java source code in * class whose fully qualified package and class name have * been provided. This method uses the deprecated JsonSchema * class in the "databind.jsonschema" package * {@see com.fasterxml.jackson.databind.jsonschema}. * * @param fullyQualifiedClassName Name of Java class upon * which JSON Schema will be extracted. */ private void writeToStandardOutputWithDeprecatedDatabindJsonSchema( final String fullyQualifiedClassName) { final ObjectMapper mapper = new ObjectMapper(); try { final com.fasterxml.jackson.databind.jsonschema.JsonSchema jsonSchema = mapper.generateJsonSchema(Class.forName(fullyQualifiedClassName)); out.println(mapper.writerWithDefaultPrettyPrinter().writeValueAsString(jsonSchema)); } catch (ClassNotFoundException cnfEx) { err.println("Unable to find class " + fullyQualifiedClassName); } catch (JsonMappingException jsonEx) { err.println("Unable to map JSON: " + jsonEx); } catch (JsonProcessingException jsonEx) { err.println("Unable to process JSON: " + jsonEx); } } /** * Write out JSON Schema based upon Java source code in * class whose fully qualified package and class name have * been provided. This method uses the newer module JsonSchema * class that replaces the deprecated databind JsonSchema. * * @param fullyQualifiedClassName Name of Java class upon * which JSON Schema will be extracted. */ private void writeToStandardOutputWithModuleJsonSchema( final String fullyQualifiedClassName) { final SchemaFactoryWrapper visitor = new SchemaFactoryWrapper(); final ObjectMapper mapper = new ObjectMapper(); try { mapper.acceptJsonFormatVisitor(mapper.constructType(Class.forName(fullyQualifiedClassName)), visitor); final com.fasterxml.jackson.module.jsonSchema.JsonSchema jsonSchema = visitor.finalSchema(); out.println(mapper.writerWithDefaultPrettyPrinter().writeValueAsString(jsonSchema)); } catch (ClassNotFoundException cnfEx) { err.println("Unable to find class " + fullyQualifiedClassName); } catch (JsonMappingException jsonEx) { err.println("Unable to map JSON: " + jsonEx); } catch (JsonProcessingException jsonEx) { err.println("Unable to process JSON: " + jsonEx); } } This blog post has shown two approaches using different versions of classes with name JsonSchema provided by Jackson to write JSON based on Java classes generated from an XSD with JAXB's xjc. The overall process demonstrated in this post is one approach for generating JSON Schema from XML Schema.

Download the MP3 Date: June 19, 2015 By: Aaron Delp and Brian Gracely Description: Aaron and Brian talk to Chip Childers (@chipchilders, VP of Technology @CloudFoundryOrg) about the current status of Cloud Foundry projects, how Microsoft .NET will be integrated, IaaS vs. PaaS, and the CF.org thinking about overall interoperability Interested in the O'Reilly OSCON? Want to register for OSCON now? Use promo code 20CLOUD for 20% off Details to win an OSCON pass coming soon! Check out the OSCON Schedule Free eBook from O'Reilly Media for Cloudcast Listeners! Check out an excerpt from the upcoming Docker Cookbook Topic 1 - From an overall project perspective, what grades would you give Cloud Foundry in terms of stability, core functionality, security, operations, etc? Topic 2 - You were previously involved (directly/indirectly)with CloudStack. As you talk to people in the marketplace, how is it different discussing IaaS vs. PaaS. Topic 3 - How much ability will you have to drive prioritization within sub-projects or new projects? (eg. Security vs. new Languages vs. Interop, etc.) Topic 4 - What’s the CF.org way of thinking about interoperability? Topic 5 - What guidance are you giving the teams in terms of expandability of Cloud Foundry? Architecturally, are there certain places you recommend over other places? Topic 6 - Is there a place for integrating SaaS applications (monitoring, logging, etc.) into Cloud Foundry?

CoreOS Linux is the operating system for “Super Massive Deployments”. We wanted to see how easily we can get CoreOS logs into Elasticsearch / ELK-powered centralized logging service. Here’s how to get your CoreOS logs into ELK in about 5 minutes, give or take. If you’re familiar with CoreOS and Logsene, you can grab CoreOS/Logsene config files from Github. Here’s an example Kibana Dashboard you can get in the end: CoreOS Kibana Dashboard CoreOS is based on the following: Docker and rkt for containers systemd for startup scripts, and restarting services automatically etcd as centralized configuration key/value store fleetd to distribute services over all machines in the cluster. Yum. journald to manage logs. Another yum. Amazingly, with CoreOS managing a cluster feels a lot like managing a single machine! We’ve come a long way since ENIAC! There’s one thing people notice when working with CoreOS – the repetitive inspection of local or remote logs using “journalctl -M machine-N -f | grep something“. It’s great to have easy access to logs from all machines in the cluster, but … grep? Really? Could this be done better? Of course, it’s 2015! Here is a quick example that shows how to centralize logging with CoreOS with just a few commands. The idea is to forward the output of “journalctl -o short” to Logsene‘s Syslog Receiver and take advantage of all its functionality – log searching, alerting, anomaly detection, integrated Kibana, even correlation of logs with Docker performance metrics — hey, why not, it’s all available right there, so we may as well make use of it all! Let’s get started! Preparation: 1) Get a list of IP addresses of your CoreOS machines fleetctl list-machines 2) Create a new Logsene App (here) 3) Change the Logsene App Settings, and authorize the CoreOS host IP Addresses from step 1) (here’s how/where) Congratulations – you just made it possible for your CoreOS machines to ship their logs to your new Logsene app! Test it by running the following on any of your CoreOS machines: journalctl -o short -f | ncat --ssl logsene-receiver-syslog.sematext.com 10514 …and check if the logs arrive in Logsene (here). If they don’t, yell at us @sematext – there’s nothing better than public shaming on Twitter to get us to fix things. :) Create a fleet unit file called logsene.service [Unit] Description=Logsene Log Forwarder [Service] Restart=always RestartSec=10s ExecStartPre=/bin/sh -c "if [ -n \"$(etcdctl get /sematext.com/logsene/`hostname`/lastlog)\" ]; then echo \"Value Exists: /sematext.com/logsene/`hostname`/lastlog $(etcdctl get /sematext.com/logsene/`hostname`/lastlog)\"; else etcdctl set /sematext.com/logsene/`hostname`/lastlog\"`date +\"%Y-%%m-%d %%H:%M:%S\"`\"; true; fi" ExecStart=/bin/sh -c "journalctl --since \"$(etcdctl get /sematext.com/logsene/`hostname`/lastlog)\" -o short -f | ncat --ssl logsene-receiver-syslog.sematext.com 10514" ExecStopPost=/bin/sh -c "export D=\"`date +\"%Y-%%m-%%d %%H:%M:%S\"`\"; /bin/etcdctl set /sematext.com/logsene/$(hostname)/lastlog \"$D\"" [Install] WantedBy=multi-user.target [X-Fleet] Global=true Activate cluster-wide logging to Logsene with fleet To start logging to Logsene from all machines activate logsene.service: fleetctl load logsene.service fleetctl start logsene.service There. That’s all there is to it! Hope this worked for you! At this point all your CoreOS logs should be going to Logsene. Now you have a central place to see all your CoreOS logs. If you want to send your app logs to Logsene, you can do that, too — anything that can send logs via Syslog or to Elasticsearch can also ship logs to Logsene. If you want some Docker containers & host monitoring to go with your CoreOS logs, just pull spm-agent-docker from Docker Registry. Enjoy!

When creating a camel route using http, the destination might require a ssl connection with a self signed certificate. Therefore on our http client we should register a TrustManager that suports the certificate. In our case we will use the https4 component of Apache Camel Therefore we should configure the routes and add them to the camel context RouteBuilder routeBuilder = new RouteBuilder() { @Override public void configure() throws Exception { from("http://localhost") .to("https4://securepage"); } }; routeBuilder.addRoutesToCamelContext(camelContext); But before we proceed on starting the camel context we should register the trust store on the component we are going to use. Therefore we should implement a function for creating an ssl context with the trustore. Supposed the jks file that has the certificate imported is located on the root of our classpath. private void registerTrustStore(CamelContext camelContext) { try { KeyStore truststore = KeyStore.getInstance("JKS"); truststore.load(getClass().getClassLoader().getResourceAsStream("example.jks"), "changeit".toCharArray()); TrustManagerFactory trustFactory = TrustManagerFactory.getInstance("SunX509"); trustFactory.init(truststore); SSLContext sslcontext = SSLContext.getInstance("TLS"); sslcontext.init(null, trustFactory.getTrustManagers(), null); SSLSocketFactory factory = new SSLSocketFactory(sslcontext, SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER); SchemeRegistry registry = new SchemeRegistry(); final Scheme scheme = new Scheme("https4", 443, factory); registry.register(scheme); HttpComponent http4 = camelContext.getComponent("https4", HttpComponent.class); http4.setHttpClientConfigurer(new HttpClientConfigurer() { @Override public void configureHttpClient(HttpClientBuilder builder) { builder.setSSLSocketFactory(factory); Registry registry = RegistryBuilder.create() .register("https", factory) .build(); HttpClientConnectionManager connectionManager = new BasicHttpClientConnectionManager(registry); builder.setConnectionManager(ccm); } }); } catch (IOException e) { e.printStackTrace(); } catch (NoSuchAlgorithmException e) { e.printStackTrace(); } catch (CertificateException e) { e.printStackTrace(); } catch (KeyStoreException e) { e.printStackTrace(); } catch (KeyManagementException e) { e.printStackTrace(); } } After that our route would be able to access the destination securely.

I have just recorded a 5 minute video that demonstrates running an out of stock example from Apache Camel release, the camel-example-servlet packaged as a docker container and running on a kubernetes platform, such as openshift 3. camel-servlet-example scaled up to 3 running containers (pods) which is easy with kubernetes and fabric8 In this video I have already deployed the example and then demonstrates how we can use the fabric8 web console to manage our application. And also connect to the running container and see inside, such as the Camel routes visually as shown above. Then I run a simple bash script from my laptop that sends a HTTP GET to the Camel example and prints the response. The script runs in a endless loop and demonstrates how kubernetes can easily scale up and down multiple Camel containers and load balance across the running containers. And at the end its even self healing when I force killing docker containers. So I suggest to grab a fresh cup of tea or coffee and sit back and play the 5 minutes video. The video is hosted on vimeo and can be seen from this link.

You are working on a big project, developing rules, events and processes at your enterprise for mission critical business needs. Part of the requirements state that a certain business unit will be providing their data model for you to leverage. This data model will not be designed in the JBoss BPM Suite Data Modeler but you need to have access to it while working on your rules, events and processes from the business central dashboard. For this article we will be using the JBoss BPM Travel Agency demo project as a reference, with it's current data model built externally to the JBoss BPM Suite business central. The external data model is called the acme-data-model and is found in the project directory: This data model is built during installation and provides you with an object data model as a Java Archive (JAR) file which is installed into the JBoss BPM Suite business central component by placing it into the following location: jboss-eap-6.4/standalone/deployments/business-central.war/WEB_INF/lib/acmeDataModel-1.0.jar Authoring --> Artifact repository. This way of deploying the data model means that it is available to all projects you work on in JBoss BPM Suite business central, something that might not always be preferable. What we need is a way to deploy external data models into JBoss BPM Suite and then selectively add them to projects as needed. Within JBoss BPM Suite there is an Artifact Repository that is made just for this purpose. We can upload through the business central dashboard UI all our models and then pick and choose from the repository artifacts (your data model is one artifact) on a per project basis. This gives you absolute control over the models that a project can access. Choose external data model file. There are a few steps involved that we will take you through here to change the current installation of JBoss BPM Travel Agency where the acmeDataModel-1.0.jar file will be removed from the previously mentioned business central component and uploaded into the Artifact Repository and added to the Special Trips Agency project. Here is how you can do it yourself: obtain and install JBoss BPM Travel Agency demo project remove current data model from global business central application: $ rm ./target/jboss-eap-6.4/standalone/deployments/business-central.war/WEB_INF/lib/acmeDataModel-1.0.jar Upload external model jar file. start JBoss BPM Suite server after installation as stated in the installation instructions login to JBoss BPM Suite at http://localhost:8080/business-centralwith: u: erics p: bpmsuite1! go to AUTHORING --> ARTIFACT REPOSITORY go to UPLOAD --> CHOOSE FILE... --> projects/acme-data-model/target/acmeDataModel-1.0.jar --> click button to UPLOAD this puts the external data model into the JBoss BPM Suite artifact repository Select dependencies to add to project. got to AUTHORING --> PROJECT AUTHORING --> OPEN PROJECT EDITOR in project editor select GENERAL PROJECT SETTINGS --> DEPENDENCIES in dependencies select ADD FROM REPOSITORY -> in pop-upSELECT entry acmeDataModel-1.0.jar This will result in the external data model being added only to the Special Trips Agency project and not available to other projects unless they add this same dependency from the JBoss BPM Suite artifact repository. If you build & deploy the project, run it as described in the project instructions you will find that the external data model is available and used by the various rules and process components that are the JBoss BPM Travel Agency. As a closing note, this works exactly the same for JBoss BRMS projects.

Written by Norberto Leite In the first part of this blog series, we covered the basic mechanics of our application and undertook some data modeling. In this second part, we will create tests that validate the behavior of our application and then describe how to set-up and run the application. Write the tests first Let’s begin by defining some small configuration libraries. file name: test/config/test_config.js Our server will be running on port 8000 on localhost. This will be fine for initial testing purposes. Later, if we change the location or port number for a production system, it would be very easy to just edit this file. To prepare for our test cases, we need to ensure that we have a good test environment. The following code achieves this for us. First, we connect to the database. file name: test/setup_tests.js Next, we drop the user collection. This ensures that our database is in a known starting state. Next, we will drop the user feed entry collection. Next, we will connect to Stormpath and delete all the users in our test application. Next, we close the database. Finally, we call async.series to ensure that all the functions run in the correct order. Frisby was briefly mentioned earlier. We will use this to define our test cases, as follows. file name: test/create_accounts_error_spec.js We will start with the enroll route in the following code. In this case we are deliberately missing the first name field, so we expect a status reply of 400 with a JSON error that we forgot to define the first name. Let’s “toss that frisby”: In the following example, we are testing a password that does not have any lower-case letters. This would actually result in an error being returned by Stormpath, and we would expect a status reply of 400. In the following example, we are testing an invalid email address. So, we can see that there is no @ sign and no domain name in the email address we are passing, and we would expect a status reply of 400. Now, let’s look at some examples of test cases that should work. Let’s start by defining 3 users. file name: test/create_accounts_spec.js In the following example, we are sending the array of the 3 users we defined above and are expecting a success status of 201. The JSON document returned would show the user object created, so we can verify that what was created matched our test data. Next, we will test for a duplicate user. In the following example, we will try to create a user where the email address already exists. One important issue is that we don’t know what API key will be returned by Stormpath a priori. So, we need to create a file dynamically that looks like the following. We can then use this file to define test cases that require us to authenticate a user. file name: /tmp/readerTestCreds.js In order to create the temporary file above, we need to connect to MongoDB and retrieve user information. This is achieved by the following code. file name: tests/writeCreds.js In the following code, we can see that the first line uses the temporary file that we created with the user information. We have also defined several feeds, such as Dilbert and the Eater Blog. file name: tests/feed_spec.js Previously, we defined some users but none of them had subscribed to any feeds. In the following code we test feed subscription. Note that authentication is required now and this is achieved using .auth with the Stormpath API keys. Our first test is to check for an empty feed list. In our next test case, we will subscribe our first test user to the Dilbert feed. In our next test case, we will try to subscribe our first test user to a feed that they are already subscribed-to. Next, we will subscribe our test user to a new feed. The result returned should confirm that the user is subscribed now to 2 feeds. Next, we will use our second test user to subscribe to a feed. The REST API Before we begin writing our REST API code, we need to define some utility libraries. First, we need to define how our application will connect to the database. Putting this information into a file gives us the flexibility to add different database URLs for development or production systems. file name: config/db.js If we wanted to turn on database authentication we could put that information in a file, as shown below. This file should not be checked into source code control for obvious reasons. file name: config/security.js We can keep Stormpath API and Secret keys in a properties file, as follows, and need to carefully manage this file as well. file name: config/stormpath_apikey.properties Express.js overview In Express.js, we create an “application” (app). This application listens on a particular port for HTTP requests to come in. When requests come in, they pass through a middleware chain. Each link in the middleware chain is given a req (request) object and a res (results) object to store the results. Each link can choose to do work, or pass it to the next link. We add new middleware via app.use(). The main middleware is called our “router”, which looks at the URL and routes each different URL/verb combination to a specific handler function. Creating our application Now we can finally see our application code, which is quite small since we can embed handlers for various routes into separate files. file name: server.js We define our own middleware at the end of the chain to handle bad URLs. Now our server application is listening on port 8000. Let’s print a message on the console to the user. Defining our Mongoose data models We use Mongoose to map objects on the Node.js side to documents inside MongoDB. Recall that earlier, we defined 4 collections: Feed collection. Feed entry collection. User collection. User feed-entry-mapping collection. So we will now define schemas for these 4 collections. Let’s begin with the user schema. Notice that we can also format the data, such as converting strings to lowercase, and remove leading or trailing whitespace using trim. file name: app/routes.js In the following code, we can also tell Mongoose what indexes need to exist. Mongoose will also ensure that these indexes are created if they do not already exist in our MongoDB database. The unique constraint ensures that duplicates are not allowed. The “email : 1” maintains email addresses in ascending order. If we used “email : -1” it would be in descending order. We repeat the process for the other 3 collections. The following is an example of a compound index on 4 fields. Each index is maintained in ascending order. Every route that comes in for GET, POST, PUT and DELETE needs to have the correct content type, which is application/json. Then the next link in the chain is called. Now we need to define handlers for each combination of URL/verb. The link to the complete code is available in the resources section and we just show a few examples below. Note the ease with which we can use Stormpath. Furthermore, notice that we have defined /api/v1.0, so the client would actually call /api/v1.0/user/enroll, for example. In the future, if we changed the API, say to 2.0, we could use /api/v2.0. This would have its own router and code, so clients using the v1.0 API would still continue to work. Starting the server and running tests Finally, here is a summary of the steps we need to follow to start the server and run the tests. Ensure that the MongoDB instance is running mongod Install the Node libraries npm install Start the REST API server node server.js Run test cases node setup_tests.js jasmine-node create_accounts_error_spec.js jasmine-node create_accounts_spec.js node write_creds.js jasmine-node feed_spec.js MongoDB University provides excellent free training. There is a course specifically aimed at Node.js developers and the link can be found in the resources section below. The resources section also contains links to good MongoDB data modeling resources. Resources HTTP status code definitions Chad Tindel’s Github Repository M101JS: MongoDB for Node.js Developers Data Models Data Modeling Considerations for MongoDB Applications

Continuing on from my blog post about scraping Neo4j’s release dates I thought it’d be even more interesting to chart the release dates of some github projects. In theory the release dates should be accessible through the github API but the few that I looked at weren’t returning any data so I scraped the data together. We’ll be using rvest again and I first wrote the following function to extract the release versions and dates from a single page: library(dplyr) library(rvest) process_page = function(releases, session) { rows = session %>% html_nodes("ul.release-timeline-tags li") for(row in rows) { date = row %>% html_node("span.date") version = row %>% html_node("div.tag-info a") if(!is.null(version) && !is.null(date)) { date = date %>% html_text() %>% str_trim() version = version %>% html_text() %>% str_trim() releases = rbind(releases, data.frame(date = date, version = version)) } } return(releases) } Let’s try it out on the Cassandra release page and see what it comes back with: > r = process_page(data.frame(), html_session("https://github.com/apache/cassandra/releases")) > r date version 1 Jun 22, 2015 cassandra-2.1.7 2 Jun 22, 2015 cassandra-2.0.16 3 Jun 8, 2015 cassandra-2.1.6 4 Jun 8, 2015 cassandra-2.2.0-rc1 5 May 19, 2015 cassandra-2.2.0-beta1 6 May 18, 2015 cassandra-2.0.15 7 Apr 29, 2015 cassandra-2.1.5 8 Apr 1, 2015 cassandra-2.0.14 9 Apr 1, 2015 cassandra-2.1.4 10 Mar 16, 2015 cassandra-2.0.13 That works pretty well but it’s only one page! To get all the pages we can use the follow_link function to follow the ‘Next’ link until there aren’t anymore pages to process. We end up with the following function to do this: find_all_releases = function(starting_page) { s = html_session(starting_page) releases = data.frame() next_page = TRUE while(next_page) { possibleError = tryCatch({ releases = process_page(releases, s) s = s %>% follow_link("Next") }, error = function(e) { e }) if(inherits(possibleError, "error")){ next_page = FALSE } } return(releases) } Let’s try it out starting from the Cassandra page: > cassandra = find_all_releases("https://github.com/apache/cassandra/releases") Navigating to https://github.com/apache/cassandra/releases?after=cassandra-2.0.13 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-2.0.10 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-2.0.8 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-1.2.13 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-2.0.0-rc1 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-1.2.3 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-1.2.0-beta2 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-1.0.10 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-1.0.6 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-1.0.0-rc2 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-0.7.7 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-0.7.4 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-0.7.0-rc3 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-0.6.4 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-0.5.0-rc3 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-0.4.0-final > cassandra %>% sample_n(10) date version 151 Mar 13, 2010 cassandra-0.5.0-rc2 25 Jul 3, 2014 cassandra-1.2.18 51 Jul 27, 2013 cassandra-1.2.8 21 Aug 19, 2014 cassandra-2.1.0-rc6 73 Sep 24, 2012 cassandra-1.2.0-beta1 158 Mar 13, 2010 cassandra-0.4.0-rc2 113 May 20, 2011 cassandra-0.7.6-2 15 Oct 24, 2014 cassandra-2.1.1 103 Sep 15, 2011 cassandra-1.0.0-beta1 93 Nov 29, 2011 cassandra-1.0.4 I want to plot when the different releases happened in time and in order to do that we need to create an extra column containing the ‘release series’ which we can do with the following transformation: series = function(version) { parts = strsplit(as.character(version), "\\.") return(unlist(lapply(parts, function(p) paste(p %>% unlist %>% head(2), collapse = ".")))) } bySeries = cassandra %>% mutate(date2 = mdy(date), series = series(version), short_version = gsub("cassandra-", "", version), short_series = series(short_version)) > bySeries %>% sample_n(10) date version date2 series short_version short_series 3 Jun 8, 2015 cassandra-2.1.6 2015-06-08 cassandra-2.1 2.1.6 2.1 161 Mar 13, 2010 cassandra-0.4.0-beta1 2010-03-13 cassandra-0.4 0.4.0-beta1 0.4 62 Feb 15, 2013 cassandra-1.1.10 2013-02-15 cassandra-1.1 1.1.10 1.1 153 Mar 13, 2010 cassandra-0.5.0-beta2 2010-03-13 cassandra-0.5 0.5.0-beta2 0.5 37 Feb 7, 2014 cassandra-2.0.5 2014-02-07 cassandra-2.0 2.0.5 2.0 36 Feb 7, 2014 cassandra-1.2.15 2014-02-07 cassandra-1.2 1.2.15 1.2 29 Jun 2, 2014 cassandra-2.1.0-rc1 2014-06-02 cassandra-2.1 2.1.0-rc1 2.1 21 Aug 19, 2014 cassandra-2.1.0-rc6 2014-08-19 cassandra-2.1 2.1.0-rc6 2.1 123 Feb 16, 2011 cassandra-0.7.2 2011-02-16 cassandra-0.7 0.7.2 0.7 135 Nov 1, 2010 cassandra-0.7.0-beta3 2010-11-01 cassandra-0.7 0.7.0-beta3 0.7 Now let’s plot those releases and see what we get: ggplot(aes(x = date2, y = short_series), data = bySeries %>% filter(!grepl("beta|rc", short_version))) + geom_text(aes(label=short_version),hjust=0.5, vjust=0.5, size = 4, angle = 90) + theme_bw() An interesting thing we can see from this visualisation is what overlap the various series of versions have. Most of the time there are only two series of versions overlapping but the 1.2, 2.0 and 2.1 series all overlap which is unusual. In this chart we excluded all beta and RC versions. Let’s bring those back in and just show the last 3 versions: ggplot(aes(x = date2, y = short_series), data = bySeries %>% filter(grepl("2\\.[012]\\.|1\\.2\\.", short_version))) + geom_text(aes(label=short_version),hjust=0.5, vjust=0.5, size = 4, angle = 90) + theme_bw() From this chart it’s clearer that the 2.0 and 2.1 series have recent releases so there will probably be three overlapping versions when the 2.2 series is released as well. The chart is still a bit cluttered although less than before. I’m not sure of a better way of visualising this type of data so if you have any ideas do let me know!

The growth of Azure has been outstanding--more than 90,000 new subscriptions every month. And the innovation is exponential with over 500 new features and services being added to the platform in the last 12 months. We're very excited to be part of this growth. As we announced yesterday, you can now access Stackato through Azure. We think it's a great way for Azure customers to get access to a Cloud Foundry and Docker based PaaS. With Azure, Microsoft provides an easy path to the cloud for their customers. All applications can be run on one cloud. Microsoft wants to dominate the cloud the same as it has with on-premise software and rarely does a day go by without reading an article about Azure. Whether it's their recent announcement to help encourage start-up's use of Azure by providing $120,000 worth of credits per year or their commitment to open source. Azure gives its customers a growing collection of integrated services that make it easier to build and manage enterprise, mobile, web and Internet of Things (IoT) apps faster. Enterprises face real complexities when building their cloud solution. Having a solid infrastructure is really just the first step in the process--companies also need the right platform to support the deployment and management of their cloud-native applications. The platform should give their developers the freedom to use the language best suited to build the application. In addition, enterprises are on more than one cloud. They need to have the versatility to scale out or move their applications to whatever cloud is appropriate in order to meet end user demand without any downtime. With Stackato, we help remove these complexities. We provide enterprises with a polyglot PaaS that supports the development of applications in virtually any language. We like to refer to Stackato as being "infrastructure-agnostic" and allow companies to deploy their applications to any cloud--private, public or hybrid--without the need to run new scripts or re-package the application in order for it to work in the new environment. The combination of Stackato on Azure gives enterprises the technology they need to streamline application delivery, drive innovation and meet the demands of their customers.

What is a Leap Second? As some know - there will be a new leap second at the end of this month. Leap seconds are inserted into the standard UTC time scale by help of label "60" either at the end of June or December (exceptionally also in March and September) in irregular intervals in order to compensate for the slightly increasing difference between an astronomical day and the pulse of modern atomic clocks we now use for time-keeping. Initial Situation When I asked myself in year 2012 (where the last leap second happened) how to watch it using Java-based software the answer was simply: Not possible. At least not possible with any kind of standard tool. As with most software today, Java is totally ignorant towards leap seconds and pretends that they don't exist. There is no way to let Java print timestamps like "2015-06-30T23:59:60Z". Note that Java-8 with its new time library (JSR-310, java.time-package) does not offer this feature, too. For standard business purposes this approach is fine. From a scientific point of view however, this is not satisfying. Of course, these rare leap seconds still exist. When it comes to clock synchronization even leap-second-ignorant software has to pay some attribution if monotonicity is important. Any clock will sooner or later be synchronized such that leap seconds will be taken into account, often by setting the clock one second back. NTP time servers will usually repeat the timestamp "2015-06-30T23:59:59Z" and send in advance a so called leap indicator flag. This flag does not directly indicate the current timestamp as leap second but is only intended to be an announcement flag. So even NTP tries to hide leap seconds in some way. And some NTP servers like those from Google apply internal smearing algorithms (by slightly prolonging the second over a day) in order to avoid reporting any leap second at all. Decision for a New Library So I decided to develop a new time library named Time4J to solve this issue. First I had to set up a new infrastructure around a built-in configurable leap second table, then a new class net.time4j.Moment capable of holding a leap second as internal state. A SNTP-based clock yielding Moment-timestamps was developed and successfully tested during the last leap second in 2012. Since leap seconds are also reported by the well-known IANA-TZDB (standard timezone repository) I decided to develop a mechanism to apply the whole TZDB and its leap seconds on the class net.time4j.Moment. Finally I had even set up a new format and parse engine from the scratch to enable formatting and parsing of leap seconds in any timezone. A lot of work but the existing Java-software was not useable at all, not even as starting point. Recently I also developed a monotonic clock which enables to watch a leap second offline. Keep in mind that any clock - even NTP - are no reliable sources to watch a leap second in live connection. Fortunately Java offers at least System.nanoTime() which accesses the monotonic clock of operating system (if available). So I used this as a base of the new monotonic clock of Time4J which can connect to a NTP time server before a leap second will happen. How Does Time4J Help? import net.time4j.Moment; import net.time4j.Month; import net.time4j.PlainDate; import net.time4j.SI; import net.time4j.SystemClock; import net.time4j.base.TimeSource; import net.time4j.clock.FixedClock; import net.time4j.format.expert.ChronoFormatter; import net.time4j.format.expert.PatternType; import net.time4j.tz.olson.AMERICA; import java.util.Locale; import java.util.concurrent.ScheduledFuture; import java.util.concurrent.ScheduledThreadPoolExecutor; import java.util.concurrent.TimeUnit; public class DZone { static TimeSource clock; static ScheduledThreadPoolExecutor executor; static ScheduledFuture future; public static void main(String[] args) throws Exception { // when is the next leap second? Moment ls = SystemClock.currentMoment().with(Moment.nextLeapSecond()); if (ls == null) { // ooops, we are now too late and after last known leap second, // so let us set it directly ls = PlainDate.of(2015, Month.JUNE, 30).atTime(23, 59, 59) .atUTC().plus(1, SI.SECONDS); } // move 5 seconds earlier ls = ls.minus(5, SI.SECONDS); // let's start our monotonic clock at determined fixed time clock = SystemClock.MONOTONIC.synchronizedWith(FixedClock.of(ls)); // finally we observe our clock every second for 10 times executor = new ScheduledThreadPoolExecutor(15); future = executor.scheduleAtFixedRate( new ClockTask(), 0, 1, TimeUnit.SECONDS); } static class ClockTask implements Runnable { private int attempt = 1; public void run() { Moment moment = clock.currentTime(); String time = ChronoFormatter.ofMomentPattern( "MMM/dd/uuuu hh:mm:ss a XXX", PatternType.CLDR, Locale.ENGLISH, AMERICA.NEW_YORK ).format(moment); String flag = moment.isLeapSecond() ? " [leap second]" : ""; System.out.println(time + flag); attempt++; if (attempt > 10) { future.cancel(false); } } } } /* output: Jun/30/2015 07:59:55 PM -04:00 Jun/30/2015 07:59:56 PM -04:00 Jun/30/2015 07:59:57 PM -04:00 Jun/30/2015 07:59:58 PM -04:00 Jun/30/2015 07:59:59 PM -04:00 Jun/30/2015 07:59:60 PM -04:00 [leap second] Jun/30/2015 08:00:00 PM -04:00 Jun/30/2015 08:00:01 PM -04:00 Jun/30/2015 08:00:02 PM -04:00 Jun/30/2015 08:00:03 PM -04:00 */ For a real-life scenario you can just replace the clock this way (SNTP-example): SntpConnector sntp = new SntpConnector("ptbtime1.ptb.de"); sntp.connect(); clock = sntp; Conclusion Time4J does not try to hide the reality but gives you also the freedom to decide if you want to handle leap seconds or not. You can even switch off this feature completely by setting an appropriate system property. As side effect, a library has been developed which does not need to be afraid of any comparison with other existing libraries like JSR-310 (java.time-package in Java-8) or Joda-Time.

Github stores its pull-request and code review data in MySql. I’d much prefer a git reperentation for both (JSON, commits, audit trail, etc). Kinda the way Github Wiki pages are stored. That’s an aside though, this article is about storing code-review comments long term. The problem I’m trying to solve is one of deletion of users thich causes their pull requenst commentary to also get deleted. Sure the commits make it back to the origin/master (in the pull request is processed), but many things are left assoctaed with the fork. If the user gets deleted such info is gone forever :( I want a permanent copy, so the interim answer is to scrape the data I fear losing, while it still exists. Hence a scrape-pull-requests.sh bash script (for Mac and maybe Linux). Github’s portal is written in Ruby on Rails. It is extremely fast which helps scraping generally. There’s not a lot of JavaScript and that means that Wget is a viable extraction tool. Anyway the script runs quickly, and leaves a decent HTML interface for easy access later. I’ve tested it, but won’t leave up a scraped set of pull-requests as our GH overlords might object on copyright grounds. They can’t object for your own GithubEnterprise instance of course. Github could change the structure of their HTML, and the script might stop work so well.If that happens I’m happy to accept back pull-requests via the usual mechanism :)

In my articles from the series “Design Patterns in Automation Testing“, I show you how to integrate the most useful code design patterns in the automation testing.