This is the third in a series of interviews for you to meet some of the committee members for the JavaOne 2015 Java EE track. The committee plays the most important part in determining the content for JavaOne. These good folks really deserve recognition as most of them devote many hours of their time helping move JavaOne forward, often as volunteers. If JavaOne matters to you, these are folks you should know about. This interview is with Johan Vos. If you are having trouble seeing the embedded video below it is available here. Johan is a Java Champion, author, speaker, blogger, member of the BeJUG steering group, member of the Devoxx steering group and a JCP member. He is a fan of Java EE, GlassFish and JavaFX. He founded LodgON, a company offering Java based solutions for social networking software. In the interview he shares his experience and expectations for the Java EE track this year. On this note, I would like to make sure you know that the JavaOne content catalog is now already live with a few preliminary fairly obvious selections we were able to make. None of the sessions accepted at this stage are from Oracle speakers on our track. The folks that we selected early for acceptance include David Blevins, Jonathan Gallimore, Mohammed Taman, Rafael Benevides and Antoine Sabot-Durand. They will be talking about Java EE Connectors (JCA), Java EE 7 real world adoption, CDI and DeltaSpike. I would encourage you to check out all the early selections in the catalog. We are working to finalize the full catalog shortly. I hope to see you at JavaOne. Do stay tuned for more interviews with committee members and some key speakers on our track.

In mathematics an automorphic number (sometimes referred to as a circular number) is a number whose square "ends" in the same digits as the number itself. For example, 52 = 25, 62 = 36, 762 = 5776, and 8906252 = 793212890625, so 5, 6, 76 and 890625 are all automorphic numbers. And the logic behind : int n=56; int d=1; int i; for(i=n;i>0;i=i/10) { d=d*10; } if((n*n)%d==n) { System.out.println(n+"\t"+"is Automorphic Number"); } else { System.out.println(n+"\t"+"is not Automorphic Number"); } } You can check full article from Geek On Java - Hub for Java and Android

Originally authored by Elizabeth Engel You are thinking about including some nice charts and graphics into your current project? Maybe you heard about D3.js, as some people claim it the universal JavaScript visualization framework. Maybe you also heard about a steep learning curve. Let’s see if this is really true! First of all, what is D3.js? D3.js is an open source JavaScript framework written by Mike Bostock helping you to manipulate documents based on data. Okay, let’s first have a look at the syntax Therefore, lets look at the following hello world example. It will append an element saying ‘Hello World!’ to the content element. As you can see the syntax is very similar to frameworks like JQuery and obviously, it saves you a lot of code lines as it offers a nice fluent API. But let’s see how we can bind data to it: d3.select('#content') .selectAll('h1') .data(['Sarah', 'Robert', 'Maria', 'Marc']) .enter() .append('h1') .text(function(name) {return 'Hello ' + name + '!'}); What happens? The data function gets our names array as parameter and for each name we append an element with a personalized greeting message. For a second, we ignore the selectAll(‘h1′) and enter() method call, as we will explore them later. Looking into the browser we can see the following: Hello Sarah! Hello Robert! Hello Maria! Hello Marc! Not bad for a start! Inspecting the element in the browser, we see the following generated markup: [...] Hello Sarah! Hello Robert! Hello Maria! Hello Marc! [...] This already shows one enourmous advantage of D3.js: You acctually see the generated code and can spot errors easily. Now, let’s have a closer look at the data-document connection As mentioned in the beginning, D3.js helps you to manipulate documents based on data. Therefore, we only take care about handing the right data over to D3.js, so the framework can do the magic for us. To understand how D3.js handles data, we’ll first have a look at how data might change over time. Let’s take the document from our last example. Every name is one data entry. Easy. Now let’s assume new data comes in: As new data is coming in, the document needs to be updated. The entries of Robert and Maria need to be removed, Sarah and Marc can stay unchanged and Mike, Sam and Nora need a new entry each. Fortunately, using D3.js we don’t have to care about finding out which nodes need to be added and removed. D3.js will take care about it. It will also reuse old nodes to improve performance. This is one key benefit of D3.js. So how can we tell D3.js what to do when? To let D3.js update our data, we initially need a data join, so D3.js knows our data. Therefore, we select all existing nodes and connect them with our data. We can also hand over a function, so D3.js knows how to identify data nodes. As we initally don’t have nodes, the selectAll function will return an empty set. var textElements = svg.selectAll('h1').data(data, function(d) { return d; }); After the first iteration, the selectAll will hand over the existing nodes, in our case Sarah, Robert, Marc and Maria. So we can now update these existing nodes. For example, we can change their CSS class to grey: textElements.attr({'class': 'grey'}); Additionally, we can tell D3.js what to do with entering nodes, in our case Mike, Sam and Nora. For example, we can add an element for each of them and set the CSS class to green for each of them: textElements.enter().append('h1').attr({'class': 'green'}); As D3.js now updated the old nodes and added the new ones, we can define what will happen to both of them. In our cases this will affect the nodes of Mike, Sarah, Sam, Mark and Nora. For example, we can rotate them: textElements.attr({'transform', rotate(30 20,40)}); Furthermore, we can specify what D3.js will do to nodes like Robert and Maria, that are not contained in the data set any more. Let’s change their CSS class to red: textElements.exit().attr({'class': 'red'}); You can find the full example code to illustrate the data-document connection of D3.js as JSFiddle here: https://jsfiddle.net/q5sgh4rs/1/ But how to visualize data with D3.js? Now that we know about the basics of D3.js, let’s go to the most interesting part of D3.js: drawing graphics. To do so, we use SVG, which stands for scalable vector graphics. Maybe you already know it from other contexts. In a nutshell, it’s a XML-based vector image language supporting animation and interaction. Fortunately, we can just add SVG tags in our HTML and all common browsers will display it directly. This also facilitates debugging, as we can inspect generated elements in the browser. In the following, we see some basic SVG elements and their attributes: To get a better understanding of how SVG looks like, we’ll have a look at it as a basic example of SVG code, generating a rectangle, a line and a circle. To generate the same code using D3.js, we need to add an SVG to our content and then append the tree elements with their attributes like this: var svg = d3.select('#content').append('svg'); svg.append('rect').attr({x: 10, y: 15, width: 60, height: 20}); svg.append('line').attr({x1: 85, y1: 35, x2: 105, y2: 15}); svg.append('circle').attr({cx: 130, cy: 25, r: 6}); Of course, for static SVG code, we wouldn’t do this, but as we already saw, D3.js can fill attributes with our data. So we are now able to create charts! Let’s see how this works: This will draw our first bar chart for us! Have a look at it: https://jsfiddle.net/tLhomz11/2/ How to turn this basic bar chart into an amazing one? Now that we started drawing charts, we can make use of all the nice features D3.js offers. First of all, we will adjust the width of each bar to fill the available space by using a linear scale, so we don’t have to scale our values by hand. Therefore, we specify the range we want to get values in and the domain we have. In our case, the data is in between 0 and 200 and we would like to scale it to a range of 0 to 400, like this: var xScale = d3.scale.linear().range([0, 400]).domain([0,200]); If we now specify x values, we just use this function and get an eqivalent value in the right range. If we don’t know our maximum value for the domain, we can use the d3.max() function to calculate it based on the data set we want to display. To add an axis to our bar chart, we can use the following function and call it on our SVG. To get it in the right position, we need to transform it below the chart. [svg from above].call(d3.svg.axis().scale(xScale).orient("bottom")); Now, we can also add interaction and react to user input. For example, we can give an alert, if someone clicks one our chart: [svg from above].on("click", function () { alert("Houston, we get attention here!"); }) Adding a text node for each line, we get the following chart rendered in the browser: If you would like to play around with it, here is the code: https://jsfiddle.net/Loco5ddt/ If you would like to see even more D3.js code, using the same data to display a pie chart and adding an update button, look at the following one: https://jsfiddle.net/4eqzyquL/ Data import Finally, we can import our data in CSV, TSV or JSON format. To import a JSON file, for example, use the following code. Of course, you can also fetch your JSON via a server call instead of importing a static file. d3.json("data.json", function(data) { [access your data using the data variable] } What else does D3,js offer? Just to name a few, D3.js helps you with layouts, geometry, scales, ranges, data transformation, array and math functions, colors, time formating and scales, geography, as well as drag & drop. There are a lot of examples online: https://github.com/mbostock/d3/wiki/Gallery TL;DR + based on web standards + totally flexible + easy to debug + many, many examples online + Libaries build on D3.js (NVD3.js, C3.js or IVML) – a lot of code compared to other libraries – for standard charts too heavy Learning more As this blog post is based on a presentation held at the MunichJS Meetup, you can find the original slides here: http://slides.com/elisabethengel/d3js#/ The recording is available on youTube: https://www.youtube.com/watch?v=EYmJEsReewo For further information, have a look at: https://github.com/mbostock/d3/wiki https://github.com/mbostock/d3/wiki/API-Reference http://bost.ocks.org/mike/ Getting Started with D3 by Mike Dewar Interactive Data Visualization for the Web by Scott Murray (online for free) https://github.com/alignedleft/d3-book If you have any questions or suggestions, feel free to comment or write me: [email protected]

the vast majority of software development companies have to place a great emphasis on the process of continuous integration and rapid delivery of new versions of their product. obviously, when supplying enterprise-level projects, such processes need to be automated as much as possible. and this is when the cloud devops tools come in handy. thus, in today’s article we’d like to pay a special attention to the devops tools that automate the continuous integration and delivery within the jelastic paas that can be installed on any bare metal or cloud infrastructure as virtual private cloud or hybrid cloud. this is a pretty complex example of enterprise application life cycle with continuous integration and seamless migration throughout devops pipeline from development to several productions (you can use simplified process if you have less complex project ). the instruction below will be useful for jelastic cluster administrators such as systems integrators, hosting service providers, enterprises, and isv customers, who can easily implement it at their jelastic cloud installations. nevertheless, this guide contains plenty of features and continuous integration tips described, which can be interesting for different developers. so, let’s get started with the first part of the instruction! setting up dedicated user groups first of all, you need to allocate separate hardware sets for all your project teams (one per each development phase, i.e. development > testing > production ) and adjust the access permissions to make them completely isolated and not influenced by others. the multi-regions for a hybrid cloud option, that became available within the recently released jelastic 3.3 version , is optimally suited for this task. to start with, create three hardware node groups (within one region) and name them after the corresponding stages for more convenience (e.g. dev , test , production ). the next step is to prepare three user groups and attach them to the corresponding hardware – in our case the dev group has access to the dev hardware node group only, qa – to the test one, and ops should work specifically with the production set. in such a way, users from the appropriate groups can use the specified sets of hardware only, but at the same time – they have a possibility to transfer their environments throughout the whole platform, between different teams’ accounts. jenkins continuous integration server configuration now we need the integration tool, that will control and perform all of the required operations automatically, i.e. build the cloud devops pipeline. our choice fell on jenkins as one of the most popular solutions used for this goal – it can be easily installed from our marketplace either at the corresponding site page or directly via the dashboard . as a result, you’ll get the pure jenkins installed, which should be properly adjusted before you start organizing your application life cycle. thus, select the open in browser button and proceed with the following configurations steps: while at the home page, click on the manage jenkins option at the left-hand menu and select the manage plugins link within the appeared list. after you’ve been redirected to the plugin manager, switch to the available tab, find the following plugins using the search filter field above and tick them for installation: git plugin – is required for building our project’s source (stored at the github repository) envfile plugin – is used for storing system environment variables (its necessity is driven by security restrictions, implemented at jelastic, which forbid the direct exporting of environment variables from the tomcat server) click install without restart when ready. during the installation process, tick the restart jenkins when installation is complete and no jobs are running option to automatically restart jenkins for enabling the chosen plugins. then, you also need to install maven, which will be used for building the project. for that, navigate to the manage jenkins > configure system menu, scroll down to the maven section and click add maven. within the expanded section, type the desired name for your maven installation (e.g. maven ) and save the changes using the same-named button at the bottom of the page. in such a way, this tool will be also automatically installed when required (i.e. during the first app build). now your jenkins server is well-staffed for the further work. add deployment process scripts to the jenkins container the next step is to upload the scripts that you are going to use for automating different organizational actions, required to be applied to your application at the intermediate development life cycle phases (like deploying, placing it to the appropriate hardware according to the stage, running auto-test, etc). the easiest way to do this is to access your jenkins container via the jelastic ssh-gateway. in the case you haven’t performed similar operations before, you need to: generate an ssh keypair add your public ssh key to the dashboard access your account via ssh protocol once inside, create a new folder for your project (we’ll use demo ) and move in there: mkdir /opt/tomcat/demo cd /opt/tomcat/demo this location can be used for storing your scripts, variables, logs etc. here, you can upload the required scripts using the command of the following type: curl -fssl {link_to_script} -o {file_name} we also provide the set of script examples, which can be used as templates for your own ones: install.sh – gets a user session and creates a new environment via the jelastic api according to the specified manifest file. it also defines, that the name of this environment will be equal to its creation date and time (as a unique name is required for every script execution, but you won’t be able to set it manually as this operation would be run automatically). however, you can set your own dynamic name pattern to be used here transfer.sh – changes the environment ownership based on the jelastic environment transferring feature migrate.sh – physically moves an environment to another hardware set (hardnode group) note: that before the appliance, each of the script templates, presented above, have to be additionally adjusted to make them work properly within a particular jelastic installation. thus, the list of parameters below should be obligatory substituted according to your platform’s settings: /path/to/scripts/ – the full path to your scripts folder (created in the previous step) {cloud_domain} – your jelastic platform domain name {jca_dashboard_appid} – your dashboard id, that could be seen within the platform.dashboard_appid parameter at the jca > about section {jca_appstore_appid} – appstore id, listed within the same section (at the platform.appstore_appid parameter) {url_to_manifest} – link to the manifest file created according to our documentation (you may also use this one as an example – it sets up two tomcat application servers with the nginx load-balancer in front of them) note: above you can see one more runtest.sh script uploaded – it simulates the testing activities for demonstration purposes, thus we don’t provide its code in this tutorial. if required, create your own one according the specifics of your application and upload it alongside the rest of the scripts. in addition, you need to create a separate file for storing the variable with environment name (as it needs to be dynamically changed each time a new environment is created): echo env_name= > /opt/tomcat/demo/variables these are the main steps of preparation to achieve automatic continuous integration and delivery of your web application with a help of jenkins within jelastic cloud platform. in the second part of these blog series, we’ll configure the set of jobs at the jenkins server, which represents the core of our automation. each of them will be devoted for a particular operation, required to be run at the corresponding application life cycle phase: create environment > build and deploy > dev tests > migrate to qa > qa tests > migrate to production stay tuned to see the next steps. if you still don’t have jelastic installation, contact us to get access to our free demo for cloud platform evaluation or just start with trial registration at one of our hosting partners .

DBmaestro, the pioneer and leading provider of DevOps for database solutions, has announced that TeamWork’s extension for Oracle SQL Developer 4.1 has been released, just weeks after Oracle released its latest version. DBmaestro TeamWork is now the only tool with an external extension that supports version 4.x of Oracle’s database development tool. TeamWork, DBmaestro’s flagship product, enables agile development and continuous integration & delivery for the database. TeamWork supports streamlining of development process management and enforcing change policy practices. Many leading enterprises use DBmaestro to facilitate DevOps for their database by executing deployment automation, enhancing and reinforcing security, and mitigating risk. DBmaestro’s extension for SQL Developer 4.x helps Oracle developers and DBAs streamline database development, collaborate across database teams, and allows for agile database development in an efficient and reliable way. Upon the release of SQL Developer 4.0, Oracle required that all extensions be updated to be compatible with the new version’s API. This is a result of the drastic changes made to JDeveloper, on which SQL Developer is built. “The SQL Developer 4.1 extension represents an important achievement for DBmaestro,” said Yaniv Yehuda, co-founder and CTO of DBmaestro. “Oracle SQL Developer has over 4 million active users and is the de facto standard database IDE tool out there. Oracle has drastically changed their API on the 4.0 version, which presented a challenge to those seeking to update third-party extensions. This integration is a statement of our commitment to our customers, and we will continue to lead the way to achieve DevOps for the databases.”

Software Driven Business is a consensus goal. But real challenges exist: the time, cost and complexity of building such apps is substantial. Business Agility – and strategic business advantage – is lost. We need another revolution – Instant Enterprise REST – that provides Business Agility using business-level specifications rather than low-level code, and delivers Enterprise-class scalability, integration, enforcement and extensibility. It’s now a reality with Instant Enterprise REST. Software Driven Business: Consensus Vision Businesses have seen the value in providing mobile and tablet apps that bring the business into the hands of customers and employees. They provide information at their finger tips – wherever they are. Industry Leaders like CA have pioneered the vision of a Software Driven Business. They argue persuasively that strategic business advantage lies in Time to Market and Time to Decision: “reveal the need for speed in the application economy. As companies transform into software-driven enterprises, bringing high-quality applications to market faster becomes one of the most critical differentiators.” The Business Agility Gap While there is consensus around this vision, there is a substantial gap in realizing the Software Driven Business. It centers around Agility – time to market. As CA argues, this drives strategic business advantage. This problem manifests both to Business Users and IT, although differently. You might have been party to a discussion like this: Business Users are frustrated about how long it takes to create systems, and revise them. They see problems that look nearly as simple as a spreadsheet take weeks… to months. How can it months for IT to build a system that takes days on a spreadsheet? IT is no less frustrated. They understand the deep technology it takes to build Enterprise-class systems: We’re working 90 hours a week. And falling behind. Gap Analysis For apps about critical corporate data, there’s general consensus that the time and cost for such systems are about evenly split between backends and front ends. And there’s nearly universal consensus that, independent of the UI technology, that RESTful APIs deliver the backend data. But the backend is far more than basic data access. A “SQL Pass-through” – simply restifying SQL data – does not meet Enterprise-class requirements to scale, integrate and enforce: Scale – APIs require Pagination to address large result sets, Nested Documents to reduce latency, Optimistic Locking to ensure concurrency. These are not provided in a simple SQL Pass-through – you must program them, by hand. Integrate – a wizard can produce an API from schema objects, but it cannot address multiple databases, or integrate non-SQL data sources such as ERP, other RESTful services, or NoSQL. Enforce – an API needs to enforce our security (down to the row level), and the integrity of the data. These are significant tasks, which are sadly often placed in client buttons where they cannot be shared. Providing these Enterprise class services takes significant time, expertise and expense. Business Agility is reduced. IT is essentially being forced to cover inadequate technology infrastructure. The Business Users are right: if the Business Specification is clear, then that ought to be enough: A clear business specification should be sufficient. Everything else is just friction. The vision of the Software Driven Business requires Business Driven Software that pre-supplies the infrastructure. We are not seeking 10 or 15%. We are looking for orders of magnitude. Our vision must be: We should be able to create RESTful APIs (mainly) from business specifications, not low level code. It should be no more difficult to create a system than it is toimagine it. Business-Driven Software: Instant Enterprise REST Business Driven Software is more than just a clever play on words. It’s a real implementation that delivers this vision, and we call it Instant Enterprise REST. It consists of 3 core technologies: Enterprise Pattern Automation – creates APIs that with Enterprise-class scalability built-in (pagination, nested documents, optimistic locking, etc) Declarative – specify your API, integration and enforcement policies with spreadsheet-like rules in a simple point-and-click UI Extensibility – enables the RESTful APIs to invoke your existing logic, inside or outside the JVM, via standard server-side JavaScript. The combination of these 3 technologies enables you to create RESTful APIs for database backends – half your system – 10 times faster. Let’s briefly examine them below. Technology 1: Enterprise Pattern Automation There are well known patterns in the data domain, describing data structure and access via SQL. There are also well-known patterns for managing SQL data in the context of RESTful services. Well known patterns can be automated. Let’s imagine a service (say, a server accessed via a browser) that automates these patterns, as described below, just by connecting the service to a database: Schema Discovery – tables, views, stored procedures: The system creates a complete (default) API for each schema object. Note this includes Stored Procedures, which often represent a significant investment. Enterprise Pattern Automation: the resultant API provides well-known services for Filter, Sort, Pagination, Optimistic Locking, handling Generated Keys and so forth. So, the service has provided a default Enterprise-class API, instantly. So, literally seconds into your project, you can test your running API: Not enough, not done, but a great start. Technology 2: Declarative Declarative is the key (“what, not how”). It has had striking impacts on domains where there are well-understood underlying patterns. Max Tardiveau has put it well: Whatever can be declarative, will be declarative. For example, spreadsheets are declarative – and they gave birth to the PC industry. And SQL is declarative – itself an industry. Two game-changers. So, the challenge is to apply the spirit of declarative to REST integration and enforcement. The stakes are high – success can deliver breathtaking agility. Declarative Integration: Multi-Database Custom API, Point and Click Enterprise Pattern Automation provides a good start, but the API is not rich. It is a flat, single-table API, really just “restified” SQL. What we really need is Nested Documents – returning multiple types (e.g., an Order, a list of Items, and a list of contact names) in a single call can reduce latency (vs. a separate call for each type). REST is perfect for this. Multi-database APIs – a RESTful server provides the opportunity to integrate multiple databases in single call, shielding clients from underlying complexity. Nested Documents are easy: define them by simply selecting tables (via a User Interface or Command Line). Foreign Keys are used to default the joins. Add the ability to choose / alias columns, and we’re on the way to a pretty good API. But what about databases that have no Foreign Keys? Or multi-database APIs? Leveraging the schema does not mean we are limited to it. All we need to do is: Provide a means to define “Virtual” Foreign Keys for the service (i.e., stored outside the schema) Extend this to Foreign Keys between databases We now have a rich, multi-database API. Defined declaratively as shown below, no code required, running in minutes, ready for client development: Declarative Enforcement: Integrity Logic, with spreadsheet-like rules So now consider enforcement, specifically database integrity. A very significant portion of any project is the multi-table validations and computations that define how the data is processed. “Your code goes here” means, well, a lot of code. We need a more powerful, more declarative, paradigm. In a spreadsheet, you assign expressions to cells. Whenever the referenced data is changed, the cell is updated. Since the cells references can chain, a series of simple expressions can solve remarkably complex problems. What if we did the same for database data? We could assign derivation expressions to columns, and validation expressions to tables. Then, the API could “watch” for requests that change the referenced column, and recompute (efficiently) the calculated column. Just as in a spreadsheet, support for chaining and proper ordering is required and implicit. To address multi-table logic, such expressions would need to address references to related tables. It’s only at this point that the logic becomes seriously powerful. Let’s take an example. To check credit in a Customer / Purchaseorder / Lineitem application, we could define spreadsheet-like expressions such as: There is actually a sub-branch of declarative that addresses this: Reactive Programming. Here it’s declarative,since you don’t need to code a Observer handler. The result is that the logic above can be fully executable. No need to code Change Detection / Change Dependency – it’s invoked and enforced automatically by the API in reaction to RESTful updates. SQL handling is also implicit, including underlying optimizations (caching, pruning etc). The impact is massive – the 5 expressions above express the same logic as hundreds of lines of code. That’s a massive 40X more concise. Game changer. And quality goes up, since the rules are applied automatically. Declarative Enforcement: Security, filter expressions for role/table We can provide an analogous approach to security: define filter expressions for roles (like SalesRep), so that when a table is accessed by the role, the API adds the filter. That way, a user with that role sees only the rows for which they are authorized. Technology 3: Standards-based Extensibility Declarative is great, but you’re probably thinking “ok, but you can’t solve every problem declaratively”. And you’re dead right. Business Value requires that we integrate a declarative approach with a procedural one that is familiar, standards-based, and enables us to integrate existing software. Automatic JavaScript Object Model The first phase of many projects is to build an ORM for natural programmatic access to data: JPA, Hibernate, Entity Framework. It’s not a small project, and cumbersome to maintain as changes occur. In fact, the Object Model can be created directly from the schema. So, you’d have an object type for Purchaseorder, for Lineitem, and so forth. The model provides access to attributes and related data, and persistence services. You could then use it as shown below. JavaScript seems like the best language choice: reasonable across technology bases (everybody uses JavaScript), and its dynamic nature eliminates code generation hassles. JavaScript Events In addition to accessors and persistence, the JavaScript objects are Logic Aware. That is, the save operation above executes any rules associated with OrderAudit (e.g., updated-by), and JavaScript Events. Here is a sample event for the PurchaseOrder object, where you access the JavaScript Object Model via the system-supplied row variable: Extensible Logic Auditing is a common pattern. It should be possible to solve this once in a genericmanner, then re-use it (e.g, to audit employees, orders and so forth). So, Instant Enterprise REST should enable you to provide Extensible Logic – load your own JavaScript code, and invoke it. So, the code above could become: MyLibrary.auditFromTo(orderRow,"OrderAudit"); where auditFromTo creates an instance of OrderAudit, sets the foreign key, sets like-named attributes, and saves it. Pluggable Authentication Most organizations have existing data stores that identify users and their roles, such as Active Directory, LDAP, OAuth, etc. Security should integrate with such systems as a function of enforcing row/column access. Standard deployment Finally, the system should deploy in a familiar manner: available on the cloud, or an on-premise virtual appliance or war file. Standards also enable integration with related critical infrastructure, such as API Management, ERP Systems, etc. See a project in 3 minutes To see how it all fits together, you can view this video to see a full project built: from concept, through initial implementation, and an iteration cycle. Actual project time was about half an hour. Instant Enterprise REST: Business Agility Instant Enterprise REST enables us to close the Agility Gap in realizing the Software Driven Business vision. We can now create important portions of our software in largely business terms, rather than technical terms. This offers major advantages: Time to Market: spreadsheet-like rules are 40X more concise. Instant REST eliminates all the SQL / REST / JSON boilerplate. Simplicity: team members can learn the basics of Espresso in days, and be as productive as rocket scientists using alternative technologies Leverage Expertise and Software: Espresso is built on standards like REST, JavaScript, and Event Oriented Programming. You can call out to existing software, and extend the rule types by identifying your own patterns and loading their implementations into Espresso. Quality: at the defect level, automatic invocation and ordering eliminate large classes of bugs. At the architectural level, centralized enforcement factors logic out of the client buttons where it can be shared, audited for compliances, etc

[This article was written by Alex Harford.] Docker APIs are a convenient way for your systems to talk to Docker infrastructure. But sometimes there are challenges associated with them. I've outlined in this blog the steps you need to take and the items you need to look out for when working with Docker APIs. Initial Docker Setup Ensure you have the latest Docker client installed. It should be v1.6 or newer. [alexh:~/work] docker pull ubuntu latest: Pulling from ubuntu 428b411c28f0: Pull complete 435050075b3f: Pull complete 9fd3c8c9af32: Pull complete 6d4946999d4f: Already exists ubuntu:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security. Digest: sha256:45e42b43f2ff4850dcf52960ee89c21cda79ec657302d36faaaa07d880215dd9 Status: Downloaded newer image for ubuntu:latest [alexh:~/work] docker run -ti ubuntu /bin/bash root@1092e8ca2ead:/# ps PID TTY TIME CMD 1 ? 00:00:00 bash 14 ? 00:00:00 ps root@1092e8ca2ead:/# exit exit Daemons, Registries, Hubs The Docker registry is used to host docker images for download. In the most simple case, it can be a process serving static images. This would be a read-only registry supporting GET operations only. If you need something more complex, you need to use a Docker registry web service. You can [a target="_blank" href="http://www.activestate.com/blog/2014/01/deploying-your-own-private-docker-registry"]run your own private Docker registry or use the public official Docker Hub. The Docker Hub contains a Docker registry, but also includes other features, like user authentication. In our examples, we will run an unauthenticated Docker registry. Setup If you are using standard Docker images, most people will pull from the Docker Hub, which is a publically accessible Docker registry. However, a more complicated service may be talking to private Docker registries running different versions of the API. Let’s assemble a test environment with both versions of the docker registry API so we can see the different ways you can access it. First, pull down two versions of the docker registry from the Docker Hub: docker pull registry:0.9.1 0.9.1: Pulling from registry e9e06b06e14c: Pull complete a82efea989f9: Pull complete 37bea4ee0c81: Pull complete 07f8e8c5e660: Pull complete 1f4ab7282e19: Pull complete 3c27027cdae8: Pull complete 7e0e5314436e: Pull complete 2696504d3685: Pull complete 012772dbb1c6: Pull complete e24d9fce1d00: Pull complete fd2726a79da8: Pull complete bffc32d7113a: Pull complete 0cd49aa0e23c: Pull complete 4e698fa80441: Already exists registry:0.9.1: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security. Digest: sha256:98937757728eecbd72c9276bf711260aa29896f15217ce05be0562287e73232d Status: Downloaded newer image for registry:0.9.1 [alexh:~/work] docker pull registry:2.0.1 2.0.1: Pulling from registry 39bb80489af7: Pull complete df2a0347c9d0: Pull complete 7a3871ba15f8: Pull complete a2703ed272d7: Pull complete 68769176e114: Pull complete ab2ab59d7d1b: Pull complete 882ecee9f360: Pull complete 40de65f8e79f: Pull complete 0c4f9c7d798f: Pull complete ca29675fe853: Pull complete 89d10e9463e5: Pull complete 1a5aa415e484: Pull complete 3ea7a9e93b04: Pull complete 769d811a57fd: Pull complete ae8a4a3af1aa: Pull complete 85cc9a791bb5: Pull complete 9cd2c8646022: Pull complete 048c32c549b9: Pull complete cbbbda28c189: Pull complete 2602c005e534: Pull complete 136beb445cfa: Pull complete 0c5e5ef1d7da: Already exists registry:2.0.1: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security. Digest: sha256:0cd177d687589aff586aa2c66c64d1c25657b8d09cff9e1492192f496e7786c3 Status: Downloaded newer image for registry:2.0.1 The next step is to start them. We will start the v1 registry on port 5000, and the v2 registry on port 6000. The v1 registry occasionally fails when starting due to a lock file race condition, so tell it to restart if necessary. [alexh:~/work] docker run -p 5000:5000 -d --restart=on-failure:3 registry:0.9.1 896c651b9bfa9780b14e3710d20428baab8497c30b9bc89946b192e1d1c145aa [alexh:~/work] docker run -p 6000:5000 -d registry:2.0.1 e09d4204921c732879ee9b7544cd40a25275e0d1f1702cacd954412cfd586ffb [alexh:~/work] docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES e09d4204921c registry:2.0.1 "registry cmd/regist 4 seconds ago Up 3 seconds 0.0.0.0:6000->5000/tcp silly_albattani 896c651b9bfa registry:0.9.1 "docker-registry" 35 seconds ago Up 34 seconds 0.0.0.0:5000->5000/tcp jovial_leakey Understanding Docker Namespaces Docker has a concept of namespaces for its repositories which can be confusing. [a target="_blank" href="https://docs.docker.com/docker-hub/official_repos/"]Official Repositories can be referred to without a username prefix: CentOS Ubuntu Internally these are prefixed by library/. This means that command like docker pull ubuntu:15.10 and docker pull library/ubuntu:15.10 are equivalent. If the name includes a '/' character (samalba/docker-registry), the left side refers to the username, and the right side refers to the image name in their public repository. It gets more complex when accessing private registries. The format becomes HOST:PORT/[USERNAME/]IMAGE. However, you should note that there is no authentication performed at this layer of our docker registry environment: anyone can push, pull, or delete from any 'user'. If the USERNAME is omitted, it is internally treated as being an 'official' image, and prefixed with library/. docker pull 127.0.0.1:5000/library/test-ubuntu Pulling repository 127.0.0.1:5000/library/test-ubuntu FATA[0004] Error: image library/test-ubuntu:latest not found [alexh:~/work] docker tag 0fe5a10d2cf8 127.0.0.1:5000/test-ubuntu [alexh:~/work] docker push 127.0.0.1:5000/test-ubuntu The push refers to a repository [127.0.0.1:5000/test-ubuntu] (len: 1) Sending image list Pushing repository 127.0.0.1:5000/test-ubuntu (1 tags) Image 5c1d0c04c3b8 already pushed, skipping Image 8c63e4ac9a5f already pushed, skipping Image 5fc05c0feaea already pushed, skipping Image 0fe5a10d2cf8 already pushed, skipping Pushing tag for rev [0fe5a10d2cf8] on {http://127.0.0.1:5000/v1/repositories/test-ubuntu/tags/latest} [alexh:~/work] docker pull 127.0.0.1:5000/library/test-ubuntu Pulling repository 127.0.0.1:5000/library/test-ubuntu 0fe5a10d2cf8: Download complete 5c1d0c04c3b8: Download complete 8c63e4ac9a5f: Download complete 5fc05c0feaea: Download complete Status: Image is up to date for 127.0.0.1:5000/library/test-ubuntu:latest In the v2 Docker registry, the [a target="_blank" href="https://docs.docker.com/registry/spec/api/#overview"]URI scheme has changed to allow the repository name to be broken up into multiple components. However, the Docker client does not yet support this flexibility. In the future, you should be able to extend the namespace of your registries, ie `redhat/centos/beta or redhat/fedora/stable. Populating the Registries We'll use Ubuntu 15.10 as our example image: docker pull ubuntu:15.10 15.10: Pulling from ubuntu 5c1d0c04c3b8: Pull complete 8c63e4ac9a5f: Pull complete 5fc05c0feaea: Pull complete 0fe5a10d2cf8: Already exists ubuntu:15.10: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security. Digest: sha256:d569b6ebfc62f35f9792392724bd4a74a4f5f5af10ccbc1880974ae2f0660898 Status: Downloaded newer image for ubuntu:15.10 It needs to be tagged with the new URL in order to push it to the private registries: [alexh:~/work] docker tag ubuntu:15.10 127.0.0.1:5000/ubuntu:15.10 [alexh:~/work] docker tag ubuntu:15.10 127.0.0.1:6000/ubuntu:15.10 [alexh:~/work] docker push 127.0.0.1:5000/ubuntu:15.10 The push refers to a repository [127.0.0.1:5000/ubuntu] (len: 1) Sending image list Pushing repository 127.0.0.1:5000/ubuntu (1 tags) 5c1d0c04c3b8: Image successfully pushed 8c63e4ac9a5f: Image successfully pushed 5fc05c0feaea: Image successfully pushed 0fe5a10d2cf8: Image successfully pushed Pushing tag for rev [0fe5a10d2cf8] on {http://127.0.0.1:5000/v1/repositories/ubuntu/tags/15.10} [alexh:~/work] docker push 127.0.0.1:6000/ubuntu:15.10 The push refers to a repository [127.0.0.1:6000/ubuntu] (len: 1) 0fe5a10d2cf8: Image already exists 5fc05c0feaea: Image successfully pushed 8c63e4ac9a5f: Image successfully pushed 5c1d0c04c3b8: Image successfully pushed Digest: sha256:1f93077ce8f2fa1da8aae87735f395eae93a1c21928d3e2d130717c9aeff177d Note that the output between the v1 registry (on port 5000) and v2 (port 6000) are slightly different, but the result is the same: the Ubuntu image is now available on each registry. Docker Registry APIs At this point, we're able to compare the different APIs. In April 2015, Docker [a target="_blank" href="http://docs.docker.com/v1.6/release-notes/"]released version 1.6 and this included v2 of the Registry. Your software should be aware of the different versions of the Docker Registry API to handle these differences. Let's look at what it takes to download the image layers through the various APIs in order to make an offline cache. First, we'll prepare our environment: [alexh:~/work] export image=ubuntu [alexh:~/work] export tag=15.10 v1 The v1 private registry can be examined at this point: [alexh:~/work] curl -s http://127.0.0.1:5000/v1/repositories/library/$image/tags/$tag | python -m json.tool "0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547" export v1_image_id=`curl -s http://127.0.0.1:5000/v1/repositories/library/$image/tags/$tag | sed 's/"//g'` [alexh:~/work] curl -s http://127.0.0.1:5000/v1/images/$v1_image_id/ancestry | python -m json.tool [ "0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547", "5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1", "8c63e4ac9a5f31e482d25a149b022209653b5948cb4f045c2ede9331a18e5824", "5c1d0c04c3b846fffd1d70886c956927a5c5f6a1c96f5e9f61c02f2ec1a45a73" ] [alexh:~/work] curl -sSL http://127.0.0.1:5000/v1/images/0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547/layer > 0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547.tar.gz [alexh:~/work] curl -sSL http://127.0.0.1:5000/v1/images/5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1/layer > 5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1.tar.gz [alexh:~/work] curl -sSL http://127.0.0.1:5000/v1/images/8c63e4ac9a5f31e482d25a149b022209653b5948cb4f045c2ede9331a18e5824/layer > 8c63e4ac9a5f31e482d25a149b022209653b5948cb4f045c2ede9331a18e5824.tar.gz [alexh:~/work] curl -sSL http://127.0.0.1:5000/v1/images/5c1d0c04c3b846fffd1d70886c956927a5c5f6a1c96f5e9f61c02f2ec1a45a73/layer > 5c1d0c04c3b846fffd1d70886c956927a5c5f6a1c96f5e9f61c02f2ec1a45a73.tar.gz v1 on Docker Hub The Docker Hub currently implements the v1 API, but requires an authentication token for certain operations. It also allows multiple endpoints to be returned by the server. We'll take the simple approach of always using the first endpoint: [alexh:~/work] export endpoint=`curl -sSL -o /dev/null -D- "https://index.docker.io/v1/repositories/$image/images" | awk '/X-Docker-Endpoints/{print $2}' | tr -d '\r' | sed 's/,//'` [alexh:~/work] echo $endpoint registry-1.docker.io [alexh:~/work] export token=`curl -sSL -o /dev/null -D- -H 'X-Docker-Token: true' "https://index.docker.io/v1/repositories/$image/images" | tr -d '\r' | awk '/X-Docker-Token/{print $2}'` The token needs to be used for authentication for the rest of the commands, but otherwise they are the same as the v1 private registry: [alexh:~/work] export v1_image_id=`curl -s -H "Authorization: Token $token" https://$endpoint/v1/repositories/library/$image/tags/$tag | sed 's/"//g'` [alexh:~/work] curl -sSL -H "Authorization: Token $token" "https://registry-1.docker.io/v1/images/$v1_image_id/ancestry" | python -m json.tool [ "0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547", "5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1", "8c63e4ac9a5f31e482d25a149b022209653b5948cb4f045c2ede9331a18e5824", "5c1d0c04c3b846fffd1d70886c956927a5c5f6a1c96f5e9f61c02f2ec1a45a73" ] [alexh:~/work] curl -sSL -H "Authorization: Token $token" https://$endpoint/v1/images/0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547/layer > 0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547.tar.gz [alexh:~/work] curl -sSL -H "Authorization: Token $token" https://$endpoint/v1/images/5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1/layer > 5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1.tar.gz [alexh:~/work] curl -sSL -H "Authorization: Token $token" https://$endpoint/v1/images/8c63e4ac9a5f31e482d25a149b022209653b5948cb4f045c2ede9331a18e5824/layer > 8c63e4ac9a5f31e482d25a149b022209653b5948cb4f045c2ede9331a18e5824.tar.gz [alexh:~/work] curl -sSL -H "Authorization: Token $token" https://$endpoint/v1/images/5c1d0c04c3b846fffd1d70886c956927a5c5f6a1c96f5e9f61c02f2ec1a45a73/layer > 5c1d0c04c3b846fffd1d70886c956927a5c5f6a1c96f5e9f61c02f2ec1a45a73.tar.gz v2 API The v2 API works with manifest files that include checksums. It's also slightly simpler. A manifest file for a tag contains all of the layer information, rather than requiring an image ID to be looked up for a tag, and then the ancestry for that image to be looked up. [alexh:~/work] curl -sSL http://127.0.0.1:6000/v2/$image/manifests/$tag | python -c 'import sys, json, pprint; pprint.pprint(json.load(sys.stdin)["fsLayers"])' [{u'blobSum': u'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'}, {u'blobSum': u'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'}, {u'blobSum': u'sha256:d4d342aa9da086ca4b7f7273858072e81021f4379c486223bc4708df6862b55d'}, {u'blobSum': u'sha256:23dc26e1038ae691b1a7e8e0152f974a358c42c929104c18c8e20b6d363c41ca'}, {u'blobSum': u'sha256:7772c716a45a828e124d20bc67199e77f2e63fb62589d0046f974f99b406e107'}] [alexh:~/work] curl -sSL http://127.0.0.1:6000/v2/$image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 > a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz [alexh:~/work] curl -sSL http://127.0.0.1:6000/v2/$image/blobs/sha256:d4d342aa9da086ca4b7f7273858072e81021f4379c486223bc4708df6862b55d > d4d342aa9da086ca4b7f7273858072e81021f4379c486223bc4708df6862b55d.tar.gz [alexh:~/work] curl -sSL http://127.0.0.1:6000/v2/$image/blobs/sha256:23dc26e1038ae691b1a7e8e0152f974a358c42c929104c18c8e20b6d363c41ca > 23dc26e1038ae691b1a7e8e0152f974a358c42c929104c18c8e20b6d363c41ca.tar.gz [alexh:~/work] curl -sSL http://127.0.0.1:6000/v2/$image/blobs/sha256:7772c716a45a828e124d20bc67199e77f2e63fb62589d0046f974f99b406e107 > 7772c716a45a828e124d20bc67199e77f2e63fb62589d0046f974f99b406e107.tar.gz We can get the checksum for these files to verify that they are what is described in the manifest file: [alexh:~/work] sha256sum *.tar.gz a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz d4d342aa9da086ca4b7f7273858072e81021f4379c486223bc4708df6862b55d d4d342aa9da086ca4b7f7273858072e81021f4379c486223bc4708df6862b55d.tar.gz 23dc26e1038ae691b1a7e8e0152f974a358c42c929104c18c8e20b6d363c41ca 23dc26e1038ae691b1a7e8e0152f974a358c42c929104c18c8e20b6d363c41ca.tar.gz 7772c716a45a828e124d20bc67199e77f2e63fb62589d0046f974f99b406e107 7772c716a45a828e124d20bc67199e77f2e63fb62589d0046f974f99b406e107.tar.gz The Remote (daemon) API Another API that is available is the Docker daemon running locally. It can be accessed over a Unix socket, or over TCP if the daemon is configured to allow it. [alexh:~/work] echo -e "GET /images/json HTTP/1.0\r\n" | nc -U /var/run/docker.sock | tail -n +6 | python -m json.tool [ { "Created": 1433116930, "Id": "0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547", "Labels": {}, "ParentId": "5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1", "RepoDigests": [], "RepoTags": [ "127.0.0.1:6000/ubuntu:15.10", "ubuntu:15.10", "127.0.0.1:5000/ubuntu:15.10" ], "Size": 0, "VirtualSize": 132392276 }, { "Created": 1432704049, "Id": "0c5e5ef1d7dac23c7164ea48faafc79f0c921f6cf87d2d8ea7469832ea31e4ca", "Labels": {}, "ParentId": "136beb445cfa7f48dbe4e36a80a83d4b7945682827fd8bfb1510ac17b6a200c0", "RepoDigests": [], "RepoTags": [ "registry:2.0.1" ], "Size": 0, "VirtualSize": 548626543 }, { "Created": 1432703977, "Id": "4e698fa804417b34b334793bab8a143403be9384e0651067b0c3933fe8d90eb2", "Labels": {}, "ParentId": "0cd49aa0e23cfe176cbea4bf622d552a6f16b21965cf52d633f8c9e27438f52c", "RepoDigests": [], "RepoTags": [ "registry:0.9.1" ], "Size": 0, "VirtualSize": 413940033 } ] A tarball containing all of the layers for a tag can be generated: [alexh:~/work] echo -e "GET /images/get?names=$image:$tag HTTP/1.0\r\n" | nc -U /var/run/docker.sock | tail -n +5 > $image-$tag.tar [alexh:~/work] mkdir tmp [alexh:~/work] tar -C tmp -xf ubuntu-15.10.tar [alexh:~/work] ls -l tmp total 20 drwxr-xr-x 2 alexh alexh 4096 Jun 2 15:33 0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547 drwxr-xr-x 2 alexh alexh 4096 Jun 2 15:33 5c1d0c04c3b846fffd1d70886c956927a5c5f6a1c96f5e9f61c02f2ec1a45a73 drwxr-xr-x 2 alexh alexh 4096 Jun 2 15:33 5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1 drwxr-xr-x 2 alexh alexh 4096 Jun 2 15:33 8c63e4ac9a5f31e482d25a149b022209653b5948cb4f045c2ede9331a18e5824 -rw-r--r-- 1 alexh alexh 87 Jun 2 15:33 repositories Conclusions Docker is a great technology and there are a lot of improvements and new features coming out at a rapid pace. Fortunately it's well documented and discussions about bugs are in the open on GitHub. However, there are still some edge cases to be aware of when talking to the Docker APIs. With some good design choices, your applications can be made backwards and forwards compatible, and will be able to use a wide range of Docker client versions and remote APIs.

Recently we started working on including support for Azure Service Bus in Cloud Portam. Prior to this, I had no experience with this service though it has been around for quite some time and I always wanted to try this out but one thing or another (oh, my stupid excuses :)!) prevented me from doing so. I learned a lot (and I am still learning) about this service while including support for it in Cloud Portam and this blog post talks about my learning. Please note that at the time of writing of all in all I have about a week of learning about this service so it is quite possible that I may be wrong about certain things. If that’s the case, please let me know and I will fix them ASAP. Now that the tone is set, let’s start! Azure Service Bus Offering The way I understand is that “Azure Service Bus” is a cloud-based messaging service that enables you to connect virtually anything – be it applications, services or devices. The beauty of Service Bus is that these things need not be in the cloud. They can run anywhere even inside the firewalled networks! Another thing I learned is that “Azure Service Bus” is essentially an umbrella service. At the time of writing of this post, there are actually four distinct services that are collectively offered under “Service Bus” umbrella – Queues, Topics & Subscriptions, Relays and Notification Hubs. Each service serves a different purpose yet the common theme is that all of them provide rich messaging infrastructure. To give you an analogy, if you have used Azure Storage Service you may already know that it offers four distinct services – Blobs, Files, Queues and Tables. It is the same with Service Bus as well. Queues Queues is the simplest of the service and kind of compares with Azure Storage Queue Service in the sense that it provides a unidirectional messaging infrastructure where a publisher publishes a message and the message is received by a receiver. There can be many receivers ready to receive the messages however one receiver can only receive a message. No two receivers can receive a single message simultaneously. For an in-depth comparison of Service Bus Queue and Storage Queues, please see this link: https://msdn.microsoft.com/en-us/library/azure/hh767287.aspx. Topics Topics are like queues in the sense that it also provides a unidirectional messaging infrastructure where a publisher publishes a message and receivers receive the message. The key difference is that same message can be received by multiple receivers (subscribers). Each subscriber can optionally specify a filter criteria so that they only receive the messages matching that criteria. To understand the difference between the two, let’s consider an example. Let’s say you run an e-commerce site and on successful completion of order, you have two tasks: 1) Send an email to customer about the order and 2) Notify the warehouse. If you were using Queues, you would either create 2 queues and put email notification message in one queue and warehouse notification message in another queue or build a workflow where you would send order confirmation message to a queue. Receiver would take that message and send out an email and then put warehouse notification message in the same queue (or other queue) and then another receiver would receive the message and notify the warehouse. However if you were using Topics, things would be much simpler logistically speaking. Essentially you would have just one message (order confirmation) but there will be two subscribers – one will be responsible for sending the email confirmation and the other will be responsible for notifying the warehouse. Relays Unlike Queues and Topics, which provide unidirectional flow of messages a Relay provides bi-directional flow. Using Relays, two disparate applications, services or devices can exchange messages. Other key difference is that a Relay doesn’t store the message like Queues and Topics. It just passes the messages from source to destination. Event Hubs Event Hubs service is meant for ingesting events and telemetry data in the cloud at massive scale (millions of events / second). Event Hubs are now more than important considering the push for connected devices (Internet-of-Things). Azure Service Bus Tiers Azure Service Bus is offered under two tiers (or SKUs if you would like): Basic and Standard. The difference is the level of functionality offered in each tier and the pricing. For example, Topics, Relays and Notification Hubs are only offered under Standard tier. Even with Queues, a limited set of functionality is exposed under Basic tier. For a list of features offered under each tier, please see this link: http://azure.microsoft.com/en-in/pricing/details/service-bus/. Summary That’s it for this post. In the next posts in this series, I will share my learnings about Queues and other Service Bus services. So stay tuned for that! Again, if you think that I have provided some incorrect information, please let me know and I will fix them ASAP.

introduction to explain why you have to use parameterized query to avoid sql injection over concatenated inline query it needs to know about sql injection. what does sql injection mean? it means when any end user send some invalid inputs to perform any crud operation or forcibly execute the wrong query into the database, those can be harmful for the database. harmful means ‘data loss’ or ‘get the data with invalid inputs. to know more, follow the below steps. step 1: create a table named ‘login’ in any database. create table user_login ( userid varchar(20), pwd varchar(20) ) now save some user credentials into the database for login purpose and select the table. insert into user_login values('rahul','bansal@123') insert into user_login values('bansal','rahul@123') step 2: create a website named ‘website1’. now i will create a login page named ‘default.aspx’ to validate the credentials from the ‘login’ table and if user is valid then redirect to it to the next page named ‘home.aspx’. add 2 textboxes for userid & password respectively and a button for login. add 2 namespaces in the .cs file of the ‘default.aspx’. using system.data.sqlclient; using system.data; now add the following code to validate the credentials from the database on click event of login button. protected void btn_login_click(object sender, eventargs e) { string constr = system.configuration.configurationmanager.connectionstrings["constr"].connectionstring; sqlconnection con = new sqlconnection(constr); string sql = "select count(userid) from user_login where userid='" + txtuserid.text + "' and pwd='" + txtpwd.text + "'"; sqlcommand cmd = new sqlcommand(sql, con); con.open(); object res = cmd.executescalar(); con.close(); if (convert.toint32(res) > 0) response.redirect("home.aspx"); else { response.write("invalid credentials"); return; } } add a new page named ‘home.aspx’. where any valid user will get welcome message. step 3: now run the ‘default’ page and log in with valid credentials. it will redirect to next page ‘home.aspx’ for valid user. note: here i have not used the textmode="password" property in password textbox to show the password. i have not used any input validations to explain my example. problem: now i will perform the sql injection with some invalid credentials with successful query execution and after that i will redirect to the next page ‘home.aspx’ as a valid user. i will enter a string in both textboxes like the following: ‘ or ‘1’=’1 now run the page and login with above string in both textboxes. it will redirect to next page name ‘home.aspx’ for valid user. see what happened. this is called sql injection in the hacking world. reason: it happened just because of the string and after filling this string in both textboxes orur sql query became like the following: select count(userid) from user_login where userid='' or '1'='1' and pwd='' or '1'='1' which will give the userid count and that is 2 in the table because 2 users are in ‘user_login’ table. it can be used in more ways like just fill the following string only in user id textbox and you will go the next page as valid user. or 1=1 - - and it will also give users count 2 because sqlquery will become like the following: select count(userid) from user_login where userid='' or 1=1 --' and pwd='' or '1'='1' note: the sign -- are for commenting the preceding text in sql. it can be more harmful or dangerous when the invalid user/hacker executes a script to drop all tables in the database or drop whole database. solution: to resolve this issue you have to do 2 things: always use parameterized query. input validations on client and server both side. sometimes if your input validation fail, then parameterized will not execute any scripted value. let’s see the example. protected void btn_login_click(object sender, eventargs e) { string constr = system.configuration.configurationmanager.connectionstrings["constr"].connectionstring; sqlconnection con = new sqlconnection(constr); string sql = "select count(userid) from user_login where userid=@userid and pwd=@pwd"; sqlcommand cmd = new sqlcommand(sql, con); sqlparameter[] param = new sqlparameter[2]; param[0] = new sqlparameter("@userid", txtuserid.text); param[1] = new sqlparameter("@pwd", txtpwd.text); cmd.parameters.add(param[0]); cmd.parameters.add(param[1]); con.open(); object res = cmd.executescalar(); con.close(); if (convert.toint32(res) > 0) response.redirect("home.aspx"); else { response.write("invalid credentials"); return; } } now if i run the page and try to login with sql scripts as done earlier. with ‘ or ‘1’=’1 with ' or 1=1 - - as you have seen parameterized didn’t execute the sql script but why? reason: the reason behind this the parameterized query would not be vulnerable and would instead look for a user id or password which literally matched the entire string. in other words ‘the sql engine checks each parameter to ensure that it is correct for its column and are treated literally, and not as part of the sql to be executed’. conclusion: always use parameterized query and input validations on client and server both side.

It’s a sad but true fact: Most data scientists spend 50-80% of their time cleaning and munging data and only a fraction of their time actually building predictive models. This is most often true in a traditional stack, where most of this data munging consists of writing lines upon lines of some flavor of SQL, leaving little time for model-building code in statistical programming languages such as R. These long, cryptic SQL queries not only slow development time but also prevent useful collaboration on analytics projects, as contributors struggle to understand each others’ SQL code. For example, in graduate school, I was on a project team where we used Oracle to store Twitter data. The kinds of queries my classmates and I were writing were unmaintainable and impossible to understand unless the author was sitting next to you. No one worked on the same queries together because they were so unwieldy. This not only hindered our collaboration efforts but also slowed our progress on the project. If we had been using an appropriate data store (like a graph database) we would have spent significantly less time pulling our hair out over the queries. Why Today’s Data Is Different This data-munging problem has persisted in the data science field because data is becoming increasingly social and highly-connected. Forcing this kind of interconnected data into an inherently tabular SQL database, where relationships are only abstract, leads to complicated schemas and overly complex queries. Yet, several NoSQL solutions – specifically in the graph database space – exist to store today’s highly-connected data. That is, data where relationships matter. A lot of data analysis today is performed in the context of better understanding people’s behavior or needs, such as: How likely is this visitor to click on advertisement X? Which products should I recommend to this user? How are User A and User B connected? Written by Nicole White People, as we know, are inherently social, so most of these questions can be answered by understanding the connections between people: User A is similar to User B, and we already know that User B likes this product, so let’s recommend this product to User A. The Good News: Data-Munging No More Data science doesn’t have to be 80% data munging. With the appropriate technology stack, a data scientist’s development process is seamless and short. It’s time to spend less time writing queries and more time building models by combining the flexibility of an open-source, NoSQL graph database with the maturity and breadth of R – an open-source statistical programming language. The combination of Neo4j’s ability to store highly-connected, possibly-unstructured data and R’s functional, ad-hoc nature creates the ideal data analysis environment. You don’t have to spend an hour writing CREATE TABLE statements. You don’t have to spend all day on StackOverflow figuring out how to traverse a tree in SQL. Just Cypher and go. Learn More at OSCON 2015 At my upcoming OSCON session we will walk through a project in which we analyze #OSCON Twitter data in a reproducible, low-effort workflow without writing a single line of SQL. For this highly-connected dataset we will use Neo4j, an open-source graph database, to store and query the data while highlighting the advantages of storing such data in a graph versus a relational schema. Finally, we will cover how to connect to Neo4j from an R environment for the purposes of performing common data science tasks, such as analysis, prediction and visualization.

It’s no surprise that many if not all programmers use GitHub today to store their codes, but it can be frustrating to keep everyone up to date with the code changes. Recently, GitHub has been integrated with Quire, a tree-structured task management tool that lets programmers to easily keep track of code changes. By linking GitHub commits to the so-called tasks (issues), users can refer to these tasks when they look at code changes, and also trace back to the codes when they look at the tasks. In a blog article, Quire goes into a bit more detail about their new integration and what exactly users can do and benefit from it. Check out the details at the link below. Hello GitHub, We’re Quire | Quire Blog

I presented a new talk at GOTO Chicago 2015 about how to change a team’s attitude towards writing automated tests. The talk covers the same case study as Groovy vs Java for Testing, adopting Spock in MongoDB, but this is a more process/agile/people perspective, not a technical look at the merits of one language over another. Slides available below. As always, the slides are not super-useful out of context, but they do contain my conclusions (also note that due to a technology fail, my hand-drawn style is even more hand-drawn than usual). Questions I sadly did not have a lot of time for questions during the presentation, but thanks to the wonders of modern technology, I have a list of unanswered questions which I will attempt to address here. Is testing to find out your system works? Or is it so you know when your system is broken? Excellent question. I would expect that if you have a system that’s in production (which is probably the large majority of the projects we work on), we can assume the system is working, for some definition of working. Automated testing is particularly good at catching when your system stops doing the things you thought it was doing when you wrote the tests (which may, or may not, mean the system is genuinely “broken”). Regression testing is to find out when your system is no longer doing what you expect, and automated tests are really good for this. But testing can also make sure you implement code that behaves the way you expect, especially if you write the tests first. Automated tests can be used to determine that your code is complete, according to some pre-agreed specification (in this case, the automated tests you wrote up front). So I guess what I’m trying to say is, when you first write the tests you have tests that, when they pass, proves the system works (assumingyour tests are testing the right things and/or not giving you false positives). Subsequent passes show that you haven’t broken anything. At what level do “tests documenting code” actually become useful? And who is/should the documentation be targeted to? In the presentation, my case study is the MongoDB Java Driver. Our users were Java programmers, who were going to be coding using our driver. So in this example, it makes a lot of sense to document the code using a language that our users understood. We started with Java, and ended up using Groovy because it was also understandable for our users and a bit more succinct. On a previous project we had different types of tests. The unit and system tests documented what the expected behaviour was at the class or module level, and was aimed at developers in the team. The acceptance tests were written in Java, but in a friendly DSL-style way. These were usually written by a triad of tester, business analyst and developer, and documented to all these guys and girls what the top-level behaviour should be. Our audience here was fairly technical though, so there was no need to go to the extent of trying to write English-language-style tests, they were readable enough for a reasonably techy (but non-programmer) audience. These were not designed to be read by “the business” - us developers might use them to answer questions about the behaviour of the system, but they didn’t document it in a way that just anyone could understand. These are two different approaches for two different-sized team/organisations, with different users. So I guess in summary the answer is “it depends”. But at the very least, developers on your own team should be able to read your tests and understand what the expected behaviour of the code is. How do you become a team champion? I.e. get authority and acceptance that people listen to you? In my case, it was just by accident - I happened to care about the tests being green and also being useful, so I moaned at people until it happened. But it’s not just about nagging, you get more buy-in if other people see you doing the right things the right way, and it’s not too painful for them to follow your example. There are going to be things that you care about that you’ll never get other people to care about, and this will be different from team to team. You have two choices here - if you care that much, and it bothers you that much, you have to do it yourself (often on your own time, especially if your boss doesn’t buy into it). Or, you have to let it go - when it comes to quality, there are so many things you could care about that it might be more beneficial to drop one cause and pick another that you can get people to care about. For example, I wanted us to use assertThat instead of assertFalse (or true, or equals, or whatever). I tried to demo the advantages (as I saw them) of my approach to the team, and tried to push this in code reviews, but in the end the other developers weren’t sold on the benefits, and from my point of view the benefits weren’t big enough to force the issue. Those of us who cared, used assertThat. For the rest, I was just happy people were writing and maintaining tests. So, pick your battles. You’ll be surprised at how many people do get on board with things. I thought implementing checkstyle and setting draconian formatting standards was going to be a tough battle, but in the end people were just happy to have any standards, especially when they were enforced by the build. Do you report test, style, coverage, etc failures separately? Why? We didn’t fail on coverage. Enforcing a coverage percentage is a really good way to end up with crappy tests, like for getters/setters and constructors (by the way, if there’s enough logic in your constructor that it needs a test, You’re Doing It Wrong). Generally different types of failures are found by different tools, so for this reason alone they will be reported separately - for example, checkstyle will fail the build if it doesn’t conform to our style standards, codenarc fails it for Groovy style failures, and Gradle will run the tests in a different task to these two. What’s actually important, though, is time-to-failure. For checkstyle, for example, it will fail on something silly like curly braces in the wrong place. You want this to fail within seconds, so you can fix the silly mistake quickly. Ideally you’d have IntelliJ (perhaps) run your checks before it even makes it into your CI environment. Compiler errors should, of course, fail things before you run a test, short-running tests should fail before long-running tests. Basically, the easier it is to fix the problem, the sooner you want to know, I guess. Our build was relatively small and not too complex, so actually we ran all our types of tests (integration and unit, both Groovy and Java) in a single task, because this turned out to be much quicker in Gradle (in our case) than splitting things up into a simple pipeline. You might have a reason to report stuff separately, but for me it’s much more important to understand how fast I need to be aware of a particular type of failure. Sometimes I find myself modifying code design and architecture to enable testing. How can I avoid damaging design? This is a great question, and a common one too. The short answer is: in general writing code that’s easier to test leads to a cleaner design anyway (for example, dependency injection at that appropriate places). If you find you need to rip your design apart to test it, there’s a smell there somewhere - either your design isn’t following SOLID principals, or you’re trying to test the wrong things. Of course, the common example here is testing private methods - how do you test these without exposing secrets? I think for me, if it’s important enough to be tested it’s important enough to be exposed in some way - it might belong in some sort of util or helper (right now I’m not going to go into whether utils or helpers are, in themselves a smell), in a smaller class that only provides this sort of functionality, or simply a protected method. Or, if you’re testing with Groovy, you can access private methods anyway so this becomes a moot point (i.e. your testing framework may be limiting you). In another story from LMAX, we found we had created methods just for testing. It seemed a bit wrong to have these methods only available for testing, but later on down the line, we needed access to many of these methods In Real Life (well, from our Admin app), so our testing had “found” a missing feature. When we came to implement it, it was pretty easy as we’d already done most of it for testing. My co-workers often point to a lack of end-to-end testing as the reason why a lot of bugs get out to production even though they don’t have much unit tests nor integration tests. What, in your experience, is a good balance between unit tests, integration tests and end-to-end testing? Hmm, sounds to me like “lack of tests” is your problem! This is a big (and contentious!) topic. Martin Fowler has written about it, Google wrote something I completely disagree with (so I’m not even going to link to it, but you’ll find references in the links in this paragraph), and my ex-colleague Adrian talks about what we, at LMAX, meant by end-to-end tests. I hope that’s enough to get you started, there’s plenty more out there too. How did you go about getting buy in from the team to use Spock? I cover this in my other presentation on the topic - the short version is, I did a week-long spike to investigate whether Spock would make testing easier for us, showed the pros and cons to the whole team, and then led by example writing tests that (I thought) were more readable than what we had before and, probably most importantly, much easier to write than what we were previously doing. I basically got buy-in by showing how much easier it was for us to use the tool than even JUnit (which we were all familiar with). It did help that we were already using Gradle, so we already had a development dependency on Groovy. It also helped that adding Spock made no changes to the dependencies of the final Jar, which was very important. Over time, further buy-in (certainly from management) came when the new tests started catching more errors - usually regressions in our code or regressions in the server’s overnight builds. I don’t think it was Spock specifically that caught more problems - I think it was writing more tests, and better tests, that caught the issues. Can we do data driven style tests in frameworks like junit or cucumber? I don’t think you can in JUnit (although maybe there’s something out there). I believe someone told me you can do it in TestNG. Are there drawbacks to having tests that only run in ci? I.e I have Java 8 on my machine, but the test requires Java 7 Yes, definitely - the drawback is Time. You have to commit your code to a branch that is being checked by CI and wait for CI to finish before you find the error. In practice, we found very little that was different between Java 7 and 8, for example, but this is a valid concern (otherwise you wouldn’t be testing a complex matrix of dependencies at all). In our case, our Java 6 driver used Netty for async capabilities, as the stuff we were using from Java 7 wasn’t available. This was clearly a different code path that wasn’t tested by us locally as we were all running Java 8. Probably more importantly for us is we were testing against at least 3 different major versions of the server, which all supported different features (and had different APIs). I would often find I’d broken the tests for version 2.2 as I’d only been running it on 2.6, and had forgotten to either turn off the new tests for the old server versions, or didn’t realise the new functionality wouldn’t work there. So the main drawback is time - it takes a lot longer to find out about these errors. There are a few ways to get around this: Commit often!! And to a branch that’s actually going to be run by CI Make your build as fast as possible, so you get failures fast (you should be doing this anyway) You could set up virtual machines locally or somewhere cloudy to run these configurations before committing, but that sounds kinda painful (and to my mind defeats a lot of the point of CI). I set up Travis on my fork of the project, so I could have that running a different version of Java and MongoDB when I committed to my own fork - I’d be able to see some errors before they made it into the “real” project. If you can, you probably want these specific tests run first so they can fail fast. E.g. if you’re running a Java 6 & MongoDB 2.2 configuration on CI, run those tests that only work in that environment first. Would probably need some Gradle magic, and/or might need you to separate these into a different set of folders. The advantage of this approach though is if you set up some aliases on your local machine you could sanity check just these special cases before checking in. For example, I had aliases to start MongoDB versions/configurations from a single command, and to set JAVA_HOME to whichever version I wanted. Do you have any tips for unit tests that pass on dev machines but not on Jenkins because it’s not as powerful as our own machines? E.g. Synchronous calls timeout on the Jenkins builds intermittently. Erk! Yes, not uncommon. No, not really. We had our timeouts set longer than I would have liked to prevent these sorts of errors, and they still intermittently failed. You can also set some sort of retry on the test, and get your build system to re-run those that fail to see if they pass later. It’s kinda nasty though. At LMAX they were able to take testing seriously enough to really invest in their testing architecture, and, of course, this is The Correct Answer. Just often very difficult to sell. If you ask where are tests and dev asks if code is correct? And you say yes. Then dev asks why you’re delaying shipping value, how do you manage that? These are my opinions: Your code is not complete without tests that show me it’s complete. Your code might do what you think it’s supposed to do right now, but given Shared Code Ownership, anyone can come in and change it at any time, you want tests in place to make sure they don’t change it to break what you thought it did The tests are not so much to show it works right now, the tests are to show it continues to work in future Having automated tests will speed you up in future. You can refactor more safely, you can fix bugs and know almost immediately if you broke something, you can read from the test what the author of the code thought the code should do, getting you up to speed faster. You don’t know you’re shipping value without tests - you’re only shipping code (to be honest, you never know if you’re shipping value until much later on when you also analyse if people are even using the feature). Testing almost never slows you down in the long run. Show me the bits of your code base which are poorly tested, and I bet I can show you the bits of your code base that frequently have bugs (either because the code is not really doing what the author thinks, or because subsequent changes break things in subtle ways). If you say code is hard to understand and dev asks if you seriously don’t understand the code, how do you explain you mean easy to understand without thinking rather than ‘can I compile this in my head’? I have zero problem with saying “I’m too stupid to understand this code, and I expect you’re much smarter than me for writing it. Can you please write it in a way so that a less smart person like myself won’t trample all over your beautiful code at a later date through lack of understanding?” By definition, code should be easy to understand by someone who’s not the author. If someone who is not the author says the code is hard to understand, then the code is hard to understand. This is not negotiable. This is what code reviews or pair programming should address. What is effective nagging like? (Whether or not you get what you want) Mmm, good question. Off the top of my head: Don’t make the people who are the target of the nagging feel stupid - they’ll get defensive. If necessary, take the burden of “stupidity” on yourself. E.g. “I’m just not smart enough to be able to tell if this test is failing because the test is bad or because the code is bad. Can you walk me through it and help me fix it?” Do at least your fair share of the work, if not more. When I wanted to get the code to a state where we could fail style errors, I fixed 99% of the problems, and delegated the handful of remaining ones that I just didn’t have the context to fix. In the face of three errors to fix each, the team could hardly say “no” after I’d fixed over 6000. Explain why things need to be done. Developers are adults and don’t want to be treated like children. Give them a good reason and they’ll follow the rules. The few times I didn’t have good reasons, I could not get the team to do what I wanted. Find carrots and sticks that work. At LMAX, a short e-mail at the start of the day summarising the errors that had happened overnight, who seemed to be responsible, and whether they looked like real errors or intermittencies, was enough to get people to fix their problems2 - they didn’t like to look bad, but they also had enough information to get right on it, they didn’t have to wade through all the build info. On occasion, when people were ignoring this, I’d turn up to work with bags of chocolate that I’d bought with my own money, offering chocolate bars to anyone who fixed up the tests. I was random with my carrot offerings so people didn’t game the system. Give up if it’s not working. If you’ve tried to phrase the “why” in a number of ways, if you’ve tried to show examples of the benefits, if you’ve tried to work the results you want into a process, but it’s still not getting done, just accept the fact that this isn’t working for the team. Move on to something else, or find a new angle. 1 I had a colleague at LMAX who was working with a hypothesis that All Private Methods Were Evil - they were clearly only sharable within single class, so provided no reuse elsewhere, and if you have the same bit of code being called multiple times from within the same class (but it’s not valuable elsewhere) then maybe your design is wrong. I’m still pondering this specific hypothesis 4 years on, and I admit I see its pros and cons. 2 This worked so well that this process was automated by one of the guys and turned into a tool called AutoTrish, which as far as I know is still used at LMAX. Dave Farley talks about it in some of hisContinuous Delivery talks. Resources My talk that specifically looks at the advantages of Spock over JUnit, plus some Spock-specific resources. I love Jay Fields book Working Effectively With Unit Tests - if I could have made the whole team read this before moving to Spock, we might have stuck with JUnit. Go read everything Adrian Sutton has written about testing at LMAX. If not everything, definitely Abstraction by DSL and Making End-to-End Tests Work If you can’t make it all the way through Dave Farley and Jez Humble’s excellent Continuous Delivery book, do take a look at one of Dave’s presentations on the subject, for example The Rationale for Continuous Delivery or The Process, Technology and Practice of Continuous Delivery - my own talk was around testing, but I’m working off the assumption that you’re at least running some sort of Continuous Integration, if not Continuous Delivery. Martin Fowler has loads of interesting and useful articles on testing. Abstract What can you do to help developers a) write tests b) write meaningful tests and c) write readable tests? Trisha will talk about her experiences of working in a team that wanted to build quality into their new software version without a painful overhead - without a QA / Testing team, without putting in place any formal processes, without slavishly improving the coverage percentage. The team had been writing automated tests and running them in a continuous integration environment, but they were simply writing tests as another tick box to check, there to verify the developer had done what the developer had aimed to do. The team needed to move to a model where tests provided more than this. The tests needed to: Demonstrate that the library code was meeting the requirements Document in a readable fashion what those requirements were, and what should happen under non-happy-path situations Provide enough coverage so a developer could confidently refactor the code This talk will cover how the team selected a new testing framework (Spock, a framework written in Groovy that can be used to test JVM code) to aid with this effort, and how they evaluated whether this tool would meet the team’s needs. And now, two years after starting to use Spock, Trisha can talk about how both the tool and the shift in the focus of the purpose of tests has affected the quality of the code. And, interestingly, the happiness of the developers.

Persistence of model objects is a part of many Java projects and a part which deserves, and often gets, high test coverage as one of the key layer integration points in the code. However, I've often felt the testing paradigms for this can be cumbersome, often involving a large amount of setup with an equivalent amount of validation. This can be tedious to both create and maintain. As a solution to this I've been testing persistence with a different pattern; by combining both the exparity-stub and the hamcrest-bean library you can thoroughly test model persistence in a few lines of test code as per the snippet below; .. User user = aRandomInstanceOf(User.class); User saved = dao.save(user); assertThat(dao.getUserById(saved.getId()), theSameBeanAs(saved)); The test snippet above is small but in those few lines will thoroughly test that all fields in a graph can be persisted and retrieved without loss, that any JPA or other mapping is valid, and that your queries are valid. For a complete example we'll work through testing a simple DAO for storing and retrieving User objects using the in-memory H2 database for simplicity. The same example will work for any persistence mechanism. Before we get started with an example lets briefly outline what the libraries are and what they do. The Exparity-Stub Library The exparity-stub libraries provides a set of static methods for creating stubs of model objects, object graphs, collections, types, and primitive types. For our example we'll be creating random stubs because we want to completely fill the graph with junk data and check it can be written down. exparity-stub offers two approaches to this, the RandomBuilder or the BeanBuilder. The RandomBuilder provides a terser notation to create random objects with less code. For example: User user = RandomBuilder.aRandomInstanceOf(User.class); List users = RandomBuilder.aRandomListOf(User.class); String anyString = RandomBuilder.aRandomString(); Whereas the BeanBuilder provides a fluent interface with finer control for building individual objects and graphs, for example; User user = BeanBuilder.aRandomInstanceOf(User.class) .excludeProperty("Id").build(); For this example i'm going to use the BeanBuilder so I can exclude the User.Id property from being populated by the random builder. The Hamcrest-Bean Library The hamcrest-bean library is an extension library to the Java Hamcrest library. The hamcrest-bean library provides a set of matchers specifically for testing Java objects and object graphs and performs deep inspections of those objects. It supports exclusions and overrides to allow fine control, if required, of how matching of any property, path, or type is handled, for example: User expected = new User("Jane", "Doe"); assertThat(new User("John", "Doe"), BeanMatchers.theSameAs(expected).excludeProperty("FirstName")); A Sample Project The sample project I'll work through is persistence of a simple User object with a child list of UserComment objects. This simple graph will be persisted to a H2 database with hibernate handling the Object-Relational Mapping (ORM) mapping, and Java Persistence Annotation (JPA) used to mark-up the model. The Model Below are the two model classes; first the User class. package org.exparity.hamcrest.bean.sample.dao; import java.util.*; import javax.persistence.*; @Entity @Table public class User { @Id @GeneratedValue(strategy = GenerationType.SEQUENCE) private Long id; private Date createTs; private String username, firstName, surname; @OneToMany(cascade = CascadeType.ALL, fetch = FetchType.EAGER) private List comments = new ArrayList<>(); public Long getId() { return id; } public void setId(Long id) { this.id = id; } public Date getCreateTs() { return createTs; } public void setCreateTs(Date createTs) { this.createTs = createTs; } public String getUsername() { return username; } public void setUsername(String username) { this.username = username; } public String getFirstName() { return firstName; } public void setFirstName(String firstName) { this.firstName = firstName; } public String getSurname() { return surname; } public void setSurname(String surname) { this.surname = surname; } public List getComments() { return comments; } public void setComments(List comments) { this.comments = comments; } } Followed by the UserComment class. package org.exparity.hamcrest.bean.sample.dao; import java.util.Date; import javax.persistence.*; @Table @Entity public class UserComment { private Long id; private Date timestamp; @Transient private String text; private String title; public Date getTimestamp() { return timestamp; } public void setTimestamp(Date timestamp) { this.timestamp = timestamp; } public String getText() { return text; } public void setText(String text) { this.text = text; } public String getTitle() { return title; } public void setTitle(String title) { this.title = title; } } Followed by the UserComment class. package org.exparity.hamcrest.bean.sample.dao; import java.util.Date; import javax.persistence.*; @Table @Entity public class UserComment { private Long id; private Date timestamp; @Transient private String text; private String title; public Date getTimestamp() { return timestamp; } public void setTimestamp(Date timestamp) { this.timestamp = timestamp; } public String getText() { return text; } public void setText(String text) { this.text = text; } public String getTitle() { return title; } public void setTitle(String title) { this.title = title; } } The Data Access Object (DAO) Next up we write our DAO layer. I've excluded the UserDAO interface from this post but it is available in the sample project ongithub .The full, if somewhat crude, implementation of the UserDAO is below. package org.exparity.hamcrest.bean.sample.dao; import org.hibernate.boot.registry.StandardServiceRegistryBuilder; import org.hibernate.cfg.Configuration; import org.hibernate.*; public class UserDAOHibernateImpl implements UserDAO { private final SessionFactory factory; public UserDAOHibernateImpl(final String resourceFile) { this.factory = new Configuration() .addAnnotatedClass(User.class) .addAnnotatedClass(UserComment.class) .buildSessionFactory( new StandardServiceRegistryBuilder() .loadProperties(resourceFile) .build()); } @Override public User save(final User user) { Session session = factory.getCurrentSession(); Transaction txn = session.beginTransaction(); try { session.save(user); txn.commit(); } catch (final Exception e) { txn.rollback(); } return user; } @Override public User getUserById(Long userId) { Session session = factory.getCurrentSession(); Transaction txn = session.beginTransaction(); try { return (User) session.get(User.class, userId); } finally { txn.rollback(); } } } Integration Test And finally, onto our integration test. The hibernate.properties will create an instance of an in-memory database and create the necessary tables on instantiation of the DAO. hibernate.dialect=org.hibernate.dialect.H2Dialect hibernate.connection.username=sa hibernate.connection.password= hibernate.connection.driver_class=org.h2.Driver hibernate.connection.url=jdbc:h2:mem:test hibernate.current_session_context_class=thread hibernate.cache.provider_class=org.hibernate.cache.internal.NoCacheProvider hibernate.show_sql=true hibernate.hbm2ddl.auto=update The integration test is below. package org.exparity.hamcrest.bean.sample.dao; import static org.exparity.hamcrest.BeanMatchers.theSameBeanAs; import static org.exparity.stub.bean.BeanBuilder.aRandomInstanceOf; import static org.hamcrest.MatcherAssert.assertThat; import static org.hamcrest.Matchers.*; import org.junit.Test; public class UserDAOHibernateImplTest { @Test public void canSaveAUser() { User user = aRandomInstanceOf(User.class).excludeProperty("Id").build(); UserDAOHibernateImpl dao = new UserDAOHibernateImpl("hibernate.properties"); User saved = dao.save(user); User loaded = dao.getUserById(saved.getId()); assertThat(loaded, not(sameInstance(user))); assertThat(loaded, theSameBeanAs(user)); } } Let's break the test down step by step to see what each step is doing and why the test is put together this way. 1) Model Setup User user = aRandomInstanceOf(User.class).excludeProperty("Id").build(); Create a random instance of the User class and it's associates using exparity-stub. The instance will be populated with random data with the exception of the Id property. I've excluded the Id property so that is left null to test that the id is being generated in the database. 2) DAO Setup UserDAOHibernateImpl dao = new UserDAOHibernateImpl("hibernate.properties") Instantiate the DAO ready to be tested, passing in the property file to use for the test. The hibernate properties used will configure an in-memory instance of H2 and create the schema automatically. 3) Exercise the DAO User saved = dao.save(user); User loaded = dao.getUserById(saved.getId()); Save the random instance of the model set up in step (1) and then query the object back out again. 4) Verify the results assertThat(loaded, not(sameInstance(user))); assertThat(loaded, theSameBeanAs(user)); The first line verifies that the loaded User instance is not the same instance as the originally saved User. This prevents false positive results when the loaded instance is returned directly from a cache. The second line uses hamcrest-bean to perform a deep comparison of the loaded User instance against the original user instance. Running the Test The first run of the test yields an error; specifically a hibernate warning because a @Id annotation has been missed on UserComment. org.hibernate.AnnotationException: No identifier specified for entity: org.exparity.hamcrest.bean.sample.dao.UserComment at org.hibernate.cfg.InheritanceState.determineDefaultAccessType(InheritanceState.java:277) at org.hibernate.cfg.InheritanceState.getElementsToProcess(InheritanceState.java:224) at org.hibernate.cfg.AnnotationBinder.bindClass(AnnotationBinder.java:775) at org.hibernate.cfg.Configuration$MetadataSourceQueue.processAnnotatedClassesQueue(Configuration.java:3845) at org.hibernate.cfg.Configuration$MetadataSourceQueue.processMetadata(Configuration.java:3799) at org.hibernate.cfg.Configuration.secondPassCompile(Configuration.java:1412) at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1846) at org.exparity.hamcrest.bean.sample.dao.UserDAOHibernateImpl.(UserDAOHibernateImpl.java:15) at org.exparity.hamcrest.bean.sample.dao.UserDAOHibernateImplTest.canSaveAUser(UserDAOHibernateImplTest.java:18) A fix to the UserComment object and we can run the test again. @Table @Entity public class UserComment { @Id @GeneratedValue(strategy = GenerationType.SEQUENCE) private Long id; private Date timestamp; @Transient private String text; private String title; ... After running the test again we get another failure. The presence of the @Transient annotation on the UserComment.text property is preventing the value being persisted java.lang.AssertionError: Expected: the same as but: User.Comments[0].Text is null instead of "mDAWDJXbheIHbbHLR1NNVJqAki49RvaVwQtKD38r79u0y3MTDD" at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8) at org.exparity.hamcrest.bean.sample.dao.UserDAOHibernateImplTest.canSaveAUser(UserDAOHibernateImplTest.java:19) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) Another change to the UserComment object to remove the @Transient annotation and we can run the test again. @Table @Entity public class UserComment { @Id @GeneratedValue(strategy = GenerationType.SEQUENCE) private Long id; private Date timestamp; private String text; private String title; ... After running the test again it all passes. Try It Out To try hamcrest-bean and exparity-stub out for yourself include the dependency in your maven pom or other dependency manager. org.exparity hamcrest-bean 1.0.10 test org.exparity exparity-stub 1.1.5 test

In this post, I demonstrate one approach for generating JSON Schema from an XML Schema (XSD). While providing an overview of an approach for creating JSON Schema from XML Schema, this post also demonstrates use of a JAXB implementation (xjc version 2.2.12-b150331.1824 bundled with JDK 9 [build 1.9.0-ea-b68]) and of a JSON/Java binding implementation (Jackson 2.5.4). The steps of this approach for generating JSON Schema from an XSD can be summarized as: Apply JAXB's xjc compiler to generate Java classes from XML Schema (XSD). Apply Jackson to generate JSON schema from JAXB-generated Java classes. Generating Java Classes from XSD with JAXB's xjc For purposes of this discussion, I'll be using the simple Food.xsd used in my previous blog post A JAXB Nuance: String Versus Enum from Enumerated Restricted XSD String. For convenience, I have reproduced that simple schema here without the XML comments specific to that earlier blog post: Food.xsd It is easy to use the xjc command line tool provided by the JDK-provided JAXB implementation to generate Java classes corresponding to this XSD. The next screen snapshot shows this process using the command: xjc -d jaxb .\Food.xsd This simple command generates Java classes corresponding to the provided Food.xsd and places those classes in the specified "jaxb" subdirectory. Generating JSON from JAXB-Generated Classes with Jackson With the JAXB-generated classes now available, Jackson can be applied to these classes to generate JSON from the Java classes. Jackson is described on its main portal page as "a multi-purpose Java library for processing" that is "inspired by the quality and variety of XML tooling available for the Java platform." The existence of Jackson and similar frameworks and libraries appears to be one of the reasons that Oracle hasdropped the JEP 198 ("Light-Weight JSON API") from Java SE 9. [It's worth noting that Java EE 7 already hasbuilt-in JSON support with its implementation of JSR 353 ("Java API for JSON Processing"), which is not associated with JEP 198).] One of the first steps of applying Jackson to generating JSON from our JAXB-generated Java classes is to acquire and configure an instance of Jackson's ObjectMapper class. One approach for accomplishing this is shown in the next code listing. Acquiring and Configuring Jackson ObjectMapper for JAXB Serialization/Deserialization /** * Create instance of ObjectMapper with JAXB introspector * and default type factory. * * @return Instance of ObjectMapper with JAXB introspector * and default type factory. */ private ObjectMapper createJaxbObjectMapper() { final ObjectMapper mapper = new ObjectMapper(); final TypeFactory typeFactory = TypeFactory.defaultInstance(); final AnnotationIntrospector introspector = new JaxbAnnotationIntrospector(typeFactory); // make deserializer use JAXB annotations (only) mapper.getDeserializationConfig().with(introspector); // make serializer use JAXB annotations (only) mapper.getSerializationConfig().with(introspector); return mapper; } The above code listing demonstrates acquiring the Jackson ObjectMapper instance and configuring it to use a default type factory and a JAXB-oriented annotation introspector. With the Jackson ObjectMapper instantiated and appropriately configured, it's easy to use thatObjectMapper instance to generate JSON from the generated JAXB classes. One way to accomplish this using the deprecated Jackson class JsonSchema is demonstrated in the next code listing. Generating JSON from Java Classes with Deprecated com.fasterxml.jackson.databind.jsonschema.JsonSchema Class /** * Write JSON Schema to standard output based upon Java source * code in class whose fully qualified package and class name * have been provided. * * @param mapper Instance of ObjectMapper from which to * invoke JSON schema generation. * @param fullyQualifiedClassName Name of Java class upon * which JSON Schema will be extracted. */ private void writeToStandardOutputWithDeprecatedJsonSchema( final ObjectMapper mapper, final String fullyQualifiedClassName) { try { final JsonSchema jsonSchema = mapper.generateJsonSchema(Class.forName(fullyQualifiedClassName)); out.println(jsonSchema); } catch (ClassNotFoundException cnfEx) { err.println("Unable to find class " + fullyQualifiedClassName); } catch (JsonMappingException jsonEx) { err.println("Unable to map JSON: " + jsonEx); } } The code in the above listing instantiates acquires the class definition of the provided Java class (the highest level Food class generated by the JAXB xjc compiler in my example) and passes that reference to the JAXB-generated class to ObjectMapper's generateJsonSchema(Class) method. The deprecated JsonSchemaclass's toString() implementation is very useful and makes it easy to write out the JSON generated from the JAXB-generated classes. For purposes of this demonstration, I provide the demonstration driver as a main(String[]) function. That function and the entire class to this point (including methods shown above) is provided in the next code listing. JsonGenerationFromJaxbClasses.java, Version 1 package dustin.examples.jackson; import com.fasterxml.jackson.databind.AnnotationIntrospector; import com.fasterxml.jackson.databind.JsonMappingException; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.databind.type.TypeFactory; import com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector; import com.fasterxml.jackson.databind.jsonschema.JsonSchema; import static java.lang.System.out; import static java.lang.System.err; /** * Generates JavaScript Object Notation (JSON) from Java classes * with Java API for XML Binding (JAXB) annotations. */ public class JsonGenerationFromJaxbClasses { /** * Create instance of ObjectMapper with JAXB introspector * and default type factory. * * @return Instance of ObjectMapper with JAXB introspector * and default type factory. */ private ObjectMapper createJaxbObjectMapper() { final ObjectMapper mapper = new ObjectMapper(); final TypeFactory typeFactory = TypeFactory.defaultInstance(); final AnnotationIntrospector introspector = new JaxbAnnotationIntrospector(typeFactory); // make deserializer use JAXB annotations (only) mapper.getDeserializationConfig().with(introspector); // make serializer use JAXB annotations (only) mapper.getSerializationConfig().with(introspector); return mapper; } /** * Write out JSON Schema based upon Java source code in * class whose fully qualified package and class name have * been provided. * * @param mapper Instance of ObjectMapper from which to * invoke JSON schema generation. * @param fullyQualifiedClassName Name of Java class upon * which JSON Schema will be extracted. */ private void writeToStandardOutputWithDeprecatedJsonSchema( final ObjectMapper mapper, final String fullyQualifiedClassName) { try { final JsonSchema jsonSchema = mapper.generateJsonSchema(Class.forName(fullyQualifiedClassName)); out.println(jsonSchema); } catch (ClassNotFoundException cnfEx) { err.println("Unable to find class " + fullyQualifiedClassName); } catch (JsonMappingException jsonEx) { err.println("Unable to map JSON: " + jsonEx); } } /** * Accepts the fully qualified (full package) name of a * Java class with JAXB annotations that will be used to * generate a JSON schema. * * @param arguments One argument expected: fully qualified * package and class name of Java class with JAXB * annotations. */ public static void main(final String[] arguments) { if (arguments.length < 1) { err.println("Need to provide the fully qualified name of the highest-level Java class with JAXB annotations."); System.exit(-1); } final JsonGenerationFromJaxbClasses instance = new JsonGenerationFromJaxbClasses(); final String fullyQualifiedClassName = arguments[0]; final ObjectMapper objectMapper = instance.createJaxbObjectMapper(); instance.writeToStandardOutputWithDeprecatedJsonSchema(objectMapper, fullyQualifiedClassName); } } To run this relatively generic code against the Java classes generated by JAXB's xjc based upon Food.xsd, I need to provide the fully qualified package name and class name of the highest-level generated class. In this case, that's com.blogspot.marxsoftware.foodxml.Food (package name is based on the XSD's namespace because I did not explicitly override that when running xjc). When I run the above code with that fully qualified class name and with the JAXB classes and Jackson libraries on the classpath, I see the following JSON written to standard output. Generated JSON {"type":"object","properties":{"vegetable":{"type":"string","enum":["CARROT","SQUASH","SPINACH","CELERY"]},"fruit":{"type":"string"},"dessert":{"type":"string","enum":["PIE","CAKE","ICE_CREAM"]}} Humans (which includes many developers) prefer prettier print than what was just shown for the generated JSON. We can tweak the implementation of the demonstration class's methodwriteToStandardOutputWithDeprecatedJsonSchema(ObjectMapper, String) as shown below to write out indented JSON that better reflects its hierarchical nature. This modified method is shown next. Modified writeToStandardOutputWithDeprecatedJsonSchema(ObjectMapper, String) to Write Indented JSON /** * Write out indented JSON Schema based upon Java source * code in class whose fully qualified package and class * name have been provided. * * @param mapper Instance of ObjectMapper from which to * invoke JSON schema generation. * @param fullyQualifiedClassName Name of Java class upon * which JSON Schema will be extracted. */ private void writeToStandardOutputWithDeprecatedJsonSchema( final ObjectMapper mapper, final String fullyQualifiedClassName) { try { final JsonSchema jsonSchema = mapper.generateJsonSchema(Class.forName(fullyQualifiedClassName)); out.println(mapper.writerWithDefaultPrettyPrinter().writeValueAsString(jsonSchema)); } catch (ClassNotFoundException cnfEx) { err.println("Unable to find class " + fullyQualifiedClassName); } catch (JsonMappingException jsonEx) { err.println("Unable to map JSON: " + jsonEx); } catch (JsonProcessingException jsonEx) { err.println("Unable to process JSON: " + jsonEx); } } When I run the demonstration class again with this modified method, the JSON output is more aesthetically pleasing: Generated JSON with Indentation Communicating Hierarchy { "type" : "object", "properties" : { "vegetable" : { "type" : "string", "enum" : [ "CARROT", "SQUASH", "SPINACH", "CELERY" ] }, "fruit" : { "type" : "string" }, "dessert" : { "type" : "string", "enum" : [ "PIE", "CAKE", "ICE_CREAM" ] } } } I have been using Jackson 2.5.4 in this post. The classcom.fasterxml.jackson.databind.jsonschema.JsonSchema is deprecated in that version with the comment, "Since 2.2, we recommend use of external JSON Schema generator module." Given that, I now look at using the new preferred approach (Jackson JSON Schema Module approach). The most significant change is to use the JsonSchema class in the com.fasterxml.jackson.module.jsonSchemapackage rather than using the JsonSchema class in the com.fasterxml.jackson.databind.jsonschema package. The approaches for obtaining instances of these different versions of JsonSchema classes are also different. The next code listing demonstrates using the newer, preferred approach for generating JSON from Java classes. Using Jackson's Newer and Preferred com.fasterxml.jackson.module.jsonSchema.JsonSchema /** * Write out JSON Schema based upon Java source code in * class whose fully qualified package and class name have * been provided. This method uses the newer module JsonSchema * class that replaces the deprecated databind JsonSchema. * * @param fullyQualifiedClassName Name of Java class upon * which JSON Schema will be extracted. */ private void writeToStandardOutputWithModuleJsonSchema( final String fullyQualifiedClassName) { final SchemaFactoryWrapper visitor = new SchemaFactoryWrapper(); final ObjectMapper mapper = new ObjectMapper(); try { mapper.acceptJsonFormatVisitor(mapper.constructType(Class.forName(fullyQualifiedClassName)), visitor); final com.fasterxml.jackson.module.jsonSchema.JsonSchema jsonSchema = visitor.finalSchema(); out.println(mapper.writerWithDefaultPrettyPrinter().writeValueAsString(jsonSchema)); } catch (ClassNotFoundException cnfEx) { err.println("Unable to find class " + fullyQualifiedClassName); } catch (JsonMappingException jsonEx) { err.println("Unable to map JSON: " + jsonEx); } catch (JsonProcessingException jsonEx) { err.println("Unable to process JSON: " + jsonEx); } } The following table compares usage of the two Jackson JsonSchema classes side-by-side with the deprecated approach shown earlier on the left (adapted a bit for this comparison) and the recommended newer approach on the right. Both generate the same output for the same given Java class from which JSON is to be written. /** * Write out JSON Schema based upon Java source code in * class whose fully qualified package and class name have * been provided. This method uses the deprecated JsonSchema * class in the "databind.jsonschema" package * {@see com.fasterxml.jackson.databind.jsonschema}. * * @param fullyQualifiedClassName Name of Java class upon * which JSON Schema will be extracted. */ private void writeToStandardOutputWithDeprecatedDatabindJsonSchema( final String fullyQualifiedClassName) { final ObjectMapper mapper = new ObjectMapper(); try { final com.fasterxml.jackson.databind.jsonschema.JsonSchema jsonSchema = mapper.generateJsonSchema(Class.forName(fullyQualifiedClassName)); out.println(mapper.writerWithDefaultPrettyPrinter().writeValueAsString(jsonSchema)); } catch (ClassNotFoundException cnfEx) { err.println("Unable to find class " + fullyQualifiedClassName); } catch (JsonMappingException jsonEx) { err.println("Unable to map JSON: " + jsonEx); } catch (JsonProcessingException jsonEx) { err.println("Unable to process JSON: " + jsonEx); } } /** * Write out JSON Schema based upon Java source code in * class whose fully qualified package and class name have * been provided. This method uses the newer module JsonSchema * class that replaces the deprecated databind JsonSchema. * * @param fullyQualifiedClassName Name of Java class upon * which JSON Schema will be extracted. */ private void writeToStandardOutputWithModuleJsonSchema( final String fullyQualifiedClassName) { final SchemaFactoryWrapper visitor = new SchemaFactoryWrapper(); final ObjectMapper mapper = new ObjectMapper(); try { mapper.acceptJsonFormatVisitor(mapper.constructType(Class.forName(fullyQualifiedClassName)), visitor); final com.fasterxml.jackson.module.jsonSchema.JsonSchema jsonSchema = visitor.finalSchema(); out.println(mapper.writerWithDefaultPrettyPrinter().writeValueAsString(jsonSchema)); } catch (ClassNotFoundException cnfEx) { err.println("Unable to find class " + fullyQualifiedClassName); } catch (JsonMappingException jsonEx) { err.println("Unable to map JSON: " + jsonEx); } catch (JsonProcessingException jsonEx) { err.println("Unable to process JSON: " + jsonEx); } } This blog post has shown two approaches using different versions of classes with name JsonSchema provided by Jackson to write JSON based on Java classes generated from an XSD with JAXB's xjc. The overall process demonstrated in this post is one approach for generating JSON Schema from XML Schema.

CoreOS Linux is the operating system for “Super Massive Deployments”. We wanted to see how easily we can get CoreOS logs into Elasticsearch / ELK-powered centralized logging service. Here’s how to get your CoreOS logs into ELK in about 5 minutes, give or take. If you’re familiar with CoreOS and Logsene, you can grab CoreOS/Logsene config files from Github. Here’s an example Kibana Dashboard you can get in the end: CoreOS Kibana Dashboard CoreOS is based on the following: Docker and rkt for containers systemd for startup scripts, and restarting services automatically etcd as centralized configuration key/value store fleetd to distribute services over all machines in the cluster. Yum. journald to manage logs. Another yum. Amazingly, with CoreOS managing a cluster feels a lot like managing a single machine! We’ve come a long way since ENIAC! There’s one thing people notice when working with CoreOS – the repetitive inspection of local or remote logs using “journalctl -M machine-N -f | grep something“. It’s great to have easy access to logs from all machines in the cluster, but … grep? Really? Could this be done better? Of course, it’s 2015! Here is a quick example that shows how to centralize logging with CoreOS with just a few commands. The idea is to forward the output of “journalctl -o short” to Logsene‘s Syslog Receiver and take advantage of all its functionality – log searching, alerting, anomaly detection, integrated Kibana, even correlation of logs with Docker performance metrics — hey, why not, it’s all available right there, so we may as well make use of it all! Let’s get started! Preparation: 1) Get a list of IP addresses of your CoreOS machines fleetctl list-machines 2) Create a new Logsene App (here) 3) Change the Logsene App Settings, and authorize the CoreOS host IP Addresses from step 1) (here’s how/where) Congratulations – you just made it possible for your CoreOS machines to ship their logs to your new Logsene app! Test it by running the following on any of your CoreOS machines: journalctl -o short -f | ncat --ssl logsene-receiver-syslog.sematext.com 10514 …and check if the logs arrive in Logsene (here). If they don’t, yell at us @sematext – there’s nothing better than public shaming on Twitter to get us to fix things. :) Create a fleet unit file called logsene.service [Unit] Description=Logsene Log Forwarder [Service] Restart=always RestartSec=10s ExecStartPre=/bin/sh -c "if [ -n \"$(etcdctl get /sematext.com/logsene/`hostname`/lastlog)\" ]; then echo \"Value Exists: /sematext.com/logsene/`hostname`/lastlog $(etcdctl get /sematext.com/logsene/`hostname`/lastlog)\"; else etcdctl set /sematext.com/logsene/`hostname`/lastlog\"`date +\"%Y-%%m-%d %%H:%M:%S\"`\"; true; fi" ExecStart=/bin/sh -c "journalctl --since \"$(etcdctl get /sematext.com/logsene/`hostname`/lastlog)\" -o short -f | ncat --ssl logsene-receiver-syslog.sematext.com 10514" ExecStopPost=/bin/sh -c "export D=\"`date +\"%Y-%%m-%%d %%H:%M:%S\"`\"; /bin/etcdctl set /sematext.com/logsene/$(hostname)/lastlog \"$D\"" [Install] WantedBy=multi-user.target [X-Fleet] Global=true Activate cluster-wide logging to Logsene with fleet To start logging to Logsene from all machines activate logsene.service: fleetctl load logsene.service fleetctl start logsene.service There. That’s all there is to it! Hope this worked for you! At this point all your CoreOS logs should be going to Logsene. Now you have a central place to see all your CoreOS logs. If you want to send your app logs to Logsene, you can do that, too — anything that can send logs via Syslog or to Elasticsearch can also ship logs to Logsene. If you want some Docker containers & host monitoring to go with your CoreOS logs, just pull spm-agent-docker from Docker Registry. Enjoy!

When creating a camel route using http, the destination might require a ssl connection with a self signed certificate. Therefore on our http client we should register a TrustManager that suports the certificate. In our case we will use the https4 component of Apache Camel Therefore we should configure the routes and add them to the camel context RouteBuilder routeBuilder = new RouteBuilder() { @Override public void configure() throws Exception { from("http://localhost") .to("https4://securepage"); } }; routeBuilder.addRoutesToCamelContext(camelContext); But before we proceed on starting the camel context we should register the trust store on the component we are going to use. Therefore we should implement a function for creating an ssl context with the trustore. Supposed the jks file that has the certificate imported is located on the root of our classpath. private void registerTrustStore(CamelContext camelContext) { try { KeyStore truststore = KeyStore.getInstance("JKS"); truststore.load(getClass().getClassLoader().getResourceAsStream("example.jks"), "changeit".toCharArray()); TrustManagerFactory trustFactory = TrustManagerFactory.getInstance("SunX509"); trustFactory.init(truststore); SSLContext sslcontext = SSLContext.getInstance("TLS"); sslcontext.init(null, trustFactory.getTrustManagers(), null); SSLSocketFactory factory = new SSLSocketFactory(sslcontext, SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER); SchemeRegistry registry = new SchemeRegistry(); final Scheme scheme = new Scheme("https4", 443, factory); registry.register(scheme); HttpComponent http4 = camelContext.getComponent("https4", HttpComponent.class); http4.setHttpClientConfigurer(new HttpClientConfigurer() { @Override public void configureHttpClient(HttpClientBuilder builder) { builder.setSSLSocketFactory(factory); Registry registry = RegistryBuilder.create() .register("https", factory) .build(); HttpClientConnectionManager connectionManager = new BasicHttpClientConnectionManager(registry); builder.setConnectionManager(ccm); } }); } catch (IOException e) { e.printStackTrace(); } catch (NoSuchAlgorithmException e) { e.printStackTrace(); } catch (CertificateException e) { e.printStackTrace(); } catch (KeyStoreException e) { e.printStackTrace(); } catch (KeyManagementException e) { e.printStackTrace(); } } After that our route would be able to access the destination securely.

I have just recorded a 5 minute video that demonstrates running an out of stock example from Apache Camel release, the camel-example-servlet packaged as a docker container and running on a kubernetes platform, such as openshift 3. camel-servlet-example scaled up to 3 running containers (pods) which is easy with kubernetes and fabric8 In this video I have already deployed the example and then demonstrates how we can use the fabric8 web console to manage our application. And also connect to the running container and see inside, such as the Camel routes visually as shown above. Then I run a simple bash script from my laptop that sends a HTTP GET to the Camel example and prints the response. The script runs in a endless loop and demonstrates how kubernetes can easily scale up and down multiple Camel containers and load balance across the running containers. And at the end its even self healing when I force killing docker containers. So I suggest to grab a fresh cup of tea or coffee and sit back and play the 5 minutes video. The video is hosted on vimeo and can be seen from this link.

You are working on a big project, developing rules, events and processes at your enterprise for mission critical business needs. Part of the requirements state that a certain business unit will be providing their data model for you to leverage. This data model will not be designed in the JBoss BPM Suite Data Modeler but you need to have access to it while working on your rules, events and processes from the business central dashboard. For this article we will be using the JBoss BPM Travel Agency demo project as a reference, with it's current data model built externally to the JBoss BPM Suite business central. The external data model is called the acme-data-model and is found in the project directory: This data model is built during installation and provides you with an object data model as a Java Archive (JAR) file which is installed into the JBoss BPM Suite business central component by placing it into the following location: jboss-eap-6.4/standalone/deployments/business-central.war/WEB_INF/lib/acmeDataModel-1.0.jar Authoring --> Artifact repository. This way of deploying the data model means that it is available to all projects you work on in JBoss BPM Suite business central, something that might not always be preferable. What we need is a way to deploy external data models into JBoss BPM Suite and then selectively add them to projects as needed. Within JBoss BPM Suite there is an Artifact Repository that is made just for this purpose. We can upload through the business central dashboard UI all our models and then pick and choose from the repository artifacts (your data model is one artifact) on a per project basis. This gives you absolute control over the models that a project can access. Choose external data model file. There are a few steps involved that we will take you through here to change the current installation of JBoss BPM Travel Agency where the acmeDataModel-1.0.jar file will be removed from the previously mentioned business central component and uploaded into the Artifact Repository and added to the Special Trips Agency project. Here is how you can do it yourself: obtain and install JBoss BPM Travel Agency demo project remove current data model from global business central application: $ rm ./target/jboss-eap-6.4/standalone/deployments/business-central.war/WEB_INF/lib/acmeDataModel-1.0.jar Upload external model jar file. start JBoss BPM Suite server after installation as stated in the installation instructions login to JBoss BPM Suite at http://localhost:8080/business-centralwith: u: erics p: bpmsuite1! go to AUTHORING --> ARTIFACT REPOSITORY go to UPLOAD --> CHOOSE FILE... --> projects/acme-data-model/target/acmeDataModel-1.0.jar --> click button to UPLOAD this puts the external data model into the JBoss BPM Suite artifact repository Select dependencies to add to project. got to AUTHORING --> PROJECT AUTHORING --> OPEN PROJECT EDITOR in project editor select GENERAL PROJECT SETTINGS --> DEPENDENCIES in dependencies select ADD FROM REPOSITORY -> in pop-upSELECT entry acmeDataModel-1.0.jar This will result in the external data model being added only to the Special Trips Agency project and not available to other projects unless they add this same dependency from the JBoss BPM Suite artifact repository. If you build & deploy the project, run it as described in the project instructions you will find that the external data model is available and used by the various rules and process components that are the JBoss BPM Travel Agency. As a closing note, this works exactly the same for JBoss BRMS projects.

Continuing on from my blog post about scraping Neo4j’s release dates I thought it’d be even more interesting to chart the release dates of some github projects. In theory the release dates should be accessible through the github API but the few that I looked at weren’t returning any data so I scraped the data together. We’ll be using rvest again and I first wrote the following function to extract the release versions and dates from a single page: library(dplyr) library(rvest) process_page = function(releases, session) { rows = session %>% html_nodes("ul.release-timeline-tags li") for(row in rows) { date = row %>% html_node("span.date") version = row %>% html_node("div.tag-info a") if(!is.null(version) && !is.null(date)) { date = date %>% html_text() %>% str_trim() version = version %>% html_text() %>% str_trim() releases = rbind(releases, data.frame(date = date, version = version)) } } return(releases) } Let’s try it out on the Cassandra release page and see what it comes back with: > r = process_page(data.frame(), html_session("https://github.com/apache/cassandra/releases")) > r date version 1 Jun 22, 2015 cassandra-2.1.7 2 Jun 22, 2015 cassandra-2.0.16 3 Jun 8, 2015 cassandra-2.1.6 4 Jun 8, 2015 cassandra-2.2.0-rc1 5 May 19, 2015 cassandra-2.2.0-beta1 6 May 18, 2015 cassandra-2.0.15 7 Apr 29, 2015 cassandra-2.1.5 8 Apr 1, 2015 cassandra-2.0.14 9 Apr 1, 2015 cassandra-2.1.4 10 Mar 16, 2015 cassandra-2.0.13 That works pretty well but it’s only one page! To get all the pages we can use the follow_link function to follow the ‘Next’ link until there aren’t anymore pages to process. We end up with the following function to do this: find_all_releases = function(starting_page) { s = html_session(starting_page) releases = data.frame() next_page = TRUE while(next_page) { possibleError = tryCatch({ releases = process_page(releases, s) s = s %>% follow_link("Next") }, error = function(e) { e }) if(inherits(possibleError, "error")){ next_page = FALSE } } return(releases) } Let’s try it out starting from the Cassandra page: > cassandra = find_all_releases("https://github.com/apache/cassandra/releases") Navigating to https://github.com/apache/cassandra/releases?after=cassandra-2.0.13 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-2.0.10 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-2.0.8 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-1.2.13 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-2.0.0-rc1 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-1.2.3 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-1.2.0-beta2 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-1.0.10 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-1.0.6 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-1.0.0-rc2 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-0.7.7 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-0.7.4 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-0.7.0-rc3 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-0.6.4 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-0.5.0-rc3 Navigating to https://github.com/apache/cassandra/releases?after=cassandra-0.4.0-final > cassandra %>% sample_n(10) date version 151 Mar 13, 2010 cassandra-0.5.0-rc2 25 Jul 3, 2014 cassandra-1.2.18 51 Jul 27, 2013 cassandra-1.2.8 21 Aug 19, 2014 cassandra-2.1.0-rc6 73 Sep 24, 2012 cassandra-1.2.0-beta1 158 Mar 13, 2010 cassandra-0.4.0-rc2 113 May 20, 2011 cassandra-0.7.6-2 15 Oct 24, 2014 cassandra-2.1.1 103 Sep 15, 2011 cassandra-1.0.0-beta1 93 Nov 29, 2011 cassandra-1.0.4 I want to plot when the different releases happened in time and in order to do that we need to create an extra column containing the ‘release series’ which we can do with the following transformation: series = function(version) { parts = strsplit(as.character(version), "\\.") return(unlist(lapply(parts, function(p) paste(p %>% unlist %>% head(2), collapse = ".")))) } bySeries = cassandra %>% mutate(date2 = mdy(date), series = series(version), short_version = gsub("cassandra-", "", version), short_series = series(short_version)) > bySeries %>% sample_n(10) date version date2 series short_version short_series 3 Jun 8, 2015 cassandra-2.1.6 2015-06-08 cassandra-2.1 2.1.6 2.1 161 Mar 13, 2010 cassandra-0.4.0-beta1 2010-03-13 cassandra-0.4 0.4.0-beta1 0.4 62 Feb 15, 2013 cassandra-1.1.10 2013-02-15 cassandra-1.1 1.1.10 1.1 153 Mar 13, 2010 cassandra-0.5.0-beta2 2010-03-13 cassandra-0.5 0.5.0-beta2 0.5 37 Feb 7, 2014 cassandra-2.0.5 2014-02-07 cassandra-2.0 2.0.5 2.0 36 Feb 7, 2014 cassandra-1.2.15 2014-02-07 cassandra-1.2 1.2.15 1.2 29 Jun 2, 2014 cassandra-2.1.0-rc1 2014-06-02 cassandra-2.1 2.1.0-rc1 2.1 21 Aug 19, 2014 cassandra-2.1.0-rc6 2014-08-19 cassandra-2.1 2.1.0-rc6 2.1 123 Feb 16, 2011 cassandra-0.7.2 2011-02-16 cassandra-0.7 0.7.2 0.7 135 Nov 1, 2010 cassandra-0.7.0-beta3 2010-11-01 cassandra-0.7 0.7.0-beta3 0.7 Now let’s plot those releases and see what we get: ggplot(aes(x = date2, y = short_series), data = bySeries %>% filter(!grepl("beta|rc", short_version))) + geom_text(aes(label=short_version),hjust=0.5, vjust=0.5, size = 4, angle = 90) + theme_bw() An interesting thing we can see from this visualisation is what overlap the various series of versions have. Most of the time there are only two series of versions overlapping but the 1.2, 2.0 and 2.1 series all overlap which is unusual. In this chart we excluded all beta and RC versions. Let’s bring those back in and just show the last 3 versions: ggplot(aes(x = date2, y = short_series), data = bySeries %>% filter(grepl("2\\.[012]\\.|1\\.2\\.", short_version))) + geom_text(aes(label=short_version),hjust=0.5, vjust=0.5, size = 4, angle = 90) + theme_bw() From this chart it’s clearer that the 2.0 and 2.1 series have recent releases so there will probably be three overlapping versions when the 2.2 series is released as well. The chart is still a bit cluttered although less than before. I’m not sure of a better way of visualising this type of data so if you have any ideas do let me know!

The growth of Azure has been outstanding--more than 90,000 new subscriptions every month. And the innovation is exponential with over 500 new features and services being added to the platform in the last 12 months. We're very excited to be part of this growth. As we announced yesterday, you can now access Stackato through Azure. We think it's a great way for Azure customers to get access to a Cloud Foundry and Docker based PaaS. With Azure, Microsoft provides an easy path to the cloud for their customers. All applications can be run on one cloud. Microsoft wants to dominate the cloud the same as it has with on-premise software and rarely does a day go by without reading an article about Azure. Whether it's their recent announcement to help encourage start-up's use of Azure by providing $120,000 worth of credits per year or their commitment to open source. Azure gives its customers a growing collection of integrated services that make it easier to build and manage enterprise, mobile, web and Internet of Things (IoT) apps faster. Enterprises face real complexities when building their cloud solution. Having a solid infrastructure is really just the first step in the process--companies also need the right platform to support the deployment and management of their cloud-native applications. The platform should give their developers the freedom to use the language best suited to build the application. In addition, enterprises are on more than one cloud. They need to have the versatility to scale out or move their applications to whatever cloud is appropriate in order to meet end user demand without any downtime. With Stackato, we help remove these complexities. We provide enterprises with a polyglot PaaS that supports the development of applications in virtually any language. We like to refer to Stackato as being "infrastructure-agnostic" and allow companies to deploy their applications to any cloud--private, public or hybrid--without the need to run new scripts or re-package the application in order for it to work in the new environment. The combination of Stackato on Azure gives enterprises the technology they need to streamline application delivery, drive innovation and meet the demands of their customers.