DBmaestro, the pioneer and leading provider of DevOps for database solutions, has announced that TeamWork’s extension for Oracle SQL Developer 4.1 has been released, just weeks after Oracle released its latest version. DBmaestro TeamWork is now the only tool with an external extension that supports version 4.x of Oracle’s database development tool. TeamWork, DBmaestro’s flagship product, enables agile development and continuous integration & delivery for the database. TeamWork supports streamlining of development process management and enforcing change policy practices. Many leading enterprises use DBmaestro to facilitate DevOps for their database by executing deployment automation, enhancing and reinforcing security, and mitigating risk. DBmaestro’s extension for SQL Developer 4.x helps Oracle developers and DBAs streamline database development, collaborate across database teams, and allows for agile database development in an efficient and reliable way. Upon the release of SQL Developer 4.0, Oracle required that all extensions be updated to be compatible with the new version’s API. This is a result of the drastic changes made to JDeveloper, on which SQL Developer is built. “The SQL Developer 4.1 extension represents an important achievement for DBmaestro,” said Yaniv Yehuda, co-founder and CTO of DBmaestro. “Oracle SQL Developer has over 4 million active users and is the de facto standard database IDE tool out there. Oracle has drastically changed their API on the 4.0 version, which presented a challenge to those seeking to update third-party extensions. This integration is a statement of our commitment to our customers, and we will continue to lead the way to achieve DevOps for the databases.”

REST is all the rage. It provides a networked approach to exposing data and services, both internally within a company and to partners. Here, we explore a novel approach to providing a RESTful API to a set of existing commands. Business Problem: expose Commands via REST Many organizations have made considerable software available through Unix-like commands. Imagine an Enterprise, well, the Enterprise, that wishes to provide RESTful APIs to commands like: OrderSickLeave , IncreaseSpeed , SelfDestruct … hundreds of them… Each of these commands is backed by some software, say a Java Class of some sort. They’d like to publish this functionality, as follows: Restify – provide a RESTful API for each command. The REST attributes are the command arguments, above Validate – before the commands is issued, validate the arguments (does exist, is the a positive number, etc) Secure – ensure the commands are available as appropriate Background: Espresso creates RESTful API for SQL data Espresso creates RESTful APIs, principally for SQL data. Some key functionality is described below. API Creation You connect Espresso to a database, and it creates a RESTful endpoint for each table, view and Stored Procedure. The API supports the usual RESTful operations of Get (including filters based on HTTP arguments), and update (Put, Post and Delete). Validation Rules You can specify validation rule expressions that are automatically executed on update requests. For example, you might specify a rule that balance <= creditLimit. Validation rules are part of the larger notion of business logic, which includes computations (the balance is the sum of the unpaid order amounts). Computations can chain, and validations work over computed data, so you can solve remarkably complicated problems with a few simple rules. Extensibility Espresso creates a server-side JavaScript Domain Object for each table, providing access to attributes (customer.balance), and persistence (customer.save()). The objects encapsulate their integrity in two ways: Rule Invocation – update requests trigger the relevant validation / computation rules Events – you can also associate server-side JavaScript event handlers with a table. These fire as update requests are processed The Event handlers can invoke anything in the JVM, such as jar file code you can load into Espresso, other RESTful services, etc. Background: Command Pattern Imagine a word processor, providing functionality to make a string of selected text bold, another bit of functionality for italics, and so forth. Now imagine we wish to provide Undo functionality. Not possible unless we save the commands. So, the Command Pattern emerged: create a class for each Command creating an instance “does” the command (e.g., make text bold), where the constructor arguments provide the necessary information (text start, length) maintain an ordered list of commands (object instances) require the Command Classes to provide an Undo method to reverse the effects of the constructor (e.g., remove the bold) Database Adaption: insert logic on a Request Table We can see this same pattern in database applications, where we wish to keep a record of transactions. For example, instead of just changing a salary (and forgetting who, when, old values etc), we create a SalaryAction(employeeId, raisePercent). We can think of this as a Request Table. The logic on the SalaryAction table might give the raise, but also validate (is the percent too big? is the resultant salary out of range?), and record who, when etc. Restify Commands with Request Tables So we can combine these notions: Create a database with table for each Command (OrderSickLeave, IncreaseSpeed, etc), with columns for command arguments. We can also provide columns for admin data (date executed, etc) Leverage Espresso to Restify these Declare Validation Logic JavaScript table events execute the command We can now use this approach to Restify our Enterprise commands, as described below. Command Requests Database The lower 3 tables are the Request Tables. They each have Foreign Keys to the Crew table, simply identifying the source of the request (this is optional). Load into Espresso We next connect Espresso, which discovers these tables and creates a Default RESTful API to them. We can view it in a test tool immediately: We can also define a Custom API, exposing just the elements we wish: Validate Command Parameters The following rules apply to Posts against our Command Request tables. Here we ensure that the IncreaseSpeed request has a argument (attribute) for warpFactor, that it be between 0 and 10, and that the optional emergencyReason parameter is supplied if the warpFactorexceeds 8 Live Browser – view, post data You can use the API tool, or this automatically constructed app: Invoke existing code Finally, we add server-side JavaScript logic (stubbed here) to execute our command. This would execute existing code in a loadable jar. Conclusion: 4 hour project This project was completed in about a half a day.

Software Driven Business is a consensus goal. But real challenges exist: the time, cost and complexity of building such apps is substantial. Business Agility – and strategic business advantage – is lost. We need another revolution – Instant Enterprise REST – that provides Business Agility using business-level specifications rather than low-level code, and delivers Enterprise-class scalability, integration, enforcement and extensibility. It’s now a reality with Instant Enterprise REST. Software Driven Business: Consensus Vision Businesses have seen the value in providing mobile and tablet apps that bring the business into the hands of customers and employees. They provide information at their finger tips – wherever they are. Industry Leaders like CA have pioneered the vision of a Software Driven Business. They argue persuasively that strategic business advantage lies in Time to Market and Time to Decision: “reveal the need for speed in the application economy. As companies transform into software-driven enterprises, bringing high-quality applications to market faster becomes one of the most critical differentiators.” The Business Agility Gap While there is consensus around this vision, there is a substantial gap in realizing the Software Driven Business. It centers around Agility – time to market. As CA argues, this drives strategic business advantage. This problem manifests both to Business Users and IT, although differently. You might have been party to a discussion like this: Business Users are frustrated about how long it takes to create systems, and revise them. They see problems that look nearly as simple as a spreadsheet take weeks… to months. How can it months for IT to build a system that takes days on a spreadsheet? IT is no less frustrated. They understand the deep technology it takes to build Enterprise-class systems: We’re working 90 hours a week. And falling behind. Gap Analysis For apps about critical corporate data, there’s general consensus that the time and cost for such systems are about evenly split between backends and front ends. And there’s nearly universal consensus that, independent of the UI technology, that RESTful APIs deliver the backend data. But the backend is far more than basic data access. A “SQL Pass-through” – simply restifying SQL data – does not meet Enterprise-class requirements to scale, integrate and enforce: Scale – APIs require Pagination to address large result sets, Nested Documents to reduce latency, Optimistic Locking to ensure concurrency. These are not provided in a simple SQL Pass-through – you must program them, by hand. Integrate – a wizard can produce an API from schema objects, but it cannot address multiple databases, or integrate non-SQL data sources such as ERP, other RESTful services, or NoSQL. Enforce – an API needs to enforce our security (down to the row level), and the integrity of the data. These are significant tasks, which are sadly often placed in client buttons where they cannot be shared. Providing these Enterprise class services takes significant time, expertise and expense. Business Agility is reduced. IT is essentially being forced to cover inadequate technology infrastructure. The Business Users are right: if the Business Specification is clear, then that ought to be enough: A clear business specification should be sufficient. Everything else is just friction. The vision of the Software Driven Business requires Business Driven Software that pre-supplies the infrastructure. We are not seeking 10 or 15%. We are looking for orders of magnitude. Our vision must be: We should be able to create RESTful APIs (mainly) from business specifications, not low level code. It should be no more difficult to create a system than it is toimagine it. Business-Driven Software: Instant Enterprise REST Business Driven Software is more than just a clever play on words. It’s a real implementation that delivers this vision, and we call it Instant Enterprise REST. It consists of 3 core technologies: Enterprise Pattern Automation – creates APIs that with Enterprise-class scalability built-in (pagination, nested documents, optimistic locking, etc) Declarative – specify your API, integration and enforcement policies with spreadsheet-like rules in a simple point-and-click UI Extensibility – enables the RESTful APIs to invoke your existing logic, inside or outside the JVM, via standard server-side JavaScript. The combination of these 3 technologies enables you to create RESTful APIs for database backends – half your system – 10 times faster. Let’s briefly examine them below. Technology 1: Enterprise Pattern Automation There are well known patterns in the data domain, describing data structure and access via SQL. There are also well-known patterns for managing SQL data in the context of RESTful services. Well known patterns can be automated. Let’s imagine a service (say, a server accessed via a browser) that automates these patterns, as described below, just by connecting the service to a database: Schema Discovery – tables, views, stored procedures: The system creates a complete (default) API for each schema object. Note this includes Stored Procedures, which often represent a significant investment. Enterprise Pattern Automation: the resultant API provides well-known services for Filter, Sort, Pagination, Optimistic Locking, handling Generated Keys and so forth. So, the service has provided a default Enterprise-class API, instantly. So, literally seconds into your project, you can test your running API: Not enough, not done, but a great start. Technology 2: Declarative Declarative is the key (“what, not how”). It has had striking impacts on domains where there are well-understood underlying patterns. Max Tardiveau has put it well: Whatever can be declarative, will be declarative. For example, spreadsheets are declarative – and they gave birth to the PC industry. And SQL is declarative – itself an industry. Two game-changers. So, the challenge is to apply the spirit of declarative to REST integration and enforcement. The stakes are high – success can deliver breathtaking agility. Declarative Integration: Multi-Database Custom API, Point and Click Enterprise Pattern Automation provides a good start, but the API is not rich. It is a flat, single-table API, really just “restified” SQL. What we really need is Nested Documents – returning multiple types (e.g., an Order, a list of Items, and a list of contact names) in a single call can reduce latency (vs. a separate call for each type). REST is perfect for this. Multi-database APIs – a RESTful server provides the opportunity to integrate multiple databases in single call, shielding clients from underlying complexity. Nested Documents are easy: define them by simply selecting tables (via a User Interface or Command Line). Foreign Keys are used to default the joins. Add the ability to choose / alias columns, and we’re on the way to a pretty good API. But what about databases that have no Foreign Keys? Or multi-database APIs? Leveraging the schema does not mean we are limited to it. All we need to do is: Provide a means to define “Virtual” Foreign Keys for the service (i.e., stored outside the schema) Extend this to Foreign Keys between databases We now have a rich, multi-database API. Defined declaratively as shown below, no code required, running in minutes, ready for client development: Declarative Enforcement: Integrity Logic, with spreadsheet-like rules So now consider enforcement, specifically database integrity. A very significant portion of any project is the multi-table validations and computations that define how the data is processed. “Your code goes here” means, well, a lot of code. We need a more powerful, more declarative, paradigm. In a spreadsheet, you assign expressions to cells. Whenever the referenced data is changed, the cell is updated. Since the cells references can chain, a series of simple expressions can solve remarkably complex problems. What if we did the same for database data? We could assign derivation expressions to columns, and validation expressions to tables. Then, the API could “watch” for requests that change the referenced column, and recompute (efficiently) the calculated column. Just as in a spreadsheet, support for chaining and proper ordering is required and implicit. To address multi-table logic, such expressions would need to address references to related tables. It’s only at this point that the logic becomes seriously powerful. Let’s take an example. To check credit in a Customer / Purchaseorder / Lineitem application, we could define spreadsheet-like expressions such as: There is actually a sub-branch of declarative that addresses this: Reactive Programming. Here it’s declarative,since you don’t need to code a Observer handler. The result is that the logic above can be fully executable. No need to code Change Detection / Change Dependency – it’s invoked and enforced automatically by the API in reaction to RESTful updates. SQL handling is also implicit, including underlying optimizations (caching, pruning etc). The impact is massive – the 5 expressions above express the same logic as hundreds of lines of code. That’s a massive 40X more concise. Game changer. And quality goes up, since the rules are applied automatically. Declarative Enforcement: Security, filter expressions for role/table We can provide an analogous approach to security: define filter expressions for roles (like SalesRep), so that when a table is accessed by the role, the API adds the filter. That way, a user with that role sees only the rows for which they are authorized. Technology 3: Standards-based Extensibility Declarative is great, but you’re probably thinking “ok, but you can’t solve every problem declaratively”. And you’re dead right. Business Value requires that we integrate a declarative approach with a procedural one that is familiar, standards-based, and enables us to integrate existing software. Automatic JavaScript Object Model The first phase of many projects is to build an ORM for natural programmatic access to data: JPA, Hibernate, Entity Framework. It’s not a small project, and cumbersome to maintain as changes occur. In fact, the Object Model can be created directly from the schema. So, you’d have an object type for Purchaseorder, for Lineitem, and so forth. The model provides access to attributes and related data, and persistence services. You could then use it as shown below. JavaScript seems like the best language choice: reasonable across technology bases (everybody uses JavaScript), and its dynamic nature eliminates code generation hassles. JavaScript Events In addition to accessors and persistence, the JavaScript objects are Logic Aware. That is, the save operation above executes any rules associated with OrderAudit (e.g., updated-by), and JavaScript Events. Here is a sample event for the PurchaseOrder object, where you access the JavaScript Object Model via the system-supplied row variable: Extensible Logic Auditing is a common pattern. It should be possible to solve this once in a genericmanner, then re-use it (e.g, to audit employees, orders and so forth). So, Instant Enterprise REST should enable you to provide Extensible Logic – load your own JavaScript code, and invoke it. So, the code above could become: MyLibrary.auditFromTo(orderRow,"OrderAudit"); where auditFromTo creates an instance of OrderAudit, sets the foreign key, sets like-named attributes, and saves it. Pluggable Authentication Most organizations have existing data stores that identify users and their roles, such as Active Directory, LDAP, OAuth, etc. Security should integrate with such systems as a function of enforcing row/column access. Standard deployment Finally, the system should deploy in a familiar manner: available on the cloud, or an on-premise virtual appliance or war file. Standards also enable integration with related critical infrastructure, such as API Management, ERP Systems, etc. See a project in 3 minutes To see how it all fits together, you can view this video to see a full project built: from concept, through initial implementation, and an iteration cycle. Actual project time was about half an hour. Instant Enterprise REST: Business Agility Instant Enterprise REST enables us to close the Agility Gap in realizing the Software Driven Business vision. We can now create important portions of our software in largely business terms, rather than technical terms. This offers major advantages: Time to Market: spreadsheet-like rules are 40X more concise. Instant REST eliminates all the SQL / REST / JSON boilerplate. Simplicity: team members can learn the basics of Espresso in days, and be as productive as rocket scientists using alternative technologies Leverage Expertise and Software: Espresso is built on standards like REST, JavaScript, and Event Oriented Programming. You can call out to existing software, and extend the rule types by identifying your own patterns and loading their implementations into Espresso. Quality: at the defect level, automatic invocation and ordering eliminate large classes of bugs. At the architectural level, centralized enforcement factors logic out of the client buttons where it can be shared, audited for compliances, etc

Recently we started working on including support for Azure Service Bus in Cloud Portam. Prior to this, I had no experience with this service though it has been around for quite some time and I always wanted to try this out but one thing or another (oh, my stupid excuses :)!) prevented me from doing so. I learned a lot (and I am still learning) about this service while including support for it in Cloud Portam and this blog post talks about my learning. Please note that at the time of writing of all in all I have about a week of learning about this service so it is quite possible that I may be wrong about certain things. If that’s the case, please let me know and I will fix them ASAP. Now that the tone is set, let’s start! Azure Service Bus Offering The way I understand is that “Azure Service Bus” is a cloud-based messaging service that enables you to connect virtually anything – be it applications, services or devices. The beauty of Service Bus is that these things need not be in the cloud. They can run anywhere even inside the firewalled networks! Another thing I learned is that “Azure Service Bus” is essentially an umbrella service. At the time of writing of this post, there are actually four distinct services that are collectively offered under “Service Bus” umbrella – Queues, Topics & Subscriptions, Relays and Notification Hubs. Each service serves a different purpose yet the common theme is that all of them provide rich messaging infrastructure. To give you an analogy, if you have used Azure Storage Service you may already know that it offers four distinct services – Blobs, Files, Queues and Tables. It is the same with Service Bus as well. Queues Queues is the simplest of the service and kind of compares with Azure Storage Queue Service in the sense that it provides a unidirectional messaging infrastructure where a publisher publishes a message and the message is received by a receiver. There can be many receivers ready to receive the messages however one receiver can only receive a message. No two receivers can receive a single message simultaneously. For an in-depth comparison of Service Bus Queue and Storage Queues, please see this link: https://msdn.microsoft.com/en-us/library/azure/hh767287.aspx. Topics Topics are like queues in the sense that it also provides a unidirectional messaging infrastructure where a publisher publishes a message and receivers receive the message. The key difference is that same message can be received by multiple receivers (subscribers). Each subscriber can optionally specify a filter criteria so that they only receive the messages matching that criteria. To understand the difference between the two, let’s consider an example. Let’s say you run an e-commerce site and on successful completion of order, you have two tasks: 1) Send an email to customer about the order and 2) Notify the warehouse. If you were using Queues, you would either create 2 queues and put email notification message in one queue and warehouse notification message in another queue or build a workflow where you would send order confirmation message to a queue. Receiver would take that message and send out an email and then put warehouse notification message in the same queue (or other queue) and then another receiver would receive the message and notify the warehouse. However if you were using Topics, things would be much simpler logistically speaking. Essentially you would have just one message (order confirmation) but there will be two subscribers – one will be responsible for sending the email confirmation and the other will be responsible for notifying the warehouse. Relays Unlike Queues and Topics, which provide unidirectional flow of messages a Relay provides bi-directional flow. Using Relays, two disparate applications, services or devices can exchange messages. Other key difference is that a Relay doesn’t store the message like Queues and Topics. It just passes the messages from source to destination. Event Hubs Event Hubs service is meant for ingesting events and telemetry data in the cloud at massive scale (millions of events / second). Event Hubs are now more than important considering the push for connected devices (Internet-of-Things). Azure Service Bus Tiers Azure Service Bus is offered under two tiers (or SKUs if you would like): Basic and Standard. The difference is the level of functionality offered in each tier and the pricing. For example, Topics, Relays and Notification Hubs are only offered under Standard tier. Even with Queues, a limited set of functionality is exposed under Basic tier. For a list of features offered under each tier, please see this link: http://azure.microsoft.com/en-in/pricing/details/service-bus/. Summary That’s it for this post. In the next posts in this series, I will share my learnings about Queues and other Service Bus services. So stay tuned for that! Again, if you think that I have provided some incorrect information, please let me know and I will fix them ASAP.

introduction to explain why you have to use parameterized query to avoid sql injection over concatenated inline query it needs to know about sql injection. what does sql injection mean? it means when any end user send some invalid inputs to perform any crud operation or forcibly execute the wrong query into the database, those can be harmful for the database. harmful means ‘data loss’ or ‘get the data with invalid inputs. to know more, follow the below steps. step 1: create a table named ‘login’ in any database. create table user_login ( userid varchar(20), pwd varchar(20) ) now save some user credentials into the database for login purpose and select the table. insert into user_login values('rahul','bansal@123') insert into user_login values('bansal','rahul@123') step 2: create a website named ‘website1’. now i will create a login page named ‘default.aspx’ to validate the credentials from the ‘login’ table and if user is valid then redirect to it to the next page named ‘home.aspx’. add 2 textboxes for userid & password respectively and a button for login. add 2 namespaces in the .cs file of the ‘default.aspx’. using system.data.sqlclient; using system.data; now add the following code to validate the credentials from the database on click event of login button. protected void btn_login_click(object sender, eventargs e) { string constr = system.configuration.configurationmanager.connectionstrings["constr"].connectionstring; sqlconnection con = new sqlconnection(constr); string sql = "select count(userid) from user_login where userid='" + txtuserid.text + "' and pwd='" + txtpwd.text + "'"; sqlcommand cmd = new sqlcommand(sql, con); con.open(); object res = cmd.executescalar(); con.close(); if (convert.toint32(res) > 0) response.redirect("home.aspx"); else { response.write("invalid credentials"); return; } } add a new page named ‘home.aspx’. where any valid user will get welcome message. step 3: now run the ‘default’ page and log in with valid credentials. it will redirect to next page ‘home.aspx’ for valid user. note: here i have not used the textmode="password" property in password textbox to show the password. i have not used any input validations to explain my example. problem: now i will perform the sql injection with some invalid credentials with successful query execution and after that i will redirect to the next page ‘home.aspx’ as a valid user. i will enter a string in both textboxes like the following: ‘ or ‘1’=’1 now run the page and login with above string in both textboxes. it will redirect to next page name ‘home.aspx’ for valid user. see what happened. this is called sql injection in the hacking world. reason: it happened just because of the string and after filling this string in both textboxes orur sql query became like the following: select count(userid) from user_login where userid='' or '1'='1' and pwd='' or '1'='1' which will give the userid count and that is 2 in the table because 2 users are in ‘user_login’ table. it can be used in more ways like just fill the following string only in user id textbox and you will go the next page as valid user. or 1=1 - - and it will also give users count 2 because sqlquery will become like the following: select count(userid) from user_login where userid='' or 1=1 --' and pwd='' or '1'='1' note: the sign -- are for commenting the preceding text in sql. it can be more harmful or dangerous when the invalid user/hacker executes a script to drop all tables in the database or drop whole database. solution: to resolve this issue you have to do 2 things: always use parameterized query. input validations on client and server both side. sometimes if your input validation fail, then parameterized will not execute any scripted value. let’s see the example. protected void btn_login_click(object sender, eventargs e) { string constr = system.configuration.configurationmanager.connectionstrings["constr"].connectionstring; sqlconnection con = new sqlconnection(constr); string sql = "select count(userid) from user_login where userid=@userid and pwd=@pwd"; sqlcommand cmd = new sqlcommand(sql, con); sqlparameter[] param = new sqlparameter[2]; param[0] = new sqlparameter("@userid", txtuserid.text); param[1] = new sqlparameter("@pwd", txtpwd.text); cmd.parameters.add(param[0]); cmd.parameters.add(param[1]); con.open(); object res = cmd.executescalar(); con.close(); if (convert.toint32(res) > 0) response.redirect("home.aspx"); else { response.write("invalid credentials"); return; } } now if i run the page and try to login with sql scripts as done earlier. with ‘ or ‘1’=’1 with ' or 1=1 - - as you have seen parameterized didn’t execute the sql script but why? reason: the reason behind this the parameterized query would not be vulnerable and would instead look for a user id or password which literally matched the entire string. in other words ‘the sql engine checks each parameter to ensure that it is correct for its column and are treated literally, and not as part of the sql to be executed’. conclusion: always use parameterized query and input validations on client and server both side.

It’s a sad but true fact: Most data scientists spend 50-80% of their time cleaning and munging data and only a fraction of their time actually building predictive models. This is most often true in a traditional stack, where most of this data munging consists of writing lines upon lines of some flavor of SQL, leaving little time for model-building code in statistical programming languages such as R. These long, cryptic SQL queries not only slow development time but also prevent useful collaboration on analytics projects, as contributors struggle to understand each others’ SQL code. For example, in graduate school, I was on a project team where we used Oracle to store Twitter data. The kinds of queries my classmates and I were writing were unmaintainable and impossible to understand unless the author was sitting next to you. No one worked on the same queries together because they were so unwieldy. This not only hindered our collaboration efforts but also slowed our progress on the project. If we had been using an appropriate data store (like a graph database) we would have spent significantly less time pulling our hair out over the queries. Why Today’s Data Is Different This data-munging problem has persisted in the data science field because data is becoming increasingly social and highly-connected. Forcing this kind of interconnected data into an inherently tabular SQL database, where relationships are only abstract, leads to complicated schemas and overly complex queries. Yet, several NoSQL solutions – specifically in the graph database space – exist to store today’s highly-connected data. That is, data where relationships matter. A lot of data analysis today is performed in the context of better understanding people’s behavior or needs, such as: How likely is this visitor to click on advertisement X? Which products should I recommend to this user? How are User A and User B connected? Written by Nicole White People, as we know, are inherently social, so most of these questions can be answered by understanding the connections between people: User A is similar to User B, and we already know that User B likes this product, so let’s recommend this product to User A. The Good News: Data-Munging No More Data science doesn’t have to be 80% data munging. With the appropriate technology stack, a data scientist’s development process is seamless and short. It’s time to spend less time writing queries and more time building models by combining the flexibility of an open-source, NoSQL graph database with the maturity and breadth of R – an open-source statistical programming language. The combination of Neo4j’s ability to store highly-connected, possibly-unstructured data and R’s functional, ad-hoc nature creates the ideal data analysis environment. You don’t have to spend an hour writing CREATE TABLE statements. You don’t have to spend all day on StackOverflow figuring out how to traverse a tree in SQL. Just Cypher and go. Learn More at OSCON 2015 At my upcoming OSCON session we will walk through a project in which we analyze #OSCON Twitter data in a reproducible, low-effort workflow without writing a single line of SQL. For this highly-connected dataset we will use Neo4j, an open-source graph database, to store and query the data while highlighting the advantages of storing such data in a graph versus a relational schema. Finally, we will cover how to connect to Neo4j from an R environment for the purposes of performing common data science tasks, such as analysis, prediction and visualization.

A huge amount has happened recently. Following the official launch of CUBA on 1st of June, we have rolled out a new release, published our first article on a few Java sites and presented the platform at the Devoxx UK сonference in London. But before the rush continues, about it is an apt time to articulate the philosophy behind CUBA. The first words associated with enterprise software development will probably be: slow, routine, complex and convoluted - nothing exciting at all! A common approach to combat these challenges is raising the level of abstraction - so that developers can operate with interfaces and tools encapsulating internal mechanisms. This enables the focus on high-level business requirements without the need to reinvent common processes for every project. Such a concept is typically implemented in frameworks, or platforms. The previous CUBA article explained why CUBA is more than just a bunch of well-known open-source frameworks comprehensively integrated together. In brief, it brings declarative UI with data aware visual components, out-of-the-box features starting from sophisticated security model to BPM, and awesome development tools to complement your chosen IDE. You can easily find more details on our Learn page, so instead of listing all of them I'll try to "raise the abstraction level" and explain the fundamental principles of CUBA. Practical The platform is a living organism, and its evolution is mostly driven by specific requests from developers. Of course, we constantly keep track of emerging technologies, but we are rather conservative and employ them only when we see that they can bring tangible value to the enterprise software development. As a result, CUBA is extremely practical; every part of it has been created to solve some real problem. Integral Apart from the obvious material features, the visual development environment provided by CUBA Studio greatly reduces the learning curve for newcomers and juniors. It is even more important that the platform brings a unified structure to your applications. When you open a CUBA-based project, you will always know where to find a screen, or a component inside of it; where the business logic is located and how is it invoked. Such an ability to quickly understand and change the code written by other developers cannot be underestimated as a significant benefit to continual enterprise development. An enterprise application lifecycle may last tens of years and your solution must constantly evolve with the business environment, regardless of any changes in your team. For this reason, the flexibility to rotate, or scale up or down the team when needed, is one of the major concerns for the companies, especially those who outsource development or have distributed teams. Open One of the key principles of CUBA is openness. This starts with the full platform source code, which you have at hand when you work on a CUBA-based project. In addition, the platform is also open in the sense that you can change almost any part of it to suit your needs. You don't need to fork it to customize some parts of the platform - it is possible to extend and modify the platform functionality right in your project. To achieve this, we usually follow the open inheritance pattern, providing access to the platform internals. We understand that this can cause issues when the project is upgraded to a newer platform version. However, from our experience, this is far less evil than maintaining a fork, or accepting the inability to adapt the tool for a particular task. We could also make a number of specific extension points, but in such case we would have to anticipate how application developers will use the platform. Such predictions always fail, sooner or later. So instead we have made the whole platform extension-friendly: you can inherit and override platform Java code including the object model, XML screens layout and configuration parameters. Transitively, this remains true for CUBA-based projects. If you follow a few simple conventions, your application becomes open for extension, allowing you to adapt the single product for many customers. Symbiotic CUBA is not positioned as a “thing-in-itself”. When a suitable and well-supported instrument already exists and we can integrate with it without sacrificing platform usability, we will integrate with it. An illustration of such integrations is full-text search and BPM engines, JavaScript charts and Google Maps API. At the same time, we have had to implement our ownreport generator from scratch, because we could not find a suitable tool (technology and license wise). The CUBA Studio follows this principle too. It is a standalone web application and it doesn't replace your preferred IDE. You can use Studio and the IDE in parallel, switching between them to accomplish different tasks. WYSIWYG approach, implemented in Studio, is great for designing the data model and screens layout, while the classic Java IDE is the best for writing code. You can change any part of your project right in the IDE, even things created by Studio. When you return to Studio, it will instantly parse all changes, allowing you to keep on developing visually. As you see, instead of competing with the power of Java IDEs, we follow a symbiotic approach. Moreover, to raise coding efficiency, we’ve developed plugins for the most popular IDEs. When we integrate with a third-party framework, we always wrap it in a higher level API. This enables replacing the underlying implementation if needed and makes the whole platform API more stable long term and less dependent on the constant changes in the integrated third-party frameworks. However, we don't restrict the direct use of underlying frameworks and libraries. It makes sense if CUBA API does not fit a particular use case. For example, if you can't do something via Generic UI, you can unwrap a visual component and get direct access to Vaadin (or Swing). The same applies for data access; if some operation is slow or not supported by ORM, just write SQL and run it via JDBC or MyBatis. Of course, such “hacks” lead to more complex and less portable application code, but they are typically very rare compared to the use of standard platform API. This knowledge of inherent flexibility and a sense of “Yes you can” adds a lot of confidence to developers. Wide use area We recommend using CUBA if you need to create an application with anything starting from 5-10 screens, as long as they consist of standard components like fields, forms, and tables. The effect from using CUBA grows exponentially with the complexity of your application, independent of the domain. We have delivered complex projects in financial, manufacturing, logistics and other areas. As an example, a non-obvious, but popular use case is using CUBA as the backend and admin UI, while creating the end-user interface with another, lighter or more customizable web technology. I hope you will see some use cases of the platform for yourself, so in the next articles we’ll focus on “what's under the hood” – as we provide a detailed overview of the different CUBA parts.

Persistence of model objects is a part of many Java projects and a part which deserves, and often gets, high test coverage as one of the key layer integration points in the code. However, I've often felt the testing paradigms for this can be cumbersome, often involving a large amount of setup with an equivalent amount of validation. This can be tedious to both create and maintain. As a solution to this I've been testing persistence with a different pattern; by combining both the exparity-stub and the hamcrest-bean library you can thoroughly test model persistence in a few lines of test code as per the snippet below; .. User user = aRandomInstanceOf(User.class); User saved = dao.save(user); assertThat(dao.getUserById(saved.getId()), theSameBeanAs(saved)); The test snippet above is small but in those few lines will thoroughly test that all fields in a graph can be persisted and retrieved without loss, that any JPA or other mapping is valid, and that your queries are valid. For a complete example we'll work through testing a simple DAO for storing and retrieving User objects using the in-memory H2 database for simplicity. The same example will work for any persistence mechanism. Before we get started with an example lets briefly outline what the libraries are and what they do. The Exparity-Stub Library The exparity-stub libraries provides a set of static methods for creating stubs of model objects, object graphs, collections, types, and primitive types. For our example we'll be creating random stubs because we want to completely fill the graph with junk data and check it can be written down. exparity-stub offers two approaches to this, the RandomBuilder or the BeanBuilder. The RandomBuilder provides a terser notation to create random objects with less code. For example: User user = RandomBuilder.aRandomInstanceOf(User.class); List users = RandomBuilder.aRandomListOf(User.class); String anyString = RandomBuilder.aRandomString(); Whereas the BeanBuilder provides a fluent interface with finer control for building individual objects and graphs, for example; User user = BeanBuilder.aRandomInstanceOf(User.class) .excludeProperty("Id").build(); For this example i'm going to use the BeanBuilder so I can exclude the User.Id property from being populated by the random builder. The Hamcrest-Bean Library The hamcrest-bean library is an extension library to the Java Hamcrest library. The hamcrest-bean library provides a set of matchers specifically for testing Java objects and object graphs and performs deep inspections of those objects. It supports exclusions and overrides to allow fine control, if required, of how matching of any property, path, or type is handled, for example: User expected = new User("Jane", "Doe"); assertThat(new User("John", "Doe"), BeanMatchers.theSameAs(expected).excludeProperty("FirstName")); A Sample Project The sample project I'll work through is persistence of a simple User object with a child list of UserComment objects. This simple graph will be persisted to a H2 database with hibernate handling the Object-Relational Mapping (ORM) mapping, and Java Persistence Annotation (JPA) used to mark-up the model. The Model Below are the two model classes; first the User class. package org.exparity.hamcrest.bean.sample.dao; import java.util.*; import javax.persistence.*; @Entity @Table public class User { @Id @GeneratedValue(strategy = GenerationType.SEQUENCE) private Long id; private Date createTs; private String username, firstName, surname; @OneToMany(cascade = CascadeType.ALL, fetch = FetchType.EAGER) private List comments = new ArrayList<>(); public Long getId() { return id; } public void setId(Long id) { this.id = id; } public Date getCreateTs() { return createTs; } public void setCreateTs(Date createTs) { this.createTs = createTs; } public String getUsername() { return username; } public void setUsername(String username) { this.username = username; } public String getFirstName() { return firstName; } public void setFirstName(String firstName) { this.firstName = firstName; } public String getSurname() { return surname; } public void setSurname(String surname) { this.surname = surname; } public List getComments() { return comments; } public void setComments(List comments) { this.comments = comments; } } Followed by the UserComment class. package org.exparity.hamcrest.bean.sample.dao; import java.util.Date; import javax.persistence.*; @Table @Entity public class UserComment { private Long id; private Date timestamp; @Transient private String text; private String title; public Date getTimestamp() { return timestamp; } public void setTimestamp(Date timestamp) { this.timestamp = timestamp; } public String getText() { return text; } public void setText(String text) { this.text = text; } public String getTitle() { return title; } public void setTitle(String title) { this.title = title; } } Followed by the UserComment class. package org.exparity.hamcrest.bean.sample.dao; import java.util.Date; import javax.persistence.*; @Table @Entity public class UserComment { private Long id; private Date timestamp; @Transient private String text; private String title; public Date getTimestamp() { return timestamp; } public void setTimestamp(Date timestamp) { this.timestamp = timestamp; } public String getText() { return text; } public void setText(String text) { this.text = text; } public String getTitle() { return title; } public void setTitle(String title) { this.title = title; } } The Data Access Object (DAO) Next up we write our DAO layer. I've excluded the UserDAO interface from this post but it is available in the sample project ongithub .The full, if somewhat crude, implementation of the UserDAO is below. package org.exparity.hamcrest.bean.sample.dao; import org.hibernate.boot.registry.StandardServiceRegistryBuilder; import org.hibernate.cfg.Configuration; import org.hibernate.*; public class UserDAOHibernateImpl implements UserDAO { private final SessionFactory factory; public UserDAOHibernateImpl(final String resourceFile) { this.factory = new Configuration() .addAnnotatedClass(User.class) .addAnnotatedClass(UserComment.class) .buildSessionFactory( new StandardServiceRegistryBuilder() .loadProperties(resourceFile) .build()); } @Override public User save(final User user) { Session session = factory.getCurrentSession(); Transaction txn = session.beginTransaction(); try { session.save(user); txn.commit(); } catch (final Exception e) { txn.rollback(); } return user; } @Override public User getUserById(Long userId) { Session session = factory.getCurrentSession(); Transaction txn = session.beginTransaction(); try { return (User) session.get(User.class, userId); } finally { txn.rollback(); } } } Integration Test And finally, onto our integration test. The hibernate.properties will create an instance of an in-memory database and create the necessary tables on instantiation of the DAO. hibernate.dialect=org.hibernate.dialect.H2Dialect hibernate.connection.username=sa hibernate.connection.password= hibernate.connection.driver_class=org.h2.Driver hibernate.connection.url=jdbc:h2:mem:test hibernate.current_session_context_class=thread hibernate.cache.provider_class=org.hibernate.cache.internal.NoCacheProvider hibernate.show_sql=true hibernate.hbm2ddl.auto=update The integration test is below. package org.exparity.hamcrest.bean.sample.dao; import static org.exparity.hamcrest.BeanMatchers.theSameBeanAs; import static org.exparity.stub.bean.BeanBuilder.aRandomInstanceOf; import static org.hamcrest.MatcherAssert.assertThat; import static org.hamcrest.Matchers.*; import org.junit.Test; public class UserDAOHibernateImplTest { @Test public void canSaveAUser() { User user = aRandomInstanceOf(User.class).excludeProperty("Id").build(); UserDAOHibernateImpl dao = new UserDAOHibernateImpl("hibernate.properties"); User saved = dao.save(user); User loaded = dao.getUserById(saved.getId()); assertThat(loaded, not(sameInstance(user))); assertThat(loaded, theSameBeanAs(user)); } } Let's break the test down step by step to see what each step is doing and why the test is put together this way. 1) Model Setup User user = aRandomInstanceOf(User.class).excludeProperty("Id").build(); Create a random instance of the User class and it's associates using exparity-stub. The instance will be populated with random data with the exception of the Id property. I've excluded the Id property so that is left null to test that the id is being generated in the database. 2) DAO Setup UserDAOHibernateImpl dao = new UserDAOHibernateImpl("hibernate.properties") Instantiate the DAO ready to be tested, passing in the property file to use for the test. The hibernate properties used will configure an in-memory instance of H2 and create the schema automatically. 3) Exercise the DAO User saved = dao.save(user); User loaded = dao.getUserById(saved.getId()); Save the random instance of the model set up in step (1) and then query the object back out again. 4) Verify the results assertThat(loaded, not(sameInstance(user))); assertThat(loaded, theSameBeanAs(user)); The first line verifies that the loaded User instance is not the same instance as the originally saved User. This prevents false positive results when the loaded instance is returned directly from a cache. The second line uses hamcrest-bean to perform a deep comparison of the loaded User instance against the original user instance. Running the Test The first run of the test yields an error; specifically a hibernate warning because a @Id annotation has been missed on UserComment. org.hibernate.AnnotationException: No identifier specified for entity: org.exparity.hamcrest.bean.sample.dao.UserComment at org.hibernate.cfg.InheritanceState.determineDefaultAccessType(InheritanceState.java:277) at org.hibernate.cfg.InheritanceState.getElementsToProcess(InheritanceState.java:224) at org.hibernate.cfg.AnnotationBinder.bindClass(AnnotationBinder.java:775) at org.hibernate.cfg.Configuration$MetadataSourceQueue.processAnnotatedClassesQueue(Configuration.java:3845) at org.hibernate.cfg.Configuration$MetadataSourceQueue.processMetadata(Configuration.java:3799) at org.hibernate.cfg.Configuration.secondPassCompile(Configuration.java:1412) at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1846) at org.exparity.hamcrest.bean.sample.dao.UserDAOHibernateImpl.(UserDAOHibernateImpl.java:15) at org.exparity.hamcrest.bean.sample.dao.UserDAOHibernateImplTest.canSaveAUser(UserDAOHibernateImplTest.java:18) A fix to the UserComment object and we can run the test again. @Table @Entity public class UserComment { @Id @GeneratedValue(strategy = GenerationType.SEQUENCE) private Long id; private Date timestamp; @Transient private String text; private String title; ... After running the test again we get another failure. The presence of the @Transient annotation on the UserComment.text property is preventing the value being persisted java.lang.AssertionError: Expected: the same as but: User.Comments[0].Text is null instead of "mDAWDJXbheIHbbHLR1NNVJqAki49RvaVwQtKD38r79u0y3MTDD" at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8) at org.exparity.hamcrest.bean.sample.dao.UserDAOHibernateImplTest.canSaveAUser(UserDAOHibernateImplTest.java:19) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) Another change to the UserComment object to remove the @Transient annotation and we can run the test again. @Table @Entity public class UserComment { @Id @GeneratedValue(strategy = GenerationType.SEQUENCE) private Long id; private Date timestamp; private String text; private String title; ... After running the test again it all passes. Try It Out To try hamcrest-bean and exparity-stub out for yourself include the dependency in your maven pom or other dependency manager. org.exparity hamcrest-bean 1.0.10 test org.exparity exparity-stub 1.1.5 test

Download the MP3 Date: June 19, 2015 By: Aaron Delp and Brian Gracely Description: Aaron and Brian talk to Chip Childers (@chipchilders, VP of Technology @CloudFoundryOrg) about the current status of Cloud Foundry projects, how Microsoft .NET will be integrated, IaaS vs. PaaS, and the CF.org thinking about overall interoperability Interested in the O'Reilly OSCON? Want to register for OSCON now? Use promo code 20CLOUD for 20% off Details to win an OSCON pass coming soon! Check out the OSCON Schedule Free eBook from O'Reilly Media for Cloudcast Listeners! Check out an excerpt from the upcoming Docker Cookbook Topic 1 - From an overall project perspective, what grades would you give Cloud Foundry in terms of stability, core functionality, security, operations, etc? Topic 2 - You were previously involved (directly/indirectly)with CloudStack. As you talk to people in the marketplace, how is it different discussing IaaS vs. PaaS. Topic 3 - How much ability will you have to drive prioritization within sub-projects or new projects? (eg. Security vs. new Languages vs. Interop, etc.) Topic 4 - What’s the CF.org way of thinking about interoperability? Topic 5 - What guidance are you giving the teams in terms of expandability of Cloud Foundry? Architecturally, are there certain places you recommend over other places? Topic 6 - Is there a place for integrating SaaS applications (monitoring, logging, etc.) into Cloud Foundry?

Written by Norberto Leite In the first part of this blog series, we covered the basic mechanics of our application and undertook some data modeling. In this second part, we will create tests that validate the behavior of our application and then describe how to set-up and run the application. Write the tests first Let’s begin by defining some small configuration libraries. file name: test/config/test_config.js Our server will be running on port 8000 on localhost. This will be fine for initial testing purposes. Later, if we change the location or port number for a production system, it would be very easy to just edit this file. To prepare for our test cases, we need to ensure that we have a good test environment. The following code achieves this for us. First, we connect to the database. file name: test/setup_tests.js Next, we drop the user collection. This ensures that our database is in a known starting state. Next, we will drop the user feed entry collection. Next, we will connect to Stormpath and delete all the users in our test application. Next, we close the database. Finally, we call async.series to ensure that all the functions run in the correct order. Frisby was briefly mentioned earlier. We will use this to define our test cases, as follows. file name: test/create_accounts_error_spec.js We will start with the enroll route in the following code. In this case we are deliberately missing the first name field, so we expect a status reply of 400 with a JSON error that we forgot to define the first name. Let’s “toss that frisby”: In the following example, we are testing a password that does not have any lower-case letters. This would actually result in an error being returned by Stormpath, and we would expect a status reply of 400. In the following example, we are testing an invalid email address. So, we can see that there is no @ sign and no domain name in the email address we are passing, and we would expect a status reply of 400. Now, let’s look at some examples of test cases that should work. Let’s start by defining 3 users. file name: test/create_accounts_spec.js In the following example, we are sending the array of the 3 users we defined above and are expecting a success status of 201. The JSON document returned would show the user object created, so we can verify that what was created matched our test data. Next, we will test for a duplicate user. In the following example, we will try to create a user where the email address already exists. One important issue is that we don’t know what API key will be returned by Stormpath a priori. So, we need to create a file dynamically that looks like the following. We can then use this file to define test cases that require us to authenticate a user. file name: /tmp/readerTestCreds.js In order to create the temporary file above, we need to connect to MongoDB and retrieve user information. This is achieved by the following code. file name: tests/writeCreds.js In the following code, we can see that the first line uses the temporary file that we created with the user information. We have also defined several feeds, such as Dilbert and the Eater Blog. file name: tests/feed_spec.js Previously, we defined some users but none of them had subscribed to any feeds. In the following code we test feed subscription. Note that authentication is required now and this is achieved using .auth with the Stormpath API keys. Our first test is to check for an empty feed list. In our next test case, we will subscribe our first test user to the Dilbert feed. In our next test case, we will try to subscribe our first test user to a feed that they are already subscribed-to. Next, we will subscribe our test user to a new feed. The result returned should confirm that the user is subscribed now to 2 feeds. Next, we will use our second test user to subscribe to a feed. The REST API Before we begin writing our REST API code, we need to define some utility libraries. First, we need to define how our application will connect to the database. Putting this information into a file gives us the flexibility to add different database URLs for development or production systems. file name: config/db.js If we wanted to turn on database authentication we could put that information in a file, as shown below. This file should not be checked into source code control for obvious reasons. file name: config/security.js We can keep Stormpath API and Secret keys in a properties file, as follows, and need to carefully manage this file as well. file name: config/stormpath_apikey.properties Express.js overview In Express.js, we create an “application” (app). This application listens on a particular port for HTTP requests to come in. When requests come in, they pass through a middleware chain. Each link in the middleware chain is given a req (request) object and a res (results) object to store the results. Each link can choose to do work, or pass it to the next link. We add new middleware via app.use(). The main middleware is called our “router”, which looks at the URL and routes each different URL/verb combination to a specific handler function. Creating our application Now we can finally see our application code, which is quite small since we can embed handlers for various routes into separate files. file name: server.js We define our own middleware at the end of the chain to handle bad URLs. Now our server application is listening on port 8000. Let’s print a message on the console to the user. Defining our Mongoose data models We use Mongoose to map objects on the Node.js side to documents inside MongoDB. Recall that earlier, we defined 4 collections: Feed collection. Feed entry collection. User collection. User feed-entry-mapping collection. So we will now define schemas for these 4 collections. Let’s begin with the user schema. Notice that we can also format the data, such as converting strings to lowercase, and remove leading or trailing whitespace using trim. file name: app/routes.js In the following code, we can also tell Mongoose what indexes need to exist. Mongoose will also ensure that these indexes are created if they do not already exist in our MongoDB database. The unique constraint ensures that duplicates are not allowed. The “email : 1” maintains email addresses in ascending order. If we used “email : -1” it would be in descending order. We repeat the process for the other 3 collections. The following is an example of a compound index on 4 fields. Each index is maintained in ascending order. Every route that comes in for GET, POST, PUT and DELETE needs to have the correct content type, which is application/json. Then the next link in the chain is called. Now we need to define handlers for each combination of URL/verb. The link to the complete code is available in the resources section and we just show a few examples below. Note the ease with which we can use Stormpath. Furthermore, notice that we have defined /api/v1.0, so the client would actually call /api/v1.0/user/enroll, for example. In the future, if we changed the API, say to 2.0, we could use /api/v2.0. This would have its own router and code, so clients using the v1.0 API would still continue to work. Starting the server and running tests Finally, here is a summary of the steps we need to follow to start the server and run the tests. Ensure that the MongoDB instance is running mongod Install the Node libraries npm install Start the REST API server node server.js Run test cases node setup_tests.js jasmine-node create_accounts_error_spec.js jasmine-node create_accounts_spec.js node write_creds.js jasmine-node feed_spec.js MongoDB University provides excellent free training. There is a course specifically aimed at Node.js developers and the link can be found in the resources section below. The resources section also contains links to good MongoDB data modeling resources. Resources HTTP status code definitions Chad Tindel’s Github Repository M101JS: MongoDB for Node.js Developers Data Models Data Modeling Considerations for MongoDB Applications

The growth of Azure has been outstanding--more than 90,000 new subscriptions every month. And the innovation is exponential with over 500 new features and services being added to the platform in the last 12 months. We're very excited to be part of this growth. As we announced yesterday, you can now access Stackato through Azure. We think it's a great way for Azure customers to get access to a Cloud Foundry and Docker based PaaS. With Azure, Microsoft provides an easy path to the cloud for their customers. All applications can be run on one cloud. Microsoft wants to dominate the cloud the same as it has with on-premise software and rarely does a day go by without reading an article about Azure. Whether it's their recent announcement to help encourage start-up's use of Azure by providing $120,000 worth of credits per year or their commitment to open source. Azure gives its customers a growing collection of integrated services that make it easier to build and manage enterprise, mobile, web and Internet of Things (IoT) apps faster. Enterprises face real complexities when building their cloud solution. Having a solid infrastructure is really just the first step in the process--companies also need the right platform to support the deployment and management of their cloud-native applications. The platform should give their developers the freedom to use the language best suited to build the application. In addition, enterprises are on more than one cloud. They need to have the versatility to scale out or move their applications to whatever cloud is appropriate in order to meet end user demand without any downtime. With Stackato, we help remove these complexities. We provide enterprises with a polyglot PaaS that supports the development of applications in virtually any language. We like to refer to Stackato as being "infrastructure-agnostic" and allow companies to deploy their applications to any cloud--private, public or hybrid--without the need to run new scripts or re-package the application in order for it to work in the new environment. The combination of Stackato on Azure gives enterprises the technology they need to streamline application delivery, drive innovation and meet the demands of their customers.

Written by Norberto Leite Introduction In this 2-part blog series, you will learn how to use MongoDB, Mongoose Object Data Mapping (ODM) with Express.js and Node.js. These technologies use a uniform language - JavaScript - providing performance gains in the software and productivity gains for developers. In this first part, we will describe the basic mechanics of our application and undertake data modeling. In the second part, we will create tests that validate the behavior of our application and then describe how to set-up and run the application. No prior experience with these technologies is assumed and developers of all skill levels should benefit from this blog series. So, if you have no previous experience using MongoDB, JavaScript or building a REST API, don’t worry - we will cover these topics with enough detail to get you past the simplistic examples one tends to find online, including authentication, structuring code in multiple files, and writing test cases. Let’s begin by defining the MEAN stack. What is the MEAN stack? The MEAN stack can be summarized as follows: M = MongoDB/Mongoose.js: the popular database, and an elegant ODM for node.js. E = Express.js: a lightweight web application framework. A = Angular.js: a robust framework for creating HTML5 and JavaScript-rich web applications. N = Node.js: a server-side JavaScript interpreter. The MEAN stack is a modern replacement for the LAMP (Linux, Apache, MySQL, PHP/Python) stack that became the popular way for building web applications in the late 1990s. In our application, we won’t be using Angular.js, as we are not building an HTML user interface. Instead, we are building a REST API which has no user interface, but could instead serve as the basis for any kind of interface, such as a website, an Android application, or an iOS application. You might say we are building our REST API on the ME(a)N stack, but we have no idea how to pronounce that! What is a REST API? REST stands for Representational State Transfer. It is a lighter weight alternative to SOAP and WSDL XML-based API protocols. REST uses a client-server model, where the server is an HTTP server and the client sends HTTP verbs (GET, POST, PUT, DELETE), along with a URL and variable parameters that are URL-encoded. The URL describes the object to act upon and the server replies with a result code and valid JavaScript Object Notation (JSON). Because the server replies with JSON, it makes the MEAN stack particularly well suited for our application, as all the components are in JavaScript and MongoDB interacts well with JSON. We will see some JSON examples later, when we start defining our Data Models. The CRUD acronym is often used to describe database operations. CRUD stands for CREATE, READ, UPDATE, and DELETE. These database operations map very nicely to the HTTP verbs, as follows: POST: A client wants to insert or create an object. GET: A client wants to read an object. PUT: A client wants to update an object. DELETE: A client wants to delete an object. These operations will become clear later when define our API. Some of the common HTTP result codes that are often used inside REST APIs are as follows: 200 - “OK”. 201 - “Created” (Used with POST). 400 - “Bad Request” (Perhaps missing required parameters). 401 - “Unauthorized” (Missing authentication parameters). 403 - “Forbidden” (You were authenticated but lacking required privileges). 404 - “Not Found”. A complete description can be found in the RFC document, listed in the resources section at the end of this blog. We will use these result codes in our application and you will see some examples shortly. Why Are We Starting with a REST API? Developing a REST API enables us to create a foundation upon which we can build all other applications. As previously mentioned, these applications may be web-based or designed for specific platforms, such as Android or iOS. Today, there are also many companies that are building applications that do not use an HTTP or web interface, such as Uber, WhatsApp, Postmates, and Wash.io. A REST API also makes it easy to implement other interfaces or applications over time, turning the initial project from a single application into a powerful platform. Creating our REST API The application that we will be building will be an RSS Aggregator, similar to Google Reader. Our application will have two main components: The REST API Feed Grabber (similar to Google Reader) In this blog series we will focus on building the REST API, and we will not cover the intricacies of RSS feeds. However, code for Feed Grabber is available in a github repository, listed in the resources section of this blog. Let’s now describe the process we will follow in building our API. We will begin by defining the data model for the following requirements: Store user information in user accounts Track RSS feeds that need to be monitored Pull feed entries into the database Track user feed subscriptions Track which feed entry a user has already read Users will need to be able to do the following: Create an account Subscribe/unsubscribe to feeds Read feed entries Mark feeds/entries as read or unread Modeling Our Data An in-depth discussion on data modeling in MongoDB is beyond the scope of this article, so see the references section for good resources on this topic. We will need 4 collections to manage this information: Feed collection Feed entry collection User collection User-feed-entry mapping collection Let’s take a closer look at each of these collections. Feed Collection Lets now look at some code. To model a feed collection, we can use the following JSON document: If you are familiar with relational database technology, then you will know about databases, tables, rows and columns. In MongoDB, there is a mapping to most of these Relational concepts. At the highest level, a MongoDB deployment supports one or more databases. A database contains one or more collections, which are the similar to tables in a relational database. Collections hold documents. Each document in a collection is, at a highest level, similar to a row in a relational table. However, documents do not follow a fixed schema with pre-defined columns of simple values. Instead, each document consists of one or more key-value pairs where the value can be simple (e.g., a date), or more sophisticated (e.g., an array of address objects). Our JSON document above is an example of one RSS feed for the Eater Blog, which tracks information about restaurants in New York City. We can see that there are a number of different fields but the key ones that our client application may be interested in include the URL of the feed and the feed description. The description is important so that if we create a mobile application, it would show a nice summary of the feed. The remaining fields in our JSON document are for internal use. A very important field is _id. In MongoDB, every document must have a field called _id. If you create a document without this field, at the point where you save the document, MongoDB will create it for you. In MongoDB, this field is a primary key and MongoDB will guarantee that within a collection, this value is unique. Feed Entry Collection After feeds, we want to track feed entries. Here is an example of a document in the feed entry collection: Again, we can see that there is a _id field. There are also some other fields, such as description, title and summary. For the content field, note that we are using an array, and the array is also storing a document. MongoDB allows us to store sub-documents in this way and this can be very useful in some situations, where we want to hold all information together. The entryID field uses the tag format to avoid duplicate feed entries. Notice also the feedID field that is of type ObjectId - the value is the _id of the Eater Blog document, described earlier. This provides a referential model, similar to a foreign key in a relational database. So, if we were interested to see the feed document associated with this ObjectId, we could take the value 523b1153a2aa6a3233a913f8 and query the feed collection on _id, and it would return the Eater Blog document. User Collection Here is the document we could use to keep track of users: A user has an email address, first name and last name. There is also an sp_api_key_id and sp_api_key_secret - we will use these later with Stormpath, a user management API. The last field, called subs, is a subscription array. The subs field tells us which feeds this user is subscribed-to. User-Feed-Entry Mapping Collection The last collection allows us to map users to feeds and to track which feeds have been read. We use a Boolean (true/false) to mark the feed as read or unread. Functional Requirements for the REST API As previously mentioned, users need to be able to do the following: Create an account. Subscribe/unsubscribe to feeds. Read feed entries. Mark feeds/entries as read or unread. Additionally, a user should be able to reset their password. The following table shows how these operations can be mapped to HTTP routes and verbs. Route Verb Description Variables /user/enroll POST Register a new user firstName lastName email password /user/resetPassword PUT Password Reset email /feeds GET Get feed subscriptions for each user with description and unread count /feeds/subscribe PUT Subscribe to a new feed feedURL /feeds/entries GET Get all entries for feeds the user is subscribed to /feeds/&ltfeedid>/entries GET Get all entries for a specific feed /feeds/&ltfeedid> PUT Mark all entries for a specific feed as read or unread read = &lttrue | false> /feeds/&ltfeedid>/entries/&ltentryid> PUT Mark a specific entry as either read or unread read = &lttrue | false> /feeds/&ltfeedid> DELETE Unsubscribe from this particular feed In a production environment, the use of secure HTTP (HTTPS) would be the standard approach when sending sensitive details, such as passwords. Real World Authentication with Stormpath In robust real-world applications it is important to provide user authentication. We need a secure approach to manage users, passwords, and password resets. There are a number of ways we could authenticate users for our application. One possibility is to use Node.js with the Passport Plugin, which could be useful if we wanted to authenticate with social media accounts, such as Facebook or Twitter. However, another possibility is to use Stormpath. Stormpath provides User Management as a Service and supports authentication and authorization through API keys. Basically, Stormpath maintains a database of user details and passwords and a client application REST API would call the Stormpath REST API to perform user authentication. The following diagram shows the flow of requests and responses using Stormpath. In detail, Stormpath will provide a secret key for each “Application” that is defined with their service. For example, we could define an application as “Reader Production” or “Reader Test”. This could be very useful when we are still developing and testing our application, as we may be frequently adding and deleting test users. Stormpath will also provide an API Key Properties file. Stormpath also allows us to define password strength requirements for each application, such as: Must have >= 8 characters. Must include lowercase and uppercase. Must include a number. Must include a non-alphabetic character Stormpath keeps track of all of our users and assigns them API keys, which we can use for our REST API authentication. This greatly simplifies the task of building our application, as we don’t have to focus on writing code for authenticating users. Node.js Node.js is a runtime environment for server-side and network applications. Node.js uses JavaScript and it is available for many different platforms, such as Linux, Microsoft Windows and Apple OS X. Node.js applications are built using many library modules and there is a very rich ecosystem of libraries available, some of which we will use to build our application. To start using Node.js, we need to define a package.json file describing our application and all of its library dependencies. The Node.js Package Manager installs copies of the libraries in a subdirectory, called node_modules/, in the application directory. This has benefits, as it isolates the library versions for each application and so avoids code compatibility problems if the libraries were to be installed in a standard system location, such as /usr/lib, for example. The command npm install will create the node_modules/ directory, with all of the required libraries. Here is the JavaScript from our package.json file: Our application is called reader-api. The main file is called server.js. Then we have a list of the dependent libraries and their versions. Some of these libraries are designed for parsing the HTTP queries. The test harness we will use is called frisby. The jasmine-node is used to run frisby scripts. One library that is particularly important is async. If you have never used node.js, it is important to understand that node.js is designed to be asynchronous. So, any function which does blocking input/output (I/O), such as reading from a socket or querying a database, will take a callback function as the last parameter, and then continue with the control flow, only returning to that callback function once the blocking operation has completed. Let’s look at the following simple example to demonstrate this. In the above example, we may think that the output would be: one two but in fact it might be: two one because the line that prints “one” might happen later, asynchronously, in the callback. We say “might” because if conditions are just right, “one” might print before “two”. This element of uncertainty in asynchronous programming is called non-deterministic execution. For many programming tasks, this is actually desirable and permits high performance, but clearly there are times when we want to execute functions in a particular order. The following example shows how we could use the async library to achieve the desired result of printing the numbers in the correct order: In the above code, we are guaranteed that function two will only be called after function one has completed. Wrapping Up Part 1 Now that we have seen the basic mechanics of node.js and async function setup, we are ready to move on. Rather than move into creating the application, we will instead start by creating tests that validate the behavior of the application. This approach is called test-driven development and has two very good features: It helps the developer really understand how data and functions are consumed and often exposes subtle needs like the ability to return 2 or more things in an array instead of just one thing. By writing tests before building the application, the paradigm becomes “broken / unimplemented until proven tested OK” instead of “assumed to be working until a test fails.” The former is a “safer” way to keep the code healthy.

Introduction As we all know about the Identity columns and Merge statement. We are not going to discuss any boring theoretical tropics related to it. Better we are discussing here with a practical scenario of merging records. Hope all of you must enjoy it and it will be informative. The Scenario We have a Table with Identity Columns named #tbl_TempStudentRecords. The table details are mentioned bellow. Column Name SrlNo StudentName StudentClass StudentSection We have another table named #tbl_TempStudrntMsrks. The table details are mentioned bellow. Column Name SrlNo StubjectName MarksObtain What we want to do is, we have another set of table called tbl_StudentDetails mentioned bellow. Column Name StdRoll (PK) StudentName StudentClass StudentSection Another table named tbl_StudentMarks Column Name IdNo (PK) StdRoll (FK) References [tbl_StudentDetails].[StdRoll] SubjectName MarksObtain Her we can insert records very easily in tbl_StudentDetails from #tbl_TempStudentRecords very easily. But the main problem is the IDENTITY columns in the Table named [tbl_StudentDetails].[ StdRoll]. When we insert records the Identity columns values generate automatically. When we are trying to insert records into the table named tbl_StudentMarks from Table named #tbl_TempStudrntMsrks we have to provide the StdRoll values, which is the Foreign Key References to the Table named [tbl_StudentDetails].[ StdRoll]. Think one minute with the case scenario. Hope you can understand the problem. Now we have to solve it and we are not using any LOOP for that and NOT even any DDL operation to change the structure of base table. We are just using the SET BASED operation to make performance high. How to Solve it Step – 1 [ Create the Base Table First ] IF OBJECT_ID('tempdb..#tbl_TempStudentRecords')IS NOT NULL BEGIN DROP TABLE #tbl_TempStudentRecords; END GO CREATE TABLE #tbl_TempStudentRecords ( SrlNo BIGINT NOT NULL, StudentName VARCHAR(50) NOT NULL, StudentClass INT NOT NULL, StudentSection CHAR(1) NOT NULL ); GO IF OBJECT_ID('tempdb..#tbl_TempStudrntMsrks')IS NOT NULL BEGIN DROP TABLE #tbl_TempStudrntMsrks; END GO CREATE TABLE #tbl_TempStudrntMsrks ( SrlNo BIGINT NOT NULL, StubjectName VARCHAR(50) NOT NULL, MarksObtain INT NOT NULL ); GO IF OBJECT_ID(N'[dbo].[tbl_StudentDetails]', N'U')IS NOT NULL BEGIN DROP TABLE [dbo].[tbl_StudentDetails]; END GO CREATE TABLE [dbo].[tbl_StudentDetails] ( StdRoll BIGINT NOT NULL IDENTITY(100,1) PRIMARY KEY, StudentName VARCHAR(50) NOT NULL, StudentClass INT NOT NULL, StudentSection CHAR(1) NOT NULL ); GO IF OBJECT_ID(N'[dbo].[tbl_StudentMarks]', N'U')IS NOT NULL BEGIN DROP TABLE [dbo].[tbl_StudentMarks]; END GO CREATE TABLE [dbo].[tbl_StudentMarks] ( IdNo BIGINT NOT NULL IDENTITY(1,1) PRIMARY KEY, StdRoll BIGINT NOT NULL, SubjectName VARCHAR(50) NOT NULL, MarksObtain INT NOT NULL ); GO ALTER TABLE [dbo].[tbl_StudentMarks] ADD CONSTRAINT FK_StdRoll_tbl_StudentMarks FOREIGN KEY(StdRoll) REFERENCES [dbo].[tbl_StudentDetails](StdRoll); Step – 2 [ Inserting Records in Temp Table ] INSERT INTO #tbl_TempStudentRecords (SrlNo, StudentName, StudentClass, StudentSection) VALUES(1, 'Joydeep Das', 1, 'A'), (2, 'Preeti Sharma', 1, 'A'), (3, 'Deepasree Das', 1, 'A'); INSERT INTO #tbl_TempStudrntMsrks (SrlNo, StubjectName, MarksObtain) VALUES (1, 'Bengali', 50), (1, 'English', 70), (1, 'Math', 80), (2, 'Bengali', 0), (2, 'English', 70), (2, 'Math', 80), (3, 'Bengali', 20), (3, 'English', 90), (3, 'Math', 95); Step – 3 [ Now Solve it By MERGE Statement ] BEGIN DECLARE @MappingTable TABLE ([NewRecordID] BIGINT, [OldRecordID] BIGINT) MERGE [dbo].[tbl_StudentDetails] AS target USING (SELECT [SrlNo] AS RecordID_Original ,[StudentName] ,[StudentClass] ,[StudentSection] FROM #tbl_TempStudentRecords ) AS source ON (target.StdRoll = NULL) WHEN NOT MATCHED THEN INSERT ([StudentName], [StudentClass], [StudentSection]) VALUES (source.[StudentName],source.[StudentClass], source.[StudentSection]) OUTPUT inserted.[StdRoll], source.[RecordID_Original] INTO @MappingTable; --- Now Map table is ready and we can use it --- INSERT INTO [dbo].[tbl_StudentMarks] (StdRoll, SubjectName, MarksObtain) SELECT b.NewRecordID, a.StubjectName, a.MarksObtain FROM #tbl_TempStudrntMsrks AS a INNER JOIN @MappingTable AS b ON a.SrlNo = b.OldRecordID; END GO Step – 4 [ Observation ] SELECT * FROM [dbo].[tbl_StudentDetails]; GO SELECT * FROM [dbo].[tbl_StudentMarks]; GO StdRoll StudentName StudentClass StudentSection 100 Joydeep Das 1 A 101 Preeti Sharma 1 A 102 Deepasree Das 1 A IdNo StdRoll SubjectName MarksObtain 1 100 Bengali 50 2 100 English 70 3 100 Math 80 4 101 Bengali 0 5 101 English 70 6 101 Math 80 7 102 Bengali 20 8 102 English 90 9 102 Math 95

If you are building mobile apps then you are connecting to some REST API. For example, if you want to resolve an address to a latitude/longitude information to display on a map, you might use the Google Geocoding API: Google Geocoding API: https://maps.googleapis.com/maps/api/geocode/json?address=San Francisco,CA&key=AIzaSyDvFMYGjeR02RH If you are invoking the API from the client, then the API key also has to be present on the client. But, this is also the problem. It’s very easy to look at the app source in the browser and get access to the API key. If someone has access to your API key, they can send requests on your behalf (without you knowing), and use up your request quota. Even if you are building a hybrid app, it’s still the same problem. A hybrid app is HTML/JavaScript inside a native wrapper, it’s possible to download the app, un-package it and gain access to API keys or any sensitive information stored in the app. Even native apps are not immune to this. For example, an Android app is just a Java application and a Java application can be de-compiled to view the original source. The next image shows how to get access to an API key in the browser: Viewing app source in browser A good solution is to never expose the API key (or any other sensitive data) on the client. How do you do that? You keep the API key and any other sensitive information on the server. Appery.io Secure Proxy Appery.io Secure Proxy (part of Backend Services) enables app developers to keep sensitive app data on the server. Your API keys or any other data is never exposed on the client. Watch this 5-minute video on how to use Secure Proxy: Before using the Secure Proxy, you need to store the data on the server. To store the data you are going to use the Appery.io Database. It’s as simple as creating a collection with two columns. The first column is the value name, the second column is the actual value. This is how the database looks when storing the API key for Google Geocoding API: Saving API key in database As this key is stored on the server, no one (but you) has access to it. You can store other data as well such as URLs, tokens or anything else that shouldn’t be exposed on the client. The next step is to setup the proxy that will use the information stored in the database. This step is also very simple, this is how it looks: Secure proxy linked to a database You give the proxy a name and then link it to a database which stores your data. The above proxy is linked to Secrets_db database, Credentials collection, and secretName, secretValue columns. The last step is to link a REST API service to the proxy. In the service editor you select the secure proxy created: REST API service using secure proxy then in the Request tab you reference the API key stored in the database (the name stored in secretName column): Request parameter substitution will happen on the server and that’s it. When the API service is invoked, the call will go through the secure proxy (server) where the API key will substituted: API key is not exposed on the client For web apps, you can add an extra layer of security by specifying from which page URLs the proxy should accept requests: URL-based security The proxy will only accept requests from page URLs listed in the table. Another option to keep API keys private is to invoke the API from the server using Server Code, I will cover this in another post. Setting up an using the Appery.io Secure Proxy is simple. It provides a very important feature by allowing to keep sensitive and private data on the server, never exposing it on the client, and adding an extra security layer to your app.

Troy Hunt (@troyhunt, blog) had a great, very hands-on 2-day workshop about webapp security at NDC Oslo. Here are my notes. Highlights – resources Personal security and privacy https://www.entropay.com/ – a Prepaid Virtual Visa Card mailinator.com – tmp email f-secure VPN https://www.netsparker.com/ – scan a site for issues (insecure cookies, framework disclosure, SQL injection, …) (lot of $k) Site security https://report-uri.io/ – get reports when CSP rules violated; also displays CSP headers for a site in a human-friendly way https://securityheaders.io/ check quality of headers wrt security free SSL – http://www.startssl.com/, https://www.cloudflare.com/ (also provides web app firewall and other protections) ; SSL quality check: https://www.ssllabs.com/ssltest/ https://letsencrypt.org/ – free, automated, open Certificate Authority (Linux Found., Mozilla) Breaches etc. http://arstechnica.com/security/2015/06/hack-of-cloud-based-lastpass-exposes-encrypted-master-passwords/ https://twitter.com/jmgosney – one of ppl behind http://passwordscon.org . http://password-hashing.net experts panel. Team Hashcat. http://arstechnica.com/security/2012/12/25-gpu-cluster-cracks-every-standard-windows-password-in-6-hours/ To follow ! http://krebsonsecurity.com/ ! http://www.troyhunt.com/ ! https://www.schneier.com/ ! https://twitter.com/mikko (of F-Secure) also great [TED] talks kevin mitnick (jailed for hacking; twitter, books) Books http://www.amazon.com/We-Are-Anonymous-LulzSec-Insurgency/dp/0316213527 – easy read, hard to put down http://www.amazon.com/Ghost-Wires-Adventures-Worlds-Wanted/dp/1441793755 – about Mitnick’s hacking, social engineering, living on the run ? http://www.amazon.com/Art-Intrusion-Exploits-Intruders-Deceivers/dp/0471782661/ Mitnick: http://www.amazon.com/Art-Deception-Controlling-Element-Security/dp/076454280X/ – social engineering Other https://www.xssposed.org/ See https://www.drupal.org/SA-CORE-2014-005 https://www.youtube.com/watch?v=Qvhdz8yE_po – Havij example http://www.troyhunt.com/2013/07/everything-you-wanted-to-know-about-sql.html, http://www.troyhunt.com/2010/05/owasp-top-10-for-net-developers-part-1.html, http://www.troyhunt.com/2012/12/stored-procedures-and-orms-wont-save.html, Googlee: find config files with SA access info: `inurl:ftp inurl:web.config filetype:config sa` https://scotthelme.co.uk/hardening-your-http-response-headers/ and https://securityheaders.io/ https://developer.mozilla.org/en-US/docs/Web/Security/Public_Key_Pinning – prevent MITM wappalyzer chrome plugin displaying info about the server and client that can be detected (jQuery, NewRelic, IIS, win OS, …) http://www.troyhunt.com/2015/05/do-you-really-want-bank-grade-security.html http://www.troyhunt.com/2012/05/everything-you-ever-wanted-to-know.html tool: https://github.com/gentilkiwi/mimikatz extract plaintexts passwords, hash, PIN code and kerberos tickets from memory on Windows Notes HackYourselfFirst.troyhunt.com – an example app with many vulnerabilities Note: maximizing your browser window will share info about your screen size, which might help to identify you haveibeenpwned.com – Troy’s online DB of hacked accounts Tips check robots.txt to know what to access Example Issues no https on login page insecure psw requirements cookies not secure flag => sent over http incl. AuthCookie) psw sent in clear text in confirm email user enumeration, f.eks. an issue with AdultFriendFinder – entry someone’s email to login to find out whether they’ve an account post illegal chars, get them displayed => injection no anti-automation (captcha) login confirm. email & autom. creating 1m accounts => sending 1m emails => pisses ppl off, likely increase one’s spam reputation (=> harder to send emails) brute-force protection? ### XSS Reflected XSS: display unescaped user input Encoding context: HTML, JS, CSS … have diff. escape sequences for the same char (e.g. <) – look at where they’re mixed Check the encoding consistency – manual encoding, omitting some chars JS => load ext resources, access cookies, manipulate the DOM Task: stal authCookie via search ### SQL injection Error-based injection: when the DB helps us by telling us what is wrong -> use ti learn more and even show some data Ex.: http://hackyourselffirst.troyhunt.com/Make/10?orderby=supercarid <—— supercarid is a column name orderby=(select * from userprofile) … learn about DB sructure, force an exception that shows the valueex.: (select top 1 cast(password) as int from userprofile) => “Conversion failed for the nvar value ‘passw0rd …’” Tips think of SQL commands that disclose structure: sys.(tables,columns), system commands enumerate records: nest queries: select top X ows asc then top 1 rows from that desc write out how you think the query works / is being constructed internally cast things to invalid types to disclose values in err msgs (or implicit cast due to -1 ..) #### Defenses whitelist input data types (id=123 => onlyallow ints) enumerable values – check against an appropr. whitelist if the value is stored – who uses it, how? making query/insertion safe permissions: give read-only permissions as much as possible; don’t use admin user from your webapp ### Mobile apps Look at HTTP req for sensitive data – creds, account, … Apps may ignore certificate validations In your app: param tampering, auth bypass, direct object refs Weak often: airlines, small scale shops, fast foods, … Tips certificate pining – the app has the fingerprint of the server cert. hardcoded and doesn’t trust even “valid” MITM certificate (banks, dropbox, …)x ### CSRF Cross-Site Request Forgery = make the user send a request => their auth cookie included async Ajax req to another site forbidden but that doesn’t apply to normal post Protection anti-forgery tags ### Understanding fwrk disclosure http://www.shodanhq.com/ -> search for “drupal 7” -> pwn How disclosed: headers familiar signs – jsessionid cookie for java, … The default error and 404 responses may help to recognize the fwr HTML code (reactid), “.do” for Sttruts implicit: order of headers (Apache x IIS), paths (capitalized?), response to improper HTTP version/protocol, => likely still possible to figure out the stack but not possible to simple search for fwrk+version ### Session hijacking Steal authentication cookie => use for illegal requests. Persistence over HTTP of auth., session: cookie, URL (but URL insecure – can be shared) Session/auth ID retrieval: insecure transport, referrer, stored in exceptions, XSS Factors limiting hijacking: short duration expiry, keyed to client device / IP (but IPs may rotate, esp, on mobile devices => be very cautious) DAY 2 ——– ### Cracking passwords Password hashing: salt: so that 2 ppl choosing the same psw will have a different hash => cracking is # salts * # passwords inst. of just N has cracking tips: character space Dictionary: passw0rd, … Mutations: manipulation and subst. of characters Tips: 1Password , LastPass, …. GPU ~ 100* faster than CPU #### Ex: Crack with hashcat common psw dict + md5-hashed passwords => crack ./hashcat-cli64.bin –hash-type=0 StratforHashes.txt hashkiller.com.dic # 23M psw dict -> Recovered.: 44 326/860 160 hashes [obs duplications] in 4 min (speed 135.35k plains) Q: What dictionary we use? Do we apply any mutations to it? ### Account enumeration = Does XY have an account? Multiple vectors (psw reset, register a new user with the same e-mail, …) Anti-automation: is there any? It may be inconsistent across vectors Does it matter? (<> privacy needs) How to “ask” the site and how to identify + and – responses? Timing attacks: distinguish positive x negative response based on the latency differing between the two ### HTTPS Confidentiality, Integrity, Authenticity Traffic hijacking: [a href="https://www.wifipineapple.com/"]https://www.wifipineapple.com/ – wifi hotspot with evil capabilities monitor probe requests (the phone looks for networks it knows), present yourself as one of those, the phone connects autom. (if no encryption) Consider everything sent over HTTP to be compromised Look at HTTPS content embedded in untrusted pages (iframes, links) – e.g. payment page embedded in http Links HSTS Preload – tell Chrome, FF that your site should only be ever loaded over HTTPS – https://hstspreload.appspot.com/ https://www.owasp.org/index.php/HTTP_Strict_Transport_Security header ### Content Scurity Policy header https://developer.chrome.com/extensions/contentSecurityPolicy See e.g. https://haveibeenpwned.com/ headers w/o CSP anything can be added to the page via a reflected XSS risk Anyth, can be added to the DOM downstream (on a proxy) With CSP the browser will only load resources you white-list; any violations can be reported Use e.g. https://report-uri.io/home/generate to create it and the report to watch for violations to fine tune it. ### SQL injection cont’d (Yesterday: Error-Based) #### Union Based SQLi Modify the query to union whatever other data and show them. More data faster than error-based inj. Ex.: http://hackyourselffirst.troyhunt.com/CarsByCylinders?Cylinders=V12 : V12 -> `V12′ union select voteid, comments collate SQL_Latin1_General_CP1_CI_AS from vote– ` #### Blind Boolean (laborious) Blind inj.: We can’t always rely on data being explicitly returned to the UI => ask a question, draw a conclusion about the data. Ex: http://hackyourselffirst.troyhunt.com/Supercar/Leaderboard?orderBy=PowerKw&asc=false -> ordedby => case when (select count(*) from userprofile) > 1 then powerkw else topspeedkm end Extract email: Is ascii of the lowercase char #1 < ascii of m ? Automation: SqlMap #### Time based blind injection When no useful output returned but yes/no responses differ significantly in how much time they take. F.ex. ask the db to delay the OK response. MS SQL: IF ‘b’ > ‘a’ WAITFOR DELAY ’00:00:05′ ### Brute force attacks Are there any defences? Often not How are defences impl? block the req resources block the src IP rate limit (by src IP) ### Automation penetration testing apps and services such as Netsparker, WhiteHatSec targets identification: shodan, googledorks, randowm crawling think aout the actions that adhere to a pattern – sql injection, fuzzing (repeat a req. trying diff. values for fields – SQLi, …), directory enumeration automation can be used for good – test your site tip: have autom. penetration testing (and perhaps static code analysis) as a part fo your build pipeline Task: Get DB schema using sqlmap (see python2.7 sqlmap.py –help) ### Protection Intrusion Detection System (IDS) – e.g. Snort Web Application Firewall (WAF) – e.g. CloudFare ($20/m)

Many applications need to be alerted when new documents are inserted or modified in the database. These include real-time apps, or features such as notifications, messaging, or chat. It is possible to build this kind of application with MongoDB, using features that are native to the database: Capped Collections and Tailable Cursors! Details and code samples that illustrate this technique are provided in How to Create a Pub/Sub application with MongoDB. There’s also a screencast that shows the application built in the blog post:

Leading today, VentureBeat reports on the quiet launch of Google’s Cloud Source Repositories. This seems like something we should have heard more about, but I don’t remember seeing anything about it. Amazon AWS announces the availability of all things Alexa, the Skills Kit, the Voice Service and a Fund. Last but not least, we have AJ Kohn, from Blind Five Year Old, talking about click-through rate being a ranking signal on Google’s search results. I don’t talk about SEO much, but reading AJ’s work is always fascinating. As always, enjoy today’s items, and please participate in the discussions on these sites. Top Stories Google has quietly launched a GitHub competitor, Cloud Source Repositories | VentureBeat Alexa Skills Kit, Alexa Voice Service, Alexa Fund | AWS Official Blog Startups, Career and Process Why offices are where work goes to die | Swizec Teller Unleashing the power of small teams | Andreas Papathanasis What Is A Tester? | Developsense Blog What happens when you stop relying on resumes | Aline Lerner Design and Development Swift 2: SIMD | Russ Bishop Why is Git better than Mercurial? | Javalobby Create a Maven archetype | Javalobby pip -t: A simple and transparent alternative to virtualenv | Zoomer Analytics Killing Off Wasabi – Part 1 | Fog Creek Blog WebAssembly- Explained | Modus Create Generating JSON Schema from XSD with JAXB and Jackson | Inspired by Actual Events AI, Machine Learning, Research and Advanced Algorithms Applying Machine Learning to Text Mining with Amazon S3 and RapidMiner | Amazon AWS Big Data, Visualization, SQL and NoSQL Is Click Through Rate A Ranking Signal? | Blind Five Year Old Cache-friendly binary search | Bannalia Discovering the Computer Science Behind Postgres Indexes | Java Code Geeks How an open-source competitive benchmark helped to improve databases | ArangoDB Security, Encryption and Cryptography Cracking JXcore… Again | Mark Haase Link Collections The Daily Six Pack: June 25, 2015 | Dirk Strauss Double Shot #1517 | A Fresh Cup Dew Drop – June 25, 2015 (#2042) | Morning Dew The Daily Six Pack: June 26, 2015 | Dirk Strauss

When a team works with a lot of maven projects it becomes quickly painful to do some basic tasks like: manage versionning and connections between the different projects. releasing and opening versions, especially when the maven-release plugin needs to be run on many projects and when versionning is not standard. managing external dependencies also can become complex, and ensuring that a single version of a dependency is used accross different projects is sometimes not a trivial question to ask. in a one word, applying transformations on a dependency graph is difficult. mind-mapping the dependency graph is difficult when the number of projects grows. That can increase the amount of time needed by new people to understand a project graph, and that also makes maintaining and changing things difficult. checking consistency and optimizing the dependency graph is not an easy task neither. having an always up-to-date build of snapshots and release is not easy when projects are distributed everywhere. Pom Explorer So there is the Pom Explorer tool which tries to address those problems by providing those functionalities : release a graph : release a pom or all poms and its/theirs dependencies and updates all dependent poms accross multiple repositories and projects, change a gav : updates a project’s gav and make all the project which depends on it follow this change. manages properties, dependency management, and so on. Pom-explorer knows what pom.xml to update and where to update. If a dependency specifies ${foobar.version}, pom-explorer will go to update the foobar.version property. query the dependency graph to retrieve pertinent information about your projects, statistics and check functions are also available, display 3d interactive graph, export graphml files, find not used dependencies and other similar problems, list java classes provided by artifacts, list java classes referenced by artifacts, runs a light and efficient web server so local and shared usage is possible. The tool will also support automatically building projects in order to always have such or such project always up to date. Use cases In this article, I will show some common use cases possible with this tool. Installation First one needs to install and run the software. Put yourself in a temporary directory and type those commands : git clone https://github.com/ltearno/pom-explorer.git cd pom-explorer java -jar target/pom-explorer.jar The program should welcome you and ask you to go to this address : http://localhost:90 This is the console to the application. You can type commands in the prompt, they will be sent to the server and it will answer. You can use up and down arrows to recall past commands. Let’s start by typing ? to get the available commands : Analyze of repositories OK. First we will analyse a directory where there are many maven projects, then we will work a bit to optimize those projects. You will have to adapt the exercise to your computer. Let’s analyse my git repositories directory : analyze directory c:\documents\repos This will analyze my projects and construct an in-memory graph of the dependency graph : Now, the program knows about everything on my projects, let’s start asking questions ! List of GAVs… Let’s get the list of all existing GAVs (groupId, artifactId, version) in the graph. There will be my projects and all the GAVs on which they depend. Type this command : gav li Note that you can type only the first letters of a command, as long as there is no ambiguity. Here li stands for list. Find dependencies on an obsolete artifact As I look through the list of GAVs, I remark that there are still an old snapshot version of the hexa.binding artifact hanging around. The latest released version is 1.3 and the working version is 1.4-SNAPSHOT so the version 1.3-SNAPSHOTshould not be used anymore. Which is the project still depending on this very deprecated this version ? Let’s ask the question : depends on fr.lteconsulting:hexa.binding:1.3-SNAPSHOT Here it is ! the project rigpa.org:regsys-clients:1.0.0-SNAPSHOT is still using an old snapshot. Let’s arrange that. Pom Explorer is able to change the pom properties and dependencies by itself. Updating this wrong dependency What we want is to change fr.lteconsulting:hexa.binding:1.3-SNAPSHOT tofr.lteconsulting:hexa.binding:1.3 so that the project uses the latest release available. We could desire to change for fr.lteconsulting:hexa.binding:1.4-SNAPSHOT which would be possible with the same command as we’ll see. For that we will use the change gav command : cha ga fr.lteconsulting:hexa.binding:1.3-SNAPSHOT fr.lteconsulting:hexa.binding:1.3 Here is what Pom Explorer answers : So first Pom Explorer finds what needs to be changed in the graph. This might be the project itself and all projects which depend on it. After that the program begins a loop in which all changes are checked and appropriately transformed when needed. For instance changing a dependency version can become changing a property value. Changes are first resolved as described before and they are then transformed in a change list to apply to be applied to pom.xml files. In the ouput, there is first a little warning saying thefr.lteconsulting:hexa.binding:1.3-SNAPSHOT project was not found. That’s normal because the project in now in version 1.4-SNAPSHOT. So there is no need to modify it. Then in the change list section, the changes that are to be applied to pom.xml files are listed. The first one says ‘project not found’ and that’s ok as seen before. The second one says to modify the C:\documents\repos\regsys-clients\pom.xmland change the dependency ([DEPENDENCY])fr.lteconsulting:hexa.binding:1.3-SNAPSHOT tofr.lteconsulting:hexa.binding:1.3. The “causes” message is useful when a change is caused by other changes (as said before a dependency change can become one or several property changes). If we had properties involved, Pom Explorer would have found them and included them in the change set. Now that we reviewed the proposed changes and agreed with them, let’s apply them by using the same command with the -apply flag : cha ga fr.lteconsulting:hexa.binding:1.3-SNAPSHOT fr.lteconsulting:hexa.binding:1.3 -apply We see that at the end of the same process, the program updated the dependency in the right pom.xml file. Let’s have a look at the file it self : fr.lteconsulting hexa.binding 1.3 compile OK, the file is correct now… Oh well no ! I just find other dependencies in SNAPSHOT versions ! Finding more duplicate and obsolete dependencies Let’s accept it, our projects are not up to date. Well let’s see how many of those artifacts there are with multiple versions used. For that i type the checkcommand : Ok there is some work to do ! Opening a version Now let’s look at another use case. Say that the hexa.binding project is in version 1.3 and i want to open the version 1.4-SNAPSHOT. I also want all the projects which depend on version 1.3to move to 1.4-SNAPSHOT. On the way, I want all modified projects still in a release version to be SNAPSHOT-ized too. And i want this to happen recursively as new projects are opened. With Pom Explorer, that’s only one command : change gav fr.lteconsulting:hexa.binding:1.3 fr.lteconsulting:hexa.binding:1.4-SNAPSHOT As you can see, warnings are generated when projects are reopened : Those are normal warnings, they are just here so that you know what happens. Then, there is a big list of changes to be made, because the hexa.bindingartifact is used in many central projects that were in a release state. Glad that we didn’t do that by hand ! Even with the maven-update-version plugin, there would have been a lot of repositories to go to open and update. Let’s apply the changes with the -apply flag : cha ga fr.lteconsulting:hexa.binding:1.3 fr.lteconsulting:hexa.binding:1.4-SNAPSHOT -apply All the changes have been made, about 30 of them. In one go ! Refresh the page so that a new session is created from the changed files. We can see that many of the projects have been reopened : You now have to commit all the repositories with this update. Pom explorer does not do that yet, but maybe in the future ! Releasing many poms Imagine the sprint is almost finished now and it’s now time to release the projects. Type the gav li fr.lteconsulting again to get the GAVs list (fr.lteconsulting is my projects package name, so I filter GAVs with that), choose one and let’s release it : fr.lteconsulting:hexa.binding.samples:1.4-SNAPSHOT The thing in the release is to have all direct and transitive dependencies released too. That’s what Pom Explorer checks. It then generates a change list to materialize your requirements. Other use cases Listing provided and referenced classes You can ask which Java classes are provided and referenced by GAVs. That’s sometimes a useful information to have. Try those commands : classes providedBy fr.lteconsulting:hexa.css:1.3 classes referencedBy fr.lteconsulting:hexa.css:1.3 Optimizing your project’s dependencies Sometimes, you ask yourself “do I still need this and that dependency ?” but you are not very sure, and since you lack time to investigate, eventually the dependency stays in your project for a long time, causing of course maintenance issues sometime. Let’s have Pom Explorer help us in the quest for the obsolete dependency. garbage dep fr.lteconsulting:carousel:1.0-SNAPSHOT This will give you something like that : You can refer to the project documentation to find how to use those informations. But sure that it can help you give away those useless dependencies ! Other goodies : graphs ! Pom Explorer can do two other things to help you visualize your dependency graph : export GraphML files so you can use them in another graph software (like yEd for instance). display an interactive 3d graph Exporting GraphML files GraphML is an open format to describe graphs. With the graph exportcommand, you can get graphml files of your working session. The program will create two files and display the links to them. Those two files are corresponding to two graphs : the dependency graph as usual, and the dependency graph between the git repositories containing your projects. Sometime one git repository can contain multiple projects and a view of the dependencies at the repository level is useful in those cases. This is the kind of picture you can get easily from editors like yEd : Interactive 3D graph Thanks to the WebGL standard which allows direct access to the 3D hardware on the running machine and thanks to libraries like three.js and ngraph.pixel, it is possible to display an interacive 3d graph. More over it is possible to customize the appearance of the graph to give account of different perspectives. Type the graph command and click on the link. This will open another tab containing the living 3d graph of your projects. When focus is given to the 3d viewport, the W, A, S, F and arrow keys allow to move in the 3d space. On the right, there is a text area where you can edit some javascript callback to customize the graph appearance. You can also stop the moving of the particle with the checkbox at the bottom right of the screen. It is not necessarilly useful, but sometimes it is relaxing to admire your work in the form of a living and moving graph ! Conclusion There are many other functions in Pom Explorer, but they are for you to discover now. This tool finds easily its place in the daily workflow because of the functions it provides. The fact that one can run it locally or on a shared server allows to use it as you wish. It is still in early development phase so many more functionalities could come up. On this subject, don’t hesitate submitting a little pull request on the GitHub repository… Pom Explorer is made with love by LTE Consulting

It’s been a year since I started the quest for a highly-effective Data Knowledge Stack and the Hibernate Master Class contains over fifty articles already. Now that I covered many aspects of database transactions, JDBC and Java Persistence, it’s time to assemble all the pieces together into the High-Performance Java Persistence book. An Agile publishing experience Writing a book is a very time-consuming and stressful process and the last thing I needed was a very tight schedule. After reading Antonio Goncalves’s story, I chose the self-publishing way. In the end, I settled for Leanpub because it allows me to publish the book incrementally. This leads to a better engagement with readers, allowing me adapt the book content on the way. The content At its core, the book is about getting the most out of your persistence layer and that can only happen when your application resonates with the database system. Because concurrency is inherent to database processing, transactions play a very important role in this regard. The first part will be about some basic performance-related database concepts such as: locking, batching, connection pooling. In the second part, I will explain how an ORM can actually improve DML performance. This part will include the Hibernate Master Class findings. The third part is about advance querying techniques with jOOQ. If you enjoy reading this article, you might want to subscribe to my newsletter and get a discount for my book as well. Get involved The Agile methodologies are not just for software development. Writing a book in a Lean style can shorten the feed-back period and readers can get involved on the way. If you have any specific request or you are interested in this project, you can join my newsletter and follow my progress. Buy it! The book is 100% done, and you can check out the full Table of Content onLeanpub. If you enjoyed this article, I bet you are going to love my book as well. The ebook The PDF, ePUB and Kindle (MOBI) versions can be bought on Leanpub. The print version The print version is available on Amazon, Amazon.co.uk, Amazon.de or Amazon.fr. Presentations If you are not convinced, then check out the following two presentations: High-Performance JDBC from Voxxed Days Bucharest High-Performance Hibernate from JavaZone

A blue-green deployment is a way to have incremental updates to your production stack without downtime and without any complexity for properly handling rolling updates (including the rollback functionality) I don’t need to repeat this wonderful explanation or Martin Fowler’s original piece. But I’ll extend on them. A blue-green deployment is one where there is an “active” and a “spare” set of servers. The active running the current version, and the spare being ready to run any newly deployed version. The “active” and “spare” is slightly different than “blue” and “green”, because one set is always “blue” and one is always “green”, while the “active” and “spare” labels change. On AWS, for example, you can script the deployment by having two child stacks of your main stacks – active and spare (indicated by a stack label), each having one (or more) auto-scaling group for your application layer, and a script that does the following (applicable to non-AWS as well): push build to an accessible location (e.g. s3) set the spare auto-scaling group size to the desired value (the spare stays at 0 when not used) make it fetch the pushed build on startup wait for it to start run sanity tests switch DNS to point to an ELB in front of the spare ASG switch the labels to make the spare one active and vice versa set the previously active ASG size to 0 The application layer is stateless, so it’s easy to do hot-replaces like that. But (as Fowler indicated) the database is the most tricky component. If you have 2 databases, where the spare one is a slave replica of the active one (and that changes every time you switch), the setup becomes more complicated. And you’ll still have to do schema changes. So using a single database, if possible, is the easier approach, regardless of whether you have a “regular” database or a schemaless one. In fact, it boils down to having your application modify the database on startup, in a way that works with both versions. This includes schema changes – table (or the relevant term in the schemaless db) creation, field addition/removal and inserting new data (e.g. enumerations). And it can go wrong in many ways, depending on the data and datatypes. Some nulls, some datatype change that makes a few values unparseable, etc. Of course, it’s harder to do it with a regular SQL database. As suggested in the post I linked earlier, you can use stored procedures (which I don’t like), or you can use a database migration tool. For a schemaless database you must do stuff manually, but but fewer actions are normally needed – you don’t have to alter tables or explicitly create new ones, as everything is handled automatically. And the most important thing is to not break the running version. But how to make sure everything works? test on staging – preferably with a replica of the production database (automatically) run your behaviour/acceptance/sanity test suites against the not-yet-active new deployment before switching the DNS to point to it. Stop the process if they fail. Only after these checks pass, switch the DNS and point your domain to the previously spare group, thus promoting it to “active”. Switching can be done manually, or automatically with the deployment script. The “switch” can be other than a DNS one (as you need a low TTL for that). It can be a load-balancer or a subnet configuration, for example – the best option depends on your setup. And while it is good to automate everything, having a few manual steps isn’t necessarily a bad thing. Overall, I’d recommend the blue-green deployment approach in order to achieve zero downtime upgrades. But always make sure your database is properly upgraded, so that it works with both the old and the new version.