I was recently asked about the benefits and wisdom of using off heap memory in Java. The answers may be of interest to others facing the same choices. Off heap memory is nothing special. The thread stacks, application code, NIO buffers are all off heap. In fact in C and C++, you only have unmanaged memory as it does not have a managed heap by default. The use of managed memory or "heap" in Java is a special feature of the language. Note: Java is not the only language to do this. new Object() vs Object pool vs Off Heap memory. new Object() Before Java 5.0, using object pools was very popular. Creating objects was still very expensive. However, from Java 5.0, object allocation and garbage cleanup was made much cheaper, and developers found they got a performance speed up and a simplification of their code by removing object pools and just creating new objects whenever needed. Before Java 5.0, almost any object pool, even an object pool which used objects provided an improvement, from Java 5.0 pooling only expensive objects obviously made sense e.g. threads, socket and database connections. Object pools In the low latency space it was still apparent that recycling mutable objects improved performance by reduced pressure on your CPU caches. These objects have to have simple life cycles and have a simple structure, but you could see significant improvements in performance and jitter by using them. Another area where it made sense to use object pools is when loading large amounts of data with many duplicate objects. With a significant reduction in memory usage and a reduction in the number of objects the GC had to manage, you saw a reduction in GC times and an increase in throughput. These object pools were designed to be more light weight than say using a synchronized HashMap, and so they still helped. Take this StringInterner class as an example. You pass it a recycled mutable StringBuilder of the text you want as a String and it will provide a String which matches. Passing a String would be inefficient as you would have already created the object. The StringBuilder can be recycled. Note: this structure has an interesting property that requires no additional thread safety features, like volatile or synchronized, other than is provided by the minimum Java guarantees. i.e. you can see the final fields in a String correctly and only read consistent references. public class StringInterner { private final String[] interner; private final int mask; public StringInterner(int capacity) { int n = Maths.nextPower2(capacity, 128); interner = new String[n]; mask = n - 1; } private static boolean isEqual(@Nullable CharSequence s, @NotNull CharSequence cs) { if (s == null) return false; if (s.length() != cs.length()) return false; for (int i = 0; i < cs.length(); i++) if (s.charAt(i) != cs.charAt(i)) return false; return true; } @NotNull public String intern(@NotNull CharSequence cs) { long hash = 0; for (int i = 0; i < cs.length(); i++) hash = 57 * hash + cs.charAt(i); int h = (int) Maths.hash(hash) & mask; String s = interner[h]; if (isEqual(s, cs)) return s; String s2 = cs.toString(); return interner[h] = s2; } } Off heap memory usage Using off heap memory and using object pools both help reduce GC pauses, this is their only similarity. Object pools are good for short lived mutable objects, expensive to create objects and long live immutable objects where there is a lot of duplication. Medium lived mutable objects, or complex objects are more likely to be better left to the GC to handle. However, medium to long lived mutable objects suffer in a number of ways which off heap memory solves. Off heap memory provides; Scalability to large memory sizes e.g. over 1 TB and larger than main memory. Notional impact on GC pause times. Sharing between processes, reducing duplication between JVMs, and making it easier to split JVMs. Persistence for faster restarts or replying of production data in test. The use of off heap memory gives you more options in terms of how you design your system. The most important improvement is not performance, but determinism. Off heap and testing One of the biggest challenges in high performance computing is reproducing obscure bugs and being able to prove you have fixed them. By storing all your input events and data off heap in a persisted way you can turn your critical systems into a series of complex state machines. (Or in simple cases, just one state machine) In this way you get reproducible behaviour and performance between test and production.A number of investment banks use this technique to replay a system reliably to any event in the day and work out exactly why that event was processed the way it was. More importantly, once you have a fix you can show that you have fixed the issue which occurred in production, instead of finding an issue and hoping this was the issue.Along with deterministic behaviour comes deterministic performance. In test environments, you can replay the events with realistic timings and show the latency distribution you expect to get in production. Some system jitter can't be reproduce esp if the hardware is not the same, but you can get pretty close when you take a statistical view. To avoid taking a day to replay a day of data you can add a threshold. e.g. if the time between events is more than 10 ms you might only wait 10 ms. This can allow you to replay a day of events with realistic timing in under an hour and see whether your changes have improved your latency distribution or not. By going more low level don't you lose some of "compile once, run anywhere"? To some degree this is true, but it is far less than you might think. When you are working closer the processor and so you are more dependant on how the processor, or OS behaves. Fortunately, most systems use AMD/Intel processors and even ARM processors are becoming more compatible in terms of the low level guarantees they provide. There is also differences in the OSes, and these techniques tend to work better on Linux than Windows. However, if you develop on MacOSX or Windows and use Linux for production, you shouldn't have any issues. This is what we do at Higher Frequency Trading. What new problems are we creating by using off heap? Nothing comes for free, and this is the case with off heap. The biggest issue with off heap is your data structures become less natural. You either need a simple data structure which can be mapped directly to off heap, or you have a complex data structure which serializes and deserializes to put it off heap. Obvious using serialization has its own headaches and performance hit. Using serialization thus much slower than on heap objects.In the financial world, most high ticking data structure are flat and simple, full of primitives which maps nicely off heap with little overhead. However, this doesn't apply in all applications and you can get complex nested data structures e.g. graphs, which you can end up having to cache some objects on-heap as well.Another problem is that the JVM limits how much of the system you can use. You don't have to worry about the JVM overloading the system so much. With off heap, some limitations are lifted and you can use data structures much larger than main memory, and you start having to worry about what kind of disk sub-system you have if you do this. For example, you don't want to be paging to a HDD which has 80 IOPS, instead you are likely to want an SSD with 80,000 IOPS (Input/Ouput Operations per Second) or better i.e. 1000x faster. How does OpenHFT help? OpenHFT has a number of libraries to hide the fact you are really using native memory to store your data. These data structures are persisted and can be used with little or no garbage. These are used in applications which run all day without a minor collection Chronicle Queue - Persisted queue of events. Supports concurrent writers across JVMs on the same machine and concurrent readers across machines. Micro-second latencies and sustained throughputs in the millions of messages per second. Chronicle Map - Native or Persisted storage of a key-value Map. Can be shared between JVMs on the same machine, replicated via UDP or TCP and/or accessed remotely via TCP. Micro-second latencies and sustained read/write rates in the millions of operations per second per machine. Thread Affinity - Binding of critical threads to isolated cores or logical cpus to minimise jitter. Can reduce jitter by a factor of 1000. Which API to use? If you need to record every event -> Chronicle Queue If you only need the latest result for a unique key -> Chronicle Map If you care about 20 micro-second jitter -> Thread Affinity Conclusion Off heap memory can have challenges but also come with a lot of benefits. Where you see the biggest gain and compares with other solutions introduced to achieve scalability. Off heap is likely to be simpler and much faster than using partitioned/sharded on heap caches, messaging solutions, or out of process databases. By being faster, you may find that some of the tricks you need to do to give you the performance you need are no longer required. e.g. off heap solutions can support synchronous writes to the OS, instead of having to perform them asynchronously with the risk of data loss.The biggest gain however, can be your startup time, giving you a production system which restarts much faster. e.g. mapping in a 1 TB data set can take 10 milli-seconds, and ease of reproducibility in test by replaying every event in order you get the same behaviour every time. This allows you to produce quality systems you can rely on.

Learn all about creating a microservices architecture on Java in this great tutorial.

{editor's note: thanks to yegor bugayenko, a new mvb at dzone. among other things, yegor blogs about java and devops. we're pleased to have him on board as a most valuable blogger. check out his blog, yegor256.com .} tl;dr orm is a terrible anti-pattern that violates all principles of object-oriented programming, tearing objects apart and turning them into dumb and passive data bags. there is no excuse for orm existence in any application, be it a small web app or an enterprise-size system with thousands of tables and crud manipulations on them. what is the alternative? sql-speaking objects . vinni-pukh (1969) by fyodor khitruk how orm works object-relational mapping (orm) is a technique (a.k.a. design pattern) of accessing a relational database from an object-oriented language (java, for example). there are multiple implementations of orm in almost every language; for example: hibernate for java, activerecord for ruby on rails, doctrine for php, and sqlalchemy for python. in java, the orm design is even standardized as jpa . first, let's see how orm works, by example. let's use java, postgresql, and hibernate. let's say we have a single table in the database, called post : +-----+------------+--------------------------+ | id | date | title | +-----+------------+--------------------------+ | 9 | 10/24/2014 | how to cook a sandwich | | 13 | 11/03/2014 | my favorite movies | | 27 | 11/17/2014 | how much i love my job | +-----+------------+--------------------------+ now we want to crud-manipulate this table from our java app (crud stands for create, read, update, and delete). first, we should create a post class (i'm sorry it's so long, but that's the best i can do): @entity @table(name = "post") public class post { private int id; private date date; private string title; @id @generatedvalue public int getid() { return this.id; } @temporal(temporaltype.timestamp) public date getdate() { return this.date; } public title gettitle() { return this.title; } public void setdate(date when) { this.date = when; } public void settitle(string txt) { this.title = txt; } } before any operation with hibernate, we have to create a session factory: sessionfactory factory = new annotationconfiguration() .configure() .addannotatedclass(post.class) .buildsessionfactory(); this factory will give us "sessions" every time we want to manipulate with post objects. every manipulation with the session should be wrapped in this code block: session session = factory.opensession(); try { transaction txn = session.begintransaction(); // your manipulations with the orm, see below txn.commit(); } catch (hibernateexception ex) { txn.rollback(); } finally { session.close(); } when the session is ready, here is how we get a list of all posts from that database table: list posts = session.createquery("from post").list(); for (post post : (list) posts){ system.out.println("title: " + post.gettitle()); } i think it's clear what's going on here. hibernate is a big, powerful engine that makes a connection to the database, executes necessary sql select requests, and retrieves the data. then it makes instances of class post and stuffs them with the data. when the object comes to us, it is filled with data, and we should use getters to take them out, like we're using gettitle() above. when we want to do a reverse operation and send an object to the database, we do all of the same but in reverse order. we make an instance of class post , stuff it with the data, and ask hibernate to save it: post post = new post(); post.setdate(new date()); post.settitle("how to cook an omelette"); session.save(post); this is how almost every orm works. the basic principle is always the same — orm objects are anemic envelopes with data. we are talking with the orm framework, and the framework is talking to the database. objects only help us send our requests to the orm framework and understand its response. besides getters and setters, objects have no other methods. they don't even know which database they came from. this is how object-relational mapping works. what's wrong with it, you may ask? everything! what's wrong with orm? seriously, what is wrong? hibernate has been one of the most popular java libraries for more than 10 years already. almost every sql-intensive application in the world is using it. each java tutorial would mention hibernate (or maybe some other orm like toplink or openjpa) for a database-connected application. it's a standard de-facto and still i'm saying that it's wrong? yes. i'm claiming that the entire idea behind orm is wrong. its invention was maybe the second big mistake in oop after null reference . actually, i'm not the only one saying something like this, and definitely not the first. a lot about this subject has already been published by very respected authors, including ormhate by martin fowler, object-relational mapping is the vietnam of computer science by jeff atwood, the vietnam of computer science by ted neward, orm is an anti-pattern by laurie voss, and many others. however, my argument is different than what they're saying. even though their reasons are practical and valid, like "orm is slow" or "database upgrades are hard", they miss the main point. you can see a very good, practical answer to these practical arguments given by bozhidar bozhanov in his orm haters don’t get it blog post. the main point is that orm, instead of encapsulating database interaction inside an object, extracts it away, literally tearing a solid and cohesive living organism apart. one part of the object keeps the data while another one, implemented inside the orm engine (session factory), knows how to deal with this data and transfers it to the relational database. look at this picture; it illustrates what orm is doing. i, being a reader of posts, have to deal with two components: 1) the orm and 2) the "obtruncated" object returned to me. the behavior i'm interacting with is supposed to be provided through a single entry point, which is an object in oop. in the case of orm, i'm getting this behavior via two entry points — the orm and the "thing", which we can't even call an object. because of this terrible and offensive violation of the object-oriented paradigm, we have a lot of practical issues already mentioned in respected publications. i can only add a few more. sql is not hidden . users of orm should speak sql (or its dialect, like hql ). see the example above; we're calling session.createquery("from post") in order to get all posts. even though it's not sql, it is very similar to it. thus, the relational model is not encapsulated inside objects. instead, it is exposed to the entire application. everybody, with each object, inevitably has to deal with a relational model in order to get or save something. thus, orm doesn't hide and wrap the sql but pollutes the entire application with it. difficult to test . when some object is working a list of posts, it needs to deal with an instance of sessionfactory . how can we mock this dependency? we have to create a mock of it? how complex is this task? look at the code above, and you will realize how verbose and cumbersome that unit test will be. instead, we can write integration tests and connect the entire application to a test version of postgresql. in that case, there is no need to mock sessionfactory , but such tests will be rather slow, and even more important, our having-nothing-to-do-with-the-database objects will be tested against the database instance. a terrible design. again, let me reiterate. practical problems of orm are just consequences. the fundamental drawback is that orm tears objects apart, terribly and offensively violating the very idea of what an object is . sql-speaking objects what is the alternative? let me show it to you by example. let's try to design that class, post , my way. we'll have to break it down into two classes: post and posts , singular and plural. i already mentioned in one of my previous articles that a good object is always an abstraction of a real-life entity. here is how this principle works in practice. we have two entities: database table and table row. that's why we'll make two classes; posts will represent the table, and post will represent the row. as i also mentioned in that article , every object should work by contract and implement an interface. let's start our design with two interfaces. of course, our objects will be immutable. here is how posts would look: @immutable interface posts { iterable iterate(); post add(date date, string title); } this is how a single post would look: @immutable interface post { int id(); date date(); string title(); } here is how we will list all posts in the database table: posts posts = // we'll discuss this right now for (post post : posts.iterate()){ system.out.println("title: " + post.title()); } here is how we will create a new post: posts posts = // we'll discuss this right now posts.add(new date(), "how to cook an omelette"); as you see, we have true objects now. they are in charge of all operations, and they perfectly hide their implementation details. there are no transactions, sessions, or factories. we don't even know whether these objects are actually talking to the postgresql or if they keep all the data in text files. all we need from posts is an ability to list all posts for us and to create a new one. implementation details are perfectly hidden inside. now let's see how we can implement these two classes. i'm going to use jcabi-jdbc as a jdbc wrapper, but you can use something else or just plain jdbc if you like. it doesn't really matter. what matters is that your database interactions are hidden inside objects. let's start with posts and implement it in class pgposts ("pg" stands for postgresql): @immutable final class pgposts implements posts { private final source dbase; public pgposts(datasource data) { this.dbase = data; } public iterable iterate() { return new jdbcsession(this.dbase) .sql("select id from post") .select( new listoutcome( new listoutcome.mapping() { @override public post map(final resultset rset) { return new pgpost(rset.getinteger(1)); } } ) ); } public post add(date date, string title) { return new pgpost( this.dbase, new jdbcsession(this.dbase) .sql("insert into post (date, title) values (?, ?)") .set(new utc(date)) .set(title) .insert(new singleoutcome(integer.class)) ); } } next, let's implement the post interface in class pgpost : @immutable final class pgpost implements post { private final source dbase; private final int number; public pgpost(datasource data, int id) { this.dbase = data; this.number = id; } public int id() { return this.number; } public date date() { return new jdbcsession(this.dbase) .sql("select date from post where id = ?") .set(this.number) .select(new singleoutcome(utc.class)); } public string title() { return new jdbcsession(this.dbase) .sql("select title from post where id = ?") .set(this.number) .select(new singleoutcome(string.class)); } } this is how a full database interaction scenario would look like using the classes we just created: posts posts = new pgposts(dbase); for (post post : posts.iterate()){ system.out.println("title: " + post.title()); } post post = posts.add(new date(), "how to cook an omelette"); system.out.println("just added post #" + post.id()); you can see a full practical example here . it's an open source web app that works with postgresql using the exact approach explained above — sql-speaking objects. what about performance? i can hear you screaming, "what about performance?" in that script a few lines above, we're making many redundant round trips to the database. first, we retrieve post ids with select id and then, in order to get their titles, we make an extra select title call for each post. this is inefficient, or simply put, too slow. no worries; this is object-oriented programming, which means it is flexible! let's create a decorator of pgpost that will accept all data in its constructor and cache it internally, forever: @immutable final class constpost implements post { private final post origin; private final date dte; private final string ttl; public constpost(post post, date date, string title) { this.origin = post; this.dte = date; this.ttl = title; } public int id() { return this.origin.id(); } public date date() { return this.dte; } public string title() { return this.ttl; } } pay attention: this decorator doesn't know anything about postgresql or jdbc. it just decorates an object of type post and pre-caches the date and title. as usual, this decorator is also immutable. now let's create another implementation of posts that will return the "constant" objects: @immutable final class constpgposts implements posts { // ... public iterable iterate() { return new jdbcsession(this.dbase) .sql("select * from post") .select( new listoutcome( new listoutcome.mapping() { @override public post map(final resultset rset) { return new constpost( new pgpost(rset.getinteger(1)), utc.gettimestamp(rset, 2), rset.getstring(3) ); } } ) ); } } now all posts returned by iterate() of this new class are pre-equipped with dates and titles fetched in one round trip to the database. using decorators and multiple implementations of the same interface, you can compose any functionality you wish. what is the most important is that while functionality is being extended, the complexity of the design is not escalating, because classes don't grow in size. instead, we're introducing new classes that stay cohesive and solid, because they are small. what about transactions? every object should deal with its own transactions and encapsulate them the same way as select or insert queries. this will lead to nested transactions, which is perfectly fine provided the database server supports them. if there is no such support, create a session-wide transaction object that will accept a "callable" class. for example: final class txn { private final datasource dbase; public t call(callable callable) { jdbcsession session = new jdbcsession(this.dbase); try { session.sql("start transaction").exec(); t result = callable.call(); session.sql("commit").exec(); return result; } catch (exception ex) { session.sql("rollback").exec(); throw ex; } } } then, when you want to wrap a few object manipulations in one transaction, do it like this: new txn(dbase).call( new callable() { @override public integer call() { posts posts = new pgposts(dbase); post post = posts.add(new date(), "how to cook an omelette"); posts.comments().post("this is my first comment!"); return post.id(); } } ); this code will create a new post and post a comment to it. if one of the calls fail, the entire transaction will be rolled back. this approach looks object-oriented to me. i'm calling it "sql-speaking objects", because they know how to speak sql with the database server. it's their skill, perfectly encapsulated inside their borders. related posts you may also find these posts interesting: how much your objects encapsulate? how an immutable object can have state and behavior? seven virtues of a good object how immutability helps paired brackets

this mobile application tutorial shows you how to create a user registration, login and logout backend using mongodb and mongoose. this article is part of a series of mobile application development tutorials that i have been publishing on my blog jorgeramon.me, which shows you how to create a meeting room booking mobile application. this app will be used to browse an inventory of meeting rooms and reserve rooms for conference calls and other types of events. the backend that we will create in this article will connect with the user account management screens that we built in a previous chapter of this series . this backend will consist of the following modules: router (using node.js and express ) controller data model to represent a user (using mongoose ) database (using mongodb ) the router receives http requests from the mobile application and forwards them to the controller, which in turn creates, reads, updates and deletes data models defined with the mongoose library . the user profiles and sessions information will reside in a mongodb database . the router also receives data from the controller and bundles it in http responses that it sends to the mobile app. in this article we will create the controller, model and database modules using mongoose and mongodb. we will test the controller and build the router in the next article of this series. let’s proceed to install the software that we will use to create the node js, mongodb and mongoose endpoint. installing node.js node.js is a platform for building network apps that you can use to build backend endpoints for your mobile applications. you can get node.js at node.js . this tutorial doesn’t require you to have extensive knowledge of node, but you should try to learn about it as much as you can in order to take full advantage of its capabilities. to start, i would recommend the tutorials over at node school . installing express express is a framework for building web applications with node.js. express’ installation page shows you how to install the framework. in this particular article we will only use the request routing capabilities of express. in the feature we will take advantage of other features. installing mongodb mongodb is a leading nosql database at the time of this writing. in the databases ecosystem, mongodb falls under the document databases category. these are databases where each record and its associated data is thought of as a “document”. document databases have characteristics that make them a good choice for storing unstructured data across multiple servers. there is abundant online documentation on this subject. if you want to learn more, you can start with the document databases page on mongodb’s website. to install mongodb you need to head to the downloads page on mongodb.org and grab the mongodb installer for your platform. if you haven’t worked with mongo, i recommend that at a minimum you go over mongodb’s interactive tutorial so you become familiar with it. installing mongoose mongoose is a javascript library that makes it easy to move data between your application and mongodb databases. it is a layer of abstraction that allows you to create schemas for the data that your application uses, and provides facilities for connecting to mongodb and validating, saving, updating, deleting and retrieving instances of these schemas. the picture below will give you an idea of where mongoose fits in our application’s architecture: you can find installation instructions and a very good introduction to the library on mongoose’s getting started page. designing the public interface of the controller the role of the controller module in our express backend will be to fulfill requests received from the mobile application: as at this point in this series of tutorials we are only concerned with the user registration, login and logout features of the application, we will create controller methods to handle these functions. we need the controller module to respond to the following requests: register user log on a user log off a user initiate a password reset for a user finalize a password reset for a user based on these requests, we will design a controller with the following public methods. controller.register(newuser, callback): this method will register a new user with the backend by saving the user’s profile in the mongodb database. parameters: newuser – the user to register in the database callback – a function that will receive the results of the registration attempt. returns: callback – the callback function passed in the arguments. controller.logon(email, password, callback): this method will logon a user if the supplied email and password are valid. if the logon attempt succeeds, the method will add the user’s profile to a private “session” variable in the controller. parameters: email – the user’s email. password – the user’s password. callback – a function that will receive the results of the logon attempt. returns: callback – the callback function passed in the arguments. controller.logoff(): this method will log off a user by delete the user’s profile data stored in the controller’s private “session” variable. controller.resetpassword(email, callback): this method will send the user an email containing a password reset link. the link will contain a unique identifier string that will be used in the controller.resetpasswordfinal method. parameters: email – the user’s email address. callback – a function that will receive the results of the reset password attempt. returns: callback – the callback function passed in the arguments. controller.resetpasswordfinal(email, newpassword, passwordresethash, callback): this method will reset a user’s password. parameters: email – the user’s email address. newpassword – the user’s new password. passwordresethash – a unique identifier sent to the user via email from the controller.resetpassword method. callback – a function that will receive the results of the reset password attempt. returns: callback – the callback function passed in the arguments. controller.setsession(session): this method will set the controller’s private “session” variable. parameters: session – the value for the controller’s “session” variable. controller.getsession(): this method will return a reference to the controller’s private session variable. returns: session – the internal “session” variable. creating a model using mongoose as explained in the mongoose documentation , the mongoose model automatically inherits a number of methods (such as create, save, remove and find) that allow us to store and retrieve model instances from a mongodb database. we will use mongoose’s help to create a model of a user. let’s create the user.js file in the model directory. in the file, we will define the following mongoose schema: var mongoose = require('mongoose'); var schema = mongoose.schema; var userschema = new schema({ email: string, firstname: string, lastname: string, passwordhash: string, passwordsalt: string }); module.exports = mongoose.model('user', userschema); the model’s properties are the user’s attributes we want to capture (email, first name and last name), as well as a hash of the user’s password and the salt value that we used to create the password’s hash. as you will see later, storing a password’s hash and salt will allow us to authenticate users without needing to store their passwords in our database. the apiresponse class i mentioned a class called apiresponse in the majority of the methods that make the controller’s public interface. this is a data transfer class that will help us move data out of the controller. let’s create the api-response.js file in the models directory. in the file, let’s type the following definition: var apiresponse = function (cnf) { this.success = cnf.success; this.extras = cnf.extras; }; module.exports = apiresponse; any request sent to the controller will eventually produce an apiresponse instance. as its name indicates, the success property of apiresponse will signal whether the request succeeded or not. the extras property will be a javascript object containing any additional data that the controller wants to send out as part of the response. the apimessages class when the success property of the apiresponse instance is false, the data sent in the extras property can include information about what caused the failure. we will define these causes in a class that we will call apimessages. let’s create the api-messages.js file in the models directory. we will define the apimessages class as follows: var apimessages = function () { }; apimessages.prototype.email_not_found = 0; apimessages.prototype.invalid_pwd = 1; apimessages.prototype.db_error = 2; apimessages.prototype.not_found = 3; apimessages.prototype.email_already_exists = 4; apimessages.prototype.could_not_create_user = 5; apimessages.prototype.password_reset_expired = 6; apimessages.prototype.password_reset_hash_mismatch = 7; apimessages.prototype.password_reset_email_mismatch = 8; apimessages.prototype.could_not_reset_password = 9; module.exports = apimessages; as the code indicates, we are defining the reasons that can cause a controller request to fail. they are basically the different error conditions that we anticipate can occur inside the controller. creating the userprofilemodel class the data sent in the extras property of an apiresponse instance can also include a read-only version of the user’s profile. we will create the userprofilemodel class to model this entity. instances of this class will help us pass user data from the database to the outer layers of the backend, and ultimately the mobile application, without exposing sensitive information such as the password hash and salt values. in the models folder, let’s create the user-profile.js file. then, type the userprofilemodel definition: var userprofilemodel = function(cnf) { this.email = cnf.email, this.firstname = cnf.firstname, this.lastname = cnf.lastname }; module.exports = userprofilemodel; in the model we defined three properties to hold the user’s first name, last name and email. this gives us a nice data transfer object that we can send from the controller out to the mobile app when the mobile app needs to display these data. we are not storing password information in instances of this model so there is no opportunity for this information to be pulled from the database and sent out as part of an http response. creating the controller it’s finally time to turn our attention to the controller itself. let’s create the account.js file in the controllers directory. we will declare the controller as follows: var accountcontroller = function (usermodel, session, mailer) { this.crypto = require('crypto'); this.uuid = require('node-uuid'); this.apiresponse = require('../models/api-response.js'); this.apimessages = require('../models/api-messages.js'); this.userprofilemodel = require('../models/user-profile.js'); this.usermodel = usermodel; this.session = session; this.mailer = mailer; }; module.exports = accountcontroller; notice that we are injecting three dependencies into the controller. the usermodel argument is an instance of the user mongoose class that we created a few minutes ago. as you already know, this is an object that knows how to save and retrieve user data from the mondodb database. the session argument is an object that the controller will use to store session data. the mailer argument is a helper object that the controller will use to send the password reset email to the user. what we are doing here is using a dependency injection approach by passing to the controller some of the entities it needs to do its job. this will make it really easy for us to test the controller using mock objects, without having to instance the database, session and mailer objects that we will use in production. in the next chapter of this tutorial you will see how this is done when we create the tests for the controller. we are also declaring a number of variables inside the controller. the crypto and uuid variables refer to the node.crypto and node-uuid modules, which we will use to generate password hashes and unique identifiers needed when we register and log on users. the apiresponse, apimessages and userprofile internal variables refer to the model classes with the same names that we created a few minutes ago. the session getter and setter methods let’s move on to implementing the controller’s public interface that we designed earlier. first, we will create the setter and getter methods for the session, immediately below the controller’s declaration: accountcontroller.prototype.getsession = function () { return this.session; }; accountcontroller.prototype.setsession = function (session) { this.session = session; }; we will use these methods to set or grab a reference to the controller’s session variable. the hashpassword method we will use this method to create a cryptographically-strong pseudo random hash of a password: accountcontroller.prototype.hashpassword = function (password, salt, callback) { // we use pbkdf2 to hash and iterate 10k times by default var iterations = 10000, keylen = 64; // 64 bit. this.crypto.pbkdf2(password, salt, iterations, keylen, callback); }; within hashpassword, we call crypto.pbkdf2, which uses a pseudorandom function to derive a key of the given length from the given password, salt and number of iterations. remember that we will save this hash in the database, instead of saving the password in clear text or encrypted. this is a good security measure because it’s very difficult to use the hash to obtain the original password without knowing the function used, salt, iteration and keylen values. the logon method next, we will create the logon method: accountcontroller.prototype.logon = function(email, password, callback) { var me = this; me.usermodel.findone({ email: email }, function (err, user) { if (err) { return callback(err, new me.apiresponse({ success: false, extras: { msg: me.apimessages.db_error } })); } if (user) { me.hashpassword(password, user.passwordsalt, function (err, passwordhash) { if (passwordhash == user.passwordhash) { var userprofilemodel = new me.userprofilemodel({ email: user.email, firstname: user.firstname, lastname: user.lastname }); me.session.userprofilemodel = userprofilemodel; return callback(err, new me.apiresponse({ success: true, extras: { userprofilemodel:userprofilemodel } })); } else { return callback(err, new me.apiresponse({ success: false, extras: { msg: me.apimessages.invalid_pwd } })); } }); } else { return callback(err, new me.apiresponse({ success: false, extras: { msg: me.apimessages.email_not_found } })); } }); }; inside logon we first create the me variable to hold a reference to the accountcontroller instance that we can use inside callback functions that we will create inline. next, we call the findone method of the usermodel instance to try to find a user with the same email in the mongodb database. the findmethod is provided by mongoose. remember that usermodel is an instance of the user model that we create with mongoose’s help. if the call to findone produces an error, we immediately invoke the callback argument, passing an apiresponse instance where the success property is set to false and the extra property contains a message that explains that there was a database error. if the call to findone produces a user, we proceed to hash the password provided by the user who is attempting to log on, and compare the hash to the password hash of the user that we found in the database. if the hashes are equal, it means that the user attempting to log on provided a valid password and we can move on to create a userprofile instance and save it to the controller’s session variable. we then invoke the callback function, setting the response’s success property to true and passing the userprofile instance in the extras property of the response. when the hashes don’t match, we invoke the callback function, setting the response’s success property to false and passing an “invalid password” reason in the extras property. finally, if the call to findone does not produce a user, we invoke the callback function with a response where the extras property contains a message indicating that the provided email was not found. the logoff method we will use the logoff method to terminate a user’s session: accountcontroller.prototype.logoff = function () { if (this.session.userprofilemodel) delete this.session.userprofilemodel; return; }; to terminate the session we simply destroy the userprofile instance that we previously saved in the controller’s session variable. the register method the controller’s register method allows a user to register with the application: accountcontroller.prototype.register = function (newuser, callback) { var me = this; me.usermodel.findone({ email: newuser.email }, function (err, user) { if (err) { return callback(err, new me.apiresponse({ success: false, extras: { msg: me.apimessages.db_error } })); } if (user) { return callback(err, new me.apiresponse({ success: false, extras: { msg: me.apimessages.email_already_exists } })); } else { newuser.save(function (err, user, numberaffected) { if (err) { return callback(err, new me.apiresponse({ success: false, extras: { msg: me.apimessages.db_error } })); } if (numberaffected === 1) { var userprofilemodel = new me.userprofilemodel({ email: user.email, firstname: user.firstname, lastname: user.lastname }); return callback(err, new me.apiresponse({ success: true, extras: { userprofilemodel: userprofilemodel } })); } else { return callback(err, new me.apiresponse({ success: false, extras: { msg: me.apimessages.could_not_create_user } })); } }); } }); }; the first step that we take in register is to check if a user with the same email address of the user that is attempting to register exists in the database. as we did in the logon method, if there is a database error we will immediately invoke the callback function and send out an apiresponse instance explaining that there was a database error. if we find an user that has the same email address of the user that is attempting to register, we also stop the registration process, as we cannot have two users with the same email address. in this case the extras property of the apiresponse instance that we send out contains a message explaining that the email address already exists. if we don’t find the email address in the database, we proceed to save the new user by invoking save method (inherited from mongooose) of the user class. the save method produces a numberaffected argument in its callback function. we check numberaffected to make sure that the new user was saved. if numberaffected is 1, we create a userprofile instance and send it out embedded in an apiresponse object. if numberaffected is not 1, we produce an apiresponse indicating that the registration failed. the resetpassword method the resetpassword method is the first step of the password reset workflow that we defined in the mobile application user registration, login and logout screens tutorial of this series. the method consists of the following code: accountcontroller.prototype.resetpassword = function (email, callback) { var me = this; me.usermodel.findone({ email: email }, function (err, user) { if (err) { return callback(err, new me.apiresponse({ success: false, extras: { msg: me.apimessages.db_error } })); } // save the user's email and a password reset hash in session. we will use var passwordresethash = me.uuid.v4(); me.session.passwordresethash = passwordresethash; me.session.emailwhorequestedpasswordreset = email; me.mailer.sendpasswordresethash(email, passwordresethash); return callback(err, new me.apiresponse({ success: true, extras: { passwordresethash: passwordresethash } })); }) }; in order to initiate a password reset sequence, users need to provide their email address. inside resetpassword we use the provided email address to retrieve the user’s record from the database. if the record exists, we create a unique identifier called passwordresethash, and pass this identifier and the user’s email address to the mailer object’s sendpasswordresethash method. this method sends a message to the user, containing the unique identifier and a password reset link that they can use to change their password. we will implement the mailer module in the next chapter of this tutorial. inside resetpassword we also save the password reset hash and the user’s email in the controller’s session variable so we can later compare them to the values provided by the user in the final step of the password reset process. if the database doesn’t have a record for the provided email address, we return an apiresponse whose extras property explains that the email was not found. the resetpasswordfinal method users will invoke this method when they access a special web page using the “password reset” link inside the email that they will receive after they perform the first step of the password reset process. here’s the code for the method: accountcontroller.prototype.resetpasswordfinal = function (email, newpassword, passwordresethash, callback) { var me = this; if (!me.session || !me.session.passwordresethash) { return callback(null, new me.apiresponse({ success: false, extras: { msg: me.apimessages.password_reset_expired } })); } if (me.session.passwordresethash !== passwordresethash) { return callback(null, new me.apiresponse({ success: false, extras: { msg: me.apimessages.password_reset_hash_mismatch } })); } if (me.session.emailwhorequestedpasswordreset !== email) { return callback(null, new me.apiresponse({ success: false, extras: { msg: me.apimessages.password_reset_email_mismatch } })); } var passwordsalt = this.uuid.v4(); me.hashpassword(newpassword, passwordsalt, function (err, passwordhash) { me.usermodel.update({ email: email }, { passwordhash: passwordhash, passwordsalt: passwordsalt }, function (err, numberaffected, raw) { if (err) { return callback(err, new me.apiresponse({ success: false, extras: { msg: me.apimessages.db_error } })); } if (numberaffected < 1) { return callback(err, new me.apiresponse({ success: false, extras: { msg: me.apimessages.could_not_reset_password } })); } else { return callback(err, new me.apiresponse({ success: true, extras: null })); } }); }); }; to reset their password a user will need to provide their email address and a new password, along with the password reset hash that we sent them in the password reset email generated from the resetpassword method. we will save the user from having to type the password reset hash by embedding the hash in the link inside the password reset email. in the next chapter of this series we will create the mailer class and implement the email features. inside resetpasswordfinal, we first check that the password reset hash is also saved in the controller’s session variable. if the hash does not exist, we return an apiresponse whose extras property explains that the password reset period expired. as a security measure, we want to limit the period of time during which a user can reset their password to the length of a session timeout period. if the password reset hash value stored in the session and the value supplied by the user do not match, we will assume that the user who requested the password reset and the user who is providing the new password are not the same. in such a case we return an apiresponse explaining that there is a mismatch of the hashes. the same logic applies when the email value stored in the session and the value supplied by the user do not match, in which case we return an apiresponse explaining that there is a mismatch of the email addresses. if the password reset hash and email address validations are successful, we proceed to hash the new password and save it by calling the user model’s update method, which is inherited from mongoose. the update method returns the number of records affected by the update operation. we check this value and return an apiresponse that signals to the outside world if the update operation succeeded or not. summary and next steps we just began building the backend for a meeting room booking application that we defined in the first chapter of this series . this is a mongodb and mongoose backend paired to a node.js and express web server. our focus in this article was building a controller module that will handle the user registration, login and logout features of the application. we implemented the controller’s public interface, along with a number of helper classes that will allow the controller to do its work. in the next chapter of this tutorial we will turn our attention to testing the controller, which will take us through choosing a testing library and implementing the tests for the controller’s features. make sure to sign up for miamicoder’s newsletter so you can be among the first to know when next part of this tutorial is available. download the source code download the mongodb and mongoose backend tutorial here: mongodb and mongoose backend for mobile application previous chapters of this series these are the previous parts of this series: mobile app tutorial: the meeting room booking app, part 1 mobile app tutorial: the meeting room booking app, part 2 mobile app tutorial: the meeting room booking app, part 3 mobile ui patterns – a flowchart for user registration, login and logout

In my previous articles I compared 4 frameworks commonly used in communicating with MongoDB from the JVM and found out that in that use-case, Spring Data for MongoDB was the easiest solution. However I did make the remark that it doesn’t use the GeoJSON format to store geolocation coordinates and geometries. I tried to add GeoJSON support before, but couldn’t get the conversion to work propertly. But after some extensive searching I found out that the reason for it not working was my use of Spring Boot: its autoconfiguration for MongoDB does not support custom conversion out of the box. Luckily, the solution was simple: provide an extra configuration that extends from AbstractMongoConfiguration and import that in the Boot application. In that configuration you can override the customConversions() and add your converters. When you compare the geo classes in Spring Data and GeoJSON, I noticed that only a subset of GeoJSON geometries can be mapped on Spring Data geo classes: Point and Polygon. Spring Boot does not support LineString, MultiLineString, MultiPolygon or MultiPoint. However, in your mapped domain classes, you won’t use these normally. Creating a converter that adheres to the GeoJSON format is quite straightforward. import com.mongodb.BasicDBObject import com.mongodb.DBObject import org.springframework.core.convert.converter.Converter import org.springframework.data.convert.ReadingConverter import org.springframework.data.convert.WritingConverter import org.springframework.data.geo.Point import org.springframework.data.geo.Polygon final class GeoJsonConverters { static List> getConvertersToRegister() { return [ GeoJsonDBObjectToPointConverter.INSTANCE, GeoJsonDBObjectToPolygonConverter.INSTANCE, GeoJsonPointToDBObjectConverter.INSTANCE, GeoJsonPolygonToDBObjectConverter.INSTANCE ] } @WritingConverter static enum GeoJsonPointToDBObjectConverter implements Converter { INSTANCE; @Override DBObject convert(Point source) { return new BasicDBObject([type: 'Point', coordinates: [source.x, source.y]]) } } @ReadingConverter static enum GeoJsonDBObjectToPointConverter implements Converter { INSTANCE; @Override Point convert(DBObject source) { def coordinates = source.coordinates as double[] return new Point(coordinates[0], coordinates[1]) } } @WritingConverter static enum GeoJsonPolygonToDBObjectConverter implements Converter { INSTANCE; @Override DBObject convert(Polygon source) { def coordinates = source.points.collect { [it.x, it.y] } return new BasicDBObject([type: 'Polygon', coordinates: coordinates]) } } @ReadingConverter static enum GeoJsonDBObjectToPolygonConverter implements Converter { INSTANCE; @Override Polygon convert(DBObject source) { def coordinates = source.coordinates as double[] return new Point(coordinates[0], coordinates[1]) } } } To add those converters to the Spring context, you’ll have to override some methods in your MongoDB spring configuration class. import com.mongodb.Mongo import org.springframework.beans.factory.annotation.* import org.springframework.boot.SpringApplication import org.springframework.boot.autoconfigure.EnableAutoConfiguration import org.springframework.context.annotation.* import org.springframework.data.mongodb.config.AbstractMongoConfiguration import org.springframework.data.mongodb.core.convert.* @EnableAutoConfiguration @ComponentScan @Configuration @Import([MongoComparisonMongoConfiguration]) class MongoComparison { static void main(String[] args) { SpringApplication.run(MongoComparison, args); } } @Configuration class MongoComparisonMongoConfiguration extends AbstractMongoConfiguration { @Autowired Mongo mongo; @Value("\${spring.data.mongodb.database}") String databaseName; @Override protected String getDatabaseName() { return databaseName } @Override Mongo mongo() throws Exception { return mongo } @Override CustomConversions customConversions() { def customConverters = [] customConverters << GeoJsonConverters.convertersToRegister return new CustomConversions(customConverters.flatten()) } } As Spring Boot already provides the configuration of the Mongo instance and the name of the database, we can reuse these in the MongoDB configuration class. The custom conversions take preference over the existing ones for Point and Polygon. I’ll be writing a library this weekend to add support for all GeoJSON geometries in Spring Data for MongoDB. However, I already noticed it’ll be very hard to provide support for those in generated query methods in repositories, but with annotated queries being possible, I don’t think this will be a big issue but we’ll see.

recently some pwc tech supremos wrote an article: agile coding in enterprise it: code small and local . subsections: moving away from the monolith why microservices? msa: a think-small approach for rapid development thinking the msa way: minimalism is a must where msa makes sense in msa, integration is the problem, not the solution conclusion msa is short for microservices architecture(s), in the above article. the article posits that microservices is the antidote to monoliths. it doesn’t mention cookie cutter scaling at all, which is another antidote to monoliths, with the right build infrastructure and devops. here’s a view of hypothetical architecture a company could deploy if they were doing microservices: w is web server. p and q don’t stand for anything in particular. here’s the same solution as cookie-cutter scaling, and the alternate (historical) choice of monolith to the right of it: the cookie cutter approach will often leverage components that are dependency injected into each other, and though monoliths might be the same today, pre 2004 they were probably hairballs of singletons (the design patten, not the springframework idiom). continuous delivery, agile? here’s one excerpt that confuses me: " … makes no sense to design and develop software over an 18-month process to accommodate all possible use cases when those use cases can change unexpectedly and the life span of code modules might be less than 18 months…. as i recall, the 18 month-delay problem was solved previously. agile methodologies principally, and continuous delivery/deployment in more recent times. it does not matter whether you’re compiling a monolith, a cookie-cutter solution, old soa services, or microservices, the 18-month fear isn’t real if you’re doing agile and/or cd. agile and cd were increasing the release cadence, and allowing the organization to pivot faster before microservices. it doesn’t matter whether you’ve got a monolith, something cookie-cutter scaled, or soa (micro or not), you’re going to be able to benefit from agile practices and devops setup that facilitates cd. in something like 30 thoughtworks client engagements since 2002, i have not seen the 18-month process at all. in fact i last encountered it in 1997 on an as/400 project, which was the last time i saw a waterfall process being championed. build(s) and trunk elsewhere there is a suggestion: “each microservice [has] its own build, to avoid trunk conflict”. that isn’t unique to microservices, of course. component based systems today also have a multiple build file (module) structure in a source tree. hopefully “trunk” mentioned is alluding to trunk based development, as i would recommend. build technologies this is a expansion on the above, and you can skip this paragraph if you want. hierarchical build systems like maven has allow you to have one build file per module (whether that’s a service or a simple jar destined for the classpath of a bigger thing). buck has a build grammar that allows for a build to grow/shrink/change based on what is being built (from implicitly shared source). maven is for the java ecosystem, while buck promises to be multi-language. both are doing multi-module builds for the sake of a composed or servicified deployment. both maven and buck are presently competing to draw the most reduced set of compile/test/deploy operations for the changes since last build for a hierarchy of modules. anyway, what is it we are striving for? what we want is to develop cheaply, and to deploy smoothly and often, without defect. we want the ability to deploy without large permanent or temporary headcount overseeing or participating in deployment. aside from development costs, and support/operation, deployment costs are a potentially big factor in total cost of ownership. what i like about cookie-cutter is the uniformity of the deployable things. the team size for deployment of such a thing doesn’t grow with the numbers of nodes that binary is being deployed to. at least, if you’re able to automate the deployment to those nodes, and have a strategy for handling the users connected to the stack at redeployment time somehow (sessions or stateless). the uniformity of the deployment is a cheapener, i think. when you have a number of dissimilar services, you might be able to minimize release personnel if you’re only doing one service. if more than one service is being updated in a particular deployment, you’re going to have to concentrate to make sure you don’t experience a multiplier effect for the participants. it is possible of course, to keep the headcount small, but the practice needed beforehand is bigger, which in turn allows for some calmness around the actual deployment. if we’ve stepped away from the project management office thinking that suggests three buggy releases a year (which is more usual than 18 month schedules of old), then we can employ continuous deployment to further eliminate personnel costs around going live. this is something that microservices does well at, but because the most adept proponents design forwards & backwards compatibility into the permutations most likely to co-exist in production. it is at least much quicker to redeploy and bounce one small service, n times than the the cookie-cutter uniform deployment.

both high availability (ha) and disaster recovery (dr) have been essential it topics. fundamentally ha is about fault tolerance relevant to the availability of an examined subject like application, database, vms, etc. while dr roots on the ability to resume operations in the aftermath of a catastrophic event. a fundamental difference of these two is that ha expects no down time and no data loss, while dr does. they are different issues and should be addressed separately. background for many it shops, either ha or dr has been a high risk and high cost item. both are essential to business continuity, while traditionally tough technical problems to solve with very significant and long-term commitments on resources. not only they are technically challenging, but a continual cost-cutting which has become an it standard practice in the past two decades makes purchasing hardware/software and constructing either ha or dr solution on premises further distant from it’s financial and technical realties. sense of urgency too often, the technical challenges and resource commitments overwhelm it and turn ha and dr into academic discussions, or symbolic items on a project checklist. at the same time, information is rapidly exploding as internet, mobility and social-network are becoming integral in our daily lives and businesses. there are progressively more data to process and store. for many businesses, the needs for ha and dr is urgent for better managing risks. and continual availability and on-demand recoverability of it are becoming increasingly critical. this is the reality, now the good news is that the recent introduction of cloud computing has fundamentally changed how an ha or dr solution can be implemented. microsoft azure is a vivid example of ha and dr solutions with significantly reduced the required financial commitment and involved technical complexities. the traditional approach by establishing redundancy and acquiring a physical dr site with long-term resources and financial commitments is now largely replaced with consumable services which can be configured in minutes by mouse-clicking and with a manageable cost structure based on usage. ha and dr have become it solutions which are financially realistic and technically feasible for businesses in all sizes. ha, redundancy, and microsoft azure lrs ha is to eliminate a single point of failure of an examined component, an application for example. it denotes a strategy to employ redundancy such that a target application can and will continue being available without downtime while experiencing a failure of hosting hardware or software. there are various and well-developed ha solutions like a hyper-v host cluster using redundant hardware to eliminate a single point of failure of hosting os or hardware, and an application cluster for eliminating a single point of failure by running the application in multiple vm instances with a synchronous state. although ha implementations may vary, the fundamental principle nevertheless remains the same. ha expects neither downtime nor data loss while experiencing an outage of a target hardware or software. ha has become dramatically simple in microsoft azure. basically, all data written to disk in microsoft azure are kept at least in the so-called lrs, locally redundant storage. lrs replicates a transaction synchronously to three different storage nodes across fault domains and upgrade domains within the same region for durability. in layman’s terms, microsoft azure by default maintains at least three copies of user data to achieve ha. dr, replication, and microsoft azure grs dr is about having a plan and backups in place to resume operations in the aftermath of a catastrophic event. unplanned outage is assumed in a dr scenario, therefore some data loss is also expected. notice that ha and dr are different business problems and addressed differently. while both ha and dr are based on applying redundancy, i.e. a source and replicas, or multiple identical nodes of an examines component like application instance, databases, or vms, there are however differences between the two. a dr solution generally employs replicas or backups, are implemented with asynchronous processes, and expects an outage of a source and with some data loss in transit while the outage occurs. while ha requires a logical representation with a real-time integrity using synchronous processes across all participating nodes, expects neither downtime nor data loss while experiencing an outage of a participating node. for a critical workload, one approach of dr is to establish geo-replication to address an outage of an entire geographic area caused by a natural disaster, for example. the concern is that a catastrophic event may impact an entire geographic area causing a datacenter where a mission critical application is being hosted becomes unavailable for an extended period of time. in microsoft azure, geo redundant storage or grs is the default and an optional setting, as shown above, while configuring a storage account. grs will queue a transaction committed to lrs as an asynchronous replication to a secondary region, a few hundreds miles away from the primary region where a storage account is originated. at the secondary region, data is also stored in lrs, i.e. made durable by replicating it to three storage nodes. specifically, a microsoft azure storage account configured with grs essentially maintains three replicas locally for high availability, and replicates the content and maintains three replicas at a secondary datacenter a few hundreds miles away for dr. so all are six copies, three locally and three remotely. all these are configured by one, yes one mouse click from a dropdown list while creating a storage account. the above is a conceptual model illustrated a data flow of grs. grs replication has little performance impact on an application since application data are committed to lrs in real-time while replication to grs is queued, i.e. asynchronously. a write to lrs is synchronous and in real-time, once committed, the changes are expected within 15 minutes to be asynchronously replicated to the secondary site. for a ra-grs storage account, in addition to one primary endpoint for read/write operations as it is in a grs, there is also one secondary endpoint as read only becomes available as shown below. the cost implications of grs or ra-grs include the additional storage and the transmission costs for egress traffic, as applicable, of the secondary datacenter. ingress traffic is free . and microsoft azure storage sla offers 99.9% availability and a cost calculator is also available. microsoft azure recovery services so far, much is about backing up or replicating data. to successfully restore, a dr plan must be put in place and ensure its availability upon a dr scenario in progress. either placing a dr plan at a primary site where the source is or a secondary site where a replica stays has some issues and concerns. keeping a dr plan at the source site where all the resources are in place and on-the-job trainings seems logical. or does it? dr is assuming a catastrophic event over an extended geographic areas where the source site is experiencing an outage. in such case, keeping a dr plan in the source site defeats the purpose. maintaining a dr plan at the secondary site is the choice then. in a dr scenario, a recovery site is to be brought on line within a expected period of time according to a dr plan, and having the dr plan right there and then at a recovery site makes all the sense. or does it? this decision introduces a number of requirements including the physical readiness, the timeliness, and the financial implications on securing and maintaining a dr plan at a remote physical facility. for a vmm server running on system center 2012 sp1 or later, an idea, reliable and straightforward way is to use azure recovery services to maintain a dr plan as shown below. and for any backup needs, using cloud as a backup site makes backing up and restoring data an anytime anywhere operation. azure site recovery vault this service essentially acts as the director of a dr process. it orchestrates and manages the protection and failover of vms in clouds managed by virtual machine manager 2012 sp1 or later. a noticeable advantage is the ability to test a recovery configuration, exercise a proactive failover and recovery, and automate recovery in the event of a site outage. the sla of site recovery services is 99.9% availability to ensure a configured dr plan is always in place with expected updates. this is a dr solution that it can implement, simulate, verify, bring online and be absolutely confident with the readiness. azure backup vault this is a reliable, scalable and inexpensive data protection solution with zero capital investment and extremely low operational expense. like other secure communication with microsoft azure, you will first upload a public certificate to microsoft azure. then download the backup agent to register a target server with the backup vault. then select what to be backed up. both microsoft azure backup sla (99.9% availability) and cost calculator are available for better assessing the solution. closing thoughts form an application’s view, ha is an on-going event while dr is an anticipation. ha and dr are different business problems and should be addressed differently. nevertheless, microsoft azure provides a single platform to gracefully address ha with lrs, dr with grs, and dr orchestration with recovery services, and all with published sla s and a predictable cost structure . going forward, it pros can now include ha and dr as a reliable, scalable and relatively inexpensive proposition by employing microsoft azure as a solution platform. call to action register at microsoft virtual academy, http://aka.ms/mva1 , and train yourself on microsoft azure by taking the track of courses. go to http://aka.ms/azure200 and acquire a free trial subscription and assess microsoft azure for ha and dr solutions. review my recommended content at http://aka.ms/recommended .

When comparing two objects, the equals method is used to return true if they are identical. Typically, this leads to the following code : if (name.equals("Jim")) { } The problem here is that whether intended or not, it is quite possible that the name value is null, in which case a null pointer exception would be thrown. A better practice is to execute the equals method of the string constant “Jim” instead : if ("Jim".equals(name)) { } Since the constant is never null, a null exception will not be thrown, and if the other value is null, the equals comparison will fail. If you are using Java 7 or above, the new Objects class has an equals static method to compare two objects while taking null values into account. if (Objects.equals(name,"Jim")) { } Alternatively if you are using a java version prior to Java 7, but using the guava library you can use the Objects class which has a static equal() method that takes two objects and handles null cases for you. It should also be noted that there are probably a number of other implementations in various libraries (i.e. Apache Commons)

When I needed to do prototyping, proof of concept or play with some new technology in free time, starting new project was always a little annoying barrier with Maven. Have to say that setting up Maven project is not hard and you can use Maven Archetypes. But Archetypes are often out of date. Who wants to play with old technologies? So I always end up wiring in dependencies I wanted to play with. Not very productive spent time. But than Spring Boot came to my way. I fell in love. In last few months I created at least 50 small playground projects, prototypes with Spring Boot. Also incorporated it at work. It’s just perfect for prototyping, learning, microservices, web, batch, enterprise, message flow or command line applications. You have to be dinosaur or be blind not to evaluate Spring Boot for your next Spring project. And when you finish evaluate it, you will go for it. I promise. I feel a need to highlight how easy is Black Box Testing of Spring Boot microservice. Black Box Testing refers to testing without any poking with application artifact. Such testing can be called also integration testing. You can also perform performance or stress testing way I am going to demonstrate. Spring Boot Microservice is usually web application with embedded Tomcat. So it is executed as JAR from command line. There is possibility to convert Spring Boot project into WAR artifact, that can be hosted on shared Servlet container. But we don’t want that now. It’s better when microservice has its own little embedded container. I used existing Spring’s REST service guide as testing target. Focus is mostly on testing project, so it is handy to use this “Hello World” REST application as example. I expect these two common tools are set up and installed on your machine: Maven 3 Git So we’ll need to download source code and install JAR artifact into our local repository. I am going to use command line to download and install the microservice. Let’s go to some directory where we download source code. Use these commands: git clone [email protected]:spring-guides/gs-rest-service.git cd gs-rest-service/complete mvn clean install If everything went OK, Spring Boot microservice JAR artifact is now installed in our local Maven repository. In serious Java development, it would be rather installed into shared repository (e.g. Artifactory, Nexus,… ). When our microservice is installed, we can focus on testing project. It is also Maven and Spring Boot based. Black box testing will be achieved by downloading the artifact from Maven repository (doesn’t matter if it is local or remote). Maven-dependency-plugin can help us this way: org.apache.maven.plugins maven-dependency-plugin copy-dependencies compile copy-dependencies gs-rest-service true It downloads microservice artifact into target/dependency directory by default. As you can see, it’s hooked to compile phase of Maven lifecycle, so that downloaded artifact is available during test phase. Artifact version is stripped from version information. We use latest version. It makes usage of JAR artifact easier during testing. Readers skilled with Maven may notice missing plugin version. Spring Boot driven project is inherited from parent Maven project called spring-boot-starter-parent. It contains versions of main Maven plugins. This is one of the Spring Boot’s opinionated aspects. I like it, because it provides stable dependencies matrix. You can change the version if you need. When we have artifact in our file system, we can start testing. We need to be able to execute JAR file from command line. I used standard JavaProcessBuilder this way: public class ProcessExecutor { public Process execute(String jarName) throws IOException { Process p = null; ProcessBuilder pb = new ProcessBuilder("java", "-jar", jarName); pb.directory(new File("target/dependency")); File log = new File("log"); pb.redirectErrorStream(true); pb.redirectOutput(Redirect.appendTo(log)); p = pb.start(); return p; } } This class executes given process JAR based on given file name. Location is hard-coded to target/dependency directory, where maven-dependency-plugin located our artifact. Standard and error outputs are redirected to file. Next class needed for testing is DTO (Data transfer object). It is simple POJO that will be used for deserialization from JSON. I use Lombok project to reduce boilerplate code needed for getters, setters, hashCode and equals. @Data @AllArgsConstructor @NoArgsConstructor public class Greeting { private long id; private String content; } Test itself looks like this: public class BlackBoxTest { private static final String RESOURCE_URL = "http://localhost:8080/greeting"; @Test public void contextLoads() throws InterruptedException, IOException { Process process = null; Greeting actualGreeting = null; try { process = new ProcessExecutor().execute("gs-rest-service.jar"); RestTemplate restTemplate = new RestTemplate(); waitForStart(restTemplate); actualGreeting = restTemplate.getForObject(RESOURCE_URL, Greeting.class); } finally { process.destroyForcibly(); } Assert.assertEquals(new Greeting(2L, "Hello, World!"), actualGreeting); } private void waitForStart(RestTemplate restTemplate) { while (true) { try { Thread.sleep(500); restTemplate.getForObject(RESOURCE_URL, String.class); return; } catch (Throwable throwable) { // ignoring errors } } } } It executes Spring Boot microservice process first and wait unit it starts. To verify if microservice is started, it sends HTTP request to URL where it’s expected. The service is ready for testing after first successful response. Microservice should send simple greeting JSON response for HTTP GET request. Deserialization from JSON into our Greeting DTO is verified at the end of the test. Source code is shared on Github.

This article represents the first Proof of Concept from series described in the previous article 4 Hands-On Approaches to Improve Your Data Access Layer Implementation and it presents how to implement Ehcache over MyBatis, how to achieve an optim configuration for it and personal opinions of the author about the chosen approach for the Data Access Layer. Throughout my research on caching over MyBatis I have discovered that Ehcache is the first option among developers when they need to implement a cache mechanism over MyBatis, using a 3rd party library. Ehcache is probably so popular because it represents an open source, java-based cache, available under an Apache 2 license. Also, it scales from in-process with one or more nodes through to a mixed in-process/out-of-process configuration with terabyte-sized caches. In addition, for those applications needing a coherent distributed cache, Ehcache uses the open source Terracotta Server Array. Last but not least, among its adopters is the Wikimedia Foundation that uses Ehcache to improve the performance of its wiki projects. Within this article, the following aspects will be addressed: 1. How will an application benefit from caching using Ehcache? Ehcache's features will be detailed in this section. 2. Hands-on implementation of the EhCachePOC project - in this section the key concepts of EhCache will be explored through a hands on implementation. 3. Summary - How has the application performance been improved after this implementation? Code of all the projects that will be implemented can be found at https://github.com/ammbra/CacherPoc or if you are interested only in the current implementation, you can access it here: https://github.com/ammbra/CacherPoc/tree/master/EhCachePoc How will an application benefit from caching using Ehcache? The time taken for an application to process a request principally depends on the speed of the CPU and main memory. In order to "speed up" your application you can perform one or more of the following: improve the algorithm performance achieve parallelisation of the computations across multiple CPUs or multiple machines upgrade the CPU speed As explained in the previous article, high availability applications should perform a small amount of actions with the database. Since the time taken to complete a computation depends principally on the rate at which data can be obtained, then the application should be able to temporarily store computations that may be reused again. Caching may be able to reduce the workload required, this means a caching mechanism should be created! Ehcache is described as : Fast and Light Weight , having a simple API and requiring only a dependency on SLF4J. Scalable to hundreds of nodes with the Terracotta Server Array, but also because provides Memory and Disk store for scalability into gigabytes Flexible because supports Object or Serializable caching; also provides LRU, LFU and FIFO cache eviction policies Standards Based having a full implementation of JSR107 JCACHE API Application Persistence Provider because it offers persistent disk store which stores data between VM restarts JMX Enabled Distributed Caching Enabler because it offers clustered caching via Terracotta and replicated caching via RMI, JGroups, or JMS Cache Server (RESTful, SOAP cache Server) Search Compatible, having a standalone and distributed search using a fluent query language Hands-on implementation of the EhCachePOC project The implementation of EhCachePoc will look as described in the diagram below: In order to test Ehcache performance through a POC(proof of concept) project the following project setup is performed: 1. Create a new Maven EJB Project from your IDE (this kind of project is platform provided by NetBeans but for those that use eclipse, here is an usefull tutorial) . In the article this project is named EhCachePOC. 2. Edit the project's pom by adding required jars : org.mybatis mybatis 3.2.6 org.mybatis.caches mybatis-ehcache 1.0.2 log4j log4j 1.2.17 net.sf.ehcache ehcache 2.7.0 org.slf4j slf4j-log4j12 1.7.5 3.Add your database connection driver, in this case apache derby: org.apache.derby derbyclient 10.11.1.1 4. Run mvn clean and mvn install commands on your project. Now the project setup is in place, let's go ahead with MyBatis implementation : 1. Configure under resources/com/tutorial/ehcachepoc/xml folder the Configuration.xml file with : 2. Create in java your own SQLSessionFactory implementation. For example, create something similar to com.tutorial.ehcachepoc.config. SQLSessionFactory : public class SQLSessionFactory { private static final SqlSessionFactory FACTORY; static { try { Reader reader = Resources.getResourceAsReader("com/tutorial/ehcachepoc/xml/Configuration.xml"); FACTORY = new SqlSessionFactoryBuilder().build(reader); } catch (Exception e){ throw new RuntimeException("Fatal Error. Cause: " + e, e); } } public static SqlSessionFactory getSqlSessionFactory() { return FACTORY; } } 3. Create the necessary bean classes, those that will map to your sql results, like Employee: public class Employee implements Serializable { private static final long serialVersionUID = 1L; private Integer id; private String firstName; private String lastName; private String adress; private Date hiringDate; private String sex; private String phone; private int positionId; private int deptId; public Employee() { } public Employee(Integer id) { this.id = id; } @Override public String toString() { return "com.tutorial.ehcachepoc.bean.Employee[ id=" + id + " ]"; } } 4. Create the IEmployeeDAO interface that will expose the ejb implementation when injected: public interface IEmployeeDAO { public List getEmployees(); } 5. Implement the above inteface and expose the implementation as a Stateless EJB (this kind of EJB preserves only its state, but there is no need to preserve its associated client state): @Stateless(name = "ehcacheDAO") @TransactionManagement(TransactionManagementType.CONTAINER) public class EmployeeDAO implements IEmployeeDAO { private static Logger logger = Logger.getLogger(EmployeeDAO.class); private SqlSessionFactory sqlSessionFactory; @PostConstruct public void init() { sqlSessionFactory = SQLSessionFactory.getSqlSessionFactory(); } @Override public List getEmployees() { logger.info("Getting employees....."); SqlSession sqlSession = sqlSessionFactory.openSession(); List results = sqlSession.selectList("retrieveEmployees"); sqlSession.close(); return results; } } 5. Create the EmployeeMapper.xml that contains the query named "retrieveEmployees" select id, first_name, last_name, hiring_date, sex, dept_id from employee If you remember the CacherPOC setup from the previously article, then you can test your implementation if you add EhCachePOC project as dependency and inject the IEmployeeDAO inside the EhCacheServlet. Your CacherPOC pom.xml file should contain : ${project.groupId} EhCachePoc ${project.version} and your servlet should look like: @WebServlet("/EhCacheServlet") public class EhCacheServlet extends HttpServlet { private static Logger logger = Logger.getLogger(EhCacheServlet.class); @EJB(beanName ="ehcacheDAO") IEmployeeDAO employeeDAO; private static final String LIST_USER = "/listEmployee.jsp"; @Override protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException { String forward= LIST_USER; List results = new ArrayList(); for (int i = 0; i < 10; i++) { for (Employee emp : employeeDAO.getEmployees()) { logger.debug(emp); results.add(emp); } try { Thread.sleep(3000); } catch (Exception e) { logger.error(e, e); } } req.setAttribute("employees", results); RequestDispatcher view = req.getRequestDispatcher(forward); view.forward(req, resp); } } Run your CacherPoc implementation to check if your Data Access Layer with MyBatis is working or download the code provided at https://github.com/ammbra/CacherPoc But if a great amount of employees is stored in database, or perhaps the retrieval of a number of 10xemployeesNo represents a lot of workload for the database. Also, can be noticed that the query from the EmployeeMapper.xml retrieves data that almost never changes (id, first_name, last_name, hiring_date, sex cannot change; the only value that might change in time is dept_id); so a caching mechanism can be used. Below is described how this can be achieved using EhCache: 1. Configure directly under the resources folder the ehcache.xml file with: This xml explains that the Memory Store is used for an LRU (Last Recently Used) caching strategy, sets the limits for the number of elements allowed for storage, their time to be idle and their time to live. The Memory Store strategy is often chosen because is fast and thread safe for use by multiple concurrent threads, being backed by LinkedHashMap. Also, all elements involved in the caching process are suitable for placement in the Memory Store. Another approach can be tried: storing cache on disk. This can be done by replacing the ehcache tag content with: diskStore path="F:\\cache" /> Unlike the memory store strategy, the disk store implementation is suitable only for elements which are serializable can be placed in the off-heap; if any non serializable elements are encountered, those will be removed and WARNING level log message emitted. The eviction is made using the LFU algorithm and it is not configurable or changeable. From persistency point of view, this method of caching allows control of the cache by the disk persistent configuration; if false or omitted, disk store will not persist between CacheManager restarts. 2. Update EmployeeMapper.xml to use the previous implemented caching strategy: select id, first_name, last_name, hiring_date, sex, dept_id from employee By adding the line and specifying on the query useCache="true" you are binding the ehcache.xml configuration to your DataAccessLayer implementation. Clean, build and redeploy both EhCachePOC and CacherPoc projects; now retrieve your employees for two times in order to allow the in-memory cache to store your values. When you run your query for the first time, your application will execute the query on the database and retrieve the results. Second time you access the employee list, your application will access the in-memory storage. Summary - How has the application performance been improved after this implementation? An application's performances depend on a multitude of factors how many times a cached piece of data can and is reduced by the application the proportion of the response time that is alleviated by caching Amdhal's law can be used to estimate the system's speed up : where P is proportion speed up and S is speed up. Let's take the application from this article as example and calculate the speed up. When the application ran the query without caching,a JDBC transaction is performed and in your log will be something similar to : INFO: 2014-11-27 18:01:30,020 [EmployeeDAO] INFO com.tutorial.hazelcastpoc.dao.EmployeeDAO:38 - Getting employees..... INFO: 2014-11-27 18:01:39,148 [JdbcTransaction] DEBUG org.apache.ibatis.transaction.jdbc.JdbcTransaction:98 - Setting autocommit to false on JDBC Connection [org.apache.derby.client.net.NetConnection40@1c374fd] INFO: 2014-11-27 18:01:39,159 [retrieveEmployees] DEBUG com.tutorial.hazelcastpoc.mapper.EmployeeMapper.retrieveEmployees:139 - ==> Preparing: select id, first_name, last_name, hiring_date, sex, dept_id from employee INFO: 2014-11-27 18:01:39,220 [retrieveEmployees] DEBUG com.tutorial.hazelcastpoc.mapper.EmployeeMapper.retrieveEmployees:139 - ==> Parameters: INFO: 2014-11-27 18:01:39,316 [retrieveEmployees] DEBUG com.tutorial.hazelcastpoc.mapper.EmployeeMapper.retrieveEmployees:139 - <== Total: 13 while running the queries with Ehcache caching the JDBC transaction is performed only once (to initialize the cache) and after that the log will look like : INFO: 2014-11-28 18:04:50,020 [EmployeeDAO] INFO com.tutorial.ehcachepoc.dao.EmployeeDAO:38 - Getting employees..... INFO: 2014-11-28 18:04:50,020 [EhCacheServlet] DEBUG com.tutorial.cacherpoc.EhCacheServlet:41 - com.tutorial.crudwithjsp.model.Employee[ id=1 ] Let's look at the time that each of our 10 times requests has scored: the first not cached version of 10 times requests took about 57 seconds and 51 milliseconds, while the cached requests scored a time of 27seconds and 86 miliseconds. In order to apply Amdhal's law for the system the following input is needed: Un-cached page time: 60 seconds Database time : 58 seconds Cache retrieval time: 28seconds Proportion: 96.6% (58/60) (P) The expected system speedup is thus: 1 / (( 1 – 0.966) + 0.966 / (58/28)) = 1 / (0.034 + 0. 966/2.07) = 2 times system speedup This result can be improved of course, but the purpose of this article was to prove that caching using Ehcache over MyBatis offers a significant improvement to what used to be available before its implementation. Learn more from: MyBatis Documentation MyBatis Ehcache Adapter EhCache website

It has been a while since my first post about the ESP8266 (see “Cheap and Simple WiFi with ESP8266 for the FRDM Board“). The ESP8266 is a new inexpensive ($4.50) WiFi module which makes it easy to connect to the network or internet. Finally this week-end I have found the time to write up a tutorial: how to implement a WiFi web server for the ESP8266 WiFi module and the Freescale FRDM-KL25Z board: WSP8266 Web Server FRDM-KL25Z with ESP8266 WiFi Module Outline In this tutorial I’m using a Freescale FRDM-KL25Z board as a web server, using theESP8266 board. The ESP8266 is a ‘less than $4.5′ WiFi board getting more and more popular as an IoT board. There is even a way to run the ESP8266 standalone (because it has a full processor on that board). However, that development is still in the flux and rather unstable. Instead, I’m using a serial connection to the ESP8266 instead. With this, any small microcontroller can send and receive data from the internet: connect that board to a microcontroller with 3.3V, GND, Tx and Rx, and you have a W-LAN connection! I’m using in this tutorial Eclipse with GNU/GDB with Processor Expert, but with the steps in this tutorial you should be able to use any other toolchain too. As things might change in the future with different firmware on the ESP8266: the firmware I’m having on the board is version 00160901. Board Connections Since my first post on the ESP8266 I have cleaned up the wiring. The pins are as below for the ESP8266: ESP8266 Pins Because the ESP8266 can take > 200 mA, I’m using a 5-to-3.3V DC-DC converter. I measured around 70 to 90 mA, so it is not (yet) really needed, but I wanted to use it to protect to board. The ESP8266 Rx and Tx are connected to the microcontroller Tx and Rx pins. A general frustration point for the ESP8266 module is the connection oft the remaining pins. What worked for me is to connect CH_PD to 3.3V and leaving RST,GPIO0 and GPIO2 unconnected/floating. Wiring Setup with FRDM-KL25Z and ESP8266 Communication Protocol I recommend to use a logic analyzer to verify the communication between the ESP8266 and the microcontroller. My module communicates with 115200, but I see reports that other modules (other firmware) can use a different baud. The module uses an AT command send. The simplest command is to send “AT\r\n” and it responds with “AT\r\r\n\r\nOK\r\n”: AT Command Sent to ESP8266 In this tutorial I’m using a command line shell (see “A Shell for the Freedom KL25Z Board“) to have a manual mode to send commands to the module. More about this later. Project Creation You can use my project and source files available on GitHub (see link at the end of this article). Or create your own project. My project is using the Kinetis Design Studio and for the FRDM-KL25Z board (MKL25Z128VLK4). I have created a project for Processor Expert, as I’m using several components of it: Processor Expert Project For the project I have several files added: ESP8266 Project in Eclipse With the following source files: Application.c/.h: This runs the application and web server program ESP8266.c/.h: Driver for the ESP8266 Events.c/.h: Processor Expert event hooks main.c: main entry point Shell.c/.h: command line interface Sources Project and Source files are available on GitHub here: https://github.com/ErichStyger/mcuoneclipse/tree/master/Examples/KDS/FRDM-KL25Z/FRDM-KL25Z_ESP8266 Please check the latest source files on GitHub. At the time of writing this article, I’m using the following: Shell.h is the interface to command line shell: view source print? 01./* 02. * Shell.h 03. * 04. * Author: Erich Styger 05. */ 06. 07.#ifndef SHELL_H_ 08.#define SHELL_H_ 09. 10./*! 11. * \brief Shell parse routine 12. */ 13.voidSHELL_Parse(void); 14. 15./*! 16. * \brief Shell initialization 17. */ 18.voidSHELL_Init(void); 19. 20.#endif /* SHELL_H_ */ Shell.c implements the application part of the shell: view source print? 01./* 02. * Shell.c 03. * 04. * Author: Erich Styger 05. */ 06. 07.#include "Shell.h" 08.#include "CLS1.h" 09.#include "ESP8266.h" 10. 11./* table with shell parser/handler */ 12.staticconstCLS1_ParseCommandCallback CmdParserTable[] = 13.{ 14. CLS1_ParseCommand, 15. ESP_ParseCommand, 16. NULL /* sentinel */ 17.}; 18. 19.staticunsigned charlocalConsole_buf[48]; /* buffer for command line */ 20. 21.voidSHELL_Parse(void) { 22. (void)CLS1_ReadAndParseWithCommandTable(localConsole_buf, sizeof(localConsole_buf), CLS1_GetStdio(), CmdParserTable); 23.} 24. 25.voidSHELL_Init(void) { 26. localConsole_buf[0] = '\0'; /* initialize buffer */ 27.} ESP8266.h is the interface to the WiFi module: view source print? 001./* 002. * ESP8266.h 003. * 004. * Author: Erich Styger 005. */ 006. 007.#ifndef ESP8266_H_ 008.#define ESP8266_H_ 009. 010.#include "CLS1.h" 011. 012.#define ESP_DEFAULT_TIMEOUT_MS (100) 013. /*!< Default timeout value in milliseconds */ 014. 015./*! 016. * \brief Command line parser routine 017. * \param cmd Pointer to command line string 018. * \param handled Return value if command has been handled 019. * \param io Standard Shell I/O handler 020. * \return Error code, ERR_OK for no failure 021. */ 022.uint8_t ESP_ParseCommand(constunsigned char*cmd, bool *handled, constCLS1_StdIOType *io); 023. 024./*! 025. * \brief Send a string to th ESP8266 module 026. * \param str String to send, "\r\n" will be appended 027. * \param io Shell I/O handler or NULL if not used 028. * \return Error code, ERR_OK for no failure 029. */ 030.uint8_t ESP_SendStr(constuint8_t *str, CLS1_ConstStdIOType *io); 031. 032./*! 033. * \brief Used to send an AT command to the ESP8266 module 034. * \param cmd Command string to send 035. * \param rxBuf Buffer for the response, can be NULL 036. * \param rxBufSize Size of response buffer 037. * \param expectedTailStr Expected response from the module, can be NULL 038. * \param msTimeout Timeout time in milliseconds 039. * \param io Shell I/O handler or NULL if not used 040. * \return Error code, ERR_OK for no failure 041. */ 042.uint8_t ESP_SendATCommand(uint8_t *cmd, uint8_t *rxBuf, size_t rxBufSize, uint8_t *expectedTailStr, uint16_t msTimeout, constCLS1_StdIOType *io); 043. 044./*! 045. * \brief Read from the serial line from the module until a sentinel char is received 046. * \param buf 047. * \param bufSize 048. * \param sentinelChar 049. * \param timeoutMs Timeout time in milliseconds 050. * \return Error code, ERR_OK for no failure 051. */ 052.uint8_t ESP_ReadCharsUntil(uint8_t *buf, size_t bufSize, uint8_t sentinelChar, uint16_t timeoutMs); 053. 054./*! 055. * \brief Sends an AT command to test the connection 056. * \return Error code, ERR_OK for no failure 057. */ 058.uint8_t ESP_TestAT(void); 059. 060./*! 061. * \brief Restarts the ESP8266 module 062. * \param io Shell I/O handler or NULL if not used 063. * \param timeoutMs Timeout time in milliseconds 064. * \return Error code, ERR_OK for no failure 065. */ 066.uint8_t ESP_Restart(constCLS1_StdIOType *io, uint16_t timeoutMs); 067. 068./*! 069. * \brief Set the current mode of the module 070. * \param mode Where is 1=Sta, 2=AP or 3=both 071. * \return Error code, ERR_OK for no failure 072. */ 073.uint8_t ESP_SelectMode(uint8_t mode); 074. 075./*! 076. * \Brief returns the firmware version string 077. * \param fwBuf Buffer for the string 078. * \param fwBufSize Size of buffer in bytes 079. * \return Error code, ERR_OK for no failure 080. */ 081.uint8_t ESP_GetFirmwareVersionString(uint8_t *fwBuf, size_t fwBufSize); 082. 083./*! 084. * \brief Join an access point. 085. * \param ssid SSID of access point 086. * \param pwd Password of access point 087. * \param nofRetries Number of connection retries 088. * \param io Shell I/O or NULL if not used 089. * \return Error code, ERR_OK for no failure 090. */ 091.uint8_t ESP_JoinAP(constuint8_t *ssid, constuint8_t *pwd, intnofRetries, CLS1_ConstStdIOType *io); 092. 093./*! 094. * \brief Scans for an IPD message sent by the module 095. * \param msgBuf Pointer to the message buffer where to store the message 096. * \param msgBufSize Size of message buffer 097. * \param ch_id Pointer to where to store the channel/id 098. * \param size Pointer where to store the size of the message 099. * \param isGet TRUE if it is a GET message, FALSE for a POST message 100. * \param timeoutMs Error code, ERR_OK for no failure 101. * \param io 102. * \return Error code, ERR_OK for no failure 103. */ 104.uint8_t ESP_GetIPD(uint8_t *msgBuf, size_t msgBufSize, uint8_t *ch_id, uint16_t *size, bool *isGet, uint16_t timeoutMs, constCLS1_StdIOType *io); 105. 106./*! 107. * \brief Closes a connection 108. * \param channel Channel ID 109. * \param io Error code, ERR_OK for no failure 110. * \param timeoutMs Error code, ERR_OK for no failure 111. * \return Error code, ERR_OK for no failure 112. */ 113.uint8_t ESP_CloseConnection(uint8_t channel, constCLS1_StdIOType *io, uint16_t timeoutMs); 114. 115./*! 116. * \brief Used to determine if the web server is running or not. 117. * \return TRUE if web server has beens started 118. */ 119.bool ESP_IsServerOn(void); 120. 121./*! 122. * \brief Driver initialization 123. */ 124.voidESP_Init(void); 125. 126./*! 127. * \brief Driver de-initialization 128. */ 129.voidESP_Deinit(void); 130. 131.#endif /* ESP8266_H_ */ And the ESP8266 driver is in ESP8266.c which implements all the low level SPI access functions, the functional implementation and a command line shell interface: view source print? 001./* 002. * ESP8266.c 003. * 004. * Author: Erich Styger 005. */ 006. 007.#include "ESP8266.h" 008.#include "Shell.h" 009.#include "UTIL1.h" 010.#include "CLS1.h" 011.#include "AS2.h" 012.#include "WAIT1.h" 013. 014.staticbool ESP_WebServerIsOn = FALSE; 015. 016.bool ESP_IsServerOn(void) { 017. returnESP_WebServerIsOn; 018.} 019. 020.staticvoidSend(unsigned char*str) { 021. while(*str!='\0') { 022. AS2_SendChar(*str); 023. str++; 024. } 025.} 026. 027.staticvoidSkipNewLines(constunsigned char**p) { 028. while(**p=='\n'|| **p=='\r') { 029. (*p)++; /* skip new lines */ 030. } 031.} 032. 033.uint8_t ESP_ReadCharsUntil(uint8_t *buf, size_t bufSize, uint8_t sentinelChar, uint16_t timeoutMs) { 034. uint8_t ch; 035. uint8_t res = ERR_OK; 036. 037. if(bufSize<=1) { 038. returnERR_OVERRUN; /* buffer to small */ 039. } 040. buf[0] = '\0'; buf[bufSize-1] = '\0'; /* always terminate */ 041. bufSize--; 042. for(;;) { /* breaks */ 043. if(bufSize==0) { 044. res = ERR_OVERRUN; 045. break; 046. } 047. if(AS2_GetCharsInRxBuf()>0) { 048. (void)AS2_RecvChar(&ch); 049. *buf = ch; 050. buf++; 051. bufSize--; 052. if(ch==sentinelChar) { 053. *buf = '\0'; /* terminate string */ 054. break; /* sentinel found */ 055. } 056. } else{ 057. if(timeoutMs>10) { 058. WAIT1_WaitOSms(5); 059. timeoutMs -= 5; 060. } else{ 061. res = ERR_NOTAVAIL; /* timeout */ 062. break; 063. } 064. } 065. } 066. returnres; 067.} 068. 069.staticuint8_t RxResponse(unsigned char*rxBuf, size_t rxBufLength, unsigned char*expectedTail, uint16_t msTimeout) { 070. unsigned charch; 071. uint8_t res = ERR_OK; 072. unsigned char*p; 073. 074. if(rxBufLength < sizeof("x\r\n")) { 075. returnERR_OVERFLOW; /* not enough space in buffer */ 076. } 077. p = rxBuf; 078. p[0] = '\0'; 079. for(;;) { /* breaks */ 080. if(msTimeout == 0) { 081. break; /* will decide outside of loop if it is a timeout. */ 082. } elseif(rxBufLength == 0) { 083. res = ERR_OVERFLOW; /* not enough space in buffer */ 084. break; 085. } elseif(AS2_GetCharsInRxBuf() > 0) { 086.#if0 087. if(AS2_RecvChar(&ch) != ERR_OK) { 088. res = ERR_RXEMPTY; 089. break; 090. } 091.#else 092. /* might get an overrun OVERRUN_ERR error here? Ignoring error for now */ 093. (void)AS2_RecvChar(&ch); 094.#endif 095. *p++ = ch; 096. *p = '\0'; /* always terminate */ 097. rxBufLength--; 098. } elseif(expectedTail!=NULL && expectedTail[0]!='\0' 099. && UTIL1_strtailcmp(rxBuf, expectedTail) == 0) { 100. break; /* finished */ 101. } else{ 102. WAIT1_WaitOSms(1); 103. msTimeout--; 104. } 105. } /* for */ 106. if(msTimeout==0) { /* timeout! */ 107. if(expectedTail[0] != '\0'/* timeout, and we expected something: an error for sure */ 108. || rxBuf[0] == '\0'/* timeout, did not know what to expect, but received nothing? There has to be a response. */ 109. ) 110. { 111. res = ERR_FAULT; 112. } 113. } 114. returnres; 115.} 116. 117.uint8_t ESP_SendATCommand(uint8_t *cmd, uint8_t *rxBuf, size_t rxBufSize, uint8_t *expectedTailStr, uint16_t msTimeout, constCLS1_StdIOType *io) { 118. uint16_t snt; 119. uint8_t res; 120. 121. if(rxBuf!=NULL) { 122. rxBuf[0] = '\0'; 123. } 124. if(io!=NULL) { 125. CLS1_SendStr("sending>>:\r\n", io->stdOut); 126. CLS1_SendStr(cmd, io->stdOut); 127. } 128. if(AS2_SendBlock(cmd, (uint16_t)UTIL1_strlen((char*)cmd), &snt) != ERR_OK) { 129. returnERR_FAILED; 130. } 131. if(rxBuf!=NULL) { 132. res = RxResponse(rxBuf, rxBufSize, expectedTailStr, msTimeout); 133. if(io!=NULL) { 134. CLS1_SendStr("received<<:\r\n", io->stdOut); 135. CLS1_SendStr(rxBuf, io->stdOut); 136. } 137. } 138. returnres; 139.} 140. 141.uint8_t ESP_TestAT(void) { 142. /* AT */ 143. uint8_t rxBuf[sizeof("AT\r\r\n\r\nOK\r\n")]; 144. uint8_t res; 145. 146. res = ESP_SendATCommand("AT\r\n", rxBuf, sizeof(rxBuf), "AT\r\r\n\r\nOK\r\n", ESP_DEFAULT_TIMEOUT_MS, NULL); 147. returnres; 148.} 149. 150.uint8_t ESP_Restart(constCLS1_StdIOType *io, uint16_t timeoutMs) { 151. /* AT+RST */ 152. uint8_t rxBuf[sizeof("AT+RST\r\r\n\r\nOK\r\n")]; 153. uint8_t res; 154. uint8_t buf[64]; 155. 156. AS2_ClearRxBuf(); /* clear buffer */ 157. res = ESP_SendATCommand("AT+RST\r\n", rxBuf, sizeof(rxBuf), "AT+RST\r\r\n\r\nOK\r\n", ESP_DEFAULT_TIMEOUT_MS, io); 158. if(res==ERR_OK) { 159. for(;;) { 160. ESP_ReadCharsUntil(buf, sizeof(buf), '\n', 1000); 161. if(io!=NULL) { 162. CLS1_SendStr(buf, io->stdOut); 163. } 164. if(UTIL1_strncmp(buf, "ready", sizeof("ready")-1)==0) { /* wait until ready message from module */ 165. break; /* module has restarted */ 166. } 167. } 168. } 169. AS2_ClearRxBuf(); /* clear buffer */ 170. returnres; 171.} 172. 173.uint8_t ESP_CloseConnection(uint8_t channel, constCLS1_StdIOType *io, uint16_t timeoutMs) { 174. /* AT+CIPCLOSE= */ 175. uint8_t res; 176. uint8_t cmd[64]; 177. 178. UTIL1_strcpy(cmd, sizeof(cmd), "AT+CIPCLOSE="); 179. UTIL1_strcatNum8u(cmd, sizeof(cmd), channel); 180. UTIL1_strcat(cmd, sizeof(cmd), "\r\n"); 181. res = ESP_SendATCommand(cmd, NULL, 0, "Unlink\r\n", timeoutMs, io); 182. returnres; 183.} 184. 185.uint8_t ESP_SetNumberOfConnections(uint8_t nof, constCLS1_StdIOType *io, uint16_t timeoutMs) { 186. /* AT+CIPMUX=, 0: single connection, 1: multiple connections */ 187. uint8_t res; 188. uint8_t cmd[sizeof("AT+CIPMUX=12\r\n")]; 189. uint8_t rxBuf[sizeof("AT+CIPMUX=12\r\n\r\nOK\r\n")+10]; 190. 191. if(nof>1) { /* only 0 and 1 allowed */ 192. if(io!=NULL) { 193. CLS1_SendStr("Wrong number of connection parameter!\r\n", io->stdErr); 194. } 195. returnERR_FAILED; 196. } 197. UTIL1_strcpy(cmd, sizeof(cmd), "AT+CIPMUX="); 198. UTIL1_strcatNum8u(cmd, sizeof(cmd), nof); 199. UTIL1_strcat(cmd, sizeof(cmd), "\r\n"); 200. res = ESP_SendATCommand(cmd, rxBuf, sizeof(rxBuf), "OK\r\n", timeoutMs, io); 201. returnres; 202.} 203. 204.uint8_t ESP_SetServer(bool startIt, uint16_t port, constCLS1_StdIOType *io, uint16_t timeoutMs) { 205. /* AT+CIPSERVER=,, where : 0: stop, 1: start */ 206. uint8_t res; 207. uint8_t cmd[sizeof("AT+CIPSERVER=1,80\r\n\r\nOK\r\n")+sizeof("no change")]; 208. uint8_t rxBuf[sizeof("AT+CIPSERVER=1,80\r\n\r\nOK\r\n")+sizeof("no change")]; 209. 210. UTIL1_strcpy(cmd, sizeof(cmd), "AT+CIPSERVER="); 211. if(startIt) { 212. UTIL1_strcat(cmd, sizeof(cmd), "1,"); 213. } else{ 214. UTIL1_strcat(cmd, sizeof(cmd), "0,"); 215. } 216. UTIL1_strcatNum16u(cmd, sizeof(cmd), port); 217. UTIL1_strcat(cmd, sizeof(cmd), "\r\n"); 218. res = ESP_SendATCommand(cmd, rxBuf, sizeof(rxBuf), "OK\r\n", timeoutMs, io); 219. if(res!=ERR_OK) { /* accept "no change" too */ 220. UTIL1_strcpy(cmd, sizeof(cmd), "AT+CIPSERVER="); 221. if(startIt) { 222. UTIL1_strcat(cmd, sizeof(cmd), "1,"); 223. } else{ 224. UTIL1_strcat(cmd, sizeof(cmd), "0,"); 225. } 226. UTIL1_strcatNum16u(cmd, sizeof(cmd), port); 227. UTIL1_strcat(cmd, sizeof(cmd), "\r\r\nno change\r\n"); 228. if(UTIL1_strcmp(rxBuf, cmd)==0) { 229. res = ERR_OK; 230. } 231. } 232. returnres; 233.} 234. 235.uint8_t ESP_SelectMode(uint8_t mode) { 236. /* AT+CWMODE=, where is 1=Sta, 2=AP or 3=both */ 237. uint8_t txBuf[sizeof("AT+CWMODE=x\r\n")]; 238. uint8_t rxBuf[sizeof("AT+CWMODE=x\r\r\nno change\r\n")]; 239. uint8_t expected[sizeof("AT+CWMODE=x\r\r\nno change\r\n")]; 240. uint8_t res; 241. 242. if(mode<1|| mode>3) { 243. returnERR_RANGE; /* only 1, 2 or 3 */ 244. } 245. UTIL1_strcpy(txBuf, sizeof(txBuf), "AT+CWMODE="); 246. UTIL1_strcatNum16u(txBuf, sizeof(txBuf), mode); 247. UTIL1_strcat(txBuf, sizeof(txBuf), "\r\n"); 248. UTIL1_strcpy(expected, sizeof(expected), "AT+CWMODE="); 249. UTIL1_strcatNum16u(expected, sizeof(expected), mode); 250. UTIL1_strcat(expected, sizeof(expected), "\r\r\n\n"); 251. res = ESP_SendATCommand(txBuf, rxBuf, sizeof(rxBuf), expected, ESP_DEFAULT_TIMEOUT_MS, NULL); 252. if(res!=ERR_OK) { 253. /* answer could be as well "AT+CWMODE=x\r\r\nno change\r\n"!! */ 254. UTIL1_strcpy(txBuf, sizeof(txBuf), "AT+CWMODE="); 255. UTIL1_strcatNum16u(txBuf, sizeof(txBuf), mode); 256. UTIL1_strcat(txBuf, sizeof(txBuf), "\r\n"); 257. UTIL1_strcpy(expected, sizeof(expected), "AT+CWMODE="); 258. UTIL1_strcatNum16u(expected, sizeof(expected), mode); 259. UTIL1_strcat(expected, sizeof(expected), "\r\r\nno change\r\n"); 260. if(UTIL1_strcmp(rxBuf, expected)==0) { 261. res = ERR_OK; 262. } 263. } 264. returnres; 265.} 266. 267.uint8_t ESP_GetFirmwareVersionString(uint8_t *fwBuf, size_t fwBufSize) { 268. /* AT+GMR */ 269. uint8_t rxBuf[32]; 270. uint8_t res; 271. constunsigned char*p; 272. 273. res = ESP_SendATCommand("AT+GMR\r\n", rxBuf, sizeof(rxBuf), "\r\n\r\nOK\r\n", ESP_DEFAULT_TIMEOUT_MS, NULL); 274. if(res!=ERR_OK) { 275. if(UTIL1_strtailcmp(rxBuf, "\r\n\r\nOK\r\n")) { 276. res = ERR_OK; 277. } 278. } 279. if(res==ERR_OK) { 280. if(UTIL1_strncmp(rxBuf, "AT+GMR\r\r\n", sizeof("AT+GMR\r\r\n")-1)==0) { /* check for beginning of response */ 281. UTIL1_strCutTail(rxBuf, "\r\n\r\nOK\r\n"); /* cut tailing response */ 282. p = rxBuf+sizeof("AT+GMR\r\r\n")-1; /* skip beginning */ 283. UTIL1_strcpy(fwBuf, fwBufSize, p); /* copy firmware information string */ 284. } else{ 285. res = ERR_FAILED; 286. } 287. } 288. if(res!=ERR_OK) { 289. UTIL1_strcpy(fwBuf, fwBufSize, "ERROR"); /* default error */ 290. } 291. returnres; 292.} 293. 294.uint8_t ESP_GetIPAddrString(uint8_t *ipBuf, size_t ipBufSize) { 295. /* AT+CIFSR */ 296. uint8_t rxBuf[32]; 297. uint8_t res; 298. constunsigned char*p; 299. 300. res = ESP_SendATCommand("AT+CIFSR\r\n", rxBuf, sizeof(rxBuf), NULL, ESP_DEFAULT_TIMEOUT_MS, NULL); 301. if(res!=ERR_OK) { 302. if(UTIL1_strtailcmp(rxBuf, "\r\n")) { 303. res = ERR_OK; 304. } 305. } 306. if(res==ERR_OK) { 307. if(UTIL1_strncmp(rxBuf, "AT+CIFSR\r\r\n", sizeof("AT+CIFSR\r\r\n")-1)==0) { /* check for beginning of response */ 308. UTIL1_strCutTail(rxBuf, "\r\n"); /* cut tailing response */ 309. p = rxBuf+sizeof("AT+CIFSR\r\r\n")-1; /* skip beginning */ 310. SkipNewLines(&p); 311. UTIL1_strcpy(ipBuf, ipBufSize, p); /* copy IP information string */ 312. } else{ 313. res = ERR_FAILED; 314. } 315. } 316. if(res!=ERR_OK) { 317. UTIL1_strcpy(ipBuf, ipBufSize, "ERROR"); 318. } 319. returnres; 320.} 321. 322.uint8_t ESP_GetModeString(uint8_t *buf, size_t bufSize) { 323. /* AT+CWMODE? */ 324. uint8_t rxBuf[32]; 325. uint8_t res; 326. constunsigned char*p; 327. 328. res = ESP_SendATCommand("AT+CWMODE?\r\n", rxBuf, sizeof(rxBuf), "\r\n\r\nOK\r\n", ESP_DEFAULT_TIMEOUT_MS, NULL); 329. if(res==ERR_OK) { 330. if(UTIL1_strncmp(rxBuf, "AT+CWMODE?\r\r\n+CWMODE:", sizeof("AT+CWMODE?\r\r\n+CWMODE:")-1)==0) { /* check for beginning of response */ 331. UTIL1_strCutTail(rxBuf, "\r\n\r\nOK\r\n"); /* cut tailing response */ 332. p = rxBuf+sizeof("AT+CWMODE?\r\r\n+CWMODE:")-1; /* skip beginning */ 333. UTIL1_strcpy(buf, bufSize, p); /* copy information string */ 334. } else{ 335. res = ERR_FAILED; 336. } 337. } 338. if(res!=ERR_OK) { 339. UTIL1_strcpy(buf, bufSize, "ERROR"); 340. } 341. returnres; 342.} 343. 344.uint8_t ESP_GetCIPMUXString(uint8_t *cipmuxBuf, size_t cipmuxBufSize) { 345. /* AT+CIPMUX? */ 346. uint8_t rxBuf[32]; 347. uint8_t res; 348. constunsigned char*p; 349. 350. res = ESP_SendATCommand("AT+CIPMUX?\r\n", rxBuf, sizeof(rxBuf), "\r\n\r\nOK\r\n", ESP_DEFAULT_TIMEOUT_MS, NULL); 351. if(res==ERR_OK) { 352. if(UTIL1_strncmp(rxBuf, "AT+CIPMUX?\r\r\n+CIPMUX:", sizeof("AT+CIPMUX?\r\r\n+CIPMUX:")-1)==0) { /* check for beginning of response */ 353. UTIL1_strCutTail(rxBuf, "\r\n\r\nOK\r\n"); /* cut tailing response */ 354. p = rxBuf+sizeof("AT+CIPMUX?\r\r\n+CIPMUX:")-1; /* skip beginning */ 355. UTIL1_strcpy(cipmuxBuf, cipmuxBufSize, p); /* copy IP information string */ 356. } else{ 357. res = ERR_FAILED; 358. } 359. } 360. if(res!=ERR_OK) { 361. UTIL1_strcpy(cipmuxBuf, cipmuxBufSize, "ERROR"); 362. } 363. returnres; 364.} 365. 366.uint8_t ESP_GetConnectedAPString(uint8_t *apBuf, size_t apBufSize) { 367. /* AT+CWJAP? */ 368. uint8_t rxBuf[48]; 369. uint8_t res; 370. constunsigned char*p; 371. 372. res = ESP_SendATCommand("AT+CWJAP?\r\n", rxBuf, sizeof(rxBuf), "\r\n\r\nOK\r\n", ESP_DEFAULT_TIMEOUT_MS, NULL); 373. if(res==ERR_OK) { 374. if(UTIL1_strncmp(rxBuf, "AT+CWJAP?\r\r\n+CWJAP:\"", sizeof("AT+CWJAP?\r\r\n+CWJAP:\"")-1)==0) { /* check for beginning of response */ 375. UTIL1_strCutTail(rxBuf, "\"\r\n\r\nOK\r\n"); /* cut tailing response */ 376. p = rxBuf+sizeof("AT+CWJAP?\r\r\n+CWJAP:\"")-1; /* skip beginning */ 377. UTIL1_strcpy(apBuf, apBufSize, p); /* copy IP information string */ 378. } else{ 379. res = ERR_FAILED; 380. } 381. } 382. if(res!=ERR_OK) { 383. UTIL1_strcpy(apBuf, apBufSize, "ERROR"); 384. } 385. returnres; 386. 387.} 388. 389.staticuint8_t JoinAccessPoint(constuint8_t *ssid, constuint8_t *pwd, CLS1_ConstStdIOType *io) { 390. /* AT+CWJAP="","" */ 391. uint8_t txBuf[48]; 392. uint8_t rxBuf[64]; 393. uint8_t expected[48]; 394. 395. UTIL1_strcpy(txBuf, sizeof(txBuf), "AT+CWJAP=\""); 396. UTIL1_strcat(txBuf, sizeof(txBuf), ssid); 397. UTIL1_strcat(txBuf, sizeof(txBuf), "\",\""); 398. UTIL1_strcat(txBuf, sizeof(txBuf), pwd); 399. UTIL1_strcat(txBuf, sizeof(txBuf), "\"\r\n"); 400. 401. UTIL1_strcpy(expected, sizeof(expected), "AT+CWJAP=\""); 402. UTIL1_strcat(expected, sizeof(expected), ssid); 403. UTIL1_strcat(expected, sizeof(expected), "\",\""); 404. UTIL1_strcat(expected, sizeof(expected), pwd); 405. UTIL1_strcat(expected, sizeof(expected), "\"\r\r\n\r\nOK\r\n"); 406. 407. returnESP_SendATCommand(txBuf, rxBuf, sizeof(rxBuf), expected, ESP_DEFAULT_TIMEOUT_MS, io); 408.} 409. 410.uint8_t ESP_JoinAP(constuint8_t *ssid, constuint8_t *pwd, intnofRetries, CLS1_ConstStdIOType *io) { 411. uint8_t buf[32]; 412. uint8_t res; 413. 414. do{ 415. res = JoinAccessPoint(ssid, pwd, io); 416. if(res==ERR_OK) { 417. break; 418. } 419. WAIT1_WaitOSms(1000); 420. nofRetries--; 421. } while(nofRetries>0); 422. returnres; 423.} 424. 425.staticuint8_t ReadIntoIPDBuffer(uint8_t *buf, size_t bufSize, uint8_t *p, uint16_t msgSize, uint16_t msTimeout, constCLS1_StdIOType *io) { 426. uint8_t ch; 427. size_t nofInBuf; 428. inttimeout; 429. 430. nofInBuf = p-buf; 431. bufSize -= nofInBuf; /* take into account what we already have in buffer */ 432. timeout = msTimeout; 433. while(msgSize>0&& bufSize>0) { 434. if(AS2_GetCharsInRxBuf()>0) { 435. (void)AS2_RecvChar(&ch); 436. *p = ch; 437. if(io!=NULL) { /* copy on console */ 438. io->stdOut(ch); 439. } 440. p++; 441. *p = '\0'; /* terminate */ 442. nofInBuf++; msgSize--; bufSize--; 443. } else{ 444. /* check in case we recveive less characters than expected, happens for POST? */ 445. if(nofInBuf>6&& UTIL1_strncmp(&p[-6], "\r\nOK\r\n", sizeof("\r\nOK\r\n")-1)==0) { 446. break; 447. } else{ 448. timeout -= 10; 449. WAIT1_WaitOSms(10); 450. if(timeout<0) { 451. returnERR_BUSY; 452. } 453. } 454. } 455. } 456. returnERR_OK; 457.} 458. 459.uint8_t ESP_GetIPD(uint8_t *msgBuf, size_t msgBufSize, uint8_t *ch_id, uint16_t *size, bool *isGet, uint16_t timeoutMs, constCLS1_StdIOType *io) { 460. /* scan e.g. for 461. * +IPD,0,404:POST / HTTP/1.1 462. * and return ch_id (0), size (404) 463. */ 464. uint8_t res = ERR_OK; 465. constuint8_t *p; 466. bool isIPD = FALSE; 467. uint8_t cmd[24], rxBuf[48]; 468. uint16_t ipdSize; 469. 470. *ch_id = 0; *size = 0; *isGet = FALSE; /* init */ 471. for(;;) { /* breaks */ 472. res = ESP_ReadCharsUntil(msgBuf, msgBufSize, '\n', timeoutMs); 473. if(res!=ERR_OK) { 474. break; /* timeout */ 475. } 476. if(res==ERR_OK) { /* line read */ 477. if(io!=NULL) { 478. CLS1_SendStr(msgBuf, io->stdOut); /* copy on console */ 479. } 480. isIPD = UTIL1_strncmp(msgBuf, "+IPD,", sizeof("+IPD,")-1)==0; 481. if(isIPD) { /* start of IPD message */ 482. p = msgBuf+sizeof("+IPD,")-1; 483. if(UTIL1_ScanDecimal8uNumber(&p, ch_id)!=ERR_OK) { 484. if(io!=NULL) { 485. CLS1_SendStr("ERR: wrong channel?\r\n", io->stdErr); /* error on console */ 486. } 487. res = ERR_FAILED; 488. break; 489. } 490. if(*p!=',') { 491. res = ERR_FAILED; 492. break; 493. } 494. p++; /* skip comma */ 495. if(UTIL1_ScanDecimal16uNumber(&p, size)!=ERR_OK) { 496. if(io!=NULL) { 497. CLS1_SendStr("ERR: wrong size?\r\n", io->stdErr); /* error on console */ 498. } 499. res = ERR_FAILED; 500. break; 501. } 502. if(*p!=':') { 503. res = ERR_FAILED; 504. break; 505. } 506. ipdSize = p-msgBuf; /* length of "+IPD,," string */ 507. p++; /* skip ':' */ 508. if(UTIL1_strncmp(p, "GET", sizeof("GET")-1)==0) { 509. *isGet = TRUE; 510. } elseif(UTIL1_strncmp(p, "POST", sizeof("POST")-1)==0) { 511. *isGet = FALSE; 512. } else{ 513. res = ERR_FAILED; 514. } 515. while(*p!='\0') { 516. p++; /* skip to the end */ 517. } 518. /* read the rest of the message */ 519. res = ReadIntoIPDBuffer(msgBuf, msgBufSize, (uint8_t*)p, (*size)-ipdSize, ESP_DEFAULT_TIMEOUT_MS, io); 520. break; 521. } 522. } 523. } 524. returnres; 525.} 526. 527.uint8_t ESP_StartWebServer(constCLS1_StdIOType *io) { 528. uint8_t buf[32]; 529. uint8_t res; 530. 531. res = ESP_SetNumberOfConnections(1, io, ESP_DEFAULT_TIMEOUT_MS); 532. if(res!=ERR_OK) { 533. CLS1_SendStr("ERR: failed to set multiple connections.\r\n", io->stdErr); 534. returnres; 535. } 536. res = ESP_SetServer(TRUE, 80, io, ESP_DEFAULT_TIMEOUT_MS); 537. if(res!=ERR_OK) { 538. CLS1_SendStr("ERR: failed to set server.\r\n", io->stdErr); 539. returnres; 540. } 541. CLS1_SendStr("INFO: Web Server started, waiting for connection on ", io->stdOut); 542. if(ESP_GetIPAddrString(buf, sizeof(buf))==ERR_OK) { 543. CLS1_SendStr(buf, io->stdOut); 544. CLS1_SendStr(":80", io->stdOut); 545. } else{ 546. CLS1_SendStr("(ERROR!)", io->stdOut); 547. } 548. CLS1_SendStr("\r\n", io->stdOut); 549. 550. returnERR_OK; 551.} 552. 553.uint8_t ESP_SendStr(constuint8_t *str, CLS1_ConstStdIOType *io) { 554. uint8_t buf[32]; 555. uint8_t rxBuf[48]; 556. uint8_t res; 557. uint16_t timeoutMs; 558. #define RX_TIMEOUT_MS 3000 559. AS2_TComData ch; 560. 561. UTIL1_strcpy(buf, sizeof(buf), str); 562. UTIL1_strcat(buf, sizeof(buf), "\r\n"); 563. res = ESP_SendATCommand(buf, rxBuf, sizeof(rxBuf), NULL, ESP_DEFAULT_TIMEOUT_MS, io); 564. timeoutMs = 0; 565. while(timeoutMs0) { 569. (void)AS2_RecvChar(&ch); 570. CLS1_SendChar(ch); 571. } 572. } 573. returnERR_OK; 574.} 575. 576.staticuint8_t ESP_PrintHelp(constCLS1_StdIOType *io) { 577. CLS1_SendHelpStr("ESP", "ESP8200 commands\r\n", io->stdOut); 578. CLS1_SendHelpStr(" help|status", "Print help or status information\r\n", io->stdOut); 579. CLS1_SendHelpStr(" send ", "Sends a string to the module\r\n", io->stdOut); 580. CLS1_SendHelpStr(" test", "Sends a test AT command\r\n", io->stdOut); 581. CLS1_SendHelpStr(" restart", "Restart module\r\n", io->stdOut); 582. CLS1_SendHelpStr(" listAP", "List available Access Points\r\n", io->stdOut); 583. CLS1_SendHelpStr(" connectAP \"ssid\",\"pwd\"", "Connect to an Access Point\r\n", io->stdOut); 584. CLS1_SendHelpStr(" server (start|stop)", "Start or stop web server\r\n", io->stdOut); 585. returnERR_OK; 586.} 587. 588.staticuint8_t ESP_PrintStatus(constCLS1_StdIOType *io) { 589. uint8_t buf[48]; 590. 591. CLS1_SendStatusStr("ESP8266", "\r\n", io->stdOut); 592. 593. CLS1_SendStatusStr(" Webserver", ESP_WebServerIsOn?"ON\r\n":"OFF\r\n", io->stdOut); 594. 595. if(ESP_GetFirmwareVersionString(buf, sizeof(buf)) != ERR_OK) { 596. UTIL1_strcpy(buf, sizeof(buf), "FAILED\r\n"); 597. } else{ 598. UTIL1_strcat(buf, sizeof(buf), "\r\n"); 599. } 600. CLS1_SendStatusStr(" AT+GMR", buf, io->stdOut); 601. 602. if(ESP_GetModeString(buf, sizeof(buf)) != ERR_OK) { 603. UTIL1_strcpy(buf, sizeof(buf), "FAILED\r\n"); 604. } else{ 605. if(UTIL1_strcmp(buf, "1")==0) { 606. UTIL1_strcat(buf, sizeof(buf), " (device)"); 607. } elseif(UTIL1_strcmp(buf, "2")==0) { 608. UTIL1_strcat(buf, sizeof(buf), " (AP)"); 609. } elseif(UTIL1_strcmp(buf, "3")==0) { 610. UTIL1_strcat(buf, sizeof(buf), " (device+AP)"); 611. } else{ 612. UTIL1_strcat(buf, sizeof(buf), " (ERROR)"); 613. } 614. UTIL1_strcat(buf, sizeof(buf), "\r\n"); 615. } 616. CLS1_SendStatusStr(" AT+CWMODE?", buf, io->stdOut); 617. 618. if(ESP_GetIPAddrString(buf, sizeof(buf)) != ERR_OK) { 619. UTIL1_strcpy(buf, sizeof(buf), "FAILED\r\n"); 620. } else{ 621. UTIL1_strcat(buf, sizeof(buf), "\r\n"); 622. } 623. CLS1_SendStatusStr(" AT+CIFSR", buf, io->stdOut); 624. 625. if(ESP_GetConnectedAPString(buf, sizeof(buf)) != ERR_OK) { 626. UTIL1_strcpy(buf, sizeof(buf), "FAILED\r\n"); 627. } else{ 628. UTIL1_strcat(buf, sizeof(buf), "\r\n"); 629. } 630. CLS1_SendStatusStr(" AT+CWJAP?", buf, io->stdOut); 631. 632. if(ESP_GetCIPMUXString(buf, sizeof(buf)) != ERR_OK) { 633. UTIL1_strcpy(buf, sizeof(buf), "FAILED\r\n"); 634. } else{ 635. if(UTIL1_strcmp(buf, "0")==0) { 636. UTIL1_strcat(buf, sizeof(buf), " (single connection)"); 637. } elseif(UTIL1_strcmp(buf, "1")==0) { 638. UTIL1_strcat(buf, sizeof(buf), " (multiple connections)"); 639. } else{ 640. UTIL1_strcat(buf, sizeof(buf), " (ERROR)"); 641. } 642. UTIL1_strcat(buf, sizeof(buf), "\r\n"); 643. } 644. CLS1_SendStatusStr(" CIPMUX", buf, io->stdOut); 645. returnERR_OK; 646.} 647. 648.uint8_t ESP_ParseCommand(constunsigned char*cmd, bool *handled, constCLS1_StdIOType *io) { 649. uint32_t val; 650. uint8_t res; 651. constunsigned char*p; 652. uint8_t pwd[24], ssid[24]; 653. 654. if(UTIL1_strcmp((char*)cmd, CLS1_CMD_HELP)==0|| UTIL1_strcmp((char*)cmd, "ESP help")==0) { 655. *handled = TRUE; 656. res = ESP_PrintHelp(io); 657. } elseif(UTIL1_strcmp((char*)cmd, CLS1_CMD_STATUS)==0|| UTIL1_strcmp((char*)cmd, "ESP status")==0) { 658. *handled = TRUE; 659. res = ESP_PrintStatus(io); 660. } elseif(UTIL1_strncmp((char*)cmd, "ESP send ", sizeof("ESP send ")-1)==0) { 661. *handled = TRUE; 662. p = cmd+sizeof("ESP send ")-1; 663. 664. (void)ESP_SendStr(p, io); 665. } elseif(UTIL1_strcmp((char*)cmd, "ESP test")==0) { 666. *handled = TRUE; 667. if(ESP_TestAT()!=ERR_OK) { 668. CLS1_SendStr("TEST failed!\r\n", io->stdErr); 669. res = ERR_FAILED; 670. } else{ 671. CLS1_SendStr("TEST ok!\r\n", io->stdOut); 672. } 673. } elseif(UTIL1_strcmp((char*)cmd, "ESP listAP")==0) { 674. *handled = TRUE; 675. (void)ESP_SendStr("AT+CWLAP", io); 676. /* AT + CWLAP 677. response 678. + CWLAP: , , [, ] 679. OK Or Fails, the return ERROR 680. 0 OPEN 681. 1 WEP 682. 2 WPA_PSK 683. 3 WPA2_PSK 684. 4 WPA_WPA2_PSK 685. string parameter, the access point name 686. signal strength 687. 0: manually connect 1: An automatic connection 688. */ 689. returnERR_OK; 690. } elseif(UTIL1_strncmp((char*)cmd, "ESP connectAP ", sizeof("ESP connectAP ")-1)==0) { 691. *handled = TRUE; 692. p = cmd+sizeof("ESP connectAP ")-1; 693. ssid[0] = '\0'; pwd[0] = '\0'; 694. res = UTIL1_ScanDoubleQuotedString(&p, ssid, sizeof(ssid)); 695. if(res==ERR_OK && *p!='\0'&& *p==',') { 696. p++; /* skip comma */ 697. res = UTIL1_ScanDoubleQuotedString(&p, pwd, sizeof(pwd)); 698. } else{ 699. CLS1_SendStr("Comma expected between strings!\r\n", io->stdErr); 700. res = ERR_FAILED; 701. } 702. if(res==ERR_OK) { 703. res = ESP_JoinAP(ssid, pwd, 3, io); 704. } else{ 705. CLS1_SendStr("Wrong command format!\r\n", io->stdErr); 706. res = ERR_FAILED; 707. } 708. } elseif(UTIL1_strcmp((char*)cmd, "ESP server start")==0) { 709. *handled = TRUE; 710. res = ESP_StartWebServer(io); 711. ESP_WebServerIsOn = res==ERR_OK; 712. } elseif(UTIL1_strcmp((char*)cmd, "ESP server stop")==0) { 713. *handled = TRUE; 714. ESP_WebServerIsOn = FALSE; 715. } elseif(UTIL1_strcmp((char*)cmd, "ESP restart")==0) { 716. *handled = TRUE; 717. ESP_Restart(io, 2000); 718. } 719. returnres; 720.} 721. 722.voidESP_Deinit(void) { 723. /* nothing to do */ 724.} 725. 726.voidESP_Init(void) { 727. AS2_ClearRxBuf(); /* clear buffer */ 728.} The application interface in Application.h is rather short :-): view source print? 01./* 02. * Application.h 03. * 04. * Author: Erich Styger 05. */ 06. 07.#ifndef APPLICATION_H_ 08.#define APPLICATION_H_ 09. 10./*! 11. * \brief Application main routine 12. */ 13.voidAPP_Run(void); 14. 15.#endif /* APPLICATION_H_ */ The main loop of the application is Application.c, along with the application specific web server code. As the SendWebPage function contains HTML code, I’m posting it here separately: view source print? 01.staticuint8_t SendWebPage(uint8_t ch_id, bool ledIsOn, uint8_t temperature, constCLS1_StdIOType *io) { 02. staticuint8_t http[1024]; 03. uint8_t cmd[24], rxBuf[48], expected[48]; 04. uint8_t buf[16]; 05. uint8_t res = ERR_OK; 06. 07. /* construct web page content */ 08. UTIL1_strcpy(http, sizeof(http), (uint8_t*)"HTTP/1.0 200 OK\r\nContent-Type: text/html\r\nPragma: no-cache\r\n\r\n"); 09. UTIL1_strcat(http, sizeof(http), (uint8_t*)"\r\n\r\n"); 10. UTIL1_strcat(http, sizeof(http), (uint8_t*)"\r\n"); 11. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Web Server using ESP8266\r\n"); 12. UTIL1_strcat(http, sizeof(http), (uint8_t*)" 13.\r\n"); 14. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Temp: OC"); 17. if(ledIsOn) { 18. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Red LED off"); 19. UTIL1_strcat(http, sizeof(http), (uint8_t*)" 20.Red LED on"); 21. } else{ 22. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Red LED off"); 23. UTIL1_strcat(http, sizeof(http), (uint8_t*)" 24.Red LED on"); 25. } 26. UTIL1_strcat(http, sizeof(http), (uint8_t*)""); 27. UTIL1_strcat(http, sizeof(http), (uint8_t*)"\r\n\r\n"); 28. 29. UTIL1_strcpy(cmd, sizeof(cmd), "AT+CIPSEND="); /* parameters are , */ 30. UTIL1_strcatNum8u(cmd, sizeof(cmd), ch_id); 31. UTIL1_chcat(cmd, sizeof(cmd), ','); 32. UTIL1_strcatNum16u(cmd, sizeof(cmd), UTIL1_strlen(http)); 33. UTIL1_strcpy(expected, sizeof(expected), cmd); /* we expect the echo of our command */ 34. UTIL1_strcat(expected, sizeof(expected), "\r\r\n> "); /* expect "> " */ 35. UTIL1_strcat(cmd, sizeof(cmd), "\r\n"); 36. res = ESP_SendATCommand(cmd, rxBuf, sizeof(rxBuf), expected, ESP_DEFAULT_TIMEOUT_MS, io); 37. if(res!=ERR_OK) { 38. if(io!=NULL) { 39. CLS1_SendStr("INFO: TIMEOUT, closing connection!\r\n", io->stdOut); 40. } 41. } else{ 42. if(io!=NULL) { 43. CLS1_SendStr("INFO: Sending http page...\r\n", io->stdOut); 44. } 45. UTIL1_strcat(http, sizeof(http), "\r\n\r\n"); /* need to add this to end the command! */ 46. res = ESP_SendATCommand(http, NULL, 0, NULL, ESP_DEFAULT_TIMEOUT_MS, io); 47. if(res!=ERR_OK) { 48. CLS1_SendStr("Sending page failed!\r\n", io->stdErr); /* copy on console */ 49. } else{ 50. for(;;) { /* breaks */ 51. res = ESP_ReadCharsUntil(buf, sizeof(buf), '\n', 1000); 52. if(res==ERR_OK) { /* line read */ 53. if(io!=NULL) { 54. CLS1_SendStr(buf, io->stdOut); /* copy on console */ 55. } 56. } 57. if(UTIL1_strncmp(buf, "SEND OK\r\n", sizeof("SEND OK\r\n")-1)==0) { /* ok from module */ 58. break; 59. } 60. } 61. } 62. } 63. returnres; 64.} The rest of Application.c is rather simple: view source print? 01./* 02. * Application.c 03. * 04. * Author: Erich Styger 05. */ 06.#include "PE_Types.h" 07.#include "CLS1.h" 08.#include "WAIT1.h" 09.#include "Shell.h" 10.#include "UTIL1.h" 11.#include "ESP8266.h" 12.#include "LEDR.h" 13.#include "LEDG.h" 14.#include "AS2.h" 15. 16.staticuint8_t APP_EspMsgBuf[512]; /* buffer for messages from ESP8266 */ 17. 18.staticvoidWebProcess(void) { 19. uint8_t res=ERR_OK; 20. bool isGet; 21. uint8_t ch_id=0; 22. uint16_t size=0; 23. constuint8_t *p; 24. constCLS1_StdIOType *io; 25. 26. if(ESP_IsServerOn()) { 27. io = CLS1_GetStdio(); 28. res = ESP_GetIPD(APP_EspMsgBuf, sizeof(APP_EspMsgBuf), &ch_id, &size, &isGet, 1000, io); 29. if(res==ERR_OK) { 30. if(isGet) { /* GET: put web page */ 31. res = SendWebPage(ch_id, LEDR_Get()!=FALSE, 21/*dummy temperature*/, io); 32. if(res!=ERR_OK && io!=NULL) { 33. CLS1_SendStr("Sending page failed!\r\n", io->stdErr); /* copy on console */ 34. } 35. } else{ /* POST: received info */ 36. intpos; 37. 38. pos = UTIL1_strFind(APP_EspMsgBuf, "radio="); 39. if(pos!=-1) { /* found */ 40. if(UTIL1_strncmp(&APP_EspMsgBuf[pos], "radio=0", sizeof("radio=0")-1)) { 41. LEDR_On(); 42. } elseif(UTIL1_strncmp(&APP_EspMsgBuf[pos], "radio=1", sizeof("radio=1")-1)) { 43. LEDR_Off(); 44. } 45. } 46. res = SendWebPage(ch_id, LEDR_Get()!=FALSE, 20/*dummy temperature*/, io); 47. if(res!=ERR_OK && io!=NULL) { 48. CLS1_SendStr("Sending page failed!\r\n", io->stdErr); /* copy on console */ 49. } 50. } 51. CLS1_SendStr("INFO: Closing connection...\r\n", io->stdOut); 52. res = ESP_CloseConnection(ch_id, io, ESP_DEFAULT_TIMEOUT_MS); 53. } 54. } else{ /* copy messages we receive to console */ 55. while(AS2_GetCharsInRxBuf()>0) { 56. uint8_t ch; 57. 58. (void)AS2_RecvChar(&ch); 59. CLS1_SendChar(ch); 60. } 61. } 62.} 63. 64.voidAPP_Run(void) { 65. CLS1_ConstStdIOType *io; 66. 67. WAIT1_Waitms(1000); /* wait after power-on */ 68. ESP_Init(); 69. SHELL_Init(); 70. io = CLS1_GetStdio(); 71. CLS1_SendStr("\r\n------------------------------------------\r\n", io->stdOut); 72. CLS1_SendStr("ESP8266 with FRDM-KL25Z\r\n", io->stdOut); 73. CLS1_SendStr("------------------------------------------\r\n", io->stdOut); 74. CLS1_PrintPrompt(io); 75. for(;;) { 76. WebProcess(); 77. SHELL_Parse(); 78. WAIT1_Waitms(10); 79. LEDG_Neg(); 80. } 81.} In main.c I call the application part: view source print? 01./* ################################################################### 02.** Filename : main.c 03.** Project : FRDM-KL25Z_ESP8266 04.** Processor : MKL25Z128VLK4 05.** Version : Driver 01.01 06.** Compiler : GNU C Compiler 07.** Date/Time : 2014-10-15, 14:28, # CodeGen: 0 08.** Abstract : 09.** Main module. 10.** This module contains user's application code. 11.** Settings : 12.** Contents : 13.** No public methods 14.** 15.** ###################################################################*/ 16./*! 17.** @file main.c 18.** @version 01.01 19.** @brief 20.** Main module. 21.** This module contains user's application code. 22.*/ 23./*! 24.** @addtogroup main_module main module documentation 25.** @{ 26.*/ 27./* MODULE main */ 28. 29./* Including needed modules to compile this module/procedure */ 30.#include "Cpu.h" 31.#include "Events.h" 32.#include "WAIT1.h" 33.#include "UTIL1.h" 34.#include "AS1.h" 35.#include "ASerialLdd1.h" 36.#include "CLS1.h" 37.#include "CS1.h" 38.#include "AS2.h" 39.#include "ASerialLdd2.h" 40.#include "LEDR.h" 41.#include "LEDpin1.h" 42.#include "BitIoLdd1.h" 43.#include "LEDG.h" 44.#include "LEDpin2.h" 45.#include "BitIoLdd2.h" 46.#include "LEDB.h" 47.#include "LEDpin3.h" 48.#include "BitIoLdd3.h" 49./* Including shared modules, which are used for whole project */ 50.#include "PE_Types.h" 51.#include "PE_Error.h" 52.#include "PE_Const.h" 53.#include "IO_Map.h" 54./* User includes (#include below this line is not maintained by Processor Expert) */ 55.#include "Application.h" 56. 57./*lint -save -e970 Disable MISRA rule (6.3) checking. */ 58.intmain(void) 59./*lint -restore Enable MISRA rule (6.3) checking. */ 60.{ 61. /* Write your local variable definition here */ 62. 63. /*** Processor Expert internal initialization. DON'T REMOVE THIS CODE!!! ***/ 64. PE_low_level_init(); 65. /*** End of Processor Expert internal initialization. ***/ 66. 67. APP_Run(); 68. 69. /*** Don't write any code pass this line, or it will be deleted during code generation. ***/ 70. /*** RTOS startup code. Macro PEX_RTOS_START is defined by the RTOS component. DON'T MODIFY THIS CODE!!! ***/ 71. #ifdef PEX_RTOS_START 72. PEX_RTOS_START(); /* Startup of the selected RTOS. Macro is defined by the RTOS component. */ 73. #endif 74. /*** End of RTOS startup code. ***/ 75. /*** Processor Expert end of main routine. DON'T MODIFY THIS CODE!!! ***/ 76. for(;;){} 77. /*** Processor Expert end of main routine. DON'T WRITE CODE BELOW!!! ***/ 78.} /*** End of main routine. DO NOT MODIFY THIS TEXT!!! ***/ 79. 80./* END main */ 81./*! 82.** @} 83.*/ 84./* 85.** ################################################################### 86.** 87.** This file was created by Processor Expert 10.4 [05.10] 88.** for the Freescale Kinetis series of microcontrollers. 89.** 90.** ################################################################### 91.*/ Processor Expert Components In addition, I’m using several Processor Expert component which are available fromSourceForge. Processor Expert Components Wait: Busy waiting component, e.g. to wait for a few milliseconds. Utility: string manipulation and utility functions. AsynchroSerial (AS1): serial interface to the host for the shell command line interface Shell: command line shell implementation CriticalSection: for creating critical sections AsynchroSerial (AS2): serial interface to the ESP8266 module LEDR, LEDG and LEDB: Red, Green and Blue LED on the FRDM-KL25Z board AS1 is configured as UART connection (over OpenSDA) for the shell: Shell UART Settings There are no special settings for the Shell component: Shell Settings Important are the correct settings to the ESP8266 UART: 115200 baud and using the correct pins on the board connected to the Rx and Tx lines of the ESP8266. I’m using rather large input and output buffers: UART connection to ESP8266 The LED components are configured for the pins used on the board: PTB18 for red, PTB19 for green and PTD1 for blue LED. Red LED for FRDM-KL25Z Sending Commands The shell implements the command ESP send which I can use to send a string or command to the module: ESP send Note that for every command a trailing “\r\n” will be sent. So instead of using the programmatic way, the shell can be used to ‘manually’ drive a web server, at least most of the part. So I’m using command line commands below to explore how the ESP8266 module works. Using the Shell With the project (link to GitHub below), I have a serial connection and command line shell interface to the module. Compile the project and download it to the FRDM-KL25Z board and use a terminal program (I use Termite) to talk with the module. It power-up, the program shows a greeting message: Greeting Message With ‘help‘ I get a list of the available commands: Help Command The ‘status‘ command gives a system status: Status Command Output With this, I’m ready to send commands to the module :-). Connection Test To test the connection I send a simple ‘AT’ command ESP send AT AT Command Output and the module should respond with AT\r\r\n\r\nOK\r\n Module Restart Sometimes the module gets stuck. What helps is a power-on reset of the module. Another way is to send the AT+RST command to reset the module. The module will boot up and print a ‘ready’ message: Reset of the ESP8266 Access Point or Device First I need to configure if the ESP is either a device or an access point. For this, theCWMODE command is used: AT+CWMODE= where is one of: 1: ‘Sta’, ESP8266 is a device, it connects to an existing access point 2: ‘AP’, ESP8266 is an access point, so other devices can connect to it 3: ‘both’. Not really clear to me, but it seems that in this mode the device is in a hybrid mode? To have the ESP as device so it can connect to an existing access point I use AT+CWMODE=1 and the module should answer with AT+CWMODE=1\r\r\n\r\nOK\r\n or with a ‘no change': AT+CWMODE=1\r\r\nno change\r\n With AT+CWMODE? I can ask for the current mode: Retrieving Current Mode List of Access Points With AT+CWLAP I get a list of access points. It reports a list like this: AT+CWLAP +CWLAP:(0,"",0) +CWLAP:(4,"APforESP",-39) +CWLAP:(4,"iza-97497",-94) OK :!: I experienced problems with that command in an environment with lots of access points visible. In this case it seems the module hands up. Try first in a place with only a few access points. For this tutorial I have configured an access point with SSID “APforESP” which shows up in my list. The list is formatted like this + CWLAP: , , [, ] With following encoding: : 0: OPEN 1: WPA_PSK 2: WPA2_PSK 4: WPA_WPA2_PSK : the SSID (string) of the access point. : Signal strength. : 0: manually connect 1: automatic connect Connecting to Access Point To connect to an access point I use the command AT+CWJAP="","" Of course replace and with your setup. The module should report back an “OK” message, and you are connected :-). :!: The module stores the ssid and password. After power-up, the module will automatically reconnect to the Access Point. IP Address Once connected I can check the IP address I have been assigned to with AT+CIFSR which should give something like AT+CIFSR 192.168.0.111 So now I know my module IP address :-). With this I can ping my module: Pinging my ESP Module Building a Web Server Now as we hav a connection, it is time to use it to run a web server :-).What I want to serve a web page which I can use to turn on or off the LEDs on the board. Number of Connections: CIPMUX Before I start the server I need to make sure it accepts multiple connections. For this I use the following command: AT+CIPMUX=1 The parameter is either 0 (single connection), or 1 (multiple connections). For a web server I need to set it up for multiple connections. The ESP module should respond with AT+CIPMUX=1\r\n\r\nOK\r\n :info: To make it clear, I have included the ‘\r’ and ‘\n’ in the responses. Starting the Server: CIPSERVER I start the server with AT+CIPSERVER=1,80 The first parameter is either 0 (close connection) or 1 (open connection), followed by the port. I use here the standard http port (80). The module should answer with: AT+CIPSERVER=1,80\r\r\n\r\nOK\r\n or if it is already running the server with a ‘no change': AT+CIPSERVER=1,80\r\r\nno change\r\n No I have a connection open on my IP address (see above: 192.168.0.111), listening to the port I have specified (80). Connecting to the Server with Browser I enter the IP address in a web browser: http://192.168.0.111:80 For clarity I have specified the standard HTTP port (80). So if you are using a different port, make sure you specify it in the address line. Connection from FireFox The browser now sends a GET request to the module, and I will see this from the message printed out from the module: First response from Module The ‘Link’ indicates that it has established a link. IPD (IP Data?) is followed by the channel number (this will the one we have to respond to), plus the size of the following data (296 bytes in that case). As I’m not responding (yet), there will be a timeout (after about 1 minute or so), with an ‘Unlink’ message from the module: Link +IPD,0,296:GET / HTTP/1.1 Host: 192.168.0.111 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: de,en-US;q=0.7,en;q=0.3 Accept-Encoding: gzip, deflate Connection: keep-alive OK Unlink Unlink message Sending Data to Server: CIPSEND Now I need to respond and send data to the browser. For this I need to know the channel number, and this is provided in the IPD message from above, right after the comma: +IPD,0 To send data, I use the command AT+CIPSEND=, So I connect again with the browser, and I send 5 bytes (“hello”) with: AT+CIPSEND=0,5 The ESP8266 responds with AT+CIPSEND=0,5\r\n> Notice the ‘>’ at the end: this is my signal to send the actual data (“hello” in my case): hello The ESP8266 now resonds with a SEND OK: Data Sent However, the browser is still busy and spins around. I already thought that I did something wrong, but after the browser run into a timeout (after about one minute), my data is there! :-) Hello in Browser Closing Connection: CIPCLOSE So things *are* working :-). The trick is that I have to close the connection after I have sent the data. There is a CIPCLOSE command I can use: AT+CIPCLOSE= which I can use to close a channel. So I close the connection with AT+CIPCLOSE=0 and now the browser shows the content right away :-). Web Server Implementation So far I have used the module in command line and manual mode. This is great for exploration of the protocol, but for building the web server I need to do this programmatically. For this I run my ‘main’ loop in APP_Run(). After printing a greeting message and initializing the sub modules, it processes the web/module responses, parses the shell command line interfaces and blinks the green (LEDG) LED. view source print? 01.voidAPP_Run(void) { 02. CLS1_ConstStdIOType *io; 03. 04. WAIT1_Waitms(1000); /* wait after power-on */ 05. ESP_Init(); 06. SHELL_Init(); 07. io = CLS1_GetStdio(); 08. CLS1_SendStr("\r\n------------------------------------------\r\n", io->stdOut); 09. CLS1_SendStr("ESP8266 with FRDM-KL25Z\r\n", io->stdOut); 10. CLS1_SendStr("------------------------------------------\r\n", io->stdOut); 11. CLS1_PrintPrompt(io); 12. for(;;) { 13. WebProcess(); 14. SHELL_Parse(); 15. WAIT1_Waitms(10); 16. LEDG_Neg(); 17. } 18.} With ESP server start I start the web server: Starting the Web Server It sends the AT+CIPMUX command followed by the AT+CIPSERVER to start the server, and then listens to the port. Reading and responding messages is done in WebProcess(): view source print? 01.staticvoidWebProcess(void) { 02. uint8_t res=ERR_OK; 03. bool isGet; 04. uint8_t ch_id=0; 05. uint16_t size=0; 06. constuint8_t *p; 07. constCLS1_StdIOType *io; 08. 09. if(ESP_IsServerOn()) { 10. io = CLS1_GetStdio(); 11. res = ESP_GetIPD(APP_EspMsgBuf, sizeof(APP_EspMsgBuf), &ch_id, &size, &isGet, 1000, io); 12. if(res==ERR_OK) { 13. if(isGet) { /* GET: put web page */ 14. res = SendWebPage(ch_id, LEDR_Get()!=FALSE, 21/*dummy temperature*/, io); 15. if(res!=ERR_OK && io!=NULL) { 16. CLS1_SendStr("Sending page failed!\r\n", io->stdErr); /* copy on console */ 17. } 18. } else{ /* POST: received info */ 19. intpos; 20. 21. pos = UTIL1_strFind(APP_EspMsgBuf, "radio="); 22. if(pos!=-1) { /* found */ 23. if(UTIL1_strncmp(&APP_EspMsgBuf[pos], "radio=0", sizeof("radio=0")-1)) { 24. LEDR_On(); 25. } elseif(UTIL1_strncmp(&APP_EspMsgBuf[pos], "radio=1", sizeof("radio=1")-1)) { 26. LEDR_Off(); 27. } 28. } 29. res = SendWebPage(ch_id, LEDR_Get()!=FALSE, 20/*dummy temperature*/, io); 30. if(res!=ERR_OK && io!=NULL) { 31. CLS1_SendStr("Sending page failed!\r\n", io->stdErr); /* copy on console */ 32. } 33. } 34. CLS1_SendStr("INFO: Closing connection...\r\n", io->stdOut); 35. res = ESP_CloseConnection(ch_id, io, ESP_DEFAULT_TIMEOUT_MS); 36. } 37. } else{ /* copy messages we receive to console */ 38. while(AS2_GetCharsInRxBuf()>0) { 39. uint8_t ch; 40. 41. (void)AS2_RecvChar(&ch); 42. CLS1_SendChar(ch); 43. } 44. } 45.} If the server is not enabled, it simply copies the received messages to the console: view source print? 1.} else{ /* copy messages we receive to console */ 2. while(AS2_GetCharsInRxBuf()>0) { 3. uint8_t ch; 4. 5. (void)AS2_RecvChar(&ch); 6. CLS1_SendChar(ch); 7. } 8. } Otherwise it scans for an IPD message (ESP_GetIPD()). This function returns the whole message, the channel, the message size and if it is a GET or POST message: 1 res = ESP_GetIPD(APP_EspMsgBuf, sizeof(APP_EspMsgBuf), &ch_id, &size, &isGet, 1000, io); If it is a GET message, then it sends a HTML page to the module: 1 res = SendWebPage(ch_id, LEDR_Get()!=FALSE, 21 /*dummy temperature*/, io); This web page shows the status of the red LED on the board, a (dummy) temperature value and a button to submit new LED values: WSP8266 Web Server The HTML code for this page is constructed in SendWebPage() and sent withAT+CIPSEND: view source print? 01.staticuint8_t SendWebPage(uint8_t ch_id, bool ledIsOn, uint8_t temperature, constCLS1_StdIOType *io) { 02. staticuint8_t http[1024]; 03. uint8_t cmd[24], rxBuf[48], expected[48]; 04. uint8_t buf[16]; 05. uint8_t res = ERR_OK; 06. 07. /* construct web page content */ 08. UTIL1_strcpy(http, sizeof(http), (uint8_t*)"HTTP/1.0 200 OK\r\nContent-Type: text/html\r\nPragma: no-cache\r\n\r\n"); 09. UTIL1_strcat(http, sizeof(http), (uint8_t*)"\r\n\r\n"); 10. UTIL1_strcat(http, sizeof(http), (uint8_t*)"\r\n"); 11. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Web Server using ESP8266\r\n"); 12. UTIL1_strcat(http, sizeof(http), (uint8_t*)" 13.\r\n"); 14. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Temp: OC"); 17. if(ledIsOn) { 18. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Red LED off"); 19. UTIL1_strcat(http, sizeof(http), (uint8_t*)" 20.Red LED on"); 21. } else{ 22. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Red LED off"); 23. UTIL1_strcat(http, sizeof(http), (uint8_t*)" 24.Red LED on"); 25. } 26. UTIL1_strcat(http, sizeof(http), (uint8_t*)""); 27. UTIL1_strcat(http, sizeof(http), (uint8_t*)"\r\n\r\n"); 28. 29. UTIL1_strcpy(cmd, sizeof(cmd), "AT+CIPSEND="); /* parameters are , */ 30. UTIL1_strcatNum8u(cmd, sizeof(cmd), ch_id); 31. UTIL1_chcat(cmd, sizeof(cmd), ','); 32. UTIL1_strcatNum16u(cmd, sizeof(cmd), UTIL1_strlen(http)); 33. UTIL1_strcpy(expected, sizeof(expected), cmd); /* we expect the echo of our command */ 34. UTIL1_strcat(expected, sizeof(expected), "\r\r\n> "); /* expect "> " */ 35. UTIL1_strcat(cmd, sizeof(cmd), "\r\n"); 36. res = ESP_SendATCommand(cmd, rxBuf, sizeof(rxBuf), expected, ESP_DEFAULT_TIMEOUT_MS, io); 37. if(res!=ERR_OK) { 38. if(io!=NULL) { 39. CLS1_SendStr("INFO: TIMEOUT, closing connection!\r\n", io->stdOut); 40. } 41. } else{ 42. if(io!=NULL) { 43. CLS1_SendStr("INFO: Sending http page...\r\n", io->stdOut); 44. } 45. UTIL1_strcat(http, sizeof(http), "\r\n\r\n"); /* need to add this to end the command! */ 46. res = ESP_SendATCommand(http, NULL, 0, NULL, ESP_DEFAULT_TIMEOUT_MS, io); 47. if(res!=ERR_OK) { 48. CLS1_SendStr("Sending page failed!\r\n", io->stdErr); /* copy on console */ 49. } else{ 50. for(;;) { /* breaks */ 51. res = ESP_ReadCharsUntil(buf, sizeof(buf), '\n', 1000); 52. if(res==ERR_OK) { /* line read */ 53. if(io!=NULL) { 54. CLS1_SendStr(buf, io->stdOut); /* copy on console */ 55. } 56. } 57. if(UTIL1_strncmp(buf, "SEND OK\r\n", sizeof("SEND OK\r\n")-1)==0) { /* ok from module */ 58. break; 59. } 60. } 61. } 62. } 63. returnres; 64.} In case of a POST message (user has pressed the button), I scan for the radio element string and turn on/off the LED accordingly, and re-submit the new web page: view source print? 01.} else{ /* POST: received info */ 02. intpos; 03. 04. pos = UTIL1_strFind(APP_EspMsgBuf, "radio="); 05. if(pos!=-1) { /* found */ 06. if(UTIL1_strncmp(&APP_EspMsgBuf[pos], "radio=0", sizeof("radio=0")-1)) { 07. LEDR_On(); 08. } elseif(UTIL1_strncmp(&APP_EspMsgBuf[pos], "radio=1", sizeof("radio=1")-1)) { 09. LEDR_Off(); 10. } 11. } 12. res = SendWebPage(ch_id, LEDR_Get()!=FALSE, 20/*dummy temperature*/, io); 13. if(res!=ERR_OK && io!=NULL) { 14. CLS1_SendStr("Sending page failed!\r\n", io->stdErr); /* copy on console */ 15. } 16. } Finally, it closes the connection at the end: view source print? 1.CLS1_SendStr("INFO: Closing connection...\r\n", io->stdOut); 2. res = ESP_CloseConnection(ch_id, io, ESP_DEFAULT_TIMEOUT_MS); With this, I handle GET and POST messages and can toggle the LED on my board :-) :-). Summary It is amazing what is possible with this tiny and inexpensive ($4.50) WiFi module. The simple AT interface allows small and tiny microprocesors to connect to the internet or the local network. With all the hype around ‘Internet of Things’ this is where things very likely will end up: small nodes connecting in an easy way to the network. The processor on that ESP8266 is probably more powerful than the KL25Z (the specs and data sheets of that ESP8266 are still evolving). Or it is possible to run that module in standalone mode too which is a very interesting approach too, see the links at the end of this article. But still having an UART way to connect to the network is very useful and powerful. Other modules costs multiple times more. I expect that many vendors will come up with similar integrated modules e.g. to combine an ARM processor with the WiFi radio, similar that ESP8266 module. For sure that ESP8266 has a head start and paved the way how WiFi connectivity should work. We all will see what the future brings. Until then, that ESP8266 module is something I can use in many projects :-). The sources and project files can be found on GitHub: https://github.com/ErichStyger/mcuoneclipse/tree/master/Examples/KDS/FRDM-KL25Z/FRDM-KL25Z_ESP8266 Happy Web-Serving :-) Useful Links: http://www.electrodragon.com/w/Wi07c http://scargill.wordpress.com/category/esp8266/ https://github.com/esp8266/esp8266-webserver http://www.cse.dmu.ac.uk/~sexton/ESP8266/ http://defcon-cc.dyndns.org/wiki/ESP8266#Update http://www.xess.com/blog/esp8266-resources/

Compared to other features, real-time search capability is undoubtedly one of the most important features in Elasticsearch. Today we’ll look closely how is provided real-time search by Elasticsearch. Real time First of all, if we need to explain the concept of real-time, in general, we can say that the delay between input and out time in the information is small at real-time systems. This means, data is taken without data accumulation, processed in real time. Today, the best solution Elasticsearch known for real-time search, when a record is added to it for storage makes it searchable in 1 second. How? As is known, the disks are able to create a risk of bottleneck for I/O operations at the data persistence step. Also some mechanisms used for prevent any loss of data increases cost of time. At this point Elasticsearch uses the file-system cache that sitting between itself and the disk for overcome the risk of bottleneck and ensure the a new document can be searched in real time. A new segment is written to the file-system cache first and only later it flushed to disk by Elasticsearch. This lightweight process of writing and opening a new segment is called a refresh in Elasticsearch. By default, all shards is refreshed automatically once every second. In this way, Elasticsearch support real-time search. Test time Above digression about the time of refresh of the shards you can bring to mind the following questions: What happens, when a new document is requested in less than 1 second time? Can be documents requested, without having to depend of the refresh period shards of managed by Elasticsearch? Short answers. Elasticsearch does not return the document. Yes. Now let’s get clarity on this issue is a simple example. hakdogan$ curl -XPUT localhost:9200/kodcucom/document/1 -d'{ > "title": "Document A" > }' We sent a document to Elasticsearch. The index name is kodcucom, type document, id value 1. The title field is only field in the document and the value of "Document A". Let’s take this document from Elasticsearch. hakdogan$ curl -XGET localhost:9200/kodcucom/document/1?pretty { "_index" : "kodcucom", "_type" : "document", "_id" : "1", "_version" : 1, "found" : true, "_source":{ "title": "Document A" } } As expected, the document was returned to us. Well, if we keep short the time between document recording and get request than default shard refresh time what will happen? Let’s see. hakdogan$ curl -XPUT localhost:9200/kodcucom/document/2 -d'{"title": "Document B"}'; curl -XGET localhost:9200/kodcucom/_search?pretty {"_index":"kodcucom","_type":"document","_id":"2","_version":1,"created":true}{ "took" : 38, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 1.0, "hits" : [ { "_index" : "kodcucom", "_type" : "document", "_id" : "1", "_score" : 1.0, "_source":{ "title": "Document A" } } ] } } As can be seen, only the previous document was returned to us by Elasticsearch when we do concurrently create and get request. Well, how can I get the document concurrently? Let’s see. hakdogan$ curl -XPUT localhost:9200/kodcucom/document/3 -d'{"title": "Document C"}'; curl -XGET localhost:9200/kodcucom/_refresh; curl -XGET localhost:9200/kodcucom/_search?pretty {"_index":"kodcucom","_type":"document","_id":"3","_version":1,"created":true}{"_shards":{"total":10,"successful":5,"failed":0}{ "took" : 3, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 3, "max_score" : 1.0, "hits" : [ { "_index" : "kodcucom", "_type" : "document", "_id" : "1", "_score" : 1.0, "_source":{ "title": "Document A" } }, { "_index" : "kodcucom", "_type" : "document", "_id" : "2", "_score" : 1.0, "_source":{"title": "Document B"} }, { "_index" : "kodcucom", "_type" : "document", "_id" : "3", "_score" : 1.0, "_source":{"title": "Document C"} } ] } } In this command, we perform to refresh operation on kodcucom index before the search request. In this way, the document was returned to us. Auto refresh time can be changed. By setting the index.refresh_interval parameter in the configuration file. Applies to all indices in the cluster. A per-index basis by updated index setting. In addition to these, you can turn off automatic refresh. An important point to keep in mind about the refresh time of the shards, the refresh operation is costly in terms of system resources. If you wished to make changes to the auto-refresh time, this situation should be taken into account. Extension of the automatic refresh time, enables faster indexing but new documents and changes made to the existing documents will not appear in searches during specified period of time.

Nowadays it has become mandatory to Gzipping the APIs response due to huge amount of data we are sending in response. It saves network bandwidth and delivery time and off course space over the internet. While using CXF; it provides an option to use the Gzip Compression in no of ways. Blueprint Annotation Blueprint: Annotation: First you need to register the GZIPOutInterceptor in out interceptors list. For that you need to hook into CXF initialization classes. public class InterceptorManager extends AbstractFeature { private static final Logger LOGGER = Logger.getLogger( "simcore" ); private static final Interceptor< Message > GZIP = new GZIPOutInterceptor(); //private static final Interceptor< Message > GZIP = new GZIPOutInterceptor(512); /* (non-Javadoc) * @see org.apache.cxf.feature.AbstractFeature#initializeProvider(org.apache.cxf.interceptor.InterceptorProvider, org.apache.cxf.Bus) */ @Override protected void initializeProvider( InterceptorProvider provider, Bus bus ) { /** * Adding Gzip interceptor to all outbound requests/responses */ LOGGER.debug( " ############## Adding Gzip as OUT Interceptor ##############" ); provider.getOutInterceptors().add( GZIP ); } } GZIPOutInterceptor comes with an option to set the Threshold value as no of Bytes. If response size will be below this threshold value then it will not be compressed. It is extremely useful when we will be sending empty lists and status messages/codes only. Because compressing those small responses will be overhead at server side. But there is another factor which is no of users requesting the response. So set this value by thinking over all the cases in mind. @GZIP Now we can use this annotation on any of our web-services controller to implement compression on all the APIs provided in that class. @WebService @Consumes ( { MediaType.TEXT_PLAIN, MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON } ) @Produces ( MediaType.APPLICATION_JSON ) @GZIP public interface WebServicesController { @GET @Path ( "/myGzipData" ) @Produces ( { MediaType.APPLICATION_JSON } ) Response getZipData( ); } Moreover we can set different parameters in Gzip annotation. @GZIP ( force = true, threshold = 512 )

there is currently a strong trend for microservice based architectures and frequent discussions comparing them to monoliths. there is much advice about breaking-up monoliths into microservices and also some amusing fights between proponents of the two paradigms - see the great microservices vs monolithic melee . the term 'monolith' is increasingly being used as a generic insult in the same way that 'legacy' is! however, i believe that there is a great deal of misunderstanding about exactly what a 'monolith' is and those discussing it are often talking about completely different things. a monolith can be considered an architectural style or a software development pattern (or anti-pattern if you view it negatively). styles and patterns usually fit into different viewtypes (a viewtype is a set, or category, of views that can be easily reconciled with each other [clements et al., 2010]) and some basic viewtypes we can discuss are: module - the code units and their relation to each other at compile time. allocation - the mapping of the software onto its environment. runtime - the static structure of the software elements and how they interact at runtime. a monolith could refer to any of the basic viewtypes above. module monolith if you have a module monolith then all of the code for a system is in a single codebase that is compiled together and produces a single artifact. the code may still be well structured (classes and packages that are coherent and decoupled at a source level rather than a big-ball-of-mud) but it is not split into separate modules for compilation. conversely a non-monolithic module design may have code split into multiple modules or libraries that can be compiled separately, stored in repositories and referenced when required. there are advantages and disadvantages to both but this tells you very little about how the code is used - it is primarily done for development management. allocation monolith for an allocation monolith, all of the code is shipped/deployed at the same time. in other words once the compiled code is 'ready for release' then a single version is shipped to all nodes. all running components have the same version of the software running at any point in time. this is independent of whether the module structure is a monolith. you may have compiled the entire codebase at once before deployment or you may have created a set of deployment artifacts from multiple sources and versions. either way this version for the system is deployed everywhere at once (often by stopping the entire system, rolling out the software and then restarting). a non-monolithic allocation would involve deploying different versions to individual nodes at different times. this is again independent of the module structure as different versions of a module monolith could be deployed individually. runtime monolith a runtime monolith will have a single application or process performing the work for the system (although the system may have multiple, external dependencies). many systems have traditionally been written like this (especially line-of-business systems such as payroll, accounts payable, cms etc). whether the runtime is a monolith is independent of whether the system code is a module monolith or not. a runtime monolith often implies an allocation monolith if there is only one main node/component to be deployed (although this is not the case if a new version of software is rolled out across regions, with separate users, over a period of time). note that my examples above are slightly forced for the viewtypes and it won't be as hard-and-fast in the real world. conclusion be very carefully when arguing about 'microservices vs monoliths'. a direct comparison is only possible when discussing the runtime viewtype and properties. you should also not assume that moving away from a module or allocation monolith will magically enable a microservice architecture (although it will probably help). if you are moving to a microservice architecture then i'd advise you to consider all these viewtypes and align your boundaries across them i.e. don't just code, build and distribute a monolith that exposes subsets of itself on different nodes.

My last post talked about what’s wrong with type classes (in general, but also specifically in Haskell). This post generated some great feedback on Reddit, including some valid criticism that I didn’t explain why I hated on newtypes so much. I took some of that feedback and incorporated it into a revised version of the post, but I have even more to say about “newtypes," so I decided to write another blog post. What’s in a Newtype Newtypes are a feature of Haskell that let you define a new type in terms of an existing type. In the following example, I create a newtype for Email, which “holds” a String. newtype Email = Email String They are similar to type synonyms (type Email = String), except that type synonyms don’t create new types, they just allow you to refer to existing types by other names. Every newtype can be easily translated into a data declaration. In fact, only the keyword changes: data Email = Email String There’s a slight semantic difference between the two, but for purposes of this blog post, any criticism I have against newtypes apply equally to similar constructs modeled with type or data. The Promise of Newtypes Newtypes are used to provide and select between alternate implementations of type classes for some base types. I think that’s a hack (albeit a necessary one), but I’ve already talked about this so I won’t belabor it here. The other promise of newtypes is that we can use them to make our code more type safe. Instead of passing around String as an email, for example, we can create a super lightweight “wrapper” around String called Email, and make it an error to use a String wherever an Email is expected. This practice isn’t restricted to Haskell. Even in Java, it’s considered good coding practice to wrap primitives with classes whose names denote the meaning of the wrapper (Email, SSN, Address, etc.). There’s a part of this promise that’s certainly true. If I have to define a function accepting four parameters, and three of them are strings, but one of those strings denotes an email, then I have two choices: Model the email parameter with a String. In this case, I may accidentally use the email where I intended to use the other two string parameters, or I may use one of the other two string parameters where I intended to use the email. Considering just these choices, there are five ways my program may go wrong if I use the wrong name in the wrong position. Model the email parameter with a newtype. In this case, I cannot use the email where I intended to use the other two string parameters, because the compiler may stop me. Similarly, I cannot use the other two string parameters where I intended to use the email, for the same reason. Looking at just these choices, there are 0 ways my program may go wrong. Thus, newtypes, like all good FP practices, reduce the number of ways my program can go wrong. Unfortunately, in my opinion, they don’t go nearly far enough. False Security For most intent and purposes, newtypes are isomorphic to the single value they hold. In my preceding example, given a String, I can get an email (Email "foo"). Given an Email, I can also get a String, e.g. by pattern matching on the Email constructor. Stated differently, and also approximately because I’m ignoring bottom: the String and Email types are isomorphic; they contain the same inhabitants, for any useful definition of “same”. The only substantive difference between the preceding String and Email is the name of the data constructor (call Email an AbergrackleFoozyWatzit, and what has changed?). Hence, my previous criticism of newtypes as “programming by name”. By themselves, newtypes don’t really reduce the number of ways my program can go wrong. They just make it a bit harder to go wrong. But any newtype is isomorphic to the value it holds, and it’s trivial to convert between the two. In fact, if my code doesn’t need to convert between the two (either directly or indirectly), then it’s better off generic. That is, if I never need to convert an Email to a String, or a String to an Email, then I should really write the code generically to work with any value (even if that means making data structures or functions more polymorphic). Parametricity provides a massive reduction in the number of ways my program can go wrong. Newtypes, on the other hand, just make it a bit harder to go wrong, by adding one layer of indirection. In this example, as with many newtypes, I’ve created a bad isomorphism. The domain model of an email is not isomorphic to the data model of a string. But by using a newtype, I have implicitly declared that they are isomorphic. Calling a string an email may make me feel better, because of the different name, but fundamentally, with a newtype, it’s still a string, and I’m only ever one more step away from going wrong. In my experience, too many newtypes create an isomorphism between things that, properly modeled, are not isomorphic. Fortunately, there’s a well-worn workaround that lets us get more mileage out of newtypes. Smart Constructors If I define Email in a module, I can make its data constructor private, and export a helper function to construct an Email. Such helper functions are called smart constructors. They can be used to break the natural isomorphisms created by newtyping. An example is shown below: newtype Email = MkEmail String mkEmail :: String -> Maybe Email mkEmail s = ... In this example, I create a smart constructor which does not promise that it can turn every string into an email. It promises only that it might be able to turn a string into an email, by returning a Maybe Email. With the smart constructor approach, I’ve modeled the fact that while every email has a string representation, not every string has an email representation. Going back to my earlier example of passing same-typed parameters to a function, if I use a smart constructor, then while I can still use an email anywhere a string is expected (by converting), I can’t use a string anywhere an email is expected. (Well, ignoring the fromJust abomination!) Smart Constructors, Dumb Data Smart constructors take us one step closer toward modeling data in a type safe fashion. Unfortunately, I still don’t think it’s far enough. With smart constructors, our data model is fundamentally underconstrained, so we patch that up by restricting who can create the data. That’s putting a band-aid on the real problem. Why not just solve the root issue — viz., that our data model is underconstrained? Dumb Constructors, Smart Data The best solution to a great many newtype problems, I believe, is creating a data model where there is a true isomorphism between the entity modeled by our data and the values passed to the data constructor. That is, creating a data model such that there exists no regions in our data’s state space which correspond to invalid states. Email is a simple example, because there are well-defined models for what constitutes a data model, which can be translated into data declarations in straightforward, if tedious fashion. (To some extent, it’s a failure of most languages I know that such specifications cannot be easily translated into code without tedious boilerplate!) When our data declaration precisely fits our data model specification, there’s no need for smart constructors, and no need for newtypes. There’s far fewer ways that code can go wrong, and because our domain model is captured precisely by our data model, we can transform that data model in ways that make semantic sense (e.g. transforming just the name part of an email, since we’re now in the realm of structured data). Summary As I’ve explained in this blog post, I don’t really hate newtypes. I think they’re very useful, and I do use them, because they make it more difficult for my programs to go wrong. Ultimately, however, I think a lot of problems solved with newtypes (modeling coordinates, positions, emails, etc.) are better solved by more precise data modeling. That is, by making our programs stop lying about isomorphisms. Precision may be tedious due to limitations of the languages we work in, but honestly, what’s more tedious than debugging broken code?

I don’t do a lot of plotting in my job, but I recently heard about a website called Plotly that provides a plotting service for anyone’s data. They even have a plotly package for Python (among others)! So in this article we will be learning how to plot with their package. Let’s have some fun making graphs! Getting Started You will need the plotly package to follow along with this article. You can use pip to get the package and install it: pip install plotly Now that you have it installed, you’ll need to go to the Plotly website and create a free account. Once that’s done, you will get an API key. To make things super simple, you can use your username and API key to create a credentials file. Here’s how to do that: import plotly.tools as tls tls.set_credentials_file( username="your_username", api_key="your_api_key") # to get your credentials credentials = tls.get_credentials_file() If you don’t want to save your credentials, then you can also sign in to their service by doing the following: import plotly.plotly as py py.sign_in('your_username','your_api_key') For the purposes of this article, I’m assuming you have created the credentials file. I found that makes interacting with their service a bit easier to use. Creating a Graph Plotly seems to default to a Scatter Plot, so we’ll start with that. I decided to grab some data from a census website. You can download any US state’s population data, along with other pieces of data. In this case, I downloaded a CSV file that contained the population of each county in the state of Iowa. Let’s take a look: import csv import plotly.plotly as py #---------------------------------------------------------------------- def plot_counties(csv_path): """ http://census.ire.org/data/bulkdata.html """ counties = {} county = [] pop = [] counter = 0 with open(csv_path) as csv_handler: reader = csv.reader(csv_handler) for row in reader: if counter == 0: counter += 1 continue county.append(row[8]) pop.append(row[9]) trace = dict(x=county, y=pop) data = [trace] py.plot(data, filename='ia_county_populations') if __name__ == '__main__': csv_path = 'ia_county_pop.csv' plot_counties(csv_path) If you run this code, you should see a graph that looks like this: <br> You can also view the graph here. Anyway, as you can see in the code above, all I did was read the CSV file and extract out the county name and the population. Then I put that data into two different Python lists. Finally I created a dictionary of those lists and then wrapped that dictionary in a list. So you end up with a list that contains a dictionary that contains two lists! To make the Scatter Plot, I passed the data to plotly’s plot method. Converting to a Bar Chart Now let’s see if we can change the ScatterPlot to a Bar Chart. First off, we’ll play around with the plot data. The following was done via the Python interpreter: >>> scatter = py.get_figure('driscollis', '0') >>> print scatter.to_string() Figure( data=Data([ Scatter( x=[u'Adair County', u'Adams County', u'Allamakee County', u'..', ], y=[u'7682', u'4029', u'14330', u'12887', u'6119', u'26076', '..' ] ) ]) ) This shows how we can grab the figure using the username and the plot’s unique number. Then we printed out the data structure. You will note that it doesn’t print out the entire data structure. Now let’s do the actual conversion to a Bar Chart: from plotly.graph_objs import Data, Figure, Layout scatter_data = scatter.get_data() trace_bar = Bar(scatter_data[0]) data = Data([trace_bar]) layout = Layout(title="IA County Populations") fig = Figure(data=data, layout=layout) py.plot(fig, filename='bar_ia_county_pop') This will create a bar chart at the following URL: https://plot.ly/~driscollis/1. Here’s the image of the graph: <br> This code is slightly different than the code we used originally. In this case, we explicitly created a Bar object and passed it the scatter plot’s data. Then we put that data into a Data object. Next we created a Layout object and gave our chart a title. Then we created a Figure object using the data and layout objects. Finally we plotted the bar chart. Saving the Graph to Disk Plotly also allows you to save your graph to your hard drive. You can save it in the following formats: png, svg, jpeg, and pdf. Assuming you still have the Figure object from the previous example handy, you can do the following: py.image.save_as(fig, filename='graph.png') If you want to save using one of the other formats, then just use that format’s extension in the filename. Wrapping Up At this point you should be able to use the plotly package pretty well. There are many other graph types available, so be sure to read Plotly’s documentation thoroughly. They also support streaming graphs. As I understand it, Plotly allows you to create 10 graphs for free. After that you would either have to delete some of your graphs or pay a monthly fee. Additional Reading Plotly Python documentation Plotly User Guide

In the modern world of microservices it's important to provide strict and polyglot clients for your service. It's better if your API is self-documented. One of the best tools for it is Apache Thrift. I want to explain how to use it with my favorite platform for microservices - Spring Boot. All project source code is available on GitHub: https://github.com/bsideup/spring-boot-thrift Project skeleton I will use Gradle to build our application. First, we need our main build.gradle file: buildscript { repositories { jcenter() } dependencies { classpath("org.springframework.boot:spring-boot-gradle-plugin:1.1.8.RELEASE") } } allprojects { repositories { jcenter() } apply plugin:'base' apply plugin: 'idea' } subprojects { apply plugin: 'java' } Nothing special for a Spring Boot project. Then we need a gradle file for thrift protocol modules (we will reuse it in next part): import org.gradle.internal.os.OperatingSystem repositories { ivy { artifactPattern "http://dl.bintray.com/bsideup/thirdparty/[artifact]-[revision](-[classifier]).[ext]" } } buildscript { repositories { jcenter() } dependencies { classpath "ru.trylogic.gradle.plugins:gradle-thrift-plugin:0.1.1" } } apply plugin: ru.trylogic.gradle.thrift.plugins.ThriftPlugin task generateThrift(type : ru.trylogic.gradle.thrift.tasks.ThriftCompileTask) { generator = 'java:beans,hashcode' destinationDir = file("generated-src/main/java") } sourceSets { main { java { srcDir generateThrift.destinationDir } } } clean { delete generateThrift.destinationDir } idea { module { sourceDirs += [file('src/main/thrift'), generateThrift.destinationDir] } } compileJava.dependsOn generateThrift dependencies { def thriftVersion = '0.9.1'; Map platformMapping = [ (OperatingSystem.WINDOWS) : 'win', (OperatingSystem.MAC_OS) : 'osx' ].withDefault { 'nix' } thrift "org.apache.thrift:thrift:$thriftVersion:${platformMapping.get(OperatingSystem.current())}@bin" compile "org.apache.thrift:libthrift:$thriftVersion" compile 'org.slf4j:slf4j-api:1.7.7' } We're using my Thrift plugin for Gradle. Thrift will generate source to the "generated-src/main/java" directory. By default, Thrift uses slf4j v1.5.8, while Spring Boot uses v1.7.7. It will cause an error in runtime when you will run your application, that's why we have to force a slf4j api dependency. Calculator service Let's start with a simple calculator service. It will have 2 modules: protocol and app.We will start with protocol. Your project should look as follows: calculator/ protocol/ src/ main/ thrift/ calculator.thrift build.gradle build.gradle settings.gradle thrift.gradle Where calculator/protocol/build.gradle contains only one line: apply from: rootProject.file('thrift.gradle') Don't forget to put these lines to settings.gradle, otherwise your modules will not be visible to Gradle: include 'calculator:protocol' include 'calculator:app' Calculator protocol Even if you're not familiar with Thrift, its protocol description file (calculator/protocol/src/main/thrift/calculator.thrift) should be very clear to you: namespace cpp com.example.calculator namespace d com.example.calculator namespace java com.example.calculator namespace php com.example.calculator namespace perl com.example.calculator namespace as3 com.example.calculator enum TOperation { ADD = 1, SUBTRACT = 2, MULTIPLY = 3, DIVIDE = 4 } exception TDivisionByZeroException { } service TCalculatorService { i32 calculate(1:i32 num1, 2:i32 num2, 3:TOperation op) throws (1:TDivisionByZeroException divisionByZero); } Here we define TCalculatorService with only one method - calculate. It can throw an exception of type TDivisionByZeroException. Note how many languages we're supporting out of the box (in this example we will use only Java as a target, though) Now run ./gradlew generateThrift, you will get generated Java protocol source in the calculator/protocol/generated-src/main/java/ folder. Calculator application Next, we need to create the service application itself. Just create calculator/app/ folder with the following structure: src/ main/ java/ com/ example/ calculator/ handler/ CalculatorServiceHandler.java service/ CalculatorService.java CalculatorApplication.java build.gradle Our build.gradle file for app module should look like this: apply plugin: 'spring-boot' dependencies { compile project(':calculator:protocol') compile 'org.springframework.boot:spring-boot-starter-web' testCompile 'org.springframework.boot:spring-boot-starter-test' } Here we have a dependency on protocol and typical starters for Spring Boot web app. CalculatorApplication is our main class. In this example I will configure Spring in the same file, but in your apps you should use another config class instead. package com.example.calculator; import com.example.calculator.handler.CalculatorServiceHandler; import org.apache.thrift.protocol.*; import org.apache.thrift.server.TServlet; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.EnableAutoConfiguration; import org.springframework.context.annotation.*; import javax.servlet.Servlet; @Configuration @EnableAutoConfiguration @ComponentScan public class CalculatorApplication { public static void main(String[] args) { SpringApplication.run(CalculatorApplication.class, args); } @Bean public TProtocolFactory tProtocolFactory() { //We will use binary protocol, but it's possible to use JSON and few others as well return new TBinaryProtocol.Factory(); } @Bean public Servlet calculator(TProtocolFactory protocolFactory, CalculatorServiceHandler handler) { return new TServlet(new TCalculatorService.Processor(handler), protocolFactory); } } You may ask why Thrift servlet bean is called "calculator". In Spring Boot, it will register your servlet bean in context of the bean name and our servlet will be available at /calculator/. After that we need a Thrift handler class: package com.example.calculator.handler; import com.example.calculator.*; import com.example.calculator.service.CalculatorService; import org.apache.thrift.TException; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Component; @Component public class CalculatorServiceHandler implements TCalculatorService.Iface { @Autowired CalculatorService calculatorService; @Override public int calculate(int num1, int num2, TOperation op) throws TException { switch(op) { case ADD: return calculatorService.add(num1, num2); case SUBTRACT: return calculatorService.subtract(num1, num2); case MULTIPLY: return calculatorService.multiply(num1, num2); case DIVIDE: try { return calculatorService.divide(num1, num2); } catch(IllegalArgumentException e) { throw new TDivisionByZeroException(); } default: throw new TException("Unknown operation " + op); } } } In this example I want to show you that Thrift handler can be a normal Spring bean and you can inject dependencies in it. Now we need to implement CalculatorService itself: package com.example.calculator.service; import org.springframework.stereotype.Component; @Component public class CalculatorService { public int add(int num1, int num2) { return num1 + num2; } public int subtract(int num1, int num2) { return num1 - num2; } public int multiply(int num1, int num2) { return num1 * num2; } public int divide(int num1, int num2) { if(num2 == 0) { throw new IllegalArgumentException("num2 must not be zero"); } return num1 / num2; } } That's it. Well... almost. We still need to test our service somehow. And it should be an integration test. Usually, even if your application is providing JSON REST API, you still have to implement a client for it. Thrift will do it for you. We don't have to care about it. Also, it will support different protocols. Let's use a generated client in our test: package com.example.calculator; import org.apache.thrift.protocol.*; import org.apache.thrift.transport.THttpClient; import org.apache.thrift.transport.TTransport; import org.junit.*; import org.junit.runner.RunWith; import org.springframework.beans.factory.annotation.*; import org.springframework.boot.test.IntegrationTest; import org.springframework.boot.test.SpringApplicationConfiguration; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; import org.springframework.test.context.web.WebAppConfiguration; import static org.junit.Assert.*; @RunWith(SpringJUnit4ClassRunner.class) @SpringApplicationConfiguration(classes = CalculatorApplication.class) @WebAppConfiguration @IntegrationTest("server.port:0") public class CalculatorApplicationTest { @Autowired protected TProtocolFactory protocolFactory; @Value("${local.server.port}") protected int port; protected TCalculatorService.Client client; @Before public void setUp() throws Exception { TTransport transport = new THttpClient("http://localhost:" + port + "/calculator/"); TProtocol protocol = protocolFactory.getProtocol(transport); client = new TCalculatorService.Client(protocol); } @Test public void testAdd() throws Exception { assertEquals(5, client.calculate(2, 3, TOperation.ADD)); } @Test public void testSubtract() throws Exception { assertEquals(3, client.calculate(5, 2, TOperation.SUBTRACT)); } @Test public void testMultiply() throws Exception { assertEquals(10, client.calculate(5, 2, TOperation.MULTIPLY)); } @Test public void testDivide() throws Exception { assertEquals(2, client.calculate(10, 5, TOperation.DIVIDE)); } @Test(expected = TDivisionByZeroException.class) public void testDivisionByZero() throws Exception { client.calculate(10, 0, TOperation.DIVIDE); } } This test will run your Spring Boot application, bind it to a random port and test it. All client-server communications will be performed in the same way real world clients are. Note how easy to use our service is from the client side. We're just calling methods and catching exceptions.

A hashmap which maintains keys as regular expressions. Any pattern matching the expression will be able to retrieve the same value. Internally it maintains two maps, one containing the regex to value, and another containing matched pattern to regex. Whenever there is a new pattern to 'get', there will be a O(n) search through the compiled regex(s) (which have been 'put' as keys) to find a match. Existing patterns will have constant time lookup through two maps. import java.util.ArrayList; import java.util.Collection; import java.util.HashMap; import java.util.Iterator; import java.util.List; import java.util.Map; import java.util.Set; import java.util.WeakHashMap; import java.util.regex.Pattern; public class RegexHashMap implements Map { private class PatternMatcher { private final String regex; private final Pattern compiled; PatternMatcher(String name) { regex = name; compiled = Pattern.compile(regex); } boolean matched(String string) { if(compiled.matcher(string).matches()) { ref.put(string, regex); return true; } return false; } } /** * Map of input to pattern */ private final Map ref; /** * Map of pattern to value */ private final Map map; /** * Compiled patterns */ private final List matchers; @Override public String toString() { return "RegexHashMap [ref=" + ref + ", map=" + map + "]"; } /** * */ public RegexHashMap() { ref = new WeakHashMap(); map = new HashMap(); matchers = new ArrayList(); } /** * Returns the value to which the specified key pattern is mapped, or null if this map contains no mapping for the key pattern */ @Override public V get(Object weakKey) { if(!ref.containsKey(weakKey)) { for(PatternMatcher matcher : matchers) { if(matcher.matched((String) weakKey)) { break; } } } if(ref.containsKey(weakKey)) { return map.get(ref.get(weakKey)); } return null; } /** * Associates a specified regular expression to a particular value */ @Override public V put(String key, V value) { V v = map.put(key, value); if (v == null) { matchers.add(new PatternMatcher(key)); } return v; } /** * Removes the regular expression key */ @Override public V remove(Object key) { V v = map.remove(key); if(v != null) { for(Iterator iter = matchers.iterator(); iter.hasNext();) { PatternMatcher matcher = iter.next(); if(matcher.regex.equals(key)) { iter.remove(); break; } } for(Iterator> iter = ref.entrySet().iterator(); iter.hasNext();) { Entry entry = iter.next(); if(entry.getValue().equals(key)) { iter.remove(); } } } return v; } /** * Set of view on the regular expression keys */ @Override public Set> entrySet() { return map.entrySet(); } @Override public void putAll(Map m) { for(Entry entry : m.entrySet()) { put(entry.getKey(), entry.getValue()); } } @Override public int size() { return map.size(); } @Override public boolean isEmpty() { return map.isEmpty(); } /** * Returns true if this map contains a mapping for the specified regular expression key. */ @Override public boolean containsKey(Object key) { return map.containsKey(key); } /** * Returns true if this map contains a mapping for the specified regular expression matched pattern. * @param key * @return */ public boolean containsKeyPattern(Object key) { return ref.containsKey(key); } @Override public boolean containsValue(Object value) { return map.containsValue(value); } @Override public void clear() { map.clear(); matchers.clear(); ref.clear(); } /** * Returns a Set view of the regular expression keys contained in this map. */ @Override public Set keySet() { return map.keySet(); } /** * Returns a Set view of the regex matched patterns contained in this map. The set is backed by the map, so changes to the map are reflected in the set, and vice-versa. * @return */ public Set keySetPattern() { return ref.keySet(); } @Override public Collection values() { return map.values(); } /** * Produces a map of patterns to values, based on the regex put in this map * @param patterns * @return */ public Map transform(List patterns) { for(String pattern : patterns) { get(pattern); } Map transformed = new HashMap(); for(Entry entry : ref.entrySet()) { transformed.put(entry.getKey(), map.get(entry.getValue())); } return transformed; } public static void main(String...strings) { RegexHashMap rh = new RegexHashMap(); rh.put("[o|O][s|S][e|E].?[1|2]", "This is a regex match"); rh.put("account", "This is a direct match"); System.out.println(rh); System.out.println("get:ose-1 -> "+rh.get("ose-1")); System.out.println("get:OSE2 -> "+rh.get("OSE2")); System.out.println("get:OSE112 -> "+rh.get("OSE112")); System.out.println("get:ose-2 -> "+rh.get("ose-2")); System.out.println("get:account -> "+rh.get("account")); System.out.println(rh); } }

daniel bryant , simon and i recently had a discussion about how to represent system communication with external apis. the requirement for integration with external apis is now extremely common but it's not immediately obvious how to clearly show them in architectural diagrams. how to represent an external system? the first thing we discussed was what symbol to use for a system supplying an api. traditionally, uml has used the actor (stick man) symbol to represent a "user or any other system that interacts with the subject" (uml superstructure specification, v2.1.2). therefore a system providing an api may look like this: i've found that this symbol tends to confuse those who aren't well versed in uml as most people assume that the actor symbol always represents a *person* rather than a system. sometimes this is stereotyped to make it more obvious e.g. however the symbol is very powerful and tends to overpower the stereotype. therefore i prefer to use a stereotyped box for an external system supplying an api. let's compare two context diagrams using boxes vs stick actors. in which diagram is it more obvious what are systems or people? note that archimate has a specific symbol for application service that can be used to represent an api: (application service notation from the open group's archimate 2.1 specification) an api or the system that supplies it? whatever symbol we choose, what we've done is to show the *system* rather than the actual api. the api is a definition of a service provided by the system in question. how should we provide more details about the api? there are a number of ways we could do this but my preference is to give details of the api on the connector (line connecting two elements/boxes). in c4 the guidelines for a container diagram includes listing protocol information on the connector and an api can be viewed as the layer above the protocol. for example: multiple apis per external system many api providers supply multiple services/apis (i'm not referring to different operations within an api but multiple sets of operations in different apis, which may even use different underlying protocols.) for example a financial marketplace may have apis that do the following: allow a bulk, batch download of static data (such as details of companies listed on a stock market) via xml over http. supply real time, low latency updates of market prices via bespoke messages over udp. allow entry of trades via industry standard fpml over a queuing system. supply a bulk, batch download of trades for end-of-day reconciliation via fpml over http. two of the services use the same protocol (xml over http) but have very different content and use. one of the apis is used to constantly supply information after user subscription (market data) and the last service involves the user supplying all the information with no acknowledgment (although it should reconcile at eod). there are multiple ways of showing this. we could: have a single service element, list the apis on it and have all components linking to it. show each service/api as a separate box and connect the components that use the individual service to the relevant box. show a single service element with multiple connections. each connection is labeled and represents an api. use a port and connector style notation to represent each api from the service provider. provide a key for the ports. use a uml style 'cup and ball' notation to define interfaces and their usage. some examples are below: a single service element and simple description in the above diagram the containers are stating what they are using but contain no information about how to use the apis. we don't know if it is a single api (with different operations) or anything about the mechanisms used to transport the data. this isn't very useful for anyone implementing a solution or resolving operational issues. single, service box with descriptive connectors in this diagram there is a single, service box with descriptive connectors. the above diagram shows all the information so is much more useful as a diagnostic or implementation tool. however it does look quite crowded. services/apis shown as separate boxes here the external system has its services/apis shown as separate boxes. this contains all the information but might be mistaken as defining the internal structure of the external system. we want to show the services it provides but we know nothing about the internal structure. using ports to represent apis in the above diagram the services/apis are shown as 'ports' on the external system and the details have been moved into a separate key/table. this is less likely to be mistaken as showing any internal structure of the external service. (note that i could have also shown outgoing rports from the brokerage system.) uml interfaces this final diagram is using a uml style interface provider and requirer. this is a clean diagram but requires the user to be aware of what the cup and ball means (although i could have explained this in the key). conclusion any of these solutions could be appropriate depending on the complexity of the api set you are trying to represent. i'd suggest starting with a simple representation (i.e. fully labeled connections) and moving to a more complex one if needed but remember to use a key to explain any elements you use!

REST Easy with SQL/NoSQL Integration and CQRS Pattern implementation New demands are being put on IT organizations everyday to deliver agile, high-performance, integrated mobile and web applications. In the meantime, the technology landscape is getting complex everyday with the advent of new technologies like REST, NoSQL, Cloud while existing technologies like SOAP and SQL still rule everyday work. Rather than taking religious side of the debate, NoSQL can successfully co-exist with SQL in this ‘polyglot’ of data storage and formats. However, this integration also adds another layer of complexity both in architecture and implementation. This document offers a guide on how some of the relatively newer technologies like REST can help bridge the gap between SQL and NoSQL with an example of a well known pattern called CQRS. This document is organized as follows: Introduction to SQL development process NoSQL Do I have to choose between SQL and NoSQL? CQRS Pattern How to implement CQRS pattern using REST services Introduction to SQL development process Developers have been using SQL Databases for decades to build and deliver enterprise business applications. The process of creating tables, attributes,and relationships is second nature for most developers. Data architects think in terms of tables and columns and navigate relationships for data. The basic concepts of delivery and transformation takes place at the web server level which means the server developer is reading and ‘binding’ to the tables and mapping attributes to a REST response. Application development lifecycle meant changes to the database schema first, followed by the bindings, then internal schema mapping, and finally the SOAP or JSON services, and eventually the client code. This all costs the project time and money. It also means that the ‘code’ (pick your language here) and the business logic would also need to be modified to handle the changes to the model. NoSQL NoSQL is gaining supporters among many SQL shops for various reasons including: Low cost Ability to handle unstructured dataa Scalability Performance The first thing database folks notice is that there is no schema. These document style storage engines can handle huge volumes of structured, semi-structured, and unstructured data. The very nature of schema-less documents allows change to a document structure without having to go through the formal change management process (or data architect). The other major difference is that NoSQL (no-schema) also means no joins or relationships. The document itself contains the embedded information by design. So an order entry would contain the customer with all the orders and line items for each order in a single document. There are many different NoSQL vendors (popular NoSQL databases include MongoDB, Casandra) that are being used for BI and Analytics (read-only) purposes. We are also seeing many customers starting to use NoSQL for auditing, logging, and archival transactions. Do I have to choose between SQL and NoSQL? The purpose of this article is to not get into the religious debate about whether to use SQL or NoSQL. Bottom line is both have their place and are suited for certain type of data – SQL for structured data and NoSQL for unstructured data. So why not have the capability to mix and match this data depending on the application. This can be done by creating a single REST API across both SQL and NoSQL databases. Why a single REST API? The answer is simple – the new agile and mobile world demands this ‘mashup’ of data into a document style JSON response. CQRS (Command Query Responsibility Segmentation) Pattern There are many design patterns for delivery of high performance RESTful services but the one that stands out was described in an article written by Martin Fowler, one of the software industry veterans. He described the pattern called CQRS that is more relevant today in a ‘polyglot’ of servers, data, services, and connections. “We may want to look at the information in a different way to the record store, perhaps collapsing multiple records into one, or forming virtual records by combining information for different places. On the update side we may find validation rules that only allow certain combinations of data to be stored, or may even infer data to be stored that’s different from that we provide.” – Martin Fowler 2011 In this design pattern, the REST API requests (GET) return documents from multiple sources (e.g. mashups). In the update process, the data is subject to business logic derivations, validations, event processing, and database transactions. This data may then be pushed back into the NoSQL using asynchronous events. With the wide-spread adoption of NoSQL databases like MongoDB and schema-less, high capacity data store; most developers are challenged with providing security, business logic, event handling, and integration to other systems. MongoDB; one the popular NoSQL databases and SQL databases share many similar concepts. However the MongoDB programming language itself is very different from the SQL we all know. How to implement CQRS pattern using a RESTFul Architecture A REST server should meet certain requirements to support the CQRS pattern. The server should run on-premise or in the cloud and appears to the mobile and web developer as an HTTP endpoint. The server architecture should implement the following: Connections and Mapping necessary for SQL and NoSQL connectivity and API services needed to create and return GET, PUT, POST, and DELETE REST responses Security Business Logic Connections and Mapping There are two main approaches to creating REST Servers and APIs for SQL and NoSQL databases: Open source frameworks like Apache Tomcat, Spring/Hibernate Commercial framework like Espresso Logic Open source Frameworks Using various open source frameworks like Tomcat, Spring/Hibernate, Node.js, JDBC and MongoDB drivers, a REST server can be created, but we would still be left with the following tasks: Creation and mapping of the necessary SQL objects Create a REST server container and configurations Create Jersey/Jackson classes and annotations Create and define REST API for tables, views, and procedures Hand write validation, event and business logic Handle persistence, optimistic locking, transaction paging Adding identity management and security by roles Now we can start down the same path to connect to MongoDB and write code to connect, select, and return data in JSON and then create the REST calls to merge these two different document styles into a single RESTful endpoint. This is a lot of work for a development team to manage and control and frankly pretty boring and repetitive and is better done by a well designed framework Commercial Frameworks Many commercial frameworks may take care of this complexity without the need to do extensive programming. Here is an example from Espresso Logic and how it handles this complexity with a point and click interface: Running REST server in the cloud or on-premise Connections to external SQL databases Object mapping to tables, views, and procedures Automatic creation of RESTful endpoints from model Reactive business rules and rich event model Integrated role-based security and authentication services. Point-and-click document API creation for SQL and MongoDB endpoints In the example below, the editor shows an SQL (customersTransactions) joined with archived details from MongoDB (archivedTransactions). The MongoDB document for each customer may include transaction details, check images, customer service notes and other relevant account information. This new mashup becomes a single REST call that can be published to mobile and web application developer. Security Security is an important part of building and delivery of RESTful services which can be broken down into two parts; authentication and access control. Authentication Before allowing anyone access to corporate data you want to use the existing corporate identity management (some call this authentication services) to capture and validate the user. This identity management service is based on using existing corporate standards such as LDAP, Windows AD, SQL Database. Role-based Access Control Each user may be assigned one or more corporate roles and these roles are then assigned specific access privileges to each resource (e.g. READ, INSERT, UPDATE, and DELETE). Role-based access should also be able to restrict permissions to specific rows and columns of the API (e.g. only sales reps can see their own orders or a manager can see and change his department salaries but cannot change his own). This restriction should be applied regardless of how or where the API is used or called. Remember, the SQL database already provides some level of security and access which must be considered when designing and delivering new front-end services to internal and external users. Business Logic for REST When data is updated to a REST Server several things need to happen. First, the authentication and access control should determine if this is a valid request and if the user has rights to the endpoint. In addition, the server may need to de-alias REST attributes back to the actual SQL column names. In a full featured business logic server, there should be a series of events and business rules to perform various calculations, validations, and fire other events on dependent tables. Finally, the entire multi-table transaction is written back to the SQL database in a single transaction. Updates are then sent asynchronously to MongoDB as part of the commit event (after the SQL transaction has completed). Conclusion In the real-world of API services, the demand for more complex document style RESTful services is a requirement. That is, the ability to create ‘mashups’ of data from multiple tables, NoSQL collections, and other external systems is a large part of this new design pattern. In addition, the ability to alias attribute names and formats from these source fields has become critical for partners and customers systems. Using REST with the CQRS pattern to blend MongoDB and SQL seamlessly to your existing data will become a major part of your future mobile strategy. To implement these REST services, one can use open source tools and spend a lot of time or select a right commercial framework. This framework should support cloud or on-premise connectivity, security, API integration, as well as business logic. This will make the design and delivery of new application services more rapid and agile in the heterogeneous world of information.