In my previous articles I compared 4 frameworks commonly used in communicating with MongoDB from the JVM and found out that in that use-case, Spring Data for MongoDB was the easiest solution. However I did make the remark that it doesn’t use the GeoJSON format to store geolocation coordinates and geometries. I tried to add GeoJSON support before, but couldn’t get the conversion to work propertly. But after some extensive searching I found out that the reason for it not working was my use of Spring Boot: its autoconfiguration for MongoDB does not support custom conversion out of the box. Luckily, the solution was simple: provide an extra configuration that extends from AbstractMongoConfiguration and import that in the Boot application. In that configuration you can override the customConversions() and add your converters. When you compare the geo classes in Spring Data and GeoJSON, I noticed that only a subset of GeoJSON geometries can be mapped on Spring Data geo classes: Point and Polygon. Spring Boot does not support LineString, MultiLineString, MultiPolygon or MultiPoint. However, in your mapped domain classes, you won’t use these normally. Creating a converter that adheres to the GeoJSON format is quite straightforward. import com.mongodb.BasicDBObject import com.mongodb.DBObject import org.springframework.core.convert.converter.Converter import org.springframework.data.convert.ReadingConverter import org.springframework.data.convert.WritingConverter import org.springframework.data.geo.Point import org.springframework.data.geo.Polygon final class GeoJsonConverters { static List> getConvertersToRegister() { return [ GeoJsonDBObjectToPointConverter.INSTANCE, GeoJsonDBObjectToPolygonConverter.INSTANCE, GeoJsonPointToDBObjectConverter.INSTANCE, GeoJsonPolygonToDBObjectConverter.INSTANCE ] } @WritingConverter static enum GeoJsonPointToDBObjectConverter implements Converter { INSTANCE; @Override DBObject convert(Point source) { return new BasicDBObject([type: 'Point', coordinates: [source.x, source.y]]) } } @ReadingConverter static enum GeoJsonDBObjectToPointConverter implements Converter { INSTANCE; @Override Point convert(DBObject source) { def coordinates = source.coordinates as double[] return new Point(coordinates[0], coordinates[1]) } } @WritingConverter static enum GeoJsonPolygonToDBObjectConverter implements Converter { INSTANCE; @Override DBObject convert(Polygon source) { def coordinates = source.points.collect { [it.x, it.y] } return new BasicDBObject([type: 'Polygon', coordinates: coordinates]) } } @ReadingConverter static enum GeoJsonDBObjectToPolygonConverter implements Converter { INSTANCE; @Override Polygon convert(DBObject source) { def coordinates = source.coordinates as double[] return new Point(coordinates[0], coordinates[1]) } } } To add those converters to the Spring context, you’ll have to override some methods in your MongoDB spring configuration class. import com.mongodb.Mongo import org.springframework.beans.factory.annotation.* import org.springframework.boot.SpringApplication import org.springframework.boot.autoconfigure.EnableAutoConfiguration import org.springframework.context.annotation.* import org.springframework.data.mongodb.config.AbstractMongoConfiguration import org.springframework.data.mongodb.core.convert.* @EnableAutoConfiguration @ComponentScan @Configuration @Import([MongoComparisonMongoConfiguration]) class MongoComparison { static void main(String[] args) { SpringApplication.run(MongoComparison, args); } } @Configuration class MongoComparisonMongoConfiguration extends AbstractMongoConfiguration { @Autowired Mongo mongo; @Value("\${spring.data.mongodb.database}") String databaseName; @Override protected String getDatabaseName() { return databaseName } @Override Mongo mongo() throws Exception { return mongo } @Override CustomConversions customConversions() { def customConverters = [] customConverters << GeoJsonConverters.convertersToRegister return new CustomConversions(customConverters.flatten()) } } As Spring Boot already provides the configuration of the Mongo instance and the name of the database, we can reuse these in the MongoDB configuration class. The custom conversions take preference over the existing ones for Point and Polygon. I’ll be writing a library this weekend to add support for all GeoJSON geometries in Spring Data for MongoDB. However, I already noticed it’ll be very hard to provide support for those in generated query methods in repositories, but with annotated queries being possible, I don’t think this will be a big issue but we’ll see.

recently some pwc tech supremos wrote an article: agile coding in enterprise it: code small and local . subsections: moving away from the monolith why microservices? msa: a think-small approach for rapid development thinking the msa way: minimalism is a must where msa makes sense in msa, integration is the problem, not the solution conclusion msa is short for microservices architecture(s), in the above article. the article posits that microservices is the antidote to monoliths. it doesn’t mention cookie cutter scaling at all, which is another antidote to monoliths, with the right build infrastructure and devops. here’s a view of hypothetical architecture a company could deploy if they were doing microservices: w is web server. p and q don’t stand for anything in particular. here’s the same solution as cookie-cutter scaling, and the alternate (historical) choice of monolith to the right of it: the cookie cutter approach will often leverage components that are dependency injected into each other, and though monoliths might be the same today, pre 2004 they were probably hairballs of singletons (the design patten, not the springframework idiom). continuous delivery, agile? here’s one excerpt that confuses me: " … makes no sense to design and develop software over an 18-month process to accommodate all possible use cases when those use cases can change unexpectedly and the life span of code modules might be less than 18 months…. as i recall, the 18 month-delay problem was solved previously. agile methodologies principally, and continuous delivery/deployment in more recent times. it does not matter whether you’re compiling a monolith, a cookie-cutter solution, old soa services, or microservices, the 18-month fear isn’t real if you’re doing agile and/or cd. agile and cd were increasing the release cadence, and allowing the organization to pivot faster before microservices. it doesn’t matter whether you’ve got a monolith, something cookie-cutter scaled, or soa (micro or not), you’re going to be able to benefit from agile practices and devops setup that facilitates cd. in something like 30 thoughtworks client engagements since 2002, i have not seen the 18-month process at all. in fact i last encountered it in 1997 on an as/400 project, which was the last time i saw a waterfall process being championed. build(s) and trunk elsewhere there is a suggestion: “each microservice [has] its own build, to avoid trunk conflict”. that isn’t unique to microservices, of course. component based systems today also have a multiple build file (module) structure in a source tree. hopefully “trunk” mentioned is alluding to trunk based development, as i would recommend. build technologies this is a expansion on the above, and you can skip this paragraph if you want. hierarchical build systems like maven has allow you to have one build file per module (whether that’s a service or a simple jar destined for the classpath of a bigger thing). buck has a build grammar that allows for a build to grow/shrink/change based on what is being built (from implicitly shared source). maven is for the java ecosystem, while buck promises to be multi-language. both are doing multi-module builds for the sake of a composed or servicified deployment. both maven and buck are presently competing to draw the most reduced set of compile/test/deploy operations for the changes since last build for a hierarchy of modules. anyway, what is it we are striving for? what we want is to develop cheaply, and to deploy smoothly and often, without defect. we want the ability to deploy without large permanent or temporary headcount overseeing or participating in deployment. aside from development costs, and support/operation, deployment costs are a potentially big factor in total cost of ownership. what i like about cookie-cutter is the uniformity of the deployable things. the team size for deployment of such a thing doesn’t grow with the numbers of nodes that binary is being deployed to. at least, if you’re able to automate the deployment to those nodes, and have a strategy for handling the users connected to the stack at redeployment time somehow (sessions or stateless). the uniformity of the deployment is a cheapener, i think. when you have a number of dissimilar services, you might be able to minimize release personnel if you’re only doing one service. if more than one service is being updated in a particular deployment, you’re going to have to concentrate to make sure you don’t experience a multiplier effect for the participants. it is possible of course, to keep the headcount small, but the practice needed beforehand is bigger, which in turn allows for some calmness around the actual deployment. if we’ve stepped away from the project management office thinking that suggests three buggy releases a year (which is more usual than 18 month schedules of old), then we can employ continuous deployment to further eliminate personnel costs around going live. this is something that microservices does well at, but because the most adept proponents design forwards & backwards compatibility into the permutations most likely to co-exist in production. it is at least much quicker to redeploy and bounce one small service, n times than the the cookie-cutter uniform deployment.

both high availability (ha) and disaster recovery (dr) have been essential it topics. fundamentally ha is about fault tolerance relevant to the availability of an examined subject like application, database, vms, etc. while dr roots on the ability to resume operations in the aftermath of a catastrophic event. a fundamental difference of these two is that ha expects no down time and no data loss, while dr does. they are different issues and should be addressed separately. background for many it shops, either ha or dr has been a high risk and high cost item. both are essential to business continuity, while traditionally tough technical problems to solve with very significant and long-term commitments on resources. not only they are technically challenging, but a continual cost-cutting which has become an it standard practice in the past two decades makes purchasing hardware/software and constructing either ha or dr solution on premises further distant from it’s financial and technical realties. sense of urgency too often, the technical challenges and resource commitments overwhelm it and turn ha and dr into academic discussions, or symbolic items on a project checklist. at the same time, information is rapidly exploding as internet, mobility and social-network are becoming integral in our daily lives and businesses. there are progressively more data to process and store. for many businesses, the needs for ha and dr is urgent for better managing risks. and continual availability and on-demand recoverability of it are becoming increasingly critical. this is the reality, now the good news is that the recent introduction of cloud computing has fundamentally changed how an ha or dr solution can be implemented. microsoft azure is a vivid example of ha and dr solutions with significantly reduced the required financial commitment and involved technical complexities. the traditional approach by establishing redundancy and acquiring a physical dr site with long-term resources and financial commitments is now largely replaced with consumable services which can be configured in minutes by mouse-clicking and with a manageable cost structure based on usage. ha and dr have become it solutions which are financially realistic and technically feasible for businesses in all sizes. ha, redundancy, and microsoft azure lrs ha is to eliminate a single point of failure of an examined component, an application for example. it denotes a strategy to employ redundancy such that a target application can and will continue being available without downtime while experiencing a failure of hosting hardware or software. there are various and well-developed ha solutions like a hyper-v host cluster using redundant hardware to eliminate a single point of failure of hosting os or hardware, and an application cluster for eliminating a single point of failure by running the application in multiple vm instances with a synchronous state. although ha implementations may vary, the fundamental principle nevertheless remains the same. ha expects neither downtime nor data loss while experiencing an outage of a target hardware or software. ha has become dramatically simple in microsoft azure. basically, all data written to disk in microsoft azure are kept at least in the so-called lrs, locally redundant storage. lrs replicates a transaction synchronously to three different storage nodes across fault domains and upgrade domains within the same region for durability. in layman’s terms, microsoft azure by default maintains at least three copies of user data to achieve ha. dr, replication, and microsoft azure grs dr is about having a plan and backups in place to resume operations in the aftermath of a catastrophic event. unplanned outage is assumed in a dr scenario, therefore some data loss is also expected. notice that ha and dr are different business problems and addressed differently. while both ha and dr are based on applying redundancy, i.e. a source and replicas, or multiple identical nodes of an examines component like application instance, databases, or vms, there are however differences between the two. a dr solution generally employs replicas or backups, are implemented with asynchronous processes, and expects an outage of a source and with some data loss in transit while the outage occurs. while ha requires a logical representation with a real-time integrity using synchronous processes across all participating nodes, expects neither downtime nor data loss while experiencing an outage of a participating node. for a critical workload, one approach of dr is to establish geo-replication to address an outage of an entire geographic area caused by a natural disaster, for example. the concern is that a catastrophic event may impact an entire geographic area causing a datacenter where a mission critical application is being hosted becomes unavailable for an extended period of time. in microsoft azure, geo redundant storage or grs is the default and an optional setting, as shown above, while configuring a storage account. grs will queue a transaction committed to lrs as an asynchronous replication to a secondary region, a few hundreds miles away from the primary region where a storage account is originated. at the secondary region, data is also stored in lrs, i.e. made durable by replicating it to three storage nodes. specifically, a microsoft azure storage account configured with grs essentially maintains three replicas locally for high availability, and replicates the content and maintains three replicas at a secondary datacenter a few hundreds miles away for dr. so all are six copies, three locally and three remotely. all these are configured by one, yes one mouse click from a dropdown list while creating a storage account. the above is a conceptual model illustrated a data flow of grs. grs replication has little performance impact on an application since application data are committed to lrs in real-time while replication to grs is queued, i.e. asynchronously. a write to lrs is synchronous and in real-time, once committed, the changes are expected within 15 minutes to be asynchronously replicated to the secondary site. for a ra-grs storage account, in addition to one primary endpoint for read/write operations as it is in a grs, there is also one secondary endpoint as read only becomes available as shown below. the cost implications of grs or ra-grs include the additional storage and the transmission costs for egress traffic, as applicable, of the secondary datacenter. ingress traffic is free . and microsoft azure storage sla offers 99.9% availability and a cost calculator is also available. microsoft azure recovery services so far, much is about backing up or replicating data. to successfully restore, a dr plan must be put in place and ensure its availability upon a dr scenario in progress. either placing a dr plan at a primary site where the source is or a secondary site where a replica stays has some issues and concerns. keeping a dr plan at the source site where all the resources are in place and on-the-job trainings seems logical. or does it? dr is assuming a catastrophic event over an extended geographic areas where the source site is experiencing an outage. in such case, keeping a dr plan in the source site defeats the purpose. maintaining a dr plan at the secondary site is the choice then. in a dr scenario, a recovery site is to be brought on line within a expected period of time according to a dr plan, and having the dr plan right there and then at a recovery site makes all the sense. or does it? this decision introduces a number of requirements including the physical readiness, the timeliness, and the financial implications on securing and maintaining a dr plan at a remote physical facility. for a vmm server running on system center 2012 sp1 or later, an idea, reliable and straightforward way is to use azure recovery services to maintain a dr plan as shown below. and for any backup needs, using cloud as a backup site makes backing up and restoring data an anytime anywhere operation. azure site recovery vault this service essentially acts as the director of a dr process. it orchestrates and manages the protection and failover of vms in clouds managed by virtual machine manager 2012 sp1 or later. a noticeable advantage is the ability to test a recovery configuration, exercise a proactive failover and recovery, and automate recovery in the event of a site outage. the sla of site recovery services is 99.9% availability to ensure a configured dr plan is always in place with expected updates. this is a dr solution that it can implement, simulate, verify, bring online and be absolutely confident with the readiness. azure backup vault this is a reliable, scalable and inexpensive data protection solution with zero capital investment and extremely low operational expense. like other secure communication with microsoft azure, you will first upload a public certificate to microsoft azure. then download the backup agent to register a target server with the backup vault. then select what to be backed up. both microsoft azure backup sla (99.9% availability) and cost calculator are available for better assessing the solution. closing thoughts form an application’s view, ha is an on-going event while dr is an anticipation. ha and dr are different business problems and should be addressed differently. nevertheless, microsoft azure provides a single platform to gracefully address ha with lrs, dr with grs, and dr orchestration with recovery services, and all with published sla s and a predictable cost structure . going forward, it pros can now include ha and dr as a reliable, scalable and relatively inexpensive proposition by employing microsoft azure as a solution platform. call to action register at microsoft virtual academy, http://aka.ms/mva1 , and train yourself on microsoft azure by taking the track of courses. go to http://aka.ms/azure200 and acquire a free trial subscription and assess microsoft azure for ha and dr solutions. review my recommended content at http://aka.ms/recommended .

When comparing two objects, the equals method is used to return true if they are identical. Typically, this leads to the following code : if (name.equals("Jim")) { } The problem here is that whether intended or not, it is quite possible that the name value is null, in which case a null pointer exception would be thrown. A better practice is to execute the equals method of the string constant “Jim” instead : if ("Jim".equals(name)) { } Since the constant is never null, a null exception will not be thrown, and if the other value is null, the equals comparison will fail. If you are using Java 7 or above, the new Objects class has an equals static method to compare two objects while taking null values into account. if (Objects.equals(name,"Jim")) { } Alternatively if you are using a java version prior to Java 7, but using the guava library you can use the Objects class which has a static equal() method that takes two objects and handles null cases for you. It should also be noted that there are probably a number of other implementations in various libraries (i.e. Apache Commons)

When I needed to do prototyping, proof of concept or play with some new technology in free time, starting new project was always a little annoying barrier with Maven. Have to say that setting up Maven project is not hard and you can use Maven Archetypes. But Archetypes are often out of date. Who wants to play with old technologies? So I always end up wiring in dependencies I wanted to play with. Not very productive spent time. But than Spring Boot came to my way. I fell in love. In last few months I created at least 50 small playground projects, prototypes with Spring Boot. Also incorporated it at work. It’s just perfect for prototyping, learning, microservices, web, batch, enterprise, message flow or command line applications. You have to be dinosaur or be blind not to evaluate Spring Boot for your next Spring project. And when you finish evaluate it, you will go for it. I promise. I feel a need to highlight how easy is Black Box Testing of Spring Boot microservice. Black Box Testing refers to testing without any poking with application artifact. Such testing can be called also integration testing. You can also perform performance or stress testing way I am going to demonstrate. Spring Boot Microservice is usually web application with embedded Tomcat. So it is executed as JAR from command line. There is possibility to convert Spring Boot project into WAR artifact, that can be hosted on shared Servlet container. But we don’t want that now. It’s better when microservice has its own little embedded container. I used existing Spring’s REST service guide as testing target. Focus is mostly on testing project, so it is handy to use this “Hello World” REST application as example. I expect these two common tools are set up and installed on your machine: Maven 3 Git So we’ll need to download source code and install JAR artifact into our local repository. I am going to use command line to download and install the microservice. Let’s go to some directory where we download source code. Use these commands: git clone [email protected]:spring-guides/gs-rest-service.git cd gs-rest-service/complete mvn clean install If everything went OK, Spring Boot microservice JAR artifact is now installed in our local Maven repository. In serious Java development, it would be rather installed into shared repository (e.g. Artifactory, Nexus,… ). When our microservice is installed, we can focus on testing project. It is also Maven and Spring Boot based. Black box testing will be achieved by downloading the artifact from Maven repository (doesn’t matter if it is local or remote). Maven-dependency-plugin can help us this way: org.apache.maven.plugins maven-dependency-plugin copy-dependencies compile copy-dependencies gs-rest-service true It downloads microservice artifact into target/dependency directory by default. As you can see, it’s hooked to compile phase of Maven lifecycle, so that downloaded artifact is available during test phase. Artifact version is stripped from version information. We use latest version. It makes usage of JAR artifact easier during testing. Readers skilled with Maven may notice missing plugin version. Spring Boot driven project is inherited from parent Maven project called spring-boot-starter-parent. It contains versions of main Maven plugins. This is one of the Spring Boot’s opinionated aspects. I like it, because it provides stable dependencies matrix. You can change the version if you need. When we have artifact in our file system, we can start testing. We need to be able to execute JAR file from command line. I used standard JavaProcessBuilder this way: public class ProcessExecutor { public Process execute(String jarName) throws IOException { Process p = null; ProcessBuilder pb = new ProcessBuilder("java", "-jar", jarName); pb.directory(new File("target/dependency")); File log = new File("log"); pb.redirectErrorStream(true); pb.redirectOutput(Redirect.appendTo(log)); p = pb.start(); return p; } } This class executes given process JAR based on given file name. Location is hard-coded to target/dependency directory, where maven-dependency-plugin located our artifact. Standard and error outputs are redirected to file. Next class needed for testing is DTO (Data transfer object). It is simple POJO that will be used for deserialization from JSON. I use Lombok project to reduce boilerplate code needed for getters, setters, hashCode and equals. @Data @AllArgsConstructor @NoArgsConstructor public class Greeting { private long id; private String content; } Test itself looks like this: public class BlackBoxTest { private static final String RESOURCE_URL = "http://localhost:8080/greeting"; @Test public void contextLoads() throws InterruptedException, IOException { Process process = null; Greeting actualGreeting = null; try { process = new ProcessExecutor().execute("gs-rest-service.jar"); RestTemplate restTemplate = new RestTemplate(); waitForStart(restTemplate); actualGreeting = restTemplate.getForObject(RESOURCE_URL, Greeting.class); } finally { process.destroyForcibly(); } Assert.assertEquals(new Greeting(2L, "Hello, World!"), actualGreeting); } private void waitForStart(RestTemplate restTemplate) { while (true) { try { Thread.sleep(500); restTemplate.getForObject(RESOURCE_URL, String.class); return; } catch (Throwable throwable) { // ignoring errors } } } } It executes Spring Boot microservice process first and wait unit it starts. To verify if microservice is started, it sends HTTP request to URL where it’s expected. The service is ready for testing after first successful response. Microservice should send simple greeting JSON response for HTTP GET request. Deserialization from JSON into our Greeting DTO is verified at the end of the test. Source code is shared on Github.

This article represents the first Proof of Concept from series described in the previous article 4 Hands-On Approaches to Improve Your Data Access Layer Implementation and it presents how to implement Ehcache over MyBatis, how to achieve an optim configuration for it and personal opinions of the author about the chosen approach for the Data Access Layer. Throughout my research on caching over MyBatis I have discovered that Ehcache is the first option among developers when they need to implement a cache mechanism over MyBatis, using a 3rd party library. Ehcache is probably so popular because it represents an open source, java-based cache, available under an Apache 2 license. Also, it scales from in-process with one or more nodes through to a mixed in-process/out-of-process configuration with terabyte-sized caches. In addition, for those applications needing a coherent distributed cache, Ehcache uses the open source Terracotta Server Array. Last but not least, among its adopters is the Wikimedia Foundation that uses Ehcache to improve the performance of its wiki projects. Within this article, the following aspects will be addressed: 1. How will an application benefit from caching using Ehcache? Ehcache's features will be detailed in this section. 2. Hands-on implementation of the EhCachePOC project - in this section the key concepts of EhCache will be explored through a hands on implementation. 3. Summary - How has the application performance been improved after this implementation? Code of all the projects that will be implemented can be found at https://github.com/ammbra/CacherPoc or if you are interested only in the current implementation, you can access it here: https://github.com/ammbra/CacherPoc/tree/master/EhCachePoc How will an application benefit from caching using Ehcache? The time taken for an application to process a request principally depends on the speed of the CPU and main memory. In order to "speed up" your application you can perform one or more of the following: improve the algorithm performance achieve parallelisation of the computations across multiple CPUs or multiple machines upgrade the CPU speed As explained in the previous article, high availability applications should perform a small amount of actions with the database. Since the time taken to complete a computation depends principally on the rate at which data can be obtained, then the application should be able to temporarily store computations that may be reused again. Caching may be able to reduce the workload required, this means a caching mechanism should be created! Ehcache is described as : Fast and Light Weight , having a simple API and requiring only a dependency on SLF4J. Scalable to hundreds of nodes with the Terracotta Server Array, but also because provides Memory and Disk store for scalability into gigabytes Flexible because supports Object or Serializable caching; also provides LRU, LFU and FIFO cache eviction policies Standards Based having a full implementation of JSR107 JCACHE API Application Persistence Provider because it offers persistent disk store which stores data between VM restarts JMX Enabled Distributed Caching Enabler because it offers clustered caching via Terracotta and replicated caching via RMI, JGroups, or JMS Cache Server (RESTful, SOAP cache Server) Search Compatible, having a standalone and distributed search using a fluent query language Hands-on implementation of the EhCachePOC project The implementation of EhCachePoc will look as described in the diagram below: In order to test Ehcache performance through a POC(proof of concept) project the following project setup is performed: 1. Create a new Maven EJB Project from your IDE (this kind of project is platform provided by NetBeans but for those that use eclipse, here is an usefull tutorial) . In the article this project is named EhCachePOC. 2. Edit the project's pom by adding required jars : org.mybatis mybatis 3.2.6 org.mybatis.caches mybatis-ehcache 1.0.2 log4j log4j 1.2.17 net.sf.ehcache ehcache 2.7.0 org.slf4j slf4j-log4j12 1.7.5 3.Add your database connection driver, in this case apache derby: org.apache.derby derbyclient 10.11.1.1 4. Run mvn clean and mvn install commands on your project. Now the project setup is in place, let's go ahead with MyBatis implementation : 1. Configure under resources/com/tutorial/ehcachepoc/xml folder the Configuration.xml file with : 2. Create in java your own SQLSessionFactory implementation. For example, create something similar to com.tutorial.ehcachepoc.config. SQLSessionFactory : public class SQLSessionFactory { private static final SqlSessionFactory FACTORY; static { try { Reader reader = Resources.getResourceAsReader("com/tutorial/ehcachepoc/xml/Configuration.xml"); FACTORY = new SqlSessionFactoryBuilder().build(reader); } catch (Exception e){ throw new RuntimeException("Fatal Error. Cause: " + e, e); } } public static SqlSessionFactory getSqlSessionFactory() { return FACTORY; } } 3. Create the necessary bean classes, those that will map to your sql results, like Employee: public class Employee implements Serializable { private static final long serialVersionUID = 1L; private Integer id; private String firstName; private String lastName; private String adress; private Date hiringDate; private String sex; private String phone; private int positionId; private int deptId; public Employee() { } public Employee(Integer id) { this.id = id; } @Override public String toString() { return "com.tutorial.ehcachepoc.bean.Employee[ id=" + id + " ]"; } } 4. Create the IEmployeeDAO interface that will expose the ejb implementation when injected: public interface IEmployeeDAO { public List getEmployees(); } 5. Implement the above inteface and expose the implementation as a Stateless EJB (this kind of EJB preserves only its state, but there is no need to preserve its associated client state): @Stateless(name = "ehcacheDAO") @TransactionManagement(TransactionManagementType.CONTAINER) public class EmployeeDAO implements IEmployeeDAO { private static Logger logger = Logger.getLogger(EmployeeDAO.class); private SqlSessionFactory sqlSessionFactory; @PostConstruct public void init() { sqlSessionFactory = SQLSessionFactory.getSqlSessionFactory(); } @Override public List getEmployees() { logger.info("Getting employees....."); SqlSession sqlSession = sqlSessionFactory.openSession(); List results = sqlSession.selectList("retrieveEmployees"); sqlSession.close(); return results; } } 5. Create the EmployeeMapper.xml that contains the query named "retrieveEmployees" select id, first_name, last_name, hiring_date, sex, dept_id from employee If you remember the CacherPOC setup from the previously article, then you can test your implementation if you add EhCachePOC project as dependency and inject the IEmployeeDAO inside the EhCacheServlet. Your CacherPOC pom.xml file should contain : ${project.groupId} EhCachePoc ${project.version} and your servlet should look like: @WebServlet("/EhCacheServlet") public class EhCacheServlet extends HttpServlet { private static Logger logger = Logger.getLogger(EhCacheServlet.class); @EJB(beanName ="ehcacheDAO") IEmployeeDAO employeeDAO; private static final String LIST_USER = "/listEmployee.jsp"; @Override protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException { String forward= LIST_USER; List results = new ArrayList(); for (int i = 0; i < 10; i++) { for (Employee emp : employeeDAO.getEmployees()) { logger.debug(emp); results.add(emp); } try { Thread.sleep(3000); } catch (Exception e) { logger.error(e, e); } } req.setAttribute("employees", results); RequestDispatcher view = req.getRequestDispatcher(forward); view.forward(req, resp); } } Run your CacherPoc implementation to check if your Data Access Layer with MyBatis is working or download the code provided at https://github.com/ammbra/CacherPoc But if a great amount of employees is stored in database, or perhaps the retrieval of a number of 10xemployeesNo represents a lot of workload for the database. Also, can be noticed that the query from the EmployeeMapper.xml retrieves data that almost never changes (id, first_name, last_name, hiring_date, sex cannot change; the only value that might change in time is dept_id); so a caching mechanism can be used. Below is described how this can be achieved using EhCache: 1. Configure directly under the resources folder the ehcache.xml file with: This xml explains that the Memory Store is used for an LRU (Last Recently Used) caching strategy, sets the limits for the number of elements allowed for storage, their time to be idle and their time to live. The Memory Store strategy is often chosen because is fast and thread safe for use by multiple concurrent threads, being backed by LinkedHashMap. Also, all elements involved in the caching process are suitable for placement in the Memory Store. Another approach can be tried: storing cache on disk. This can be done by replacing the ehcache tag content with: diskStore path="F:\\cache" /> Unlike the memory store strategy, the disk store implementation is suitable only for elements which are serializable can be placed in the off-heap; if any non serializable elements are encountered, those will be removed and WARNING level log message emitted. The eviction is made using the LFU algorithm and it is not configurable or changeable. From persistency point of view, this method of caching allows control of the cache by the disk persistent configuration; if false or omitted, disk store will not persist between CacheManager restarts. 2. Update EmployeeMapper.xml to use the previous implemented caching strategy: select id, first_name, last_name, hiring_date, sex, dept_id from employee By adding the line and specifying on the query useCache="true" you are binding the ehcache.xml configuration to your DataAccessLayer implementation. Clean, build and redeploy both EhCachePOC and CacherPoc projects; now retrieve your employees for two times in order to allow the in-memory cache to store your values. When you run your query for the first time, your application will execute the query on the database and retrieve the results. Second time you access the employee list, your application will access the in-memory storage. Summary - How has the application performance been improved after this implementation? An application's performances depend on a multitude of factors how many times a cached piece of data can and is reduced by the application the proportion of the response time that is alleviated by caching Amdhal's law can be used to estimate the system's speed up : where P is proportion speed up and S is speed up. Let's take the application from this article as example and calculate the speed up. When the application ran the query without caching,a JDBC transaction is performed and in your log will be something similar to : INFO: 2014-11-27 18:01:30,020 [EmployeeDAO] INFO com.tutorial.hazelcastpoc.dao.EmployeeDAO:38 - Getting employees..... INFO: 2014-11-27 18:01:39,148 [JdbcTransaction] DEBUG org.apache.ibatis.transaction.jdbc.JdbcTransaction:98 - Setting autocommit to false on JDBC Connection [org.apache.derby.client.net.NetConnection40@1c374fd] INFO: 2014-11-27 18:01:39,159 [retrieveEmployees] DEBUG com.tutorial.hazelcastpoc.mapper.EmployeeMapper.retrieveEmployees:139 - ==> Preparing: select id, first_name, last_name, hiring_date, sex, dept_id from employee INFO: 2014-11-27 18:01:39,220 [retrieveEmployees] DEBUG com.tutorial.hazelcastpoc.mapper.EmployeeMapper.retrieveEmployees:139 - ==> Parameters: INFO: 2014-11-27 18:01:39,316 [retrieveEmployees] DEBUG com.tutorial.hazelcastpoc.mapper.EmployeeMapper.retrieveEmployees:139 - <== Total: 13 while running the queries with Ehcache caching the JDBC transaction is performed only once (to initialize the cache) and after that the log will look like : INFO: 2014-11-28 18:04:50,020 [EmployeeDAO] INFO com.tutorial.ehcachepoc.dao.EmployeeDAO:38 - Getting employees..... INFO: 2014-11-28 18:04:50,020 [EhCacheServlet] DEBUG com.tutorial.cacherpoc.EhCacheServlet:41 - com.tutorial.crudwithjsp.model.Employee[ id=1 ] Let's look at the time that each of our 10 times requests has scored: the first not cached version of 10 times requests took about 57 seconds and 51 milliseconds, while the cached requests scored a time of 27seconds and 86 miliseconds. In order to apply Amdhal's law for the system the following input is needed: Un-cached page time: 60 seconds Database time : 58 seconds Cache retrieval time: 28seconds Proportion: 96.6% (58/60) (P) The expected system speedup is thus: 1 / (( 1 – 0.966) + 0.966 / (58/28)) = 1 / (0.034 + 0. 966/2.07) = 2 times system speedup This result can be improved of course, but the purpose of this article was to prove that caching using Ehcache over MyBatis offers a significant improvement to what used to be available before its implementation. Learn more from: MyBatis Documentation MyBatis Ehcache Adapter EhCache website

It has been a while since my first post about the ESP8266 (see “Cheap and Simple WiFi with ESP8266 for the FRDM Board“). The ESP8266 is a new inexpensive ($4.50) WiFi module which makes it easy to connect to the network or internet. Finally this week-end I have found the time to write up a tutorial: how to implement a WiFi web server for the ESP8266 WiFi module and the Freescale FRDM-KL25Z board: WSP8266 Web Server FRDM-KL25Z with ESP8266 WiFi Module Outline In this tutorial I’m using a Freescale FRDM-KL25Z board as a web server, using theESP8266 board. The ESP8266 is a ‘less than $4.5′ WiFi board getting more and more popular as an IoT board. There is even a way to run the ESP8266 standalone (because it has a full processor on that board). However, that development is still in the flux and rather unstable. Instead, I’m using a serial connection to the ESP8266 instead. With this, any small microcontroller can send and receive data from the internet: connect that board to a microcontroller with 3.3V, GND, Tx and Rx, and you have a W-LAN connection! I’m using in this tutorial Eclipse with GNU/GDB with Processor Expert, but with the steps in this tutorial you should be able to use any other toolchain too. As things might change in the future with different firmware on the ESP8266: the firmware I’m having on the board is version 00160901. Board Connections Since my first post on the ESP8266 I have cleaned up the wiring. The pins are as below for the ESP8266: ESP8266 Pins Because the ESP8266 can take > 200 mA, I’m using a 5-to-3.3V DC-DC converter. I measured around 70 to 90 mA, so it is not (yet) really needed, but I wanted to use it to protect to board. The ESP8266 Rx and Tx are connected to the microcontroller Tx and Rx pins. A general frustration point for the ESP8266 module is the connection oft the remaining pins. What worked for me is to connect CH_PD to 3.3V and leaving RST,GPIO0 and GPIO2 unconnected/floating. Wiring Setup with FRDM-KL25Z and ESP8266 Communication Protocol I recommend to use a logic analyzer to verify the communication between the ESP8266 and the microcontroller. My module communicates with 115200, but I see reports that other modules (other firmware) can use a different baud. The module uses an AT command send. The simplest command is to send “AT\r\n” and it responds with “AT\r\r\n\r\nOK\r\n”: AT Command Sent to ESP8266 In this tutorial I’m using a command line shell (see “A Shell for the Freedom KL25Z Board“) to have a manual mode to send commands to the module. More about this later. Project Creation You can use my project and source files available on GitHub (see link at the end of this article). Or create your own project. My project is using the Kinetis Design Studio and for the FRDM-KL25Z board (MKL25Z128VLK4). I have created a project for Processor Expert, as I’m using several components of it: Processor Expert Project For the project I have several files added: ESP8266 Project in Eclipse With the following source files: Application.c/.h: This runs the application and web server program ESP8266.c/.h: Driver for the ESP8266 Events.c/.h: Processor Expert event hooks main.c: main entry point Shell.c/.h: command line interface Sources Project and Source files are available on GitHub here: https://github.com/ErichStyger/mcuoneclipse/tree/master/Examples/KDS/FRDM-KL25Z/FRDM-KL25Z_ESP8266 Please check the latest source files on GitHub. At the time of writing this article, I’m using the following: Shell.h is the interface to command line shell: view source print? 01./* 02. * Shell.h 03. * 04. * Author: Erich Styger 05. */ 06. 07.#ifndef SHELL_H_ 08.#define SHELL_H_ 09. 10./*! 11. * \brief Shell parse routine 12. */ 13.voidSHELL_Parse(void); 14. 15./*! 16. * \brief Shell initialization 17. */ 18.voidSHELL_Init(void); 19. 20.#endif /* SHELL_H_ */ Shell.c implements the application part of the shell: view source print? 01./* 02. * Shell.c 03. * 04. * Author: Erich Styger 05. */ 06. 07.#include "Shell.h" 08.#include "CLS1.h" 09.#include "ESP8266.h" 10. 11./* table with shell parser/handler */ 12.staticconstCLS1_ParseCommandCallback CmdParserTable[] = 13.{ 14. CLS1_ParseCommand, 15. ESP_ParseCommand, 16. NULL /* sentinel */ 17.}; 18. 19.staticunsigned charlocalConsole_buf[48]; /* buffer for command line */ 20. 21.voidSHELL_Parse(void) { 22. (void)CLS1_ReadAndParseWithCommandTable(localConsole_buf, sizeof(localConsole_buf), CLS1_GetStdio(), CmdParserTable); 23.} 24. 25.voidSHELL_Init(void) { 26. localConsole_buf[0] = '\0'; /* initialize buffer */ 27.} ESP8266.h is the interface to the WiFi module: view source print? 001./* 002. * ESP8266.h 003. * 004. * Author: Erich Styger 005. */ 006. 007.#ifndef ESP8266_H_ 008.#define ESP8266_H_ 009. 010.#include "CLS1.h" 011. 012.#define ESP_DEFAULT_TIMEOUT_MS (100) 013. /*!< Default timeout value in milliseconds */ 014. 015./*! 016. * \brief Command line parser routine 017. * \param cmd Pointer to command line string 018. * \param handled Return value if command has been handled 019. * \param io Standard Shell I/O handler 020. * \return Error code, ERR_OK for no failure 021. */ 022.uint8_t ESP_ParseCommand(constunsigned char*cmd, bool *handled, constCLS1_StdIOType *io); 023. 024./*! 025. * \brief Send a string to th ESP8266 module 026. * \param str String to send, "\r\n" will be appended 027. * \param io Shell I/O handler or NULL if not used 028. * \return Error code, ERR_OK for no failure 029. */ 030.uint8_t ESP_SendStr(constuint8_t *str, CLS1_ConstStdIOType *io); 031. 032./*! 033. * \brief Used to send an AT command to the ESP8266 module 034. * \param cmd Command string to send 035. * \param rxBuf Buffer for the response, can be NULL 036. * \param rxBufSize Size of response buffer 037. * \param expectedTailStr Expected response from the module, can be NULL 038. * \param msTimeout Timeout time in milliseconds 039. * \param io Shell I/O handler or NULL if not used 040. * \return Error code, ERR_OK for no failure 041. */ 042.uint8_t ESP_SendATCommand(uint8_t *cmd, uint8_t *rxBuf, size_t rxBufSize, uint8_t *expectedTailStr, uint16_t msTimeout, constCLS1_StdIOType *io); 043. 044./*! 045. * \brief Read from the serial line from the module until a sentinel char is received 046. * \param buf 047. * \param bufSize 048. * \param sentinelChar 049. * \param timeoutMs Timeout time in milliseconds 050. * \return Error code, ERR_OK for no failure 051. */ 052.uint8_t ESP_ReadCharsUntil(uint8_t *buf, size_t bufSize, uint8_t sentinelChar, uint16_t timeoutMs); 053. 054./*! 055. * \brief Sends an AT command to test the connection 056. * \return Error code, ERR_OK for no failure 057. */ 058.uint8_t ESP_TestAT(void); 059. 060./*! 061. * \brief Restarts the ESP8266 module 062. * \param io Shell I/O handler or NULL if not used 063. * \param timeoutMs Timeout time in milliseconds 064. * \return Error code, ERR_OK for no failure 065. */ 066.uint8_t ESP_Restart(constCLS1_StdIOType *io, uint16_t timeoutMs); 067. 068./*! 069. * \brief Set the current mode of the module 070. * \param mode Where is 1=Sta, 2=AP or 3=both 071. * \return Error code, ERR_OK for no failure 072. */ 073.uint8_t ESP_SelectMode(uint8_t mode); 074. 075./*! 076. * \Brief returns the firmware version string 077. * \param fwBuf Buffer for the string 078. * \param fwBufSize Size of buffer in bytes 079. * \return Error code, ERR_OK for no failure 080. */ 081.uint8_t ESP_GetFirmwareVersionString(uint8_t *fwBuf, size_t fwBufSize); 082. 083./*! 084. * \brief Join an access point. 085. * \param ssid SSID of access point 086. * \param pwd Password of access point 087. * \param nofRetries Number of connection retries 088. * \param io Shell I/O or NULL if not used 089. * \return Error code, ERR_OK for no failure 090. */ 091.uint8_t ESP_JoinAP(constuint8_t *ssid, constuint8_t *pwd, intnofRetries, CLS1_ConstStdIOType *io); 092. 093./*! 094. * \brief Scans for an IPD message sent by the module 095. * \param msgBuf Pointer to the message buffer where to store the message 096. * \param msgBufSize Size of message buffer 097. * \param ch_id Pointer to where to store the channel/id 098. * \param size Pointer where to store the size of the message 099. * \param isGet TRUE if it is a GET message, FALSE for a POST message 100. * \param timeoutMs Error code, ERR_OK for no failure 101. * \param io 102. * \return Error code, ERR_OK for no failure 103. */ 104.uint8_t ESP_GetIPD(uint8_t *msgBuf, size_t msgBufSize, uint8_t *ch_id, uint16_t *size, bool *isGet, uint16_t timeoutMs, constCLS1_StdIOType *io); 105. 106./*! 107. * \brief Closes a connection 108. * \param channel Channel ID 109. * \param io Error code, ERR_OK for no failure 110. * \param timeoutMs Error code, ERR_OK for no failure 111. * \return Error code, ERR_OK for no failure 112. */ 113.uint8_t ESP_CloseConnection(uint8_t channel, constCLS1_StdIOType *io, uint16_t timeoutMs); 114. 115./*! 116. * \brief Used to determine if the web server is running or not. 117. * \return TRUE if web server has beens started 118. */ 119.bool ESP_IsServerOn(void); 120. 121./*! 122. * \brief Driver initialization 123. */ 124.voidESP_Init(void); 125. 126./*! 127. * \brief Driver de-initialization 128. */ 129.voidESP_Deinit(void); 130. 131.#endif /* ESP8266_H_ */ And the ESP8266 driver is in ESP8266.c which implements all the low level SPI access functions, the functional implementation and a command line shell interface: view source print? 001./* 002. * ESP8266.c 003. * 004. * Author: Erich Styger 005. */ 006. 007.#include "ESP8266.h" 008.#include "Shell.h" 009.#include "UTIL1.h" 010.#include "CLS1.h" 011.#include "AS2.h" 012.#include "WAIT1.h" 013. 014.staticbool ESP_WebServerIsOn = FALSE; 015. 016.bool ESP_IsServerOn(void) { 017. returnESP_WebServerIsOn; 018.} 019. 020.staticvoidSend(unsigned char*str) { 021. while(*str!='\0') { 022. AS2_SendChar(*str); 023. str++; 024. } 025.} 026. 027.staticvoidSkipNewLines(constunsigned char**p) { 028. while(**p=='\n'|| **p=='\r') { 029. (*p)++; /* skip new lines */ 030. } 031.} 032. 033.uint8_t ESP_ReadCharsUntil(uint8_t *buf, size_t bufSize, uint8_t sentinelChar, uint16_t timeoutMs) { 034. uint8_t ch; 035. uint8_t res = ERR_OK; 036. 037. if(bufSize<=1) { 038. returnERR_OVERRUN; /* buffer to small */ 039. } 040. buf[0] = '\0'; buf[bufSize-1] = '\0'; /* always terminate */ 041. bufSize--; 042. for(;;) { /* breaks */ 043. if(bufSize==0) { 044. res = ERR_OVERRUN; 045. break; 046. } 047. if(AS2_GetCharsInRxBuf()>0) { 048. (void)AS2_RecvChar(&ch); 049. *buf = ch; 050. buf++; 051. bufSize--; 052. if(ch==sentinelChar) { 053. *buf = '\0'; /* terminate string */ 054. break; /* sentinel found */ 055. } 056. } else{ 057. if(timeoutMs>10) { 058. WAIT1_WaitOSms(5); 059. timeoutMs -= 5; 060. } else{ 061. res = ERR_NOTAVAIL; /* timeout */ 062. break; 063. } 064. } 065. } 066. returnres; 067.} 068. 069.staticuint8_t RxResponse(unsigned char*rxBuf, size_t rxBufLength, unsigned char*expectedTail, uint16_t msTimeout) { 070. unsigned charch; 071. uint8_t res = ERR_OK; 072. unsigned char*p; 073. 074. if(rxBufLength < sizeof("x\r\n")) { 075. returnERR_OVERFLOW; /* not enough space in buffer */ 076. } 077. p = rxBuf; 078. p[0] = '\0'; 079. for(;;) { /* breaks */ 080. if(msTimeout == 0) { 081. break; /* will decide outside of loop if it is a timeout. */ 082. } elseif(rxBufLength == 0) { 083. res = ERR_OVERFLOW; /* not enough space in buffer */ 084. break; 085. } elseif(AS2_GetCharsInRxBuf() > 0) { 086.#if0 087. if(AS2_RecvChar(&ch) != ERR_OK) { 088. res = ERR_RXEMPTY; 089. break; 090. } 091.#else 092. /* might get an overrun OVERRUN_ERR error here? Ignoring error for now */ 093. (void)AS2_RecvChar(&ch); 094.#endif 095. *p++ = ch; 096. *p = '\0'; /* always terminate */ 097. rxBufLength--; 098. } elseif(expectedTail!=NULL && expectedTail[0]!='\0' 099. && UTIL1_strtailcmp(rxBuf, expectedTail) == 0) { 100. break; /* finished */ 101. } else{ 102. WAIT1_WaitOSms(1); 103. msTimeout--; 104. } 105. } /* for */ 106. if(msTimeout==0) { /* timeout! */ 107. if(expectedTail[0] != '\0'/* timeout, and we expected something: an error for sure */ 108. || rxBuf[0] == '\0'/* timeout, did not know what to expect, but received nothing? There has to be a response. */ 109. ) 110. { 111. res = ERR_FAULT; 112. } 113. } 114. returnres; 115.} 116. 117.uint8_t ESP_SendATCommand(uint8_t *cmd, uint8_t *rxBuf, size_t rxBufSize, uint8_t *expectedTailStr, uint16_t msTimeout, constCLS1_StdIOType *io) { 118. uint16_t snt; 119. uint8_t res; 120. 121. if(rxBuf!=NULL) { 122. rxBuf[0] = '\0'; 123. } 124. if(io!=NULL) { 125. CLS1_SendStr("sending>>:\r\n", io->stdOut); 126. CLS1_SendStr(cmd, io->stdOut); 127. } 128. if(AS2_SendBlock(cmd, (uint16_t)UTIL1_strlen((char*)cmd), &snt) != ERR_OK) { 129. returnERR_FAILED; 130. } 131. if(rxBuf!=NULL) { 132. res = RxResponse(rxBuf, rxBufSize, expectedTailStr, msTimeout); 133. if(io!=NULL) { 134. CLS1_SendStr("received<<:\r\n", io->stdOut); 135. CLS1_SendStr(rxBuf, io->stdOut); 136. } 137. } 138. returnres; 139.} 140. 141.uint8_t ESP_TestAT(void) { 142. /* AT */ 143. uint8_t rxBuf[sizeof("AT\r\r\n\r\nOK\r\n")]; 144. uint8_t res; 145. 146. res = ESP_SendATCommand("AT\r\n", rxBuf, sizeof(rxBuf), "AT\r\r\n\r\nOK\r\n", ESP_DEFAULT_TIMEOUT_MS, NULL); 147. returnres; 148.} 149. 150.uint8_t ESP_Restart(constCLS1_StdIOType *io, uint16_t timeoutMs) { 151. /* AT+RST */ 152. uint8_t rxBuf[sizeof("AT+RST\r\r\n\r\nOK\r\n")]; 153. uint8_t res; 154. uint8_t buf[64]; 155. 156. AS2_ClearRxBuf(); /* clear buffer */ 157. res = ESP_SendATCommand("AT+RST\r\n", rxBuf, sizeof(rxBuf), "AT+RST\r\r\n\r\nOK\r\n", ESP_DEFAULT_TIMEOUT_MS, io); 158. if(res==ERR_OK) { 159. for(;;) { 160. ESP_ReadCharsUntil(buf, sizeof(buf), '\n', 1000); 161. if(io!=NULL) { 162. CLS1_SendStr(buf, io->stdOut); 163. } 164. if(UTIL1_strncmp(buf, "ready", sizeof("ready")-1)==0) { /* wait until ready message from module */ 165. break; /* module has restarted */ 166. } 167. } 168. } 169. AS2_ClearRxBuf(); /* clear buffer */ 170. returnres; 171.} 172. 173.uint8_t ESP_CloseConnection(uint8_t channel, constCLS1_StdIOType *io, uint16_t timeoutMs) { 174. /* AT+CIPCLOSE= */ 175. uint8_t res; 176. uint8_t cmd[64]; 177. 178. UTIL1_strcpy(cmd, sizeof(cmd), "AT+CIPCLOSE="); 179. UTIL1_strcatNum8u(cmd, sizeof(cmd), channel); 180. UTIL1_strcat(cmd, sizeof(cmd), "\r\n"); 181. res = ESP_SendATCommand(cmd, NULL, 0, "Unlink\r\n", timeoutMs, io); 182. returnres; 183.} 184. 185.uint8_t ESP_SetNumberOfConnections(uint8_t nof, constCLS1_StdIOType *io, uint16_t timeoutMs) { 186. /* AT+CIPMUX=, 0: single connection, 1: multiple connections */ 187. uint8_t res; 188. uint8_t cmd[sizeof("AT+CIPMUX=12\r\n")]; 189. uint8_t rxBuf[sizeof("AT+CIPMUX=12\r\n\r\nOK\r\n")+10]; 190. 191. if(nof>1) { /* only 0 and 1 allowed */ 192. if(io!=NULL) { 193. CLS1_SendStr("Wrong number of connection parameter!\r\n", io->stdErr); 194. } 195. returnERR_FAILED; 196. } 197. UTIL1_strcpy(cmd, sizeof(cmd), "AT+CIPMUX="); 198. UTIL1_strcatNum8u(cmd, sizeof(cmd), nof); 199. UTIL1_strcat(cmd, sizeof(cmd), "\r\n"); 200. res = ESP_SendATCommand(cmd, rxBuf, sizeof(rxBuf), "OK\r\n", timeoutMs, io); 201. returnres; 202.} 203. 204.uint8_t ESP_SetServer(bool startIt, uint16_t port, constCLS1_StdIOType *io, uint16_t timeoutMs) { 205. /* AT+CIPSERVER=,, where : 0: stop, 1: start */ 206. uint8_t res; 207. uint8_t cmd[sizeof("AT+CIPSERVER=1,80\r\n\r\nOK\r\n")+sizeof("no change")]; 208. uint8_t rxBuf[sizeof("AT+CIPSERVER=1,80\r\n\r\nOK\r\n")+sizeof("no change")]; 209. 210. UTIL1_strcpy(cmd, sizeof(cmd), "AT+CIPSERVER="); 211. if(startIt) { 212. UTIL1_strcat(cmd, sizeof(cmd), "1,"); 213. } else{ 214. UTIL1_strcat(cmd, sizeof(cmd), "0,"); 215. } 216. UTIL1_strcatNum16u(cmd, sizeof(cmd), port); 217. UTIL1_strcat(cmd, sizeof(cmd), "\r\n"); 218. res = ESP_SendATCommand(cmd, rxBuf, sizeof(rxBuf), "OK\r\n", timeoutMs, io); 219. if(res!=ERR_OK) { /* accept "no change" too */ 220. UTIL1_strcpy(cmd, sizeof(cmd), "AT+CIPSERVER="); 221. if(startIt) { 222. UTIL1_strcat(cmd, sizeof(cmd), "1,"); 223. } else{ 224. UTIL1_strcat(cmd, sizeof(cmd), "0,"); 225. } 226. UTIL1_strcatNum16u(cmd, sizeof(cmd), port); 227. UTIL1_strcat(cmd, sizeof(cmd), "\r\r\nno change\r\n"); 228. if(UTIL1_strcmp(rxBuf, cmd)==0) { 229. res = ERR_OK; 230. } 231. } 232. returnres; 233.} 234. 235.uint8_t ESP_SelectMode(uint8_t mode) { 236. /* AT+CWMODE=, where is 1=Sta, 2=AP or 3=both */ 237. uint8_t txBuf[sizeof("AT+CWMODE=x\r\n")]; 238. uint8_t rxBuf[sizeof("AT+CWMODE=x\r\r\nno change\r\n")]; 239. uint8_t expected[sizeof("AT+CWMODE=x\r\r\nno change\r\n")]; 240. uint8_t res; 241. 242. if(mode<1|| mode>3) { 243. returnERR_RANGE; /* only 1, 2 or 3 */ 244. } 245. UTIL1_strcpy(txBuf, sizeof(txBuf), "AT+CWMODE="); 246. UTIL1_strcatNum16u(txBuf, sizeof(txBuf), mode); 247. UTIL1_strcat(txBuf, sizeof(txBuf), "\r\n"); 248. UTIL1_strcpy(expected, sizeof(expected), "AT+CWMODE="); 249. UTIL1_strcatNum16u(expected, sizeof(expected), mode); 250. UTIL1_strcat(expected, sizeof(expected), "\r\r\n\n"); 251. res = ESP_SendATCommand(txBuf, rxBuf, sizeof(rxBuf), expected, ESP_DEFAULT_TIMEOUT_MS, NULL); 252. if(res!=ERR_OK) { 253. /* answer could be as well "AT+CWMODE=x\r\r\nno change\r\n"!! */ 254. UTIL1_strcpy(txBuf, sizeof(txBuf), "AT+CWMODE="); 255. UTIL1_strcatNum16u(txBuf, sizeof(txBuf), mode); 256. UTIL1_strcat(txBuf, sizeof(txBuf), "\r\n"); 257. UTIL1_strcpy(expected, sizeof(expected), "AT+CWMODE="); 258. UTIL1_strcatNum16u(expected, sizeof(expected), mode); 259. UTIL1_strcat(expected, sizeof(expected), "\r\r\nno change\r\n"); 260. if(UTIL1_strcmp(rxBuf, expected)==0) { 261. res = ERR_OK; 262. } 263. } 264. returnres; 265.} 266. 267.uint8_t ESP_GetFirmwareVersionString(uint8_t *fwBuf, size_t fwBufSize) { 268. /* AT+GMR */ 269. uint8_t rxBuf[32]; 270. uint8_t res; 271. constunsigned char*p; 272. 273. res = ESP_SendATCommand("AT+GMR\r\n", rxBuf, sizeof(rxBuf), "\r\n\r\nOK\r\n", ESP_DEFAULT_TIMEOUT_MS, NULL); 274. if(res!=ERR_OK) { 275. if(UTIL1_strtailcmp(rxBuf, "\r\n\r\nOK\r\n")) { 276. res = ERR_OK; 277. } 278. } 279. if(res==ERR_OK) { 280. if(UTIL1_strncmp(rxBuf, "AT+GMR\r\r\n", sizeof("AT+GMR\r\r\n")-1)==0) { /* check for beginning of response */ 281. UTIL1_strCutTail(rxBuf, "\r\n\r\nOK\r\n"); /* cut tailing response */ 282. p = rxBuf+sizeof("AT+GMR\r\r\n")-1; /* skip beginning */ 283. UTIL1_strcpy(fwBuf, fwBufSize, p); /* copy firmware information string */ 284. } else{ 285. res = ERR_FAILED; 286. } 287. } 288. if(res!=ERR_OK) { 289. UTIL1_strcpy(fwBuf, fwBufSize, "ERROR"); /* default error */ 290. } 291. returnres; 292.} 293. 294.uint8_t ESP_GetIPAddrString(uint8_t *ipBuf, size_t ipBufSize) { 295. /* AT+CIFSR */ 296. uint8_t rxBuf[32]; 297. uint8_t res; 298. constunsigned char*p; 299. 300. res = ESP_SendATCommand("AT+CIFSR\r\n", rxBuf, sizeof(rxBuf), NULL, ESP_DEFAULT_TIMEOUT_MS, NULL); 301. if(res!=ERR_OK) { 302. if(UTIL1_strtailcmp(rxBuf, "\r\n")) { 303. res = ERR_OK; 304. } 305. } 306. if(res==ERR_OK) { 307. if(UTIL1_strncmp(rxBuf, "AT+CIFSR\r\r\n", sizeof("AT+CIFSR\r\r\n")-1)==0) { /* check for beginning of response */ 308. UTIL1_strCutTail(rxBuf, "\r\n"); /* cut tailing response */ 309. p = rxBuf+sizeof("AT+CIFSR\r\r\n")-1; /* skip beginning */ 310. SkipNewLines(&p); 311. UTIL1_strcpy(ipBuf, ipBufSize, p); /* copy IP information string */ 312. } else{ 313. res = ERR_FAILED; 314. } 315. } 316. if(res!=ERR_OK) { 317. UTIL1_strcpy(ipBuf, ipBufSize, "ERROR"); 318. } 319. returnres; 320.} 321. 322.uint8_t ESP_GetModeString(uint8_t *buf, size_t bufSize) { 323. /* AT+CWMODE? */ 324. uint8_t rxBuf[32]; 325. uint8_t res; 326. constunsigned char*p; 327. 328. res = ESP_SendATCommand("AT+CWMODE?\r\n", rxBuf, sizeof(rxBuf), "\r\n\r\nOK\r\n", ESP_DEFAULT_TIMEOUT_MS, NULL); 329. if(res==ERR_OK) { 330. if(UTIL1_strncmp(rxBuf, "AT+CWMODE?\r\r\n+CWMODE:", sizeof("AT+CWMODE?\r\r\n+CWMODE:")-1)==0) { /* check for beginning of response */ 331. UTIL1_strCutTail(rxBuf, "\r\n\r\nOK\r\n"); /* cut tailing response */ 332. p = rxBuf+sizeof("AT+CWMODE?\r\r\n+CWMODE:")-1; /* skip beginning */ 333. UTIL1_strcpy(buf, bufSize, p); /* copy information string */ 334. } else{ 335. res = ERR_FAILED; 336. } 337. } 338. if(res!=ERR_OK) { 339. UTIL1_strcpy(buf, bufSize, "ERROR"); 340. } 341. returnres; 342.} 343. 344.uint8_t ESP_GetCIPMUXString(uint8_t *cipmuxBuf, size_t cipmuxBufSize) { 345. /* AT+CIPMUX? */ 346. uint8_t rxBuf[32]; 347. uint8_t res; 348. constunsigned char*p; 349. 350. res = ESP_SendATCommand("AT+CIPMUX?\r\n", rxBuf, sizeof(rxBuf), "\r\n\r\nOK\r\n", ESP_DEFAULT_TIMEOUT_MS, NULL); 351. if(res==ERR_OK) { 352. if(UTIL1_strncmp(rxBuf, "AT+CIPMUX?\r\r\n+CIPMUX:", sizeof("AT+CIPMUX?\r\r\n+CIPMUX:")-1)==0) { /* check for beginning of response */ 353. UTIL1_strCutTail(rxBuf, "\r\n\r\nOK\r\n"); /* cut tailing response */ 354. p = rxBuf+sizeof("AT+CIPMUX?\r\r\n+CIPMUX:")-1; /* skip beginning */ 355. UTIL1_strcpy(cipmuxBuf, cipmuxBufSize, p); /* copy IP information string */ 356. } else{ 357. res = ERR_FAILED; 358. } 359. } 360. if(res!=ERR_OK) { 361. UTIL1_strcpy(cipmuxBuf, cipmuxBufSize, "ERROR"); 362. } 363. returnres; 364.} 365. 366.uint8_t ESP_GetConnectedAPString(uint8_t *apBuf, size_t apBufSize) { 367. /* AT+CWJAP? */ 368. uint8_t rxBuf[48]; 369. uint8_t res; 370. constunsigned char*p; 371. 372. res = ESP_SendATCommand("AT+CWJAP?\r\n", rxBuf, sizeof(rxBuf), "\r\n\r\nOK\r\n", ESP_DEFAULT_TIMEOUT_MS, NULL); 373. if(res==ERR_OK) { 374. if(UTIL1_strncmp(rxBuf, "AT+CWJAP?\r\r\n+CWJAP:\"", sizeof("AT+CWJAP?\r\r\n+CWJAP:\"")-1)==0) { /* check for beginning of response */ 375. UTIL1_strCutTail(rxBuf, "\"\r\n\r\nOK\r\n"); /* cut tailing response */ 376. p = rxBuf+sizeof("AT+CWJAP?\r\r\n+CWJAP:\"")-1; /* skip beginning */ 377. UTIL1_strcpy(apBuf, apBufSize, p); /* copy IP information string */ 378. } else{ 379. res = ERR_FAILED; 380. } 381. } 382. if(res!=ERR_OK) { 383. UTIL1_strcpy(apBuf, apBufSize, "ERROR"); 384. } 385. returnres; 386. 387.} 388. 389.staticuint8_t JoinAccessPoint(constuint8_t *ssid, constuint8_t *pwd, CLS1_ConstStdIOType *io) { 390. /* AT+CWJAP="","" */ 391. uint8_t txBuf[48]; 392. uint8_t rxBuf[64]; 393. uint8_t expected[48]; 394. 395. UTIL1_strcpy(txBuf, sizeof(txBuf), "AT+CWJAP=\""); 396. UTIL1_strcat(txBuf, sizeof(txBuf), ssid); 397. UTIL1_strcat(txBuf, sizeof(txBuf), "\",\""); 398. UTIL1_strcat(txBuf, sizeof(txBuf), pwd); 399. UTIL1_strcat(txBuf, sizeof(txBuf), "\"\r\n"); 400. 401. UTIL1_strcpy(expected, sizeof(expected), "AT+CWJAP=\""); 402. UTIL1_strcat(expected, sizeof(expected), ssid); 403. UTIL1_strcat(expected, sizeof(expected), "\",\""); 404. UTIL1_strcat(expected, sizeof(expected), pwd); 405. UTIL1_strcat(expected, sizeof(expected), "\"\r\r\n\r\nOK\r\n"); 406. 407. returnESP_SendATCommand(txBuf, rxBuf, sizeof(rxBuf), expected, ESP_DEFAULT_TIMEOUT_MS, io); 408.} 409. 410.uint8_t ESP_JoinAP(constuint8_t *ssid, constuint8_t *pwd, intnofRetries, CLS1_ConstStdIOType *io) { 411. uint8_t buf[32]; 412. uint8_t res; 413. 414. do{ 415. res = JoinAccessPoint(ssid, pwd, io); 416. if(res==ERR_OK) { 417. break; 418. } 419. WAIT1_WaitOSms(1000); 420. nofRetries--; 421. } while(nofRetries>0); 422. returnres; 423.} 424. 425.staticuint8_t ReadIntoIPDBuffer(uint8_t *buf, size_t bufSize, uint8_t *p, uint16_t msgSize, uint16_t msTimeout, constCLS1_StdIOType *io) { 426. uint8_t ch; 427. size_t nofInBuf; 428. inttimeout; 429. 430. nofInBuf = p-buf; 431. bufSize -= nofInBuf; /* take into account what we already have in buffer */ 432. timeout = msTimeout; 433. while(msgSize>0&& bufSize>0) { 434. if(AS2_GetCharsInRxBuf()>0) { 435. (void)AS2_RecvChar(&ch); 436. *p = ch; 437. if(io!=NULL) { /* copy on console */ 438. io->stdOut(ch); 439. } 440. p++; 441. *p = '\0'; /* terminate */ 442. nofInBuf++; msgSize--; bufSize--; 443. } else{ 444. /* check in case we recveive less characters than expected, happens for POST? */ 445. if(nofInBuf>6&& UTIL1_strncmp(&p[-6], "\r\nOK\r\n", sizeof("\r\nOK\r\n")-1)==0) { 446. break; 447. } else{ 448. timeout -= 10; 449. WAIT1_WaitOSms(10); 450. if(timeout<0) { 451. returnERR_BUSY; 452. } 453. } 454. } 455. } 456. returnERR_OK; 457.} 458. 459.uint8_t ESP_GetIPD(uint8_t *msgBuf, size_t msgBufSize, uint8_t *ch_id, uint16_t *size, bool *isGet, uint16_t timeoutMs, constCLS1_StdIOType *io) { 460. /* scan e.g. for 461. * +IPD,0,404:POST / HTTP/1.1 462. * and return ch_id (0), size (404) 463. */ 464. uint8_t res = ERR_OK; 465. constuint8_t *p; 466. bool isIPD = FALSE; 467. uint8_t cmd[24], rxBuf[48]; 468. uint16_t ipdSize; 469. 470. *ch_id = 0; *size = 0; *isGet = FALSE; /* init */ 471. for(;;) { /* breaks */ 472. res = ESP_ReadCharsUntil(msgBuf, msgBufSize, '\n', timeoutMs); 473. if(res!=ERR_OK) { 474. break; /* timeout */ 475. } 476. if(res==ERR_OK) { /* line read */ 477. if(io!=NULL) { 478. CLS1_SendStr(msgBuf, io->stdOut); /* copy on console */ 479. } 480. isIPD = UTIL1_strncmp(msgBuf, "+IPD,", sizeof("+IPD,")-1)==0; 481. if(isIPD) { /* start of IPD message */ 482. p = msgBuf+sizeof("+IPD,")-1; 483. if(UTIL1_ScanDecimal8uNumber(&p, ch_id)!=ERR_OK) { 484. if(io!=NULL) { 485. CLS1_SendStr("ERR: wrong channel?\r\n", io->stdErr); /* error on console */ 486. } 487. res = ERR_FAILED; 488. break; 489. } 490. if(*p!=',') { 491. res = ERR_FAILED; 492. break; 493. } 494. p++; /* skip comma */ 495. if(UTIL1_ScanDecimal16uNumber(&p, size)!=ERR_OK) { 496. if(io!=NULL) { 497. CLS1_SendStr("ERR: wrong size?\r\n", io->stdErr); /* error on console */ 498. } 499. res = ERR_FAILED; 500. break; 501. } 502. if(*p!=':') { 503. res = ERR_FAILED; 504. break; 505. } 506. ipdSize = p-msgBuf; /* length of "+IPD,," string */ 507. p++; /* skip ':' */ 508. if(UTIL1_strncmp(p, "GET", sizeof("GET")-1)==0) { 509. *isGet = TRUE; 510. } elseif(UTIL1_strncmp(p, "POST", sizeof("POST")-1)==0) { 511. *isGet = FALSE; 512. } else{ 513. res = ERR_FAILED; 514. } 515. while(*p!='\0') { 516. p++; /* skip to the end */ 517. } 518. /* read the rest of the message */ 519. res = ReadIntoIPDBuffer(msgBuf, msgBufSize, (uint8_t*)p, (*size)-ipdSize, ESP_DEFAULT_TIMEOUT_MS, io); 520. break; 521. } 522. } 523. } 524. returnres; 525.} 526. 527.uint8_t ESP_StartWebServer(constCLS1_StdIOType *io) { 528. uint8_t buf[32]; 529. uint8_t res; 530. 531. res = ESP_SetNumberOfConnections(1, io, ESP_DEFAULT_TIMEOUT_MS); 532. if(res!=ERR_OK) { 533. CLS1_SendStr("ERR: failed to set multiple connections.\r\n", io->stdErr); 534. returnres; 535. } 536. res = ESP_SetServer(TRUE, 80, io, ESP_DEFAULT_TIMEOUT_MS); 537. if(res!=ERR_OK) { 538. CLS1_SendStr("ERR: failed to set server.\r\n", io->stdErr); 539. returnres; 540. } 541. CLS1_SendStr("INFO: Web Server started, waiting for connection on ", io->stdOut); 542. if(ESP_GetIPAddrString(buf, sizeof(buf))==ERR_OK) { 543. CLS1_SendStr(buf, io->stdOut); 544. CLS1_SendStr(":80", io->stdOut); 545. } else{ 546. CLS1_SendStr("(ERROR!)", io->stdOut); 547. } 548. CLS1_SendStr("\r\n", io->stdOut); 549. 550. returnERR_OK; 551.} 552. 553.uint8_t ESP_SendStr(constuint8_t *str, CLS1_ConstStdIOType *io) { 554. uint8_t buf[32]; 555. uint8_t rxBuf[48]; 556. uint8_t res; 557. uint16_t timeoutMs; 558. #define RX_TIMEOUT_MS 3000 559. AS2_TComData ch; 560. 561. UTIL1_strcpy(buf, sizeof(buf), str); 562. UTIL1_strcat(buf, sizeof(buf), "\r\n"); 563. res = ESP_SendATCommand(buf, rxBuf, sizeof(rxBuf), NULL, ESP_DEFAULT_TIMEOUT_MS, io); 564. timeoutMs = 0; 565. while(timeoutMs0) { 569. (void)AS2_RecvChar(&ch); 570. CLS1_SendChar(ch); 571. } 572. } 573. returnERR_OK; 574.} 575. 576.staticuint8_t ESP_PrintHelp(constCLS1_StdIOType *io) { 577. CLS1_SendHelpStr("ESP", "ESP8200 commands\r\n", io->stdOut); 578. CLS1_SendHelpStr(" help|status", "Print help or status information\r\n", io->stdOut); 579. CLS1_SendHelpStr(" send ", "Sends a string to the module\r\n", io->stdOut); 580. CLS1_SendHelpStr(" test", "Sends a test AT command\r\n", io->stdOut); 581. CLS1_SendHelpStr(" restart", "Restart module\r\n", io->stdOut); 582. CLS1_SendHelpStr(" listAP", "List available Access Points\r\n", io->stdOut); 583. CLS1_SendHelpStr(" connectAP \"ssid\",\"pwd\"", "Connect to an Access Point\r\n", io->stdOut); 584. CLS1_SendHelpStr(" server (start|stop)", "Start or stop web server\r\n", io->stdOut); 585. returnERR_OK; 586.} 587. 588.staticuint8_t ESP_PrintStatus(constCLS1_StdIOType *io) { 589. uint8_t buf[48]; 590. 591. CLS1_SendStatusStr("ESP8266", "\r\n", io->stdOut); 592. 593. CLS1_SendStatusStr(" Webserver", ESP_WebServerIsOn?"ON\r\n":"OFF\r\n", io->stdOut); 594. 595. if(ESP_GetFirmwareVersionString(buf, sizeof(buf)) != ERR_OK) { 596. UTIL1_strcpy(buf, sizeof(buf), "FAILED\r\n"); 597. } else{ 598. UTIL1_strcat(buf, sizeof(buf), "\r\n"); 599. } 600. CLS1_SendStatusStr(" AT+GMR", buf, io->stdOut); 601. 602. if(ESP_GetModeString(buf, sizeof(buf)) != ERR_OK) { 603. UTIL1_strcpy(buf, sizeof(buf), "FAILED\r\n"); 604. } else{ 605. if(UTIL1_strcmp(buf, "1")==0) { 606. UTIL1_strcat(buf, sizeof(buf), " (device)"); 607. } elseif(UTIL1_strcmp(buf, "2")==0) { 608. UTIL1_strcat(buf, sizeof(buf), " (AP)"); 609. } elseif(UTIL1_strcmp(buf, "3")==0) { 610. UTIL1_strcat(buf, sizeof(buf), " (device+AP)"); 611. } else{ 612. UTIL1_strcat(buf, sizeof(buf), " (ERROR)"); 613. } 614. UTIL1_strcat(buf, sizeof(buf), "\r\n"); 615. } 616. CLS1_SendStatusStr(" AT+CWMODE?", buf, io->stdOut); 617. 618. if(ESP_GetIPAddrString(buf, sizeof(buf)) != ERR_OK) { 619. UTIL1_strcpy(buf, sizeof(buf), "FAILED\r\n"); 620. } else{ 621. UTIL1_strcat(buf, sizeof(buf), "\r\n"); 622. } 623. CLS1_SendStatusStr(" AT+CIFSR", buf, io->stdOut); 624. 625. if(ESP_GetConnectedAPString(buf, sizeof(buf)) != ERR_OK) { 626. UTIL1_strcpy(buf, sizeof(buf), "FAILED\r\n"); 627. } else{ 628. UTIL1_strcat(buf, sizeof(buf), "\r\n"); 629. } 630. CLS1_SendStatusStr(" AT+CWJAP?", buf, io->stdOut); 631. 632. if(ESP_GetCIPMUXString(buf, sizeof(buf)) != ERR_OK) { 633. UTIL1_strcpy(buf, sizeof(buf), "FAILED\r\n"); 634. } else{ 635. if(UTIL1_strcmp(buf, "0")==0) { 636. UTIL1_strcat(buf, sizeof(buf), " (single connection)"); 637. } elseif(UTIL1_strcmp(buf, "1")==0) { 638. UTIL1_strcat(buf, sizeof(buf), " (multiple connections)"); 639. } else{ 640. UTIL1_strcat(buf, sizeof(buf), " (ERROR)"); 641. } 642. UTIL1_strcat(buf, sizeof(buf), "\r\n"); 643. } 644. CLS1_SendStatusStr(" CIPMUX", buf, io->stdOut); 645. returnERR_OK; 646.} 647. 648.uint8_t ESP_ParseCommand(constunsigned char*cmd, bool *handled, constCLS1_StdIOType *io) { 649. uint32_t val; 650. uint8_t res; 651. constunsigned char*p; 652. uint8_t pwd[24], ssid[24]; 653. 654. if(UTIL1_strcmp((char*)cmd, CLS1_CMD_HELP)==0|| UTIL1_strcmp((char*)cmd, "ESP help")==0) { 655. *handled = TRUE; 656. res = ESP_PrintHelp(io); 657. } elseif(UTIL1_strcmp((char*)cmd, CLS1_CMD_STATUS)==0|| UTIL1_strcmp((char*)cmd, "ESP status")==0) { 658. *handled = TRUE; 659. res = ESP_PrintStatus(io); 660. } elseif(UTIL1_strncmp((char*)cmd, "ESP send ", sizeof("ESP send ")-1)==0) { 661. *handled = TRUE; 662. p = cmd+sizeof("ESP send ")-1; 663. 664. (void)ESP_SendStr(p, io); 665. } elseif(UTIL1_strcmp((char*)cmd, "ESP test")==0) { 666. *handled = TRUE; 667. if(ESP_TestAT()!=ERR_OK) { 668. CLS1_SendStr("TEST failed!\r\n", io->stdErr); 669. res = ERR_FAILED; 670. } else{ 671. CLS1_SendStr("TEST ok!\r\n", io->stdOut); 672. } 673. } elseif(UTIL1_strcmp((char*)cmd, "ESP listAP")==0) { 674. *handled = TRUE; 675. (void)ESP_SendStr("AT+CWLAP", io); 676. /* AT + CWLAP 677. response 678. + CWLAP: , , [, ] 679. OK Or Fails, the return ERROR 680. 0 OPEN 681. 1 WEP 682. 2 WPA_PSK 683. 3 WPA2_PSK 684. 4 WPA_WPA2_PSK 685. string parameter, the access point name 686. signal strength 687. 0: manually connect 1: An automatic connection 688. */ 689. returnERR_OK; 690. } elseif(UTIL1_strncmp((char*)cmd, "ESP connectAP ", sizeof("ESP connectAP ")-1)==0) { 691. *handled = TRUE; 692. p = cmd+sizeof("ESP connectAP ")-1; 693. ssid[0] = '\0'; pwd[0] = '\0'; 694. res = UTIL1_ScanDoubleQuotedString(&p, ssid, sizeof(ssid)); 695. if(res==ERR_OK && *p!='\0'&& *p==',') { 696. p++; /* skip comma */ 697. res = UTIL1_ScanDoubleQuotedString(&p, pwd, sizeof(pwd)); 698. } else{ 699. CLS1_SendStr("Comma expected between strings!\r\n", io->stdErr); 700. res = ERR_FAILED; 701. } 702. if(res==ERR_OK) { 703. res = ESP_JoinAP(ssid, pwd, 3, io); 704. } else{ 705. CLS1_SendStr("Wrong command format!\r\n", io->stdErr); 706. res = ERR_FAILED; 707. } 708. } elseif(UTIL1_strcmp((char*)cmd, "ESP server start")==0) { 709. *handled = TRUE; 710. res = ESP_StartWebServer(io); 711. ESP_WebServerIsOn = res==ERR_OK; 712. } elseif(UTIL1_strcmp((char*)cmd, "ESP server stop")==0) { 713. *handled = TRUE; 714. ESP_WebServerIsOn = FALSE; 715. } elseif(UTIL1_strcmp((char*)cmd, "ESP restart")==0) { 716. *handled = TRUE; 717. ESP_Restart(io, 2000); 718. } 719. returnres; 720.} 721. 722.voidESP_Deinit(void) { 723. /* nothing to do */ 724.} 725. 726.voidESP_Init(void) { 727. AS2_ClearRxBuf(); /* clear buffer */ 728.} The application interface in Application.h is rather short :-): view source print? 01./* 02. * Application.h 03. * 04. * Author: Erich Styger 05. */ 06. 07.#ifndef APPLICATION_H_ 08.#define APPLICATION_H_ 09. 10./*! 11. * \brief Application main routine 12. */ 13.voidAPP_Run(void); 14. 15.#endif /* APPLICATION_H_ */ The main loop of the application is Application.c, along with the application specific web server code. As the SendWebPage function contains HTML code, I’m posting it here separately: view source print? 01.staticuint8_t SendWebPage(uint8_t ch_id, bool ledIsOn, uint8_t temperature, constCLS1_StdIOType *io) { 02. staticuint8_t http[1024]; 03. uint8_t cmd[24], rxBuf[48], expected[48]; 04. uint8_t buf[16]; 05. uint8_t res = ERR_OK; 06. 07. /* construct web page content */ 08. UTIL1_strcpy(http, sizeof(http), (uint8_t*)"HTTP/1.0 200 OK\r\nContent-Type: text/html\r\nPragma: no-cache\r\n\r\n"); 09. UTIL1_strcat(http, sizeof(http), (uint8_t*)"\r\n\r\n"); 10. UTIL1_strcat(http, sizeof(http), (uint8_t*)"\r\n"); 11. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Web Server using ESP8266\r\n"); 12. UTIL1_strcat(http, sizeof(http), (uint8_t*)" 13.\r\n"); 14. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Temp: OC"); 17. if(ledIsOn) { 18. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Red LED off"); 19. UTIL1_strcat(http, sizeof(http), (uint8_t*)" 20.Red LED on"); 21. } else{ 22. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Red LED off"); 23. UTIL1_strcat(http, sizeof(http), (uint8_t*)" 24.Red LED on"); 25. } 26. UTIL1_strcat(http, sizeof(http), (uint8_t*)""); 27. UTIL1_strcat(http, sizeof(http), (uint8_t*)"\r\n\r\n"); 28. 29. UTIL1_strcpy(cmd, sizeof(cmd), "AT+CIPSEND="); /* parameters are , */ 30. UTIL1_strcatNum8u(cmd, sizeof(cmd), ch_id); 31. UTIL1_chcat(cmd, sizeof(cmd), ','); 32. UTIL1_strcatNum16u(cmd, sizeof(cmd), UTIL1_strlen(http)); 33. UTIL1_strcpy(expected, sizeof(expected), cmd); /* we expect the echo of our command */ 34. UTIL1_strcat(expected, sizeof(expected), "\r\r\n> "); /* expect "> " */ 35. UTIL1_strcat(cmd, sizeof(cmd), "\r\n"); 36. res = ESP_SendATCommand(cmd, rxBuf, sizeof(rxBuf), expected, ESP_DEFAULT_TIMEOUT_MS, io); 37. if(res!=ERR_OK) { 38. if(io!=NULL) { 39. CLS1_SendStr("INFO: TIMEOUT, closing connection!\r\n", io->stdOut); 40. } 41. } else{ 42. if(io!=NULL) { 43. CLS1_SendStr("INFO: Sending http page...\r\n", io->stdOut); 44. } 45. UTIL1_strcat(http, sizeof(http), "\r\n\r\n"); /* need to add this to end the command! */ 46. res = ESP_SendATCommand(http, NULL, 0, NULL, ESP_DEFAULT_TIMEOUT_MS, io); 47. if(res!=ERR_OK) { 48. CLS1_SendStr("Sending page failed!\r\n", io->stdErr); /* copy on console */ 49. } else{ 50. for(;;) { /* breaks */ 51. res = ESP_ReadCharsUntil(buf, sizeof(buf), '\n', 1000); 52. if(res==ERR_OK) { /* line read */ 53. if(io!=NULL) { 54. CLS1_SendStr(buf, io->stdOut); /* copy on console */ 55. } 56. } 57. if(UTIL1_strncmp(buf, "SEND OK\r\n", sizeof("SEND OK\r\n")-1)==0) { /* ok from module */ 58. break; 59. } 60. } 61. } 62. } 63. returnres; 64.} The rest of Application.c is rather simple: view source print? 01./* 02. * Application.c 03. * 04. * Author: Erich Styger 05. */ 06.#include "PE_Types.h" 07.#include "CLS1.h" 08.#include "WAIT1.h" 09.#include "Shell.h" 10.#include "UTIL1.h" 11.#include "ESP8266.h" 12.#include "LEDR.h" 13.#include "LEDG.h" 14.#include "AS2.h" 15. 16.staticuint8_t APP_EspMsgBuf[512]; /* buffer for messages from ESP8266 */ 17. 18.staticvoidWebProcess(void) { 19. uint8_t res=ERR_OK; 20. bool isGet; 21. uint8_t ch_id=0; 22. uint16_t size=0; 23. constuint8_t *p; 24. constCLS1_StdIOType *io; 25. 26. if(ESP_IsServerOn()) { 27. io = CLS1_GetStdio(); 28. res = ESP_GetIPD(APP_EspMsgBuf, sizeof(APP_EspMsgBuf), &ch_id, &size, &isGet, 1000, io); 29. if(res==ERR_OK) { 30. if(isGet) { /* GET: put web page */ 31. res = SendWebPage(ch_id, LEDR_Get()!=FALSE, 21/*dummy temperature*/, io); 32. if(res!=ERR_OK && io!=NULL) { 33. CLS1_SendStr("Sending page failed!\r\n", io->stdErr); /* copy on console */ 34. } 35. } else{ /* POST: received info */ 36. intpos; 37. 38. pos = UTIL1_strFind(APP_EspMsgBuf, "radio="); 39. if(pos!=-1) { /* found */ 40. if(UTIL1_strncmp(&APP_EspMsgBuf[pos], "radio=0", sizeof("radio=0")-1)) { 41. LEDR_On(); 42. } elseif(UTIL1_strncmp(&APP_EspMsgBuf[pos], "radio=1", sizeof("radio=1")-1)) { 43. LEDR_Off(); 44. } 45. } 46. res = SendWebPage(ch_id, LEDR_Get()!=FALSE, 20/*dummy temperature*/, io); 47. if(res!=ERR_OK && io!=NULL) { 48. CLS1_SendStr("Sending page failed!\r\n", io->stdErr); /* copy on console */ 49. } 50. } 51. CLS1_SendStr("INFO: Closing connection...\r\n", io->stdOut); 52. res = ESP_CloseConnection(ch_id, io, ESP_DEFAULT_TIMEOUT_MS); 53. } 54. } else{ /* copy messages we receive to console */ 55. while(AS2_GetCharsInRxBuf()>0) { 56. uint8_t ch; 57. 58. (void)AS2_RecvChar(&ch); 59. CLS1_SendChar(ch); 60. } 61. } 62.} 63. 64.voidAPP_Run(void) { 65. CLS1_ConstStdIOType *io; 66. 67. WAIT1_Waitms(1000); /* wait after power-on */ 68. ESP_Init(); 69. SHELL_Init(); 70. io = CLS1_GetStdio(); 71. CLS1_SendStr("\r\n------------------------------------------\r\n", io->stdOut); 72. CLS1_SendStr("ESP8266 with FRDM-KL25Z\r\n", io->stdOut); 73. CLS1_SendStr("------------------------------------------\r\n", io->stdOut); 74. CLS1_PrintPrompt(io); 75. for(;;) { 76. WebProcess(); 77. SHELL_Parse(); 78. WAIT1_Waitms(10); 79. LEDG_Neg(); 80. } 81.} In main.c I call the application part: view source print? 01./* ################################################################### 02.** Filename : main.c 03.** Project : FRDM-KL25Z_ESP8266 04.** Processor : MKL25Z128VLK4 05.** Version : Driver 01.01 06.** Compiler : GNU C Compiler 07.** Date/Time : 2014-10-15, 14:28, # CodeGen: 0 08.** Abstract : 09.** Main module. 10.** This module contains user's application code. 11.** Settings : 12.** Contents : 13.** No public methods 14.** 15.** ###################################################################*/ 16./*! 17.** @file main.c 18.** @version 01.01 19.** @brief 20.** Main module. 21.** This module contains user's application code. 22.*/ 23./*! 24.** @addtogroup main_module main module documentation 25.** @{ 26.*/ 27./* MODULE main */ 28. 29./* Including needed modules to compile this module/procedure */ 30.#include "Cpu.h" 31.#include "Events.h" 32.#include "WAIT1.h" 33.#include "UTIL1.h" 34.#include "AS1.h" 35.#include "ASerialLdd1.h" 36.#include "CLS1.h" 37.#include "CS1.h" 38.#include "AS2.h" 39.#include "ASerialLdd2.h" 40.#include "LEDR.h" 41.#include "LEDpin1.h" 42.#include "BitIoLdd1.h" 43.#include "LEDG.h" 44.#include "LEDpin2.h" 45.#include "BitIoLdd2.h" 46.#include "LEDB.h" 47.#include "LEDpin3.h" 48.#include "BitIoLdd3.h" 49./* Including shared modules, which are used for whole project */ 50.#include "PE_Types.h" 51.#include "PE_Error.h" 52.#include "PE_Const.h" 53.#include "IO_Map.h" 54./* User includes (#include below this line is not maintained by Processor Expert) */ 55.#include "Application.h" 56. 57./*lint -save -e970 Disable MISRA rule (6.3) checking. */ 58.intmain(void) 59./*lint -restore Enable MISRA rule (6.3) checking. */ 60.{ 61. /* Write your local variable definition here */ 62. 63. /*** Processor Expert internal initialization. DON'T REMOVE THIS CODE!!! ***/ 64. PE_low_level_init(); 65. /*** End of Processor Expert internal initialization. ***/ 66. 67. APP_Run(); 68. 69. /*** Don't write any code pass this line, or it will be deleted during code generation. ***/ 70. /*** RTOS startup code. Macro PEX_RTOS_START is defined by the RTOS component. DON'T MODIFY THIS CODE!!! ***/ 71. #ifdef PEX_RTOS_START 72. PEX_RTOS_START(); /* Startup of the selected RTOS. Macro is defined by the RTOS component. */ 73. #endif 74. /*** End of RTOS startup code. ***/ 75. /*** Processor Expert end of main routine. DON'T MODIFY THIS CODE!!! ***/ 76. for(;;){} 77. /*** Processor Expert end of main routine. DON'T WRITE CODE BELOW!!! ***/ 78.} /*** End of main routine. DO NOT MODIFY THIS TEXT!!! ***/ 79. 80./* END main */ 81./*! 82.** @} 83.*/ 84./* 85.** ################################################################### 86.** 87.** This file was created by Processor Expert 10.4 [05.10] 88.** for the Freescale Kinetis series of microcontrollers. 89.** 90.** ################################################################### 91.*/ Processor Expert Components In addition, I’m using several Processor Expert component which are available fromSourceForge. Processor Expert Components Wait: Busy waiting component, e.g. to wait for a few milliseconds. Utility: string manipulation and utility functions. AsynchroSerial (AS1): serial interface to the host for the shell command line interface Shell: command line shell implementation CriticalSection: for creating critical sections AsynchroSerial (AS2): serial interface to the ESP8266 module LEDR, LEDG and LEDB: Red, Green and Blue LED on the FRDM-KL25Z board AS1 is configured as UART connection (over OpenSDA) for the shell: Shell UART Settings There are no special settings for the Shell component: Shell Settings Important are the correct settings to the ESP8266 UART: 115200 baud and using the correct pins on the board connected to the Rx and Tx lines of the ESP8266. I’m using rather large input and output buffers: UART connection to ESP8266 The LED components are configured for the pins used on the board: PTB18 for red, PTB19 for green and PTD1 for blue LED. Red LED for FRDM-KL25Z Sending Commands The shell implements the command ESP send which I can use to send a string or command to the module: ESP send Note that for every command a trailing “\r\n” will be sent. So instead of using the programmatic way, the shell can be used to ‘manually’ drive a web server, at least most of the part. So I’m using command line commands below to explore how the ESP8266 module works. Using the Shell With the project (link to GitHub below), I have a serial connection and command line shell interface to the module. Compile the project and download it to the FRDM-KL25Z board and use a terminal program (I use Termite) to talk with the module. It power-up, the program shows a greeting message: Greeting Message With ‘help‘ I get a list of the available commands: Help Command The ‘status‘ command gives a system status: Status Command Output With this, I’m ready to send commands to the module :-). Connection Test To test the connection I send a simple ‘AT’ command ESP send AT AT Command Output and the module should respond with AT\r\r\n\r\nOK\r\n Module Restart Sometimes the module gets stuck. What helps is a power-on reset of the module. Another way is to send the AT+RST command to reset the module. The module will boot up and print a ‘ready’ message: Reset of the ESP8266 Access Point or Device First I need to configure if the ESP is either a device or an access point. For this, theCWMODE command is used: AT+CWMODE= where is one of: 1: ‘Sta’, ESP8266 is a device, it connects to an existing access point 2: ‘AP’, ESP8266 is an access point, so other devices can connect to it 3: ‘both’. Not really clear to me, but it seems that in this mode the device is in a hybrid mode? To have the ESP as device so it can connect to an existing access point I use AT+CWMODE=1 and the module should answer with AT+CWMODE=1\r\r\n\r\nOK\r\n or with a ‘no change': AT+CWMODE=1\r\r\nno change\r\n With AT+CWMODE? I can ask for the current mode: Retrieving Current Mode List of Access Points With AT+CWLAP I get a list of access points. It reports a list like this: AT+CWLAP +CWLAP:(0,"",0) +CWLAP:(4,"APforESP",-39) +CWLAP:(4,"iza-97497",-94) OK :!: I experienced problems with that command in an environment with lots of access points visible. In this case it seems the module hands up. Try first in a place with only a few access points. For this tutorial I have configured an access point with SSID “APforESP” which shows up in my list. The list is formatted like this + CWLAP: , , [, ] With following encoding: : 0: OPEN 1: WPA_PSK 2: WPA2_PSK 4: WPA_WPA2_PSK : the SSID (string) of the access point. : Signal strength. : 0: manually connect 1: automatic connect Connecting to Access Point To connect to an access point I use the command AT+CWJAP="","" Of course replace and with your setup. The module should report back an “OK” message, and you are connected :-). :!: The module stores the ssid and password. After power-up, the module will automatically reconnect to the Access Point. IP Address Once connected I can check the IP address I have been assigned to with AT+CIFSR which should give something like AT+CIFSR 192.168.0.111 So now I know my module IP address :-). With this I can ping my module: Pinging my ESP Module Building a Web Server Now as we hav a connection, it is time to use it to run a web server :-).What I want to serve a web page which I can use to turn on or off the LEDs on the board. Number of Connections: CIPMUX Before I start the server I need to make sure it accepts multiple connections. For this I use the following command: AT+CIPMUX=1 The parameter is either 0 (single connection), or 1 (multiple connections). For a web server I need to set it up for multiple connections. The ESP module should respond with AT+CIPMUX=1\r\n\r\nOK\r\n :info: To make it clear, I have included the ‘\r’ and ‘\n’ in the responses. Starting the Server: CIPSERVER I start the server with AT+CIPSERVER=1,80 The first parameter is either 0 (close connection) or 1 (open connection), followed by the port. I use here the standard http port (80). The module should answer with: AT+CIPSERVER=1,80\r\r\n\r\nOK\r\n or if it is already running the server with a ‘no change': AT+CIPSERVER=1,80\r\r\nno change\r\n No I have a connection open on my IP address (see above: 192.168.0.111), listening to the port I have specified (80). Connecting to the Server with Browser I enter the IP address in a web browser: http://192.168.0.111:80 For clarity I have specified the standard HTTP port (80). So if you are using a different port, make sure you specify it in the address line. Connection from FireFox The browser now sends a GET request to the module, and I will see this from the message printed out from the module: First response from Module The ‘Link’ indicates that it has established a link. IPD (IP Data?) is followed by the channel number (this will the one we have to respond to), plus the size of the following data (296 bytes in that case). As I’m not responding (yet), there will be a timeout (after about 1 minute or so), with an ‘Unlink’ message from the module: Link +IPD,0,296:GET / HTTP/1.1 Host: 192.168.0.111 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: de,en-US;q=0.7,en;q=0.3 Accept-Encoding: gzip, deflate Connection: keep-alive OK Unlink Unlink message Sending Data to Server: CIPSEND Now I need to respond and send data to the browser. For this I need to know the channel number, and this is provided in the IPD message from above, right after the comma: +IPD,0 To send data, I use the command AT+CIPSEND=, So I connect again with the browser, and I send 5 bytes (“hello”) with: AT+CIPSEND=0,5 The ESP8266 responds with AT+CIPSEND=0,5\r\n> Notice the ‘>’ at the end: this is my signal to send the actual data (“hello” in my case): hello The ESP8266 now resonds with a SEND OK: Data Sent However, the browser is still busy and spins around. I already thought that I did something wrong, but after the browser run into a timeout (after about one minute), my data is there! :-) Hello in Browser Closing Connection: CIPCLOSE So things *are* working :-). The trick is that I have to close the connection after I have sent the data. There is a CIPCLOSE command I can use: AT+CIPCLOSE= which I can use to close a channel. So I close the connection with AT+CIPCLOSE=0 and now the browser shows the content right away :-). Web Server Implementation So far I have used the module in command line and manual mode. This is great for exploration of the protocol, but for building the web server I need to do this programmatically. For this I run my ‘main’ loop in APP_Run(). After printing a greeting message and initializing the sub modules, it processes the web/module responses, parses the shell command line interfaces and blinks the green (LEDG) LED. view source print? 01.voidAPP_Run(void) { 02. CLS1_ConstStdIOType *io; 03. 04. WAIT1_Waitms(1000); /* wait after power-on */ 05. ESP_Init(); 06. SHELL_Init(); 07. io = CLS1_GetStdio(); 08. CLS1_SendStr("\r\n------------------------------------------\r\n", io->stdOut); 09. CLS1_SendStr("ESP8266 with FRDM-KL25Z\r\n", io->stdOut); 10. CLS1_SendStr("------------------------------------------\r\n", io->stdOut); 11. CLS1_PrintPrompt(io); 12. for(;;) { 13. WebProcess(); 14. SHELL_Parse(); 15. WAIT1_Waitms(10); 16. LEDG_Neg(); 17. } 18.} With ESP server start I start the web server: Starting the Web Server It sends the AT+CIPMUX command followed by the AT+CIPSERVER to start the server, and then listens to the port. Reading and responding messages is done in WebProcess(): view source print? 01.staticvoidWebProcess(void) { 02. uint8_t res=ERR_OK; 03. bool isGet; 04. uint8_t ch_id=0; 05. uint16_t size=0; 06. constuint8_t *p; 07. constCLS1_StdIOType *io; 08. 09. if(ESP_IsServerOn()) { 10. io = CLS1_GetStdio(); 11. res = ESP_GetIPD(APP_EspMsgBuf, sizeof(APP_EspMsgBuf), &ch_id, &size, &isGet, 1000, io); 12. if(res==ERR_OK) { 13. if(isGet) { /* GET: put web page */ 14. res = SendWebPage(ch_id, LEDR_Get()!=FALSE, 21/*dummy temperature*/, io); 15. if(res!=ERR_OK && io!=NULL) { 16. CLS1_SendStr("Sending page failed!\r\n", io->stdErr); /* copy on console */ 17. } 18. } else{ /* POST: received info */ 19. intpos; 20. 21. pos = UTIL1_strFind(APP_EspMsgBuf, "radio="); 22. if(pos!=-1) { /* found */ 23. if(UTIL1_strncmp(&APP_EspMsgBuf[pos], "radio=0", sizeof("radio=0")-1)) { 24. LEDR_On(); 25. } elseif(UTIL1_strncmp(&APP_EspMsgBuf[pos], "radio=1", sizeof("radio=1")-1)) { 26. LEDR_Off(); 27. } 28. } 29. res = SendWebPage(ch_id, LEDR_Get()!=FALSE, 20/*dummy temperature*/, io); 30. if(res!=ERR_OK && io!=NULL) { 31. CLS1_SendStr("Sending page failed!\r\n", io->stdErr); /* copy on console */ 32. } 33. } 34. CLS1_SendStr("INFO: Closing connection...\r\n", io->stdOut); 35. res = ESP_CloseConnection(ch_id, io, ESP_DEFAULT_TIMEOUT_MS); 36. } 37. } else{ /* copy messages we receive to console */ 38. while(AS2_GetCharsInRxBuf()>0) { 39. uint8_t ch; 40. 41. (void)AS2_RecvChar(&ch); 42. CLS1_SendChar(ch); 43. } 44. } 45.} If the server is not enabled, it simply copies the received messages to the console: view source print? 1.} else{ /* copy messages we receive to console */ 2. while(AS2_GetCharsInRxBuf()>0) { 3. uint8_t ch; 4. 5. (void)AS2_RecvChar(&ch); 6. CLS1_SendChar(ch); 7. } 8. } Otherwise it scans for an IPD message (ESP_GetIPD()). This function returns the whole message, the channel, the message size and if it is a GET or POST message: 1 res = ESP_GetIPD(APP_EspMsgBuf, sizeof(APP_EspMsgBuf), &ch_id, &size, &isGet, 1000, io); If it is a GET message, then it sends a HTML page to the module: 1 res = SendWebPage(ch_id, LEDR_Get()!=FALSE, 21 /*dummy temperature*/, io); This web page shows the status of the red LED on the board, a (dummy) temperature value and a button to submit new LED values: WSP8266 Web Server The HTML code for this page is constructed in SendWebPage() and sent withAT+CIPSEND: view source print? 01.staticuint8_t SendWebPage(uint8_t ch_id, bool ledIsOn, uint8_t temperature, constCLS1_StdIOType *io) { 02. staticuint8_t http[1024]; 03. uint8_t cmd[24], rxBuf[48], expected[48]; 04. uint8_t buf[16]; 05. uint8_t res = ERR_OK; 06. 07. /* construct web page content */ 08. UTIL1_strcpy(http, sizeof(http), (uint8_t*)"HTTP/1.0 200 OK\r\nContent-Type: text/html\r\nPragma: no-cache\r\n\r\n"); 09. UTIL1_strcat(http, sizeof(http), (uint8_t*)"\r\n\r\n"); 10. UTIL1_strcat(http, sizeof(http), (uint8_t*)"\r\n"); 11. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Web Server using ESP8266\r\n"); 12. UTIL1_strcat(http, sizeof(http), (uint8_t*)" 13.\r\n"); 14. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Temp: OC"); 17. if(ledIsOn) { 18. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Red LED off"); 19. UTIL1_strcat(http, sizeof(http), (uint8_t*)" 20.Red LED on"); 21. } else{ 22. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Red LED off"); 23. UTIL1_strcat(http, sizeof(http), (uint8_t*)" 24.Red LED on"); 25. } 26. UTIL1_strcat(http, sizeof(http), (uint8_t*)""); 27. UTIL1_strcat(http, sizeof(http), (uint8_t*)"\r\n\r\n"); 28. 29. UTIL1_strcpy(cmd, sizeof(cmd), "AT+CIPSEND="); /* parameters are , */ 30. UTIL1_strcatNum8u(cmd, sizeof(cmd), ch_id); 31. UTIL1_chcat(cmd, sizeof(cmd), ','); 32. UTIL1_strcatNum16u(cmd, sizeof(cmd), UTIL1_strlen(http)); 33. UTIL1_strcpy(expected, sizeof(expected), cmd); /* we expect the echo of our command */ 34. UTIL1_strcat(expected, sizeof(expected), "\r\r\n> "); /* expect "> " */ 35. UTIL1_strcat(cmd, sizeof(cmd), "\r\n"); 36. res = ESP_SendATCommand(cmd, rxBuf, sizeof(rxBuf), expected, ESP_DEFAULT_TIMEOUT_MS, io); 37. if(res!=ERR_OK) { 38. if(io!=NULL) { 39. CLS1_SendStr("INFO: TIMEOUT, closing connection!\r\n", io->stdOut); 40. } 41. } else{ 42. if(io!=NULL) { 43. CLS1_SendStr("INFO: Sending http page...\r\n", io->stdOut); 44. } 45. UTIL1_strcat(http, sizeof(http), "\r\n\r\n"); /* need to add this to end the command! */ 46. res = ESP_SendATCommand(http, NULL, 0, NULL, ESP_DEFAULT_TIMEOUT_MS, io); 47. if(res!=ERR_OK) { 48. CLS1_SendStr("Sending page failed!\r\n", io->stdErr); /* copy on console */ 49. } else{ 50. for(;;) { /* breaks */ 51. res = ESP_ReadCharsUntil(buf, sizeof(buf), '\n', 1000); 52. if(res==ERR_OK) { /* line read */ 53. if(io!=NULL) { 54. CLS1_SendStr(buf, io->stdOut); /* copy on console */ 55. } 56. } 57. if(UTIL1_strncmp(buf, "SEND OK\r\n", sizeof("SEND OK\r\n")-1)==0) { /* ok from module */ 58. break; 59. } 60. } 61. } 62. } 63. returnres; 64.} In case of a POST message (user has pressed the button), I scan for the radio element string and turn on/off the LED accordingly, and re-submit the new web page: view source print? 01.} else{ /* POST: received info */ 02. intpos; 03. 04. pos = UTIL1_strFind(APP_EspMsgBuf, "radio="); 05. if(pos!=-1) { /* found */ 06. if(UTIL1_strncmp(&APP_EspMsgBuf[pos], "radio=0", sizeof("radio=0")-1)) { 07. LEDR_On(); 08. } elseif(UTIL1_strncmp(&APP_EspMsgBuf[pos], "radio=1", sizeof("radio=1")-1)) { 09. LEDR_Off(); 10. } 11. } 12. res = SendWebPage(ch_id, LEDR_Get()!=FALSE, 20/*dummy temperature*/, io); 13. if(res!=ERR_OK && io!=NULL) { 14. CLS1_SendStr("Sending page failed!\r\n", io->stdErr); /* copy on console */ 15. } 16. } Finally, it closes the connection at the end: view source print? 1.CLS1_SendStr("INFO: Closing connection...\r\n", io->stdOut); 2. res = ESP_CloseConnection(ch_id, io, ESP_DEFAULT_TIMEOUT_MS); With this, I handle GET and POST messages and can toggle the LED on my board :-) :-). Summary It is amazing what is possible with this tiny and inexpensive ($4.50) WiFi module. The simple AT interface allows small and tiny microprocesors to connect to the internet or the local network. With all the hype around ‘Internet of Things’ this is where things very likely will end up: small nodes connecting in an easy way to the network. The processor on that ESP8266 is probably more powerful than the KL25Z (the specs and data sheets of that ESP8266 are still evolving). Or it is possible to run that module in standalone mode too which is a very interesting approach too, see the links at the end of this article. But still having an UART way to connect to the network is very useful and powerful. Other modules costs multiple times more. I expect that many vendors will come up with similar integrated modules e.g. to combine an ARM processor with the WiFi radio, similar that ESP8266 module. For sure that ESP8266 has a head start and paved the way how WiFi connectivity should work. We all will see what the future brings. Until then, that ESP8266 module is something I can use in many projects :-). The sources and project files can be found on GitHub: https://github.com/ErichStyger/mcuoneclipse/tree/master/Examples/KDS/FRDM-KL25Z/FRDM-KL25Z_ESP8266 Happy Web-Serving :-) Useful Links: http://www.electrodragon.com/w/Wi07c http://scargill.wordpress.com/category/esp8266/ https://github.com/esp8266/esp8266-webserver http://www.cse.dmu.ac.uk/~sexton/ESP8266/ http://defcon-cc.dyndns.org/wiki/ESP8266#Update http://www.xess.com/blog/esp8266-resources/

Compared to other features, real-time search capability is undoubtedly one of the most important features in Elasticsearch. Today we’ll look closely how is provided real-time search by Elasticsearch. Real time First of all, if we need to explain the concept of real-time, in general, we can say that the delay between input and out time in the information is small at real-time systems. This means, data is taken without data accumulation, processed in real time. Today, the best solution Elasticsearch known for real-time search, when a record is added to it for storage makes it searchable in 1 second. How? As is known, the disks are able to create a risk of bottleneck for I/O operations at the data persistence step. Also some mechanisms used for prevent any loss of data increases cost of time. At this point Elasticsearch uses the file-system cache that sitting between itself and the disk for overcome the risk of bottleneck and ensure the a new document can be searched in real time. A new segment is written to the file-system cache first and only later it flushed to disk by Elasticsearch. This lightweight process of writing and opening a new segment is called a refresh in Elasticsearch. By default, all shards is refreshed automatically once every second. In this way, Elasticsearch support real-time search. Test time Above digression about the time of refresh of the shards you can bring to mind the following questions: What happens, when a new document is requested in less than 1 second time? Can be documents requested, without having to depend of the refresh period shards of managed by Elasticsearch? Short answers. Elasticsearch does not return the document. Yes. Now let’s get clarity on this issue is a simple example. hakdogan$ curl -XPUT localhost:9200/kodcucom/document/1 -d'{ > "title": "Document A" > }' We sent a document to Elasticsearch. The index name is kodcucom, type document, id value 1. The title field is only field in the document and the value of "Document A". Let’s take this document from Elasticsearch. hakdogan$ curl -XGET localhost:9200/kodcucom/document/1?pretty { "_index" : "kodcucom", "_type" : "document", "_id" : "1", "_version" : 1, "found" : true, "_source":{ "title": "Document A" } } As expected, the document was returned to us. Well, if we keep short the time between document recording and get request than default shard refresh time what will happen? Let’s see. hakdogan$ curl -XPUT localhost:9200/kodcucom/document/2 -d'{"title": "Document B"}'; curl -XGET localhost:9200/kodcucom/_search?pretty {"_index":"kodcucom","_type":"document","_id":"2","_version":1,"created":true}{ "took" : 38, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 1.0, "hits" : [ { "_index" : "kodcucom", "_type" : "document", "_id" : "1", "_score" : 1.0, "_source":{ "title": "Document A" } } ] } } As can be seen, only the previous document was returned to us by Elasticsearch when we do concurrently create and get request. Well, how can I get the document concurrently? Let’s see. hakdogan$ curl -XPUT localhost:9200/kodcucom/document/3 -d'{"title": "Document C"}'; curl -XGET localhost:9200/kodcucom/_refresh; curl -XGET localhost:9200/kodcucom/_search?pretty {"_index":"kodcucom","_type":"document","_id":"3","_version":1,"created":true}{"_shards":{"total":10,"successful":5,"failed":0}{ "took" : 3, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 3, "max_score" : 1.0, "hits" : [ { "_index" : "kodcucom", "_type" : "document", "_id" : "1", "_score" : 1.0, "_source":{ "title": "Document A" } }, { "_index" : "kodcucom", "_type" : "document", "_id" : "2", "_score" : 1.0, "_source":{"title": "Document B"} }, { "_index" : "kodcucom", "_type" : "document", "_id" : "3", "_score" : 1.0, "_source":{"title": "Document C"} } ] } } In this command, we perform to refresh operation on kodcucom index before the search request. In this way, the document was returned to us. Auto refresh time can be changed. By setting the index.refresh_interval parameter in the configuration file. Applies to all indices in the cluster. A per-index basis by updated index setting. In addition to these, you can turn off automatic refresh. An important point to keep in mind about the refresh time of the shards, the refresh operation is costly in terms of system resources. If you wished to make changes to the auto-refresh time, this situation should be taken into account. Extension of the automatic refresh time, enables faster indexing but new documents and changes made to the existing documents will not appear in searches during specified period of time.

Nowadays it has become mandatory to Gzipping the APIs response due to huge amount of data we are sending in response. It saves network bandwidth and delivery time and off course space over the internet. While using CXF; it provides an option to use the Gzip Compression in no of ways. Blueprint Annotation Blueprint: Annotation: First you need to register the GZIPOutInterceptor in out interceptors list. For that you need to hook into CXF initialization classes. public class InterceptorManager extends AbstractFeature { private static final Logger LOGGER = Logger.getLogger( "simcore" ); private static final Interceptor< Message > GZIP = new GZIPOutInterceptor(); //private static final Interceptor< Message > GZIP = new GZIPOutInterceptor(512); /* (non-Javadoc) * @see org.apache.cxf.feature.AbstractFeature#initializeProvider(org.apache.cxf.interceptor.InterceptorProvider, org.apache.cxf.Bus) */ @Override protected void initializeProvider( InterceptorProvider provider, Bus bus ) { /** * Adding Gzip interceptor to all outbound requests/responses */ LOGGER.debug( " ############## Adding Gzip as OUT Interceptor ##############" ); provider.getOutInterceptors().add( GZIP ); } } GZIPOutInterceptor comes with an option to set the Threshold value as no of Bytes. If response size will be below this threshold value then it will not be compressed. It is extremely useful when we will be sending empty lists and status messages/codes only. Because compressing those small responses will be overhead at server side. But there is another factor which is no of users requesting the response. So set this value by thinking over all the cases in mind. @GZIP Now we can use this annotation on any of our web-services controller to implement compression on all the APIs provided in that class. @WebService @Consumes ( { MediaType.TEXT_PLAIN, MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON } ) @Produces ( MediaType.APPLICATION_JSON ) @GZIP public interface WebServicesController { @GET @Path ( "/myGzipData" ) @Produces ( { MediaType.APPLICATION_JSON } ) Response getZipData( ); } Moreover we can set different parameters in Gzip annotation. @GZIP ( force = true, threshold = 512 )

there is currently a strong trend for microservice based architectures and frequent discussions comparing them to monoliths. there is much advice about breaking-up monoliths into microservices and also some amusing fights between proponents of the two paradigms - see the great microservices vs monolithic melee . the term 'monolith' is increasingly being used as a generic insult in the same way that 'legacy' is! however, i believe that there is a great deal of misunderstanding about exactly what a 'monolith' is and those discussing it are often talking about completely different things. a monolith can be considered an architectural style or a software development pattern (or anti-pattern if you view it negatively). styles and patterns usually fit into different viewtypes (a viewtype is a set, or category, of views that can be easily reconciled with each other [clements et al., 2010]) and some basic viewtypes we can discuss are: module - the code units and their relation to each other at compile time. allocation - the mapping of the software onto its environment. runtime - the static structure of the software elements and how they interact at runtime. a monolith could refer to any of the basic viewtypes above. module monolith if you have a module monolith then all of the code for a system is in a single codebase that is compiled together and produces a single artifact. the code may still be well structured (classes and packages that are coherent and decoupled at a source level rather than a big-ball-of-mud) but it is not split into separate modules for compilation. conversely a non-monolithic module design may have code split into multiple modules or libraries that can be compiled separately, stored in repositories and referenced when required. there are advantages and disadvantages to both but this tells you very little about how the code is used - it is primarily done for development management. allocation monolith for an allocation monolith, all of the code is shipped/deployed at the same time. in other words once the compiled code is 'ready for release' then a single version is shipped to all nodes. all running components have the same version of the software running at any point in time. this is independent of whether the module structure is a monolith. you may have compiled the entire codebase at once before deployment or you may have created a set of deployment artifacts from multiple sources and versions. either way this version for the system is deployed everywhere at once (often by stopping the entire system, rolling out the software and then restarting). a non-monolithic allocation would involve deploying different versions to individual nodes at different times. this is again independent of the module structure as different versions of a module monolith could be deployed individually. runtime monolith a runtime monolith will have a single application or process performing the work for the system (although the system may have multiple, external dependencies). many systems have traditionally been written like this (especially line-of-business systems such as payroll, accounts payable, cms etc). whether the runtime is a monolith is independent of whether the system code is a module monolith or not. a runtime monolith often implies an allocation monolith if there is only one main node/component to be deployed (although this is not the case if a new version of software is rolled out across regions, with separate users, over a period of time). note that my examples above are slightly forced for the viewtypes and it won't be as hard-and-fast in the real world. conclusion be very carefully when arguing about 'microservices vs monoliths'. a direct comparison is only possible when discussing the runtime viewtype and properties. you should also not assume that moving away from a module or allocation monolith will magically enable a microservice architecture (although it will probably help). if you are moving to a microservice architecture then i'd advise you to consider all these viewtypes and align your boundaries across them i.e. don't just code, build and distribute a monolith that exposes subsets of itself on different nodes.

My last post talked about what’s wrong with type classes (in general, but also specifically in Haskell). This post generated some great feedback on Reddit, including some valid criticism that I didn’t explain why I hated on newtypes so much. I took some of that feedback and incorporated it into a revised version of the post, but I have even more to say about “newtypes," so I decided to write another blog post. What’s in a Newtype Newtypes are a feature of Haskell that let you define a new type in terms of an existing type. In the following example, I create a newtype for Email, which “holds” a String. newtype Email = Email String They are similar to type synonyms (type Email = String), except that type synonyms don’t create new types, they just allow you to refer to existing types by other names. Every newtype can be easily translated into a data declaration. In fact, only the keyword changes: data Email = Email String There’s a slight semantic difference between the two, but for purposes of this blog post, any criticism I have against newtypes apply equally to similar constructs modeled with type or data. The Promise of Newtypes Newtypes are used to provide and select between alternate implementations of type classes for some base types. I think that’s a hack (albeit a necessary one), but I’ve already talked about this so I won’t belabor it here. The other promise of newtypes is that we can use them to make our code more type safe. Instead of passing around String as an email, for example, we can create a super lightweight “wrapper” around String called Email, and make it an error to use a String wherever an Email is expected. This practice isn’t restricted to Haskell. Even in Java, it’s considered good coding practice to wrap primitives with classes whose names denote the meaning of the wrapper (Email, SSN, Address, etc.). There’s a part of this promise that’s certainly true. If I have to define a function accepting four parameters, and three of them are strings, but one of those strings denotes an email, then I have two choices: Model the email parameter with a String. In this case, I may accidentally use the email where I intended to use the other two string parameters, or I may use one of the other two string parameters where I intended to use the email. Considering just these choices, there are five ways my program may go wrong if I use the wrong name in the wrong position. Model the email parameter with a newtype. In this case, I cannot use the email where I intended to use the other two string parameters, because the compiler may stop me. Similarly, I cannot use the other two string parameters where I intended to use the email, for the same reason. Looking at just these choices, there are 0 ways my program may go wrong. Thus, newtypes, like all good FP practices, reduce the number of ways my program can go wrong. Unfortunately, in my opinion, they don’t go nearly far enough. False Security For most intent and purposes, newtypes are isomorphic to the single value they hold. In my preceding example, given a String, I can get an email (Email "foo"). Given an Email, I can also get a String, e.g. by pattern matching on the Email constructor. Stated differently, and also approximately because I’m ignoring bottom: the String and Email types are isomorphic; they contain the same inhabitants, for any useful definition of “same”. The only substantive difference between the preceding String and Email is the name of the data constructor (call Email an AbergrackleFoozyWatzit, and what has changed?). Hence, my previous criticism of newtypes as “programming by name”. By themselves, newtypes don’t really reduce the number of ways my program can go wrong. They just make it a bit harder to go wrong. But any newtype is isomorphic to the value it holds, and it’s trivial to convert between the two. In fact, if my code doesn’t need to convert between the two (either directly or indirectly), then it’s better off generic. That is, if I never need to convert an Email to a String, or a String to an Email, then I should really write the code generically to work with any value (even if that means making data structures or functions more polymorphic). Parametricity provides a massive reduction in the number of ways my program can go wrong. Newtypes, on the other hand, just make it a bit harder to go wrong, by adding one layer of indirection. In this example, as with many newtypes, I’ve created a bad isomorphism. The domain model of an email is not isomorphic to the data model of a string. But by using a newtype, I have implicitly declared that they are isomorphic. Calling a string an email may make me feel better, because of the different name, but fundamentally, with a newtype, it’s still a string, and I’m only ever one more step away from going wrong. In my experience, too many newtypes create an isomorphism between things that, properly modeled, are not isomorphic. Fortunately, there’s a well-worn workaround that lets us get more mileage out of newtypes. Smart Constructors If I define Email in a module, I can make its data constructor private, and export a helper function to construct an Email. Such helper functions are called smart constructors. They can be used to break the natural isomorphisms created by newtyping. An example is shown below: newtype Email = MkEmail String mkEmail :: String -> Maybe Email mkEmail s = ... In this example, I create a smart constructor which does not promise that it can turn every string into an email. It promises only that it might be able to turn a string into an email, by returning a Maybe Email. With the smart constructor approach, I’ve modeled the fact that while every email has a string representation, not every string has an email representation. Going back to my earlier example of passing same-typed parameters to a function, if I use a smart constructor, then while I can still use an email anywhere a string is expected (by converting), I can’t use a string anywhere an email is expected. (Well, ignoring the fromJust abomination!) Smart Constructors, Dumb Data Smart constructors take us one step closer toward modeling data in a type safe fashion. Unfortunately, I still don’t think it’s far enough. With smart constructors, our data model is fundamentally underconstrained, so we patch that up by restricting who can create the data. That’s putting a band-aid on the real problem. Why not just solve the root issue — viz., that our data model is underconstrained? Dumb Constructors, Smart Data The best solution to a great many newtype problems, I believe, is creating a data model where there is a true isomorphism between the entity modeled by our data and the values passed to the data constructor. That is, creating a data model such that there exists no regions in our data’s state space which correspond to invalid states. Email is a simple example, because there are well-defined models for what constitutes a data model, which can be translated into data declarations in straightforward, if tedious fashion. (To some extent, it’s a failure of most languages I know that such specifications cannot be easily translated into code without tedious boilerplate!) When our data declaration precisely fits our data model specification, there’s no need for smart constructors, and no need for newtypes. There’s far fewer ways that code can go wrong, and because our domain model is captured precisely by our data model, we can transform that data model in ways that make semantic sense (e.g. transforming just the name part of an email, since we’re now in the realm of structured data). Summary As I’ve explained in this blog post, I don’t really hate newtypes. I think they’re very useful, and I do use them, because they make it more difficult for my programs to go wrong. Ultimately, however, I think a lot of problems solved with newtypes (modeling coordinates, positions, emails, etc.) are better solved by more precise data modeling. That is, by making our programs stop lying about isomorphisms. Precision may be tedious due to limitations of the languages we work in, but honestly, what’s more tedious than debugging broken code?

I don’t do a lot of plotting in my job, but I recently heard about a website called Plotly that provides a plotting service for anyone’s data. They even have a plotly package for Python (among others)! So in this article we will be learning how to plot with their package. Let’s have some fun making graphs! Getting Started You will need the plotly package to follow along with this article. You can use pip to get the package and install it: pip install plotly Now that you have it installed, you’ll need to go to the Plotly website and create a free account. Once that’s done, you will get an API key. To make things super simple, you can use your username and API key to create a credentials file. Here’s how to do that: import plotly.tools as tls tls.set_credentials_file( username="your_username", api_key="your_api_key") # to get your credentials credentials = tls.get_credentials_file() If you don’t want to save your credentials, then you can also sign in to their service by doing the following: import plotly.plotly as py py.sign_in('your_username','your_api_key') For the purposes of this article, I’m assuming you have created the credentials file. I found that makes interacting with their service a bit easier to use. Creating a Graph Plotly seems to default to a Scatter Plot, so we’ll start with that. I decided to grab some data from a census website. You can download any US state’s population data, along with other pieces of data. In this case, I downloaded a CSV file that contained the population of each county in the state of Iowa. Let’s take a look: import csv import plotly.plotly as py #---------------------------------------------------------------------- def plot_counties(csv_path): """ http://census.ire.org/data/bulkdata.html """ counties = {} county = [] pop = [] counter = 0 with open(csv_path) as csv_handler: reader = csv.reader(csv_handler) for row in reader: if counter == 0: counter += 1 continue county.append(row[8]) pop.append(row[9]) trace = dict(x=county, y=pop) data = [trace] py.plot(data, filename='ia_county_populations') if __name__ == '__main__': csv_path = 'ia_county_pop.csv' plot_counties(csv_path) If you run this code, you should see a graph that looks like this: <br> You can also view the graph here. Anyway, as you can see in the code above, all I did was read the CSV file and extract out the county name and the population. Then I put that data into two different Python lists. Finally I created a dictionary of those lists and then wrapped that dictionary in a list. So you end up with a list that contains a dictionary that contains two lists! To make the Scatter Plot, I passed the data to plotly’s plot method. Converting to a Bar Chart Now let’s see if we can change the ScatterPlot to a Bar Chart. First off, we’ll play around with the plot data. The following was done via the Python interpreter: >>> scatter = py.get_figure('driscollis', '0') >>> print scatter.to_string() Figure( data=Data([ Scatter( x=[u'Adair County', u'Adams County', u'Allamakee County', u'..', ], y=[u'7682', u'4029', u'14330', u'12887', u'6119', u'26076', '..' ] ) ]) ) This shows how we can grab the figure using the username and the plot’s unique number. Then we printed out the data structure. You will note that it doesn’t print out the entire data structure. Now let’s do the actual conversion to a Bar Chart: from plotly.graph_objs import Data, Figure, Layout scatter_data = scatter.get_data() trace_bar = Bar(scatter_data[0]) data = Data([trace_bar]) layout = Layout(title="IA County Populations") fig = Figure(data=data, layout=layout) py.plot(fig, filename='bar_ia_county_pop') This will create a bar chart at the following URL: https://plot.ly/~driscollis/1. Here’s the image of the graph: <br> This code is slightly different than the code we used originally. In this case, we explicitly created a Bar object and passed it the scatter plot’s data. Then we put that data into a Data object. Next we created a Layout object and gave our chart a title. Then we created a Figure object using the data and layout objects. Finally we plotted the bar chart. Saving the Graph to Disk Plotly also allows you to save your graph to your hard drive. You can save it in the following formats: png, svg, jpeg, and pdf. Assuming you still have the Figure object from the previous example handy, you can do the following: py.image.save_as(fig, filename='graph.png') If you want to save using one of the other formats, then just use that format’s extension in the filename. Wrapping Up At this point you should be able to use the plotly package pretty well. There are many other graph types available, so be sure to read Plotly’s documentation thoroughly. They also support streaming graphs. As I understand it, Plotly allows you to create 10 graphs for free. After that you would either have to delete some of your graphs or pay a monthly fee. Additional Reading Plotly Python documentation Plotly User Guide

In the modern world of microservices it's important to provide strict and polyglot clients for your service. It's better if your API is self-documented. One of the best tools for it is Apache Thrift. I want to explain how to use it with my favorite platform for microservices - Spring Boot. All project source code is available on GitHub: https://github.com/bsideup/spring-boot-thrift Project skeleton I will use Gradle to build our application. First, we need our main build.gradle file: buildscript { repositories { jcenter() } dependencies { classpath("org.springframework.boot:spring-boot-gradle-plugin:1.1.8.RELEASE") } } allprojects { repositories { jcenter() } apply plugin:'base' apply plugin: 'idea' } subprojects { apply plugin: 'java' } Nothing special for a Spring Boot project. Then we need a gradle file for thrift protocol modules (we will reuse it in next part): import org.gradle.internal.os.OperatingSystem repositories { ivy { artifactPattern "http://dl.bintray.com/bsideup/thirdparty/[artifact]-[revision](-[classifier]).[ext]" } } buildscript { repositories { jcenter() } dependencies { classpath "ru.trylogic.gradle.plugins:gradle-thrift-plugin:0.1.1" } } apply plugin: ru.trylogic.gradle.thrift.plugins.ThriftPlugin task generateThrift(type : ru.trylogic.gradle.thrift.tasks.ThriftCompileTask) { generator = 'java:beans,hashcode' destinationDir = file("generated-src/main/java") } sourceSets { main { java { srcDir generateThrift.destinationDir } } } clean { delete generateThrift.destinationDir } idea { module { sourceDirs += [file('src/main/thrift'), generateThrift.destinationDir] } } compileJava.dependsOn generateThrift dependencies { def thriftVersion = '0.9.1'; Map platformMapping = [ (OperatingSystem.WINDOWS) : 'win', (OperatingSystem.MAC_OS) : 'osx' ].withDefault { 'nix' } thrift "org.apache.thrift:thrift:$thriftVersion:${platformMapping.get(OperatingSystem.current())}@bin" compile "org.apache.thrift:libthrift:$thriftVersion" compile 'org.slf4j:slf4j-api:1.7.7' } We're using my Thrift plugin for Gradle. Thrift will generate source to the "generated-src/main/java" directory. By default, Thrift uses slf4j v1.5.8, while Spring Boot uses v1.7.7. It will cause an error in runtime when you will run your application, that's why we have to force a slf4j api dependency. Calculator service Let's start with a simple calculator service. It will have 2 modules: protocol and app.We will start with protocol. Your project should look as follows: calculator/ protocol/ src/ main/ thrift/ calculator.thrift build.gradle build.gradle settings.gradle thrift.gradle Where calculator/protocol/build.gradle contains only one line: apply from: rootProject.file('thrift.gradle') Don't forget to put these lines to settings.gradle, otherwise your modules will not be visible to Gradle: include 'calculator:protocol' include 'calculator:app' Calculator protocol Even if you're not familiar with Thrift, its protocol description file (calculator/protocol/src/main/thrift/calculator.thrift) should be very clear to you: namespace cpp com.example.calculator namespace d com.example.calculator namespace java com.example.calculator namespace php com.example.calculator namespace perl com.example.calculator namespace as3 com.example.calculator enum TOperation { ADD = 1, SUBTRACT = 2, MULTIPLY = 3, DIVIDE = 4 } exception TDivisionByZeroException { } service TCalculatorService { i32 calculate(1:i32 num1, 2:i32 num2, 3:TOperation op) throws (1:TDivisionByZeroException divisionByZero); } Here we define TCalculatorService with only one method - calculate. It can throw an exception of type TDivisionByZeroException. Note how many languages we're supporting out of the box (in this example we will use only Java as a target, though) Now run ./gradlew generateThrift, you will get generated Java protocol source in the calculator/protocol/generated-src/main/java/ folder. Calculator application Next, we need to create the service application itself. Just create calculator/app/ folder with the following structure: src/ main/ java/ com/ example/ calculator/ handler/ CalculatorServiceHandler.java service/ CalculatorService.java CalculatorApplication.java build.gradle Our build.gradle file for app module should look like this: apply plugin: 'spring-boot' dependencies { compile project(':calculator:protocol') compile 'org.springframework.boot:spring-boot-starter-web' testCompile 'org.springframework.boot:spring-boot-starter-test' } Here we have a dependency on protocol and typical starters for Spring Boot web app. CalculatorApplication is our main class. In this example I will configure Spring in the same file, but in your apps you should use another config class instead. package com.example.calculator; import com.example.calculator.handler.CalculatorServiceHandler; import org.apache.thrift.protocol.*; import org.apache.thrift.server.TServlet; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.EnableAutoConfiguration; import org.springframework.context.annotation.*; import javax.servlet.Servlet; @Configuration @EnableAutoConfiguration @ComponentScan public class CalculatorApplication { public static void main(String[] args) { SpringApplication.run(CalculatorApplication.class, args); } @Bean public TProtocolFactory tProtocolFactory() { //We will use binary protocol, but it's possible to use JSON and few others as well return new TBinaryProtocol.Factory(); } @Bean public Servlet calculator(TProtocolFactory protocolFactory, CalculatorServiceHandler handler) { return new TServlet(new TCalculatorService.Processor(handler), protocolFactory); } } You may ask why Thrift servlet bean is called "calculator". In Spring Boot, it will register your servlet bean in context of the bean name and our servlet will be available at /calculator/. After that we need a Thrift handler class: package com.example.calculator.handler; import com.example.calculator.*; import com.example.calculator.service.CalculatorService; import org.apache.thrift.TException; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Component; @Component public class CalculatorServiceHandler implements TCalculatorService.Iface { @Autowired CalculatorService calculatorService; @Override public int calculate(int num1, int num2, TOperation op) throws TException { switch(op) { case ADD: return calculatorService.add(num1, num2); case SUBTRACT: return calculatorService.subtract(num1, num2); case MULTIPLY: return calculatorService.multiply(num1, num2); case DIVIDE: try { return calculatorService.divide(num1, num2); } catch(IllegalArgumentException e) { throw new TDivisionByZeroException(); } default: throw new TException("Unknown operation " + op); } } } In this example I want to show you that Thrift handler can be a normal Spring bean and you can inject dependencies in it. Now we need to implement CalculatorService itself: package com.example.calculator.service; import org.springframework.stereotype.Component; @Component public class CalculatorService { public int add(int num1, int num2) { return num1 + num2; } public int subtract(int num1, int num2) { return num1 - num2; } public int multiply(int num1, int num2) { return num1 * num2; } public int divide(int num1, int num2) { if(num2 == 0) { throw new IllegalArgumentException("num2 must not be zero"); } return num1 / num2; } } That's it. Well... almost. We still need to test our service somehow. And it should be an integration test. Usually, even if your application is providing JSON REST API, you still have to implement a client for it. Thrift will do it for you. We don't have to care about it. Also, it will support different protocols. Let's use a generated client in our test: package com.example.calculator; import org.apache.thrift.protocol.*; import org.apache.thrift.transport.THttpClient; import org.apache.thrift.transport.TTransport; import org.junit.*; import org.junit.runner.RunWith; import org.springframework.beans.factory.annotation.*; import org.springframework.boot.test.IntegrationTest; import org.springframework.boot.test.SpringApplicationConfiguration; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; import org.springframework.test.context.web.WebAppConfiguration; import static org.junit.Assert.*; @RunWith(SpringJUnit4ClassRunner.class) @SpringApplicationConfiguration(classes = CalculatorApplication.class) @WebAppConfiguration @IntegrationTest("server.port:0") public class CalculatorApplicationTest { @Autowired protected TProtocolFactory protocolFactory; @Value("${local.server.port}") protected int port; protected TCalculatorService.Client client; @Before public void setUp() throws Exception { TTransport transport = new THttpClient("http://localhost:" + port + "/calculator/"); TProtocol protocol = protocolFactory.getProtocol(transport); client = new TCalculatorService.Client(protocol); } @Test public void testAdd() throws Exception { assertEquals(5, client.calculate(2, 3, TOperation.ADD)); } @Test public void testSubtract() throws Exception { assertEquals(3, client.calculate(5, 2, TOperation.SUBTRACT)); } @Test public void testMultiply() throws Exception { assertEquals(10, client.calculate(5, 2, TOperation.MULTIPLY)); } @Test public void testDivide() throws Exception { assertEquals(2, client.calculate(10, 5, TOperation.DIVIDE)); } @Test(expected = TDivisionByZeroException.class) public void testDivisionByZero() throws Exception { client.calculate(10, 0, TOperation.DIVIDE); } } This test will run your Spring Boot application, bind it to a random port and test it. All client-server communications will be performed in the same way real world clients are. Note how easy to use our service is from the client side. We're just calling methods and catching exceptions.

A hashmap which maintains keys as regular expressions. Any pattern matching the expression will be able to retrieve the same value. Internally it maintains two maps, one containing the regex to value, and another containing matched pattern to regex. Whenever there is a new pattern to 'get', there will be a O(n) search through the compiled regex(s) (which have been 'put' as keys) to find a match. Existing patterns will have constant time lookup through two maps. import java.util.ArrayList; import java.util.Collection; import java.util.HashMap; import java.util.Iterator; import java.util.List; import java.util.Map; import java.util.Set; import java.util.WeakHashMap; import java.util.regex.Pattern; public class RegexHashMap implements Map { private class PatternMatcher { private final String regex; private final Pattern compiled; PatternMatcher(String name) { regex = name; compiled = Pattern.compile(regex); } boolean matched(String string) { if(compiled.matcher(string).matches()) { ref.put(string, regex); return true; } return false; } } /** * Map of input to pattern */ private final Map ref; /** * Map of pattern to value */ private final Map map; /** * Compiled patterns */ private final List matchers; @Override public String toString() { return "RegexHashMap [ref=" + ref + ", map=" + map + "]"; } /** * */ public RegexHashMap() { ref = new WeakHashMap(); map = new HashMap(); matchers = new ArrayList(); } /** * Returns the value to which the specified key pattern is mapped, or null if this map contains no mapping for the key pattern */ @Override public V get(Object weakKey) { if(!ref.containsKey(weakKey)) { for(PatternMatcher matcher : matchers) { if(matcher.matched((String) weakKey)) { break; } } } if(ref.containsKey(weakKey)) { return map.get(ref.get(weakKey)); } return null; } /** * Associates a specified regular expression to a particular value */ @Override public V put(String key, V value) { V v = map.put(key, value); if (v == null) { matchers.add(new PatternMatcher(key)); } return v; } /** * Removes the regular expression key */ @Override public V remove(Object key) { V v = map.remove(key); if(v != null) { for(Iterator iter = matchers.iterator(); iter.hasNext();) { PatternMatcher matcher = iter.next(); if(matcher.regex.equals(key)) { iter.remove(); break; } } for(Iterator> iter = ref.entrySet().iterator(); iter.hasNext();) { Entry entry = iter.next(); if(entry.getValue().equals(key)) { iter.remove(); } } } return v; } /** * Set of view on the regular expression keys */ @Override public Set> entrySet() { return map.entrySet(); } @Override public void putAll(Map m) { for(Entry entry : m.entrySet()) { put(entry.getKey(), entry.getValue()); } } @Override public int size() { return map.size(); } @Override public boolean isEmpty() { return map.isEmpty(); } /** * Returns true if this map contains a mapping for the specified regular expression key. */ @Override public boolean containsKey(Object key) { return map.containsKey(key); } /** * Returns true if this map contains a mapping for the specified regular expression matched pattern. * @param key * @return */ public boolean containsKeyPattern(Object key) { return ref.containsKey(key); } @Override public boolean containsValue(Object value) { return map.containsValue(value); } @Override public void clear() { map.clear(); matchers.clear(); ref.clear(); } /** * Returns a Set view of the regular expression keys contained in this map. */ @Override public Set keySet() { return map.keySet(); } /** * Returns a Set view of the regex matched patterns contained in this map. The set is backed by the map, so changes to the map are reflected in the set, and vice-versa. * @return */ public Set keySetPattern() { return ref.keySet(); } @Override public Collection values() { return map.values(); } /** * Produces a map of patterns to values, based on the regex put in this map * @param patterns * @return */ public Map transform(List patterns) { for(String pattern : patterns) { get(pattern); } Map transformed = new HashMap(); for(Entry entry : ref.entrySet()) { transformed.put(entry.getKey(), map.get(entry.getValue())); } return transformed; } public static void main(String...strings) { RegexHashMap rh = new RegexHashMap(); rh.put("[o|O][s|S][e|E].?[1|2]", "This is a regex match"); rh.put("account", "This is a direct match"); System.out.println(rh); System.out.println("get:ose-1 -> "+rh.get("ose-1")); System.out.println("get:OSE2 -> "+rh.get("OSE2")); System.out.println("get:OSE112 -> "+rh.get("OSE112")); System.out.println("get:ose-2 -> "+rh.get("ose-2")); System.out.println("get:account -> "+rh.get("account")); System.out.println(rh); } }

daniel bryant , simon and i recently had a discussion about how to represent system communication with external apis. the requirement for integration with external apis is now extremely common but it's not immediately obvious how to clearly show them in architectural diagrams. how to represent an external system? the first thing we discussed was what symbol to use for a system supplying an api. traditionally, uml has used the actor (stick man) symbol to represent a "user or any other system that interacts with the subject" (uml superstructure specification, v2.1.2). therefore a system providing an api may look like this: i've found that this symbol tends to confuse those who aren't well versed in uml as most people assume that the actor symbol always represents a *person* rather than a system. sometimes this is stereotyped to make it more obvious e.g. however the symbol is very powerful and tends to overpower the stereotype. therefore i prefer to use a stereotyped box for an external system supplying an api. let's compare two context diagrams using boxes vs stick actors. in which diagram is it more obvious what are systems or people? note that archimate has a specific symbol for application service that can be used to represent an api: (application service notation from the open group's archimate 2.1 specification) an api or the system that supplies it? whatever symbol we choose, what we've done is to show the *system* rather than the actual api. the api is a definition of a service provided by the system in question. how should we provide more details about the api? there are a number of ways we could do this but my preference is to give details of the api on the connector (line connecting two elements/boxes). in c4 the guidelines for a container diagram includes listing protocol information on the connector and an api can be viewed as the layer above the protocol. for example: multiple apis per external system many api providers supply multiple services/apis (i'm not referring to different operations within an api but multiple sets of operations in different apis, which may even use different underlying protocols.) for example a financial marketplace may have apis that do the following: allow a bulk, batch download of static data (such as details of companies listed on a stock market) via xml over http. supply real time, low latency updates of market prices via bespoke messages over udp. allow entry of trades via industry standard fpml over a queuing system. supply a bulk, batch download of trades for end-of-day reconciliation via fpml over http. two of the services use the same protocol (xml over http) but have very different content and use. one of the apis is used to constantly supply information after user subscription (market data) and the last service involves the user supplying all the information with no acknowledgment (although it should reconcile at eod). there are multiple ways of showing this. we could: have a single service element, list the apis on it and have all components linking to it. show each service/api as a separate box and connect the components that use the individual service to the relevant box. show a single service element with multiple connections. each connection is labeled and represents an api. use a port and connector style notation to represent each api from the service provider. provide a key for the ports. use a uml style 'cup and ball' notation to define interfaces and their usage. some examples are below: a single service element and simple description in the above diagram the containers are stating what they are using but contain no information about how to use the apis. we don't know if it is a single api (with different operations) or anything about the mechanisms used to transport the data. this isn't very useful for anyone implementing a solution or resolving operational issues. single, service box with descriptive connectors in this diagram there is a single, service box with descriptive connectors. the above diagram shows all the information so is much more useful as a diagnostic or implementation tool. however it does look quite crowded. services/apis shown as separate boxes here the external system has its services/apis shown as separate boxes. this contains all the information but might be mistaken as defining the internal structure of the external system. we want to show the services it provides but we know nothing about the internal structure. using ports to represent apis in the above diagram the services/apis are shown as 'ports' on the external system and the details have been moved into a separate key/table. this is less likely to be mistaken as showing any internal structure of the external service. (note that i could have also shown outgoing rports from the brokerage system.) uml interfaces this final diagram is using a uml style interface provider and requirer. this is a clean diagram but requires the user to be aware of what the cup and ball means (although i could have explained this in the key). conclusion any of these solutions could be appropriate depending on the complexity of the api set you are trying to represent. i'd suggest starting with a simple representation (i.e. fully labeled connections) and moving to a more complex one if needed but remember to use a key to explain any elements you use!

REST Easy with SQL/NoSQL Integration and CQRS Pattern implementation New demands are being put on IT organizations everyday to deliver agile, high-performance, integrated mobile and web applications. In the meantime, the technology landscape is getting complex everyday with the advent of new technologies like REST, NoSQL, Cloud while existing technologies like SOAP and SQL still rule everyday work. Rather than taking religious side of the debate, NoSQL can successfully co-exist with SQL in this ‘polyglot’ of data storage and formats. However, this integration also adds another layer of complexity both in architecture and implementation. This document offers a guide on how some of the relatively newer technologies like REST can help bridge the gap between SQL and NoSQL with an example of a well known pattern called CQRS. This document is organized as follows: Introduction to SQL development process NoSQL Do I have to choose between SQL and NoSQL? CQRS Pattern How to implement CQRS pattern using REST services Introduction to SQL development process Developers have been using SQL Databases for decades to build and deliver enterprise business applications. The process of creating tables, attributes,and relationships is second nature for most developers. Data architects think in terms of tables and columns and navigate relationships for data. The basic concepts of delivery and transformation takes place at the web server level which means the server developer is reading and ‘binding’ to the tables and mapping attributes to a REST response. Application development lifecycle meant changes to the database schema first, followed by the bindings, then internal schema mapping, and finally the SOAP or JSON services, and eventually the client code. This all costs the project time and money. It also means that the ‘code’ (pick your language here) and the business logic would also need to be modified to handle the changes to the model. NoSQL NoSQL is gaining supporters among many SQL shops for various reasons including: Low cost Ability to handle unstructured dataa Scalability Performance The first thing database folks notice is that there is no schema. These document style storage engines can handle huge volumes of structured, semi-structured, and unstructured data. The very nature of schema-less documents allows change to a document structure without having to go through the formal change management process (or data architect). The other major difference is that NoSQL (no-schema) also means no joins or relationships. The document itself contains the embedded information by design. So an order entry would contain the customer with all the orders and line items for each order in a single document. There are many different NoSQL vendors (popular NoSQL databases include MongoDB, Casandra) that are being used for BI and Analytics (read-only) purposes. We are also seeing many customers starting to use NoSQL for auditing, logging, and archival transactions. Do I have to choose between SQL and NoSQL? The purpose of this article is to not get into the religious debate about whether to use SQL or NoSQL. Bottom line is both have their place and are suited for certain type of data – SQL for structured data and NoSQL for unstructured data. So why not have the capability to mix and match this data depending on the application. This can be done by creating a single REST API across both SQL and NoSQL databases. Why a single REST API? The answer is simple – the new agile and mobile world demands this ‘mashup’ of data into a document style JSON response. CQRS (Command Query Responsibility Segmentation) Pattern There are many design patterns for delivery of high performance RESTful services but the one that stands out was described in an article written by Martin Fowler, one of the software industry veterans. He described the pattern called CQRS that is more relevant today in a ‘polyglot’ of servers, data, services, and connections. “We may want to look at the information in a different way to the record store, perhaps collapsing multiple records into one, or forming virtual records by combining information for different places. On the update side we may find validation rules that only allow certain combinations of data to be stored, or may even infer data to be stored that’s different from that we provide.” – Martin Fowler 2011 In this design pattern, the REST API requests (GET) return documents from multiple sources (e.g. mashups). In the update process, the data is subject to business logic derivations, validations, event processing, and database transactions. This data may then be pushed back into the NoSQL using asynchronous events. With the wide-spread adoption of NoSQL databases like MongoDB and schema-less, high capacity data store; most developers are challenged with providing security, business logic, event handling, and integration to other systems. MongoDB; one the popular NoSQL databases and SQL databases share many similar concepts. However the MongoDB programming language itself is very different from the SQL we all know. How to implement CQRS pattern using a RESTFul Architecture A REST server should meet certain requirements to support the CQRS pattern. The server should run on-premise or in the cloud and appears to the mobile and web developer as an HTTP endpoint. The server architecture should implement the following: Connections and Mapping necessary for SQL and NoSQL connectivity and API services needed to create and return GET, PUT, POST, and DELETE REST responses Security Business Logic Connections and Mapping There are two main approaches to creating REST Servers and APIs for SQL and NoSQL databases: Open source frameworks like Apache Tomcat, Spring/Hibernate Commercial framework like Espresso Logic Open source Frameworks Using various open source frameworks like Tomcat, Spring/Hibernate, Node.js, JDBC and MongoDB drivers, a REST server can be created, but we would still be left with the following tasks: Creation and mapping of the necessary SQL objects Create a REST server container and configurations Create Jersey/Jackson classes and annotations Create and define REST API for tables, views, and procedures Hand write validation, event and business logic Handle persistence, optimistic locking, transaction paging Adding identity management and security by roles Now we can start down the same path to connect to MongoDB and write code to connect, select, and return data in JSON and then create the REST calls to merge these two different document styles into a single RESTful endpoint. This is a lot of work for a development team to manage and control and frankly pretty boring and repetitive and is better done by a well designed framework Commercial Frameworks Many commercial frameworks may take care of this complexity without the need to do extensive programming. Here is an example from Espresso Logic and how it handles this complexity with a point and click interface: Running REST server in the cloud or on-premise Connections to external SQL databases Object mapping to tables, views, and procedures Automatic creation of RESTful endpoints from model Reactive business rules and rich event model Integrated role-based security and authentication services. Point-and-click document API creation for SQL and MongoDB endpoints In the example below, the editor shows an SQL (customersTransactions) joined with archived details from MongoDB (archivedTransactions). The MongoDB document for each customer may include transaction details, check images, customer service notes and other relevant account information. This new mashup becomes a single REST call that can be published to mobile and web application developer. Security Security is an important part of building and delivery of RESTful services which can be broken down into two parts; authentication and access control. Authentication Before allowing anyone access to corporate data you want to use the existing corporate identity management (some call this authentication services) to capture and validate the user. This identity management service is based on using existing corporate standards such as LDAP, Windows AD, SQL Database. Role-based Access Control Each user may be assigned one or more corporate roles and these roles are then assigned specific access privileges to each resource (e.g. READ, INSERT, UPDATE, and DELETE). Role-based access should also be able to restrict permissions to specific rows and columns of the API (e.g. only sales reps can see their own orders or a manager can see and change his department salaries but cannot change his own). This restriction should be applied regardless of how or where the API is used or called. Remember, the SQL database already provides some level of security and access which must be considered when designing and delivering new front-end services to internal and external users. Business Logic for REST When data is updated to a REST Server several things need to happen. First, the authentication and access control should determine if this is a valid request and if the user has rights to the endpoint. In addition, the server may need to de-alias REST attributes back to the actual SQL column names. In a full featured business logic server, there should be a series of events and business rules to perform various calculations, validations, and fire other events on dependent tables. Finally, the entire multi-table transaction is written back to the SQL database in a single transaction. Updates are then sent asynchronously to MongoDB as part of the commit event (after the SQL transaction has completed). Conclusion In the real-world of API services, the demand for more complex document style RESTful services is a requirement. That is, the ability to create ‘mashups’ of data from multiple tables, NoSQL collections, and other external systems is a large part of this new design pattern. In addition, the ability to alias attribute names and formats from these source fields has become critical for partners and customers systems. Using REST with the CQRS pattern to blend MongoDB and SQL seamlessly to your existing data will become a major part of your future mobile strategy. To implement these REST services, one can use open source tools and spend a lot of time or select a right commercial framework. This framework should support cloud or on-premise connectivity, security, API integration, as well as business logic. This will make the design and delivery of new application services more rapid and agile in the heterogeneous world of information.

Spring provides a great out of the box support for caching expensive method calls. The caching abstraction is covered in a great detail here. My objective here is to cover one of the newer cache implementations that Spring now provides with 4.0+ version of the framework - using Google Guava Cache In brief, consider a service which has a few slow methods: public class DummyBookService implements BookService { @Override public Book loadBook(String isbn) { // Slow method 1. } @Override public List loadBookByAuthor(String author) { // Slow method 2 } } With Spring Caching abstraction, repeated calls with the same parameter can be sped up by an annotation on the method along these lines - here the result of loadBook is being cached in to a "book" cache and listing of books cached into another "books" cache: public class DummyBookService implements BookService { @Override @Cacheable("book") public Book loadBook(String isbn) { // slow response time.. } @Override @Cacheable("books") public List loadBookByAuthor(String author) { // Slow listing } } Now, Caching abstraction support requires a CacheManager to be available which is responsible for managing the underlying caches to store the cached results, with the new Guava Cache support the CacheManager is along these lines: @Bean public CacheManager cacheManager() { return new GuavaCacheManager("books", "book"); } Google Guava Cache provides a rich API to be able to pre-load the cache, set eviction duration based on last access or created time, set the size of the cache etc, if the cache is to be customized then a guava CacheBuilder can be passed to the CacheManager for this customization: @Bean public CacheManager cacheManager() { GuavaCacheManager guavaCacheManager = new GuavaCacheManager(); guavaCacheManager.setCacheBuilder(CacheBuilder.newBuilder().expireAfterAccess(30, TimeUnit.MINUTES)); return guavaCacheManager; } This works well if all the caches have a similar configuration, what if the caches need to be configured differently - for eg. in the sample above, I may want the "book" cache to never expire but the "books" cache to have an expiration of 30 mins, then the GuavaCacheManager abstraction does not work well, instead a better solution is actually to use a SimpleCacheManager which provides a more direct way to get to the cache and can be configured this way: @Bean public CacheManager cacheManager() { SimpleCacheManager simpleCacheManager = new SimpleCacheManager(); GuavaCache cache1 = new GuavaCache("book", CacheBuilder.newBuilder().build()); GuavaCache cache2 = new GuavaCache("books", CacheBuilder.newBuilder() .expireAfterAccess(30, TimeUnit.MINUTES) .build()); simpleCacheManager.setCaches(Arrays.asList(cache1, cache2)); return simpleCacheManager; } This approach works very nicely, if required certain caches can be configured to be backed by a different caching engines itself, say a simple hashmap, some by Guava or EhCache some by distributed caches like Gemfire.

As memory gets cheaper and cheaper, our applications can keep more data readily available in main memory, or even all as in case of in-memory databases. To make real use of the growing heap memory, appropriate data structures must be used. Interesting enough, there seem to be no specialized implementations for lists - by far the most used collection. This article introduces BigList, a list designed for handling large collections where large means that all data still fit completely in the heap memory. The article will show the special requirements for handling large collections, how BigList is implemented and how it compares to other list implementations. 1. Requirements What are the special requirements we need to handle large collections efficiently? Memory: Sparing use of memory: The list should need little memory for its own implementation so memory can be used for storing application data. Specialized versions for primitives: It must be possible to store common primitives like ints in a memory saving way. Avoid copying large data blocks: If the list grows or shrinks, only a small part of the data must be copied around, as this operation becomes expensive and needs the same amount of memory again. Data sharing: copying collections is a frequent operation which should be efficiently possible even if the collection is large. An efficient implementation requires some sort of data sharing as copying all elements is per se a costly operation. Performance: Good performance for normal operations like reading, storing, adding or removing single elements. Great performance for bulk operations like adding or removing multiple elements. Predictable overhead of operations, so similar operations should need a similar amount of time without excessive worst case scenarios. If an implementation does not offer these features, some operations will not only be slow for really large collections, but will becomse just not feasible because memory or CPU usage will be too exhaustive. Introduction to BigList BigList is a member of the Brownies Collections library which also includes GapList, the fastest list implementation known. GapList is a drop-in replacement for ArrayList, LinkedList, or ArrayDequeue and offers fast access by index and fast insertion/removal at the beginning and at the end at the same time. GapList however has not been designed to cope with large collections, so adding or removing elements can make it necessary to copy a lot of elements around which will lead to performance problems. Also copying a large collection becomes an expensive operation, both in term of time and memory consumption. It will simply not be possible to make a copy of a large collections if not the same amount of memory is available a second time. And this is a common operation as you often want to return a copy of an internal list through your API which has no reference the original list. BigList addresses both problems. The first problem is solved by storing the collection elements in fixed size blocks. Add or remove operations are then implemented to use only data from one block. The copying problem is solved by maintaining a reference count on the fixed size blocks which allows to implement a copy-on-write approach. For efficient access to the fixed size blocks, they are maintained in a specialized tree structure. 2. BigList Details Each BigList instance stores the following information: Elements are stored in in fixed-size blocks A single block is implemented as GapList with a reference count for sharing All blocks are maintained in a tree for fast access Access information for the current block is cached for better performance The following illustration shows these details for two instances of BigList which share one block. 2.1 Use of Blocks Elements are stored in in fixed-size blocks with a default block size of 1000. Where this default may look pretty small, it is most of the time a good choice because it guarantees that write operation only need to move few elements. Read operations will profit from locality of reference by using the currently cached block to be fast. It is however possible to specify the block size for each created BigList instance. All blocks except the first are allocated with this fixed size and will not grow or shrink. The first block will grow to the specified block size to save memory for small lists. If a block has reached its maximum size and more data must be stored within, the block needs to be split up in two blocks before more elements can be stored. If elements are added to the head or tail of the list, the block will only be filled up to a threshold of 95%. This allows inserts into the block without the immediate need for split operations. To save memory, blocks are also merged. This happens automatically if two adjacent blocks are both filled less than 35% after a remove operation. 2.2 Locality of Reference For each operation on BigList, the affected block must be determined first. As predicted by locality of reference, most of the time the affected block will be the same as for the last operation. The implementation of BigList has therefore been designed to profit from locality of reference which makes common operations like iterating over a list very efficient. Instead of always traversing the block tree to determine the block needed for an operation, lower and upper index of the last used block are cached. So if the next operation happens near to the previous one, the same block can be used again without need to traverse the tree. 2.3 Reference Counting To support a copy-on-write approach, BigList stores a reference count for each fixed size blocks indicating whether this block is private or shared. Initially all lists are private having a reference count of 0, so that modification are allowed. If a list is copied, the reference count is incremented which prohibits further modifications. Before a modification then can be made, the block must be copied decrementing the block's reference count and setting the reference count of the copy to 0. The reference count of a block is then decremented by the finalizer of BigList. 3. Benchmarks To prove the excellence of BigList in time and memory consumption, we compare it with some other List implementations. And here are the nominees: Type Library Description BigList brownie-collections List optimized for storing large number of elements. Elements stored in fixed size blocks which are maintained in a tree. GapList brownie-collections Fastest list implementation known. Fast access by index and fast insertion/removal at end and at beginning. ArrayList JDK Maintains elements in a single array. Fast access by index, fast insertion/removal at end, but slow at beginning. LinkedList JDK Elements stored using a linked list. Slow access by index. Memory overhead for each element stored. TreeList commons-collections Elements stored in a tree. All operations are not really fast, but there are no very slow operations. Memory overhead for each element stored. FastTable javolution Elements stored in a "fractal"-like data structure. Good performance and use of memory. However no bulk operations and collection does not shrink. 3.1 Handling Objects In the first part of the benchmark, we compare memory consumption and performance of the different list implementations. Let's first have a look at the memory consumption. The following table shows the bytes used to hold a list with 1'000'000 null elements: BigList GapList ArrayList LinkedList TreeList FastTable 32 bit 4'298'466 4'021'296 4'861'992 8'000'028 18'000'028 4'142'892 64 bit 8'544'254 8'042'552 9'723'964 16'000'044 26'000'044 8'222'988 We can see that BigList, GapList, ArrayList, and FastTable only add small overhead to the stored elements, where as Linkedlist needs twice the memory and TreeList even more. Now to the performance. Here are the results of 9 benchmarks which have been run for each of the 6 candidates with JDK 8 in a 32 bit Windows environment and a list of 1'000'000 elements: The result table can be read as follows: the fastest candidate for each test has a relative performance indicator of 1 the value for the other candidates indicate how many times they have been slower, so a factor of 3 means that this implementation was 3 times slower than the best one The different factor are colored like this: 
factor 1: green (best)
 factor <5: blue (good) 
factor <25: yellow (moderate) 
factor >25: red (poor) If we look the benchmark result, we can see that the performance of BigList is best for all expect two benchmarks. The only moderate result is produces in getting elements in a totally random order. This could be expected as there is no locality of reference which can be exploited, so for each access, the block tree must be traversed to find the correct block. Luckily this is a rare use case in real applications. And the benchmark "Get local" shows that performance is back to good as soon as elements next to each other must be retrieved - as it is the case if we iterate over a range. 3.2 Handling Primitives In the second part of the benchmark, we want see how big the savings are if we use a data structure specialized for storing primitives compared to strong wrapped objects. For this reason, we compare IntBigList and BigList. The following table shows memory needed to store 1'000'000 integer values: BigList IntBigList 32 bit 16'298'454 4'534'840 64 bit 28'544'234 4'570'432 Obviously it is easy to save a lot of memory. In a 32 bit environment, IntBigList just needs 25% percent of memory, in a 64 bit environment only 14%! These figures become plausible if you recall that a simple object needs 8 bytes in a 32 bit, but already 16 bytes in a 64 bit environment, where as a primitive integer value always only needs 4 bytes. The measurable performance gain is not so impressive, it is something below 10% for simple get operations and something above 10% for add and remove operations. These numbers show that the JVM is impressively fast in creating wrapper objects and boxing and unboxing primitive values. We must however also consider that each created object will need to be garbage collected once and therefore adds to the total load of the JVM. 4. Summary BigList is a scalable high-performance list for storing large collections. Its design guarantees that all operations will be predictable and efficient both in term of performance and memory consumption, even copying large collections is tremendous fast. Benchmarks haven proven this and shown that BigList outperform other known list implementations. The library also offers specialized implementations for primitive types like IntBigList which save much memory and provide superior performance. BigList for handling objects and the specializations for handling primitives are part of the Brownies Collections library and can be downloaded from http://www.magicwerk.org/collections.

if someone asked you to develop a rest api on the jvm, which frameworks would you use? i was recently tasked with such a project. my client asked me to implement a rest api to ingest requests from a 3rd party. the project entailed consuming xml requests, storing the data in a database, then exposing the data to internal application with a json endpoint. finally, it would allow taking in a json request and turning it into an xml request back to the 3rd party. with the recent release of apache camel 2.14 and my success using it , i started by copying my apache camel / cxf / spring boot project and trimming it down to the bare essentials. i whipped together a simple hello world service using camel and spring mvc. i also integrated swagger into both. both implementations were pretty easy to create ( sample code ), but i decided to use spring mvc. my reasons were simple: its rest support was more mature, i knew it well, and spring mvc test makes it easy to test apis. camel's swagger support without web.xml as part of the aforementioned spike, i learned out how to configure camel's rest and swagger support using spring's javaconfig and no web.xml. i made this into a sample project and put it on github as camel-rest-swagger . this article shows how i built a rest api with java 8, spring boot/mvc, jaxb and spring data (jpa and rest components). i stumbled a few times while developing this project, but figured out how to get over all the hurdles. i hope this helps the team that's now maintaining this project (my last day was friday) and those that are trying to do something similar. xml to java with jaxb the data we needed to ingest from a 3rd party was based on the ncpdp standards. as a member, we were able to download a number of xsd files, put them in our project and generate java classes to handle the incoming/outgoing requests. i used the maven-jaxb2-plugin to generate the java classes. org.jvnet.jaxb2.maven2 maven-jaxb2-plugin 0.8.3 generate -xtostring -xequals -xhashcode -xcopyable org.jvnet.jaxb2_commons jaxb2-basics 0.6.4 src/main/resources/schemas/ncpdp the first error i ran into was about a property already being defined. [info] --- maven-jaxb2-plugin:0.8.3:generate (default) @ spring-app --- [error] error while parsing schema(s).location [ file:/users/mraible/dev/spring-app/src/main/resources/schemas/ncpdp/structures.xsd{1811,48}]. com.sun.istack.saxparseexception2; systemid: file:/users/mraible/dev/spring-app/src/main/resources/schemas/ncpdp/structures.xsd; linenumber: 1811; columnnumber: 48; property "multipletimingmodifierandtimingandduration" is already defined. use to resolve this conflict. at com.sun.tools.xjc.errorreceiver.error(errorreceiver.java:86) i was able to workaround this by upgrading to maven-jaxb2-plugin version 0.9.1. i created a controller and stubbed out a response with hard-coded data. i confirmed the incoming xml-to-java marshalling worked by testing with a sample request provided by our 3rd party customer. i started with a curl command, because it was easy to use and could be run by anyone with the file and curl installed. curl -x post -h 'accept: application/xml' -h 'content-type: application/xml' \ --data-binary @sample-request.xml http://localhost:8080/api/message -v this is when i ran into another stumbling block: the response wasn't getting marshalled back to xml correctly. after some research, i found out this was caused by the lack of @xmlrootelement annotations on my generated classes. i posted a question to stack overflow titled returning jaxb-generated elements from spring boot controller . after banging my head against the wall for a couple days, i figured out the solution . i created a bindings.xjb file in the same directory as my schemas. this causes jaxb to generate @xmlrootelement on classes. to add namespaces prefixes to the returned xml, i had to modify the maven-jaxb2-plugin to add a couple arguments. -extension -xnamespace-prefix and add a dependency: org.jvnet.jaxb2_commons jaxb2-namespace-prefix 1.1 then i modified bindings.xjb to include the package and prefix settings. i also moved into a global setting. i eventually had to add prefixes for all schemas and their packages. i learned how to add prefixes from the namespace-prefix plugins page . finally, i customized the code-generation process to generate joda-time's datetime instead of the default xmlgregoriancalendar . this involved a couple custom xmladapters and a couple additional lines in bindings.xjb . you can see the adapters and bindings.xjb with all necessary prefixes in this gist . nicolas fränkel's customize your jaxb bindings was a great resource for making all this work. i wrote a test to prove that the ingest api worked as desired. @runwith(springjunit4classrunner.class) @springapplicationconfiguration(classes = application.class) @webappconfiguration @dirtiescontext(classmode = dirtiescontext.classmode.after_class) public class initiaterequestcontrollertest { @inject private initiaterequestcontroller controller; private mockmvc mockmvc; @before public void setup() { mockitoannotations.initmocks(this); this.mockmvc = mockmvcbuilders.standalonesetup(controller).build(); } @test public void testgetnotallowedonmessagesapi() throws exception { mockmvc.perform(get("/api/initiate") .accept(mediatype.application_xml)) .andexpect(status().ismethodnotallowed()); } @test public void testpostpainitiationrequest() throws exception { string request = new scanner(new classpathresource("sample-request.xml").getfile()).usedelimiter("\\z").next(); mockmvc.perform(post("/api/initiate") .accept(mediatype.application_xml) .contenttype(mediatype.application_xml) .content(request)) .andexpect(status().isok()) .andexpect(content().contenttype(mediatype.application_xml)) .andexpect(xpath("/message/header/to").string("3rdparty")) .andexpect(xpath("/message/header/sendersoftware/sendersoftwaredeveloper").string("hid")) .andexpect(xpath("/message/body/status/code").string("010")); } } spring data for jpa and rest with jaxb out of the way, i turned to creating an internal api that could be used by another application. spring data was fresh in my mind after reading about it last summer. i created classes for entities i wanted to persist, using lombok's @data to reduce boilerplate. i read the accessing data with jpa guide, created a couple repositories and wrote some tests to prove they worked. i ran into an issue trying to persist joda's datetime and found jadira provided a solution. i added its usertype.core as a dependency to my pom.xml: org.jadira.usertype usertype.core 3.2.0.ga ... and annotated datetime variables accordingly. @column(name = "last_modified", nullable = false) @type(type="org.jadira.usertype.dateandtime.joda.persistentdatetime") private datetime lastmodified; with jpa working, i turned to exposing rest endpoints. i used accessing jpa data with rest as a guide and was looking at json in my browser in a matter of minutes. i was surprised to see a "profile" service listed next to mine, and posted a question to the spring boot team. oliver gierke provided an excellent answer . swagger spring mvc's integration for swagger has greatly improved since i last wrote about it . now you can enable it with a @enableswagger annotation. below is the swaggerconfig class i used to configure swagger and read properties from application.yml . @configuration @enableswagger public class swaggerconfig implements environmentaware { public static final string default_include_pattern = "/api/.*"; private relaxedpropertyresolver propertyresolver; @override public void setenvironment(environment environment) { this.propertyresolver = new relaxedpropertyresolver(environment, "swagger."); } /** * swagger spring mvc configuration */ @bean public swaggerspringmvcplugin swaggerspringmvcplugin(springswaggerconfig springswaggerconfig) { return new swaggerspringmvcplugin(springswaggerconfig) .apiinfo(apiinfo()) .genericmodelsubstitutes(responseentity.class) .includepatterns(default_include_pattern); } /** * api info as it appears on the swagger-ui page */ private apiinfo apiinfo() { return new apiinfo( propertyresolver.getproperty("title"), propertyresolver.getproperty("description"), propertyresolver.getproperty("termsofserviceurl"), propertyresolver.getproperty("contact"), propertyresolver.getproperty("license"), propertyresolver.getproperty("licenseurl")); } } after getting swagger to work, i discovered that endpoints published with @repositoryrestresource aren't picked up by swagger. there is an open issue for spring data support in the swagger-springmvc project. liquibase integration i configured this project to use h2 in development and postgresql in production. i used spring profiles to do this and copied xml/yaml (for maven and application*.yml files) from a previously created jhipster project. next, i needed to create a database. i decided to use liquibase to create tables, rather than hibernate's schema-export. i chose liquibase over flyway based of discussions in the jhipster project . to use liquibase with spring boot is dead simple: add the following dependency to pom.xml, then place changelog files in src/main/resources/db/changelog . org.liquibase liquibase-core i started by using hibernate's schema-export and changing hibernate.ddl-auto to "create-drop" in application-dev.yml . i also commented out the liquibase-core dependency. then i setup a postgresql database and started the app with "mvn spring-boot:run -pprod". i generated the liquibase changelog from an existing schema using the following command (after downloading and installing liquibase). liquibase --driver=org.postgresql.driver --classpath="/users/mraible/.m2/repository/org/postgresql/postgresql/9.3-1102-jdbc41/postgresql-9.3-1102-jdbc41.jar:/users/mraible/snakeyaml-1.11.jar" --changelogfile=/users/mraible/dev/spring-app/src/main/resources/db/changelog/db.changelog-02.yaml --url="jdbc:postgresql://localhost:5432/mydb" --username=user --password=pass generatechangelog i did find one bug - the generatechangelog command generates too many constraints in version 3.2.2 . i was able to fix this by manually editing the generated yaml file. tip: if you want to drop all tables in your database to verify liquibase creation is working in postgesql, run the following commands: psql -d mydb drop schema public cascade; create schema public; after writing minimal code for spring data and configuring liquibase to create tables/relationships, i relaxed a bit, documented how everything worked and added a loggingfilter . the loggingfilter was handy for viewing api requests and responses. @bean public filterregistrationbean loggingfilter() { loggingfilter filter = new loggingfilter(); filterregistrationbean registrationbean = new filterregistrationbean(); registrationbean.setfilter(filter); registrationbean.seturlpatterns(arrays.aslist("/api/*")); return registrationbean; } accessing api with resttemplate the final step i needed to do was figure out how to access my new and fancy api with resttemplate . at first, i thought it would be easy. then i realized that spring data produces a hal -compliant api, so its content is embedded inside an "_embedded" json key. after much trial and error, i discovered i needed to create a resttemplate with hal and joda-time awareness. @bean public resttemplate resttemplate() { objectmapper mapper = new objectmapper(); mapper.configure(deserializationfeature.fail_on_unknown_properties, false); mapper.registermodule(new jackson2halmodule()); mapper.registermodule(new jodamodule()); mappingjackson2httpmessageconverter converter = new mappingjackson2httpmessageconverter(); converter.setsupportedmediatypes(mediatype.parsemediatypes("application/hal+json")); converter.setobjectmapper(mapper); stringhttpmessageconverter stringconverter = new stringhttpmessageconverter(); stringconverter.setsupportedmediatypes(mediatype.parsemediatypes("application/xml")); list> converters = new arraylist<>(); converters.add(converter); converters.add(stringconverter); return new resttemplate(converters); } the jodamodule was provided by the following dependency: com.fasterxml.jackson.datatype jackson-datatype-joda with the configuration complete, i was able to write a messagesapiitest integration test that posts a request and retrieves it using the api. the api was secured using basic authentication, so it took me a bit to figure out how to make that work with resttemplate. willie wheeler's basic authentication with spring resttemplate was a big help. @runwith(springjunit4classrunner.class) @contextconfiguration(classes = integrationtestconfig.class) public class messagesapiitest { private final static log log = logfactory.getlog(messagesapiitest.class); @value("http://${app.host}/api/initiate") private string initiateapi; @value("http://${app.host}/api/messages") private string messagesapi; @value("${app.host}") private string host; @inject private resttemplate resttemplate; @before public void setup() throws exception { string request = new scanner(new classpathresource("sample-request.xml").getfile()).usedelimiter("\\z").next(); responseentity response = resttemplate.exchange(gettesturl(initiateapi), httpmethod.post, getbasicauthheaders(request), org.ncpdp.schema.transport.message.class, collections.emptymap()); assertequals(httpstatus.ok, response.getstatuscode()); } @test public void testgetmessages() { httpentity request = getbasicauthheaders(null); responseentity> result = resttemplate.exchange(gettesturl(messagesapi), httpmethod.get, request, new parameterizedtypereference>() {}); httpstatus status = result.getstatuscode(); collection messages = result.getbody().getcontent(); log.debug("messages found: " + messages.size()); assertequals(httpstatus.ok, status); for (message message : messages) { log.debug("message.id: " + message.getid()); log.debug("message.datecreated: " + message.getdatecreated()); } } private httpentity getbasicauthheaders(string body) { string plaincreds = "user:pass"; byte[] plaincredsbytes = plaincreds.getbytes(); byte[] base64credsbytes = base64.encodebase64(plaincredsbytes); string base64creds = new string(base64credsbytes); httpheaders headers = new httpheaders(); headers.add("authorization", "basic " + base64creds); headers.add("content-type", "application/xml"); if (body == null) { return new httpentity<>(headers); } else { return new httpentity<>(body, headers); } } } to get spring data to populate the message id, i created a custom restconfig class to expose it. i learned how to do this from tommy ziegler . /** * used to expose ids for resources. */ @configuration public class restconfig extends repositoryrestmvcconfiguration { @override protected void configurerepositoryrestconfiguration(repositoryrestconfiguration config) { config.exposeidsfor(message.class); config.setbaseuri("/api"); } } summary this article explains how i built a rest api using jaxb, spring boot, spring data and liquibase. it was relatively easy to build, but required some tricks to access it with spring's resttemplate. figuring out how to customize jaxb's code generation was also essential to make things work. i started developing the project with spring boot 1.1.7, but upgraded to 1.2.0.m2 after i found it supported log4j2 and configuring spring data rest's base uri in application.yml. when i handed the project off to my client last week, it was using 1.2.0.build-snapshot because of a bug when running in tomcat . this was an enjoyable project to work on. i especially liked how easy spring data makes it to expose jpa entities in an api. spring boot made things easy to configure once again and liquibase seems like a nice tool for database migrations. if someone asked me to develop a rest api on the jvm, which frameworks would i use? spring boot, spring data, jackson, joda-time, lombok and liquibase. these frameworks worked really well for me on this particular project.

In Parts 1 and 2 we have covered a number of common issues people run into when managing a sharded MongoDB cluster. In this final post of the series we will cover a subtle, but important distinction in terms of balancing a sharded cluster as well as an interesting limitation that can be worked around relatively easily, but is nonetheless surprising when it comes up. 6. Chunk balancing != data balancing != traffic balancing The balancer in a sharded cluster cares about just one thing: Are chunks for a given collection evenly balanced across all shards? If they are not, then it will take steps to rectify that imbalance. This all sounds perfectly logical, and even with extra complexity like tagging involved the logic is pretty straight forward. If we assume that all chunks are equal, then we can rest assured that our data is being evenly balanced across all the shards in our cluster and rest easy at night. Although that is sometimes, perhaps even frequently, the case it is not always true - chunks are not always equal. There can be massive “jumbo” chunks that exceed the maximum chunk size (64MiB), completely empty chunks and everything in between. Let’s use an example from our first pitfall, the monotonically increasing shard key. For our example, we have picked just such a key to shard on (date), and up until this point we have had just one shard and had not sharded the collection. We are about to add a second shard to our cluster and so we enable sharding on the collection and do the necessary admin work to add the new shard into the cluster. Once the collection is enabled for sharding, the first shard contains all the newly minted chunks. Let’s represent them in a simplified table of 10 chunks. This is not representative of a real data set, but it will do for illustrative purposes: Table 1 - Initial Chunk Layout Now we add our second shard. The balancer will kick in and attempt to distribute the chunks evenly. It will do this by moving the lowest range chunks to the new shard until the counts are identical. Once it is finished balancing, our table now looks like this: Table 2 - Balanced Chunk Layout That looks pretty good at the moment, but lets imagine that more recent chunks are more likely to have more activity (updates say) than older chunks. Adding the traffic share estimates for each chunk shows that shard1 is taking far more traffic (72%) than shard2 (28%) despite the chunks seeming balanced overall based on the approximate size. Hence, chunk balancing is not equal to traffic balancing. Using that same example, let’s add another wrinkle - periodic deletion of old data. Every 3 months we run a job to delete any data older than 12 months. Let’s look at the impact of that on our table after we run it for the first time (assuming the first run happens on July 1st 2015). Table 3 - Post-Delete Chunk Layout The distribution of data is now completely skewed toward shard1 - shard2 is in fact empty! However, the balancer is completely unaware of this imbalance - the chunk count has remained the same the entire time, and as far as it is concerned the system is in a steady state. With no data on shard2, our traffic imbalance as seen above will be even worse, and we have essentially negated the benefit of having a second shard for this collection. Possible Mitigation Strategies If data and traffic balance are important, select an appropriate shard key Move chunks manually to address the imbalances - swap “hot” chunks for “cool” chunks, empty chunks for larger chunks 7. Waiting too long to shard a collection (collection too large) This is not very common, but when it falls on your shoulders, it can be quite challenging to solve. There is a maximum data size for a collection when when it is initially split which is a function of the chunk size and data size as noted on the limits page. If your collection contains less than 256GiB of data, then there will be no issue. If the collection size exceeds 256GiB but is less than 400GiB, then MongoDB may be able to do an initial split without any special measures being taken. Otherwise, with larger initial data sizes and the default settings, the initial split will fail. It is worth noting that once split the collection may grow as needed and without any real limitations as long as you can continue to add shards as data size grows. Possible Mitigation Strategies Since the limit is dictated by the chunk size and the data size, and assuming there is not much to be done about the data size, then the remaining variable is the chunk size. This is adjustable (default is 64MiB) and can be raised in order to let a large collection split initially and then reduced once that has been completed. The required chunk size increase will depend on the actual data size. However, this is relatively easy to work out - simply divide your data size by 256GB and then multiply that figure by 64MiB (and round up if it is not a nice even number). As an example, let’s consider a 4TiB collection: 4TiB divided by 256GiB = 16 64MiB x 16 = 1024MiB Hence, set the max chunk size to 1024MiB, then perform the initial sharding of the collection, and then finally reduce the chunk size back to 64MiB using the same procedure. . Thanks for reading through the Sharding Pitfall series! If you want to learn more about managing MongoDB deployments at scale, sign up for my online education course, MongoDB Advanced Deployment and Operations. Planning for scale? No problem: MongoDB is here to help. Get a preview of what it’s like to work with MongoDB’s Technical Services Team. Give us some details on your deployment and we can set you up with an expert who can provide detailed guidance on all aspects of scaling with MongoDB, based on our experience with hundreds of deployments.