It has been a while since my first post about the ESP8266 (see “Cheap and Simple WiFi with ESP8266 for the FRDM Board“). The ESP8266 is a new inexpensive ($4.50) WiFi module which makes it easy to connect to the network or internet. Finally this week-end I have found the time to write up a tutorial: how to implement a WiFi web server for the ESP8266 WiFi module and the Freescale FRDM-KL25Z board: WSP8266 Web Server FRDM-KL25Z with ESP8266 WiFi Module Outline In this tutorial I’m using a Freescale FRDM-KL25Z board as a web server, using theESP8266 board. The ESP8266 is a ‘less than $4.5′ WiFi board getting more and more popular as an IoT board. There is even a way to run the ESP8266 standalone (because it has a full processor on that board). However, that development is still in the flux and rather unstable. Instead, I’m using a serial connection to the ESP8266 instead. With this, any small microcontroller can send and receive data from the internet: connect that board to a microcontroller with 3.3V, GND, Tx and Rx, and you have a W-LAN connection! I’m using in this tutorial Eclipse with GNU/GDB with Processor Expert, but with the steps in this tutorial you should be able to use any other toolchain too. As things might change in the future with different firmware on the ESP8266: the firmware I’m having on the board is version 00160901. Board Connections Since my first post on the ESP8266 I have cleaned up the wiring. The pins are as below for the ESP8266: ESP8266 Pins Because the ESP8266 can take > 200 mA, I’m using a 5-to-3.3V DC-DC converter. I measured around 70 to 90 mA, so it is not (yet) really needed, but I wanted to use it to protect to board. The ESP8266 Rx and Tx are connected to the microcontroller Tx and Rx pins. A general frustration point for the ESP8266 module is the connection oft the remaining pins. What worked for me is to connect CH_PD to 3.3V and leaving RST,GPIO0 and GPIO2 unconnected/floating. Wiring Setup with FRDM-KL25Z and ESP8266 Communication Protocol I recommend to use a logic analyzer to verify the communication between the ESP8266 and the microcontroller. My module communicates with 115200, but I see reports that other modules (other firmware) can use a different baud. The module uses an AT command send. The simplest command is to send “AT\r\n” and it responds with “AT\r\r\n\r\nOK\r\n”: AT Command Sent to ESP8266 In this tutorial I’m using a command line shell (see “A Shell for the Freedom KL25Z Board“) to have a manual mode to send commands to the module. More about this later. Project Creation You can use my project and source files available on GitHub (see link at the end of this article). Or create your own project. My project is using the Kinetis Design Studio and for the FRDM-KL25Z board (MKL25Z128VLK4). I have created a project for Processor Expert, as I’m using several components of it: Processor Expert Project For the project I have several files added: ESP8266 Project in Eclipse With the following source files: Application.c/.h: This runs the application and web server program ESP8266.c/.h: Driver for the ESP8266 Events.c/.h: Processor Expert event hooks main.c: main entry point Shell.c/.h: command line interface Sources Project and Source files are available on GitHub here: https://github.com/ErichStyger/mcuoneclipse/tree/master/Examples/KDS/FRDM-KL25Z/FRDM-KL25Z_ESP8266 Please check the latest source files on GitHub. At the time of writing this article, I’m using the following: Shell.h is the interface to command line shell: view source print? 01./* 02. * Shell.h 03. * 04. * Author: Erich Styger 05. */ 06. 07.#ifndef SHELL_H_ 08.#define SHELL_H_ 09. 10./*! 11. * \brief Shell parse routine 12. */ 13.voidSHELL_Parse(void); 14. 15./*! 16. * \brief Shell initialization 17. */ 18.voidSHELL_Init(void); 19. 20.#endif /* SHELL_H_ */ Shell.c implements the application part of the shell: view source print? 01./* 02. * Shell.c 03. * 04. * Author: Erich Styger 05. */ 06. 07.#include "Shell.h" 08.#include "CLS1.h" 09.#include "ESP8266.h" 10. 11./* table with shell parser/handler */ 12.staticconstCLS1_ParseCommandCallback CmdParserTable[] = 13.{ 14. CLS1_ParseCommand, 15. ESP_ParseCommand, 16. NULL /* sentinel */ 17.}; 18. 19.staticunsigned charlocalConsole_buf[48]; /* buffer for command line */ 20. 21.voidSHELL_Parse(void) { 22. (void)CLS1_ReadAndParseWithCommandTable(localConsole_buf, sizeof(localConsole_buf), CLS1_GetStdio(), CmdParserTable); 23.} 24. 25.voidSHELL_Init(void) { 26. localConsole_buf[0] = '\0'; /* initialize buffer */ 27.} ESP8266.h is the interface to the WiFi module: view source print? 001./* 002. * ESP8266.h 003. * 004. * Author: Erich Styger 005. */ 006. 007.#ifndef ESP8266_H_ 008.#define ESP8266_H_ 009. 010.#include "CLS1.h" 011. 012.#define ESP_DEFAULT_TIMEOUT_MS (100) 013. /*!< Default timeout value in milliseconds */ 014. 015./*! 016. * \brief Command line parser routine 017. * \param cmd Pointer to command line string 018. * \param handled Return value if command has been handled 019. * \param io Standard Shell I/O handler 020. * \return Error code, ERR_OK for no failure 021. */ 022.uint8_t ESP_ParseCommand(constunsigned char*cmd, bool *handled, constCLS1_StdIOType *io); 023. 024./*! 025. * \brief Send a string to th ESP8266 module 026. * \param str String to send, "\r\n" will be appended 027. * \param io Shell I/O handler or NULL if not used 028. * \return Error code, ERR_OK for no failure 029. */ 030.uint8_t ESP_SendStr(constuint8_t *str, CLS1_ConstStdIOType *io); 031. 032./*! 033. * \brief Used to send an AT command to the ESP8266 module 034. * \param cmd Command string to send 035. * \param rxBuf Buffer for the response, can be NULL 036. * \param rxBufSize Size of response buffer 037. * \param expectedTailStr Expected response from the module, can be NULL 038. * \param msTimeout Timeout time in milliseconds 039. * \param io Shell I/O handler or NULL if not used 040. * \return Error code, ERR_OK for no failure 041. */ 042.uint8_t ESP_SendATCommand(uint8_t *cmd, uint8_t *rxBuf, size_t rxBufSize, uint8_t *expectedTailStr, uint16_t msTimeout, constCLS1_StdIOType *io); 043. 044./*! 045. * \brief Read from the serial line from the module until a sentinel char is received 046. * \param buf 047. * \param bufSize 048. * \param sentinelChar 049. * \param timeoutMs Timeout time in milliseconds 050. * \return Error code, ERR_OK for no failure 051. */ 052.uint8_t ESP_ReadCharsUntil(uint8_t *buf, size_t bufSize, uint8_t sentinelChar, uint16_t timeoutMs); 053. 054./*! 055. * \brief Sends an AT command to test the connection 056. * \return Error code, ERR_OK for no failure 057. */ 058.uint8_t ESP_TestAT(void); 059. 060./*! 061. * \brief Restarts the ESP8266 module 062. * \param io Shell I/O handler or NULL if not used 063. * \param timeoutMs Timeout time in milliseconds 064. * \return Error code, ERR_OK for no failure 065. */ 066.uint8_t ESP_Restart(constCLS1_StdIOType *io, uint16_t timeoutMs); 067. 068./*! 069. * \brief Set the current mode of the module 070. * \param mode Where is 1=Sta, 2=AP or 3=both 071. * \return Error code, ERR_OK for no failure 072. */ 073.uint8_t ESP_SelectMode(uint8_t mode); 074. 075./*! 076. * \Brief returns the firmware version string 077. * \param fwBuf Buffer for the string 078. * \param fwBufSize Size of buffer in bytes 079. * \return Error code, ERR_OK for no failure 080. */ 081.uint8_t ESP_GetFirmwareVersionString(uint8_t *fwBuf, size_t fwBufSize); 082. 083./*! 084. * \brief Join an access point. 085. * \param ssid SSID of access point 086. * \param pwd Password of access point 087. * \param nofRetries Number of connection retries 088. * \param io Shell I/O or NULL if not used 089. * \return Error code, ERR_OK for no failure 090. */ 091.uint8_t ESP_JoinAP(constuint8_t *ssid, constuint8_t *pwd, intnofRetries, CLS1_ConstStdIOType *io); 092. 093./*! 094. * \brief Scans for an IPD message sent by the module 095. * \param msgBuf Pointer to the message buffer where to store the message 096. * \param msgBufSize Size of message buffer 097. * \param ch_id Pointer to where to store the channel/id 098. * \param size Pointer where to store the size of the message 099. * \param isGet TRUE if it is a GET message, FALSE for a POST message 100. * \param timeoutMs Error code, ERR_OK for no failure 101. * \param io 102. * \return Error code, ERR_OK for no failure 103. */ 104.uint8_t ESP_GetIPD(uint8_t *msgBuf, size_t msgBufSize, uint8_t *ch_id, uint16_t *size, bool *isGet, uint16_t timeoutMs, constCLS1_StdIOType *io); 105. 106./*! 107. * \brief Closes a connection 108. * \param channel Channel ID 109. * \param io Error code, ERR_OK for no failure 110. * \param timeoutMs Error code, ERR_OK for no failure 111. * \return Error code, ERR_OK for no failure 112. */ 113.uint8_t ESP_CloseConnection(uint8_t channel, constCLS1_StdIOType *io, uint16_t timeoutMs); 114. 115./*! 116. * \brief Used to determine if the web server is running or not. 117. * \return TRUE if web server has beens started 118. */ 119.bool ESP_IsServerOn(void); 120. 121./*! 122. * \brief Driver initialization 123. */ 124.voidESP_Init(void); 125. 126./*! 127. * \brief Driver de-initialization 128. */ 129.voidESP_Deinit(void); 130. 131.#endif /* ESP8266_H_ */ And the ESP8266 driver is in ESP8266.c which implements all the low level SPI access functions, the functional implementation and a command line shell interface: view source print? 001./* 002. * ESP8266.c 003. * 004. * Author: Erich Styger 005. */ 006. 007.#include "ESP8266.h" 008.#include "Shell.h" 009.#include "UTIL1.h" 010.#include "CLS1.h" 011.#include "AS2.h" 012.#include "WAIT1.h" 013. 014.staticbool ESP_WebServerIsOn = FALSE; 015. 016.bool ESP_IsServerOn(void) { 017. returnESP_WebServerIsOn; 018.} 019. 020.staticvoidSend(unsigned char*str) { 021. while(*str!='\0') { 022. AS2_SendChar(*str); 023. str++; 024. } 025.} 026. 027.staticvoidSkipNewLines(constunsigned char**p) { 028. while(**p=='\n'|| **p=='\r') { 029. (*p)++; /* skip new lines */ 030. } 031.} 032. 033.uint8_t ESP_ReadCharsUntil(uint8_t *buf, size_t bufSize, uint8_t sentinelChar, uint16_t timeoutMs) { 034. uint8_t ch; 035. uint8_t res = ERR_OK; 036. 037. if(bufSize<=1) { 038. returnERR_OVERRUN; /* buffer to small */ 039. } 040. buf[0] = '\0'; buf[bufSize-1] = '\0'; /* always terminate */ 041. bufSize--; 042. for(;;) { /* breaks */ 043. if(bufSize==0) { 044. res = ERR_OVERRUN; 045. break; 046. } 047. if(AS2_GetCharsInRxBuf()>0) { 048. (void)AS2_RecvChar(&ch); 049. *buf = ch; 050. buf++; 051. bufSize--; 052. if(ch==sentinelChar) { 053. *buf = '\0'; /* terminate string */ 054. break; /* sentinel found */ 055. } 056. } else{ 057. if(timeoutMs>10) { 058. WAIT1_WaitOSms(5); 059. timeoutMs -= 5; 060. } else{ 061. res = ERR_NOTAVAIL; /* timeout */ 062. break; 063. } 064. } 065. } 066. returnres; 067.} 068. 069.staticuint8_t RxResponse(unsigned char*rxBuf, size_t rxBufLength, unsigned char*expectedTail, uint16_t msTimeout) { 070. unsigned charch; 071. uint8_t res = ERR_OK; 072. unsigned char*p; 073. 074. if(rxBufLength < sizeof("x\r\n")) { 075. returnERR_OVERFLOW; /* not enough space in buffer */ 076. } 077. p = rxBuf; 078. p[0] = '\0'; 079. for(;;) { /* breaks */ 080. if(msTimeout == 0) { 081. break; /* will decide outside of loop if it is a timeout. */ 082. } elseif(rxBufLength == 0) { 083. res = ERR_OVERFLOW; /* not enough space in buffer */ 084. break; 085. } elseif(AS2_GetCharsInRxBuf() > 0) { 086.#if0 087. if(AS2_RecvChar(&ch) != ERR_OK) { 088. res = ERR_RXEMPTY; 089. break; 090. } 091.#else 092. /* might get an overrun OVERRUN_ERR error here? Ignoring error for now */ 093. (void)AS2_RecvChar(&ch); 094.#endif 095. *p++ = ch; 096. *p = '\0'; /* always terminate */ 097. rxBufLength--; 098. } elseif(expectedTail!=NULL && expectedTail[0]!='\0' 099. && UTIL1_strtailcmp(rxBuf, expectedTail) == 0) { 100. break; /* finished */ 101. } else{ 102. WAIT1_WaitOSms(1); 103. msTimeout--; 104. } 105. } /* for */ 106. if(msTimeout==0) { /* timeout! */ 107. if(expectedTail[0] != '\0'/* timeout, and we expected something: an error for sure */ 108. || rxBuf[0] == '\0'/* timeout, did not know what to expect, but received nothing? There has to be a response. */ 109. ) 110. { 111. res = ERR_FAULT; 112. } 113. } 114. returnres; 115.} 116. 117.uint8_t ESP_SendATCommand(uint8_t *cmd, uint8_t *rxBuf, size_t rxBufSize, uint8_t *expectedTailStr, uint16_t msTimeout, constCLS1_StdIOType *io) { 118. uint16_t snt; 119. uint8_t res; 120. 121. if(rxBuf!=NULL) { 122. rxBuf[0] = '\0'; 123. } 124. if(io!=NULL) { 125. CLS1_SendStr("sending>>:\r\n", io->stdOut); 126. CLS1_SendStr(cmd, io->stdOut); 127. } 128. if(AS2_SendBlock(cmd, (uint16_t)UTIL1_strlen((char*)cmd), &snt) != ERR_OK) { 129. returnERR_FAILED; 130. } 131. if(rxBuf!=NULL) { 132. res = RxResponse(rxBuf, rxBufSize, expectedTailStr, msTimeout); 133. if(io!=NULL) { 134. CLS1_SendStr("received<<:\r\n", io->stdOut); 135. CLS1_SendStr(rxBuf, io->stdOut); 136. } 137. } 138. returnres; 139.} 140. 141.uint8_t ESP_TestAT(void) { 142. /* AT */ 143. uint8_t rxBuf[sizeof("AT\r\r\n\r\nOK\r\n")]; 144. uint8_t res; 145. 146. res = ESP_SendATCommand("AT\r\n", rxBuf, sizeof(rxBuf), "AT\r\r\n\r\nOK\r\n", ESP_DEFAULT_TIMEOUT_MS, NULL); 147. returnres; 148.} 149. 150.uint8_t ESP_Restart(constCLS1_StdIOType *io, uint16_t timeoutMs) { 151. /* AT+RST */ 152. uint8_t rxBuf[sizeof("AT+RST\r\r\n\r\nOK\r\n")]; 153. uint8_t res; 154. uint8_t buf[64]; 155. 156. AS2_ClearRxBuf(); /* clear buffer */ 157. res = ESP_SendATCommand("AT+RST\r\n", rxBuf, sizeof(rxBuf), "AT+RST\r\r\n\r\nOK\r\n", ESP_DEFAULT_TIMEOUT_MS, io); 158. if(res==ERR_OK) { 159. for(;;) { 160. ESP_ReadCharsUntil(buf, sizeof(buf), '\n', 1000); 161. if(io!=NULL) { 162. CLS1_SendStr(buf, io->stdOut); 163. } 164. if(UTIL1_strncmp(buf, "ready", sizeof("ready")-1)==0) { /* wait until ready message from module */ 165. break; /* module has restarted */ 166. } 167. } 168. } 169. AS2_ClearRxBuf(); /* clear buffer */ 170. returnres; 171.} 172. 173.uint8_t ESP_CloseConnection(uint8_t channel, constCLS1_StdIOType *io, uint16_t timeoutMs) { 174. /* AT+CIPCLOSE= */ 175. uint8_t res; 176. uint8_t cmd[64]; 177. 178. UTIL1_strcpy(cmd, sizeof(cmd), "AT+CIPCLOSE="); 179. UTIL1_strcatNum8u(cmd, sizeof(cmd), channel); 180. UTIL1_strcat(cmd, sizeof(cmd), "\r\n"); 181. res = ESP_SendATCommand(cmd, NULL, 0, "Unlink\r\n", timeoutMs, io); 182. returnres; 183.} 184. 185.uint8_t ESP_SetNumberOfConnections(uint8_t nof, constCLS1_StdIOType *io, uint16_t timeoutMs) { 186. /* AT+CIPMUX=, 0: single connection, 1: multiple connections */ 187. uint8_t res; 188. uint8_t cmd[sizeof("AT+CIPMUX=12\r\n")]; 189. uint8_t rxBuf[sizeof("AT+CIPMUX=12\r\n\r\nOK\r\n")+10]; 190. 191. if(nof>1) { /* only 0 and 1 allowed */ 192. if(io!=NULL) { 193. CLS1_SendStr("Wrong number of connection parameter!\r\n", io->stdErr); 194. } 195. returnERR_FAILED; 196. } 197. UTIL1_strcpy(cmd, sizeof(cmd), "AT+CIPMUX="); 198. UTIL1_strcatNum8u(cmd, sizeof(cmd), nof); 199. UTIL1_strcat(cmd, sizeof(cmd), "\r\n"); 200. res = ESP_SendATCommand(cmd, rxBuf, sizeof(rxBuf), "OK\r\n", timeoutMs, io); 201. returnres; 202.} 203. 204.uint8_t ESP_SetServer(bool startIt, uint16_t port, constCLS1_StdIOType *io, uint16_t timeoutMs) { 205. /* AT+CIPSERVER=,, where : 0: stop, 1: start */ 206. uint8_t res; 207. uint8_t cmd[sizeof("AT+CIPSERVER=1,80\r\n\r\nOK\r\n")+sizeof("no change")]; 208. uint8_t rxBuf[sizeof("AT+CIPSERVER=1,80\r\n\r\nOK\r\n")+sizeof("no change")]; 209. 210. UTIL1_strcpy(cmd, sizeof(cmd), "AT+CIPSERVER="); 211. if(startIt) { 212. UTIL1_strcat(cmd, sizeof(cmd), "1,"); 213. } else{ 214. UTIL1_strcat(cmd, sizeof(cmd), "0,"); 215. } 216. UTIL1_strcatNum16u(cmd, sizeof(cmd), port); 217. UTIL1_strcat(cmd, sizeof(cmd), "\r\n"); 218. res = ESP_SendATCommand(cmd, rxBuf, sizeof(rxBuf), "OK\r\n", timeoutMs, io); 219. if(res!=ERR_OK) { /* accept "no change" too */ 220. UTIL1_strcpy(cmd, sizeof(cmd), "AT+CIPSERVER="); 221. if(startIt) { 222. UTIL1_strcat(cmd, sizeof(cmd), "1,"); 223. } else{ 224. UTIL1_strcat(cmd, sizeof(cmd), "0,"); 225. } 226. UTIL1_strcatNum16u(cmd, sizeof(cmd), port); 227. UTIL1_strcat(cmd, sizeof(cmd), "\r\r\nno change\r\n"); 228. if(UTIL1_strcmp(rxBuf, cmd)==0) { 229. res = ERR_OK; 230. } 231. } 232. returnres; 233.} 234. 235.uint8_t ESP_SelectMode(uint8_t mode) { 236. /* AT+CWMODE=, where is 1=Sta, 2=AP or 3=both */ 237. uint8_t txBuf[sizeof("AT+CWMODE=x\r\n")]; 238. uint8_t rxBuf[sizeof("AT+CWMODE=x\r\r\nno change\r\n")]; 239. uint8_t expected[sizeof("AT+CWMODE=x\r\r\nno change\r\n")]; 240. uint8_t res; 241. 242. if(mode<1|| mode>3) { 243. returnERR_RANGE; /* only 1, 2 or 3 */ 244. } 245. UTIL1_strcpy(txBuf, sizeof(txBuf), "AT+CWMODE="); 246. UTIL1_strcatNum16u(txBuf, sizeof(txBuf), mode); 247. UTIL1_strcat(txBuf, sizeof(txBuf), "\r\n"); 248. UTIL1_strcpy(expected, sizeof(expected), "AT+CWMODE="); 249. UTIL1_strcatNum16u(expected, sizeof(expected), mode); 250. UTIL1_strcat(expected, sizeof(expected), "\r\r\n\n"); 251. res = ESP_SendATCommand(txBuf, rxBuf, sizeof(rxBuf), expected, ESP_DEFAULT_TIMEOUT_MS, NULL); 252. if(res!=ERR_OK) { 253. /* answer could be as well "AT+CWMODE=x\r\r\nno change\r\n"!! */ 254. UTIL1_strcpy(txBuf, sizeof(txBuf), "AT+CWMODE="); 255. UTIL1_strcatNum16u(txBuf, sizeof(txBuf), mode); 256. UTIL1_strcat(txBuf, sizeof(txBuf), "\r\n"); 257. UTIL1_strcpy(expected, sizeof(expected), "AT+CWMODE="); 258. UTIL1_strcatNum16u(expected, sizeof(expected), mode); 259. UTIL1_strcat(expected, sizeof(expected), "\r\r\nno change\r\n"); 260. if(UTIL1_strcmp(rxBuf, expected)==0) { 261. res = ERR_OK; 262. } 263. } 264. returnres; 265.} 266. 267.uint8_t ESP_GetFirmwareVersionString(uint8_t *fwBuf, size_t fwBufSize) { 268. /* AT+GMR */ 269. uint8_t rxBuf[32]; 270. uint8_t res; 271. constunsigned char*p; 272. 273. res = ESP_SendATCommand("AT+GMR\r\n", rxBuf, sizeof(rxBuf), "\r\n\r\nOK\r\n", ESP_DEFAULT_TIMEOUT_MS, NULL); 274. if(res!=ERR_OK) { 275. if(UTIL1_strtailcmp(rxBuf, "\r\n\r\nOK\r\n")) { 276. res = ERR_OK; 277. } 278. } 279. if(res==ERR_OK) { 280. if(UTIL1_strncmp(rxBuf, "AT+GMR\r\r\n", sizeof("AT+GMR\r\r\n")-1)==0) { /* check for beginning of response */ 281. UTIL1_strCutTail(rxBuf, "\r\n\r\nOK\r\n"); /* cut tailing response */ 282. p = rxBuf+sizeof("AT+GMR\r\r\n")-1; /* skip beginning */ 283. UTIL1_strcpy(fwBuf, fwBufSize, p); /* copy firmware information string */ 284. } else{ 285. res = ERR_FAILED; 286. } 287. } 288. if(res!=ERR_OK) { 289. UTIL1_strcpy(fwBuf, fwBufSize, "ERROR"); /* default error */ 290. } 291. returnres; 292.} 293. 294.uint8_t ESP_GetIPAddrString(uint8_t *ipBuf, size_t ipBufSize) { 295. /* AT+CIFSR */ 296. uint8_t rxBuf[32]; 297. uint8_t res; 298. constunsigned char*p; 299. 300. res = ESP_SendATCommand("AT+CIFSR\r\n", rxBuf, sizeof(rxBuf), NULL, ESP_DEFAULT_TIMEOUT_MS, NULL); 301. if(res!=ERR_OK) { 302. if(UTIL1_strtailcmp(rxBuf, "\r\n")) { 303. res = ERR_OK; 304. } 305. } 306. if(res==ERR_OK) { 307. if(UTIL1_strncmp(rxBuf, "AT+CIFSR\r\r\n", sizeof("AT+CIFSR\r\r\n")-1)==0) { /* check for beginning of response */ 308. UTIL1_strCutTail(rxBuf, "\r\n"); /* cut tailing response */ 309. p = rxBuf+sizeof("AT+CIFSR\r\r\n")-1; /* skip beginning */ 310. SkipNewLines(&p); 311. UTIL1_strcpy(ipBuf, ipBufSize, p); /* copy IP information string */ 312. } else{ 313. res = ERR_FAILED; 314. } 315. } 316. if(res!=ERR_OK) { 317. UTIL1_strcpy(ipBuf, ipBufSize, "ERROR"); 318. } 319. returnres; 320.} 321. 322.uint8_t ESP_GetModeString(uint8_t *buf, size_t bufSize) { 323. /* AT+CWMODE? */ 324. uint8_t rxBuf[32]; 325. uint8_t res; 326. constunsigned char*p; 327. 328. res = ESP_SendATCommand("AT+CWMODE?\r\n", rxBuf, sizeof(rxBuf), "\r\n\r\nOK\r\n", ESP_DEFAULT_TIMEOUT_MS, NULL); 329. if(res==ERR_OK) { 330. if(UTIL1_strncmp(rxBuf, "AT+CWMODE?\r\r\n+CWMODE:", sizeof("AT+CWMODE?\r\r\n+CWMODE:")-1)==0) { /* check for beginning of response */ 331. UTIL1_strCutTail(rxBuf, "\r\n\r\nOK\r\n"); /* cut tailing response */ 332. p = rxBuf+sizeof("AT+CWMODE?\r\r\n+CWMODE:")-1; /* skip beginning */ 333. UTIL1_strcpy(buf, bufSize, p); /* copy information string */ 334. } else{ 335. res = ERR_FAILED; 336. } 337. } 338. if(res!=ERR_OK) { 339. UTIL1_strcpy(buf, bufSize, "ERROR"); 340. } 341. returnres; 342.} 343. 344.uint8_t ESP_GetCIPMUXString(uint8_t *cipmuxBuf, size_t cipmuxBufSize) { 345. /* AT+CIPMUX? */ 346. uint8_t rxBuf[32]; 347. uint8_t res; 348. constunsigned char*p; 349. 350. res = ESP_SendATCommand("AT+CIPMUX?\r\n", rxBuf, sizeof(rxBuf), "\r\n\r\nOK\r\n", ESP_DEFAULT_TIMEOUT_MS, NULL); 351. if(res==ERR_OK) { 352. if(UTIL1_strncmp(rxBuf, "AT+CIPMUX?\r\r\n+CIPMUX:", sizeof("AT+CIPMUX?\r\r\n+CIPMUX:")-1)==0) { /* check for beginning of response */ 353. UTIL1_strCutTail(rxBuf, "\r\n\r\nOK\r\n"); /* cut tailing response */ 354. p = rxBuf+sizeof("AT+CIPMUX?\r\r\n+CIPMUX:")-1; /* skip beginning */ 355. UTIL1_strcpy(cipmuxBuf, cipmuxBufSize, p); /* copy IP information string */ 356. } else{ 357. res = ERR_FAILED; 358. } 359. } 360. if(res!=ERR_OK) { 361. UTIL1_strcpy(cipmuxBuf, cipmuxBufSize, "ERROR"); 362. } 363. returnres; 364.} 365. 366.uint8_t ESP_GetConnectedAPString(uint8_t *apBuf, size_t apBufSize) { 367. /* AT+CWJAP? */ 368. uint8_t rxBuf[48]; 369. uint8_t res; 370. constunsigned char*p; 371. 372. res = ESP_SendATCommand("AT+CWJAP?\r\n", rxBuf, sizeof(rxBuf), "\r\n\r\nOK\r\n", ESP_DEFAULT_TIMEOUT_MS, NULL); 373. if(res==ERR_OK) { 374. if(UTIL1_strncmp(rxBuf, "AT+CWJAP?\r\r\n+CWJAP:\"", sizeof("AT+CWJAP?\r\r\n+CWJAP:\"")-1)==0) { /* check for beginning of response */ 375. UTIL1_strCutTail(rxBuf, "\"\r\n\r\nOK\r\n"); /* cut tailing response */ 376. p = rxBuf+sizeof("AT+CWJAP?\r\r\n+CWJAP:\"")-1; /* skip beginning */ 377. UTIL1_strcpy(apBuf, apBufSize, p); /* copy IP information string */ 378. } else{ 379. res = ERR_FAILED; 380. } 381. } 382. if(res!=ERR_OK) { 383. UTIL1_strcpy(apBuf, apBufSize, "ERROR"); 384. } 385. returnres; 386. 387.} 388. 389.staticuint8_t JoinAccessPoint(constuint8_t *ssid, constuint8_t *pwd, CLS1_ConstStdIOType *io) { 390. /* AT+CWJAP="","" */ 391. uint8_t txBuf[48]; 392. uint8_t rxBuf[64]; 393. uint8_t expected[48]; 394. 395. UTIL1_strcpy(txBuf, sizeof(txBuf), "AT+CWJAP=\""); 396. UTIL1_strcat(txBuf, sizeof(txBuf), ssid); 397. UTIL1_strcat(txBuf, sizeof(txBuf), "\",\""); 398. UTIL1_strcat(txBuf, sizeof(txBuf), pwd); 399. UTIL1_strcat(txBuf, sizeof(txBuf), "\"\r\n"); 400. 401. UTIL1_strcpy(expected, sizeof(expected), "AT+CWJAP=\""); 402. UTIL1_strcat(expected, sizeof(expected), ssid); 403. UTIL1_strcat(expected, sizeof(expected), "\",\""); 404. UTIL1_strcat(expected, sizeof(expected), pwd); 405. UTIL1_strcat(expected, sizeof(expected), "\"\r\r\n\r\nOK\r\n"); 406. 407. returnESP_SendATCommand(txBuf, rxBuf, sizeof(rxBuf), expected, ESP_DEFAULT_TIMEOUT_MS, io); 408.} 409. 410.uint8_t ESP_JoinAP(constuint8_t *ssid, constuint8_t *pwd, intnofRetries, CLS1_ConstStdIOType *io) { 411. uint8_t buf[32]; 412. uint8_t res; 413. 414. do{ 415. res = JoinAccessPoint(ssid, pwd, io); 416. if(res==ERR_OK) { 417. break; 418. } 419. WAIT1_WaitOSms(1000); 420. nofRetries--; 421. } while(nofRetries>0); 422. returnres; 423.} 424. 425.staticuint8_t ReadIntoIPDBuffer(uint8_t *buf, size_t bufSize, uint8_t *p, uint16_t msgSize, uint16_t msTimeout, constCLS1_StdIOType *io) { 426. uint8_t ch; 427. size_t nofInBuf; 428. inttimeout; 429. 430. nofInBuf = p-buf; 431. bufSize -= nofInBuf; /* take into account what we already have in buffer */ 432. timeout = msTimeout; 433. while(msgSize>0&& bufSize>0) { 434. if(AS2_GetCharsInRxBuf()>0) { 435. (void)AS2_RecvChar(&ch); 436. *p = ch; 437. if(io!=NULL) { /* copy on console */ 438. io->stdOut(ch); 439. } 440. p++; 441. *p = '\0'; /* terminate */ 442. nofInBuf++; msgSize--; bufSize--; 443. } else{ 444. /* check in case we recveive less characters than expected, happens for POST? */ 445. if(nofInBuf>6&& UTIL1_strncmp(&p[-6], "\r\nOK\r\n", sizeof("\r\nOK\r\n")-1)==0) { 446. break; 447. } else{ 448. timeout -= 10; 449. WAIT1_WaitOSms(10); 450. if(timeout<0) { 451. returnERR_BUSY; 452. } 453. } 454. } 455. } 456. returnERR_OK; 457.} 458. 459.uint8_t ESP_GetIPD(uint8_t *msgBuf, size_t msgBufSize, uint8_t *ch_id, uint16_t *size, bool *isGet, uint16_t timeoutMs, constCLS1_StdIOType *io) { 460. /* scan e.g. for 461. * +IPD,0,404:POST / HTTP/1.1 462. * and return ch_id (0), size (404) 463. */ 464. uint8_t res = ERR_OK; 465. constuint8_t *p; 466. bool isIPD = FALSE; 467. uint8_t cmd[24], rxBuf[48]; 468. uint16_t ipdSize; 469. 470. *ch_id = 0; *size = 0; *isGet = FALSE; /* init */ 471. for(;;) { /* breaks */ 472. res = ESP_ReadCharsUntil(msgBuf, msgBufSize, '\n', timeoutMs); 473. if(res!=ERR_OK) { 474. break; /* timeout */ 475. } 476. if(res==ERR_OK) { /* line read */ 477. if(io!=NULL) { 478. CLS1_SendStr(msgBuf, io->stdOut); /* copy on console */ 479. } 480. isIPD = UTIL1_strncmp(msgBuf, "+IPD,", sizeof("+IPD,")-1)==0; 481. if(isIPD) { /* start of IPD message */ 482. p = msgBuf+sizeof("+IPD,")-1; 483. if(UTIL1_ScanDecimal8uNumber(&p, ch_id)!=ERR_OK) { 484. if(io!=NULL) { 485. CLS1_SendStr("ERR: wrong channel?\r\n", io->stdErr); /* error on console */ 486. } 487. res = ERR_FAILED; 488. break; 489. } 490. if(*p!=',') { 491. res = ERR_FAILED; 492. break; 493. } 494. p++; /* skip comma */ 495. if(UTIL1_ScanDecimal16uNumber(&p, size)!=ERR_OK) { 496. if(io!=NULL) { 497. CLS1_SendStr("ERR: wrong size?\r\n", io->stdErr); /* error on console */ 498. } 499. res = ERR_FAILED; 500. break; 501. } 502. if(*p!=':') { 503. res = ERR_FAILED; 504. break; 505. } 506. ipdSize = p-msgBuf; /* length of "+IPD,," string */ 507. p++; /* skip ':' */ 508. if(UTIL1_strncmp(p, "GET", sizeof("GET")-1)==0) { 509. *isGet = TRUE; 510. } elseif(UTIL1_strncmp(p, "POST", sizeof("POST")-1)==0) { 511. *isGet = FALSE; 512. } else{ 513. res = ERR_FAILED; 514. } 515. while(*p!='\0') { 516. p++; /* skip to the end */ 517. } 518. /* read the rest of the message */ 519. res = ReadIntoIPDBuffer(msgBuf, msgBufSize, (uint8_t*)p, (*size)-ipdSize, ESP_DEFAULT_TIMEOUT_MS, io); 520. break; 521. } 522. } 523. } 524. returnres; 525.} 526. 527.uint8_t ESP_StartWebServer(constCLS1_StdIOType *io) { 528. uint8_t buf[32]; 529. uint8_t res; 530. 531. res = ESP_SetNumberOfConnections(1, io, ESP_DEFAULT_TIMEOUT_MS); 532. if(res!=ERR_OK) { 533. CLS1_SendStr("ERR: failed to set multiple connections.\r\n", io->stdErr); 534. returnres; 535. } 536. res = ESP_SetServer(TRUE, 80, io, ESP_DEFAULT_TIMEOUT_MS); 537. if(res!=ERR_OK) { 538. CLS1_SendStr("ERR: failed to set server.\r\n", io->stdErr); 539. returnres; 540. } 541. CLS1_SendStr("INFO: Web Server started, waiting for connection on ", io->stdOut); 542. if(ESP_GetIPAddrString(buf, sizeof(buf))==ERR_OK) { 543. CLS1_SendStr(buf, io->stdOut); 544. CLS1_SendStr(":80", io->stdOut); 545. } else{ 546. CLS1_SendStr("(ERROR!)", io->stdOut); 547. } 548. CLS1_SendStr("\r\n", io->stdOut); 549. 550. returnERR_OK; 551.} 552. 553.uint8_t ESP_SendStr(constuint8_t *str, CLS1_ConstStdIOType *io) { 554. uint8_t buf[32]; 555. uint8_t rxBuf[48]; 556. uint8_t res; 557. uint16_t timeoutMs; 558. #define RX_TIMEOUT_MS 3000 559. AS2_TComData ch; 560. 561. UTIL1_strcpy(buf, sizeof(buf), str); 562. UTIL1_strcat(buf, sizeof(buf), "\r\n"); 563. res = ESP_SendATCommand(buf, rxBuf, sizeof(rxBuf), NULL, ESP_DEFAULT_TIMEOUT_MS, io); 564. timeoutMs = 0; 565. while(timeoutMs0) { 569. (void)AS2_RecvChar(&ch); 570. CLS1_SendChar(ch); 571. } 572. } 573. returnERR_OK; 574.} 575. 576.staticuint8_t ESP_PrintHelp(constCLS1_StdIOType *io) { 577. CLS1_SendHelpStr("ESP", "ESP8200 commands\r\n", io->stdOut); 578. CLS1_SendHelpStr(" help|status", "Print help or status information\r\n", io->stdOut); 579. CLS1_SendHelpStr(" send ", "Sends a string to the module\r\n", io->stdOut); 580. CLS1_SendHelpStr(" test", "Sends a test AT command\r\n", io->stdOut); 581. CLS1_SendHelpStr(" restart", "Restart module\r\n", io->stdOut); 582. CLS1_SendHelpStr(" listAP", "List available Access Points\r\n", io->stdOut); 583. CLS1_SendHelpStr(" connectAP \"ssid\",\"pwd\"", "Connect to an Access Point\r\n", io->stdOut); 584. CLS1_SendHelpStr(" server (start|stop)", "Start or stop web server\r\n", io->stdOut); 585. returnERR_OK; 586.} 587. 588.staticuint8_t ESP_PrintStatus(constCLS1_StdIOType *io) { 589. uint8_t buf[48]; 590. 591. CLS1_SendStatusStr("ESP8266", "\r\n", io->stdOut); 592. 593. CLS1_SendStatusStr(" Webserver", ESP_WebServerIsOn?"ON\r\n":"OFF\r\n", io->stdOut); 594. 595. if(ESP_GetFirmwareVersionString(buf, sizeof(buf)) != ERR_OK) { 596. UTIL1_strcpy(buf, sizeof(buf), "FAILED\r\n"); 597. } else{ 598. UTIL1_strcat(buf, sizeof(buf), "\r\n"); 599. } 600. CLS1_SendStatusStr(" AT+GMR", buf, io->stdOut); 601. 602. if(ESP_GetModeString(buf, sizeof(buf)) != ERR_OK) { 603. UTIL1_strcpy(buf, sizeof(buf), "FAILED\r\n"); 604. } else{ 605. if(UTIL1_strcmp(buf, "1")==0) { 606. UTIL1_strcat(buf, sizeof(buf), " (device)"); 607. } elseif(UTIL1_strcmp(buf, "2")==0) { 608. UTIL1_strcat(buf, sizeof(buf), " (AP)"); 609. } elseif(UTIL1_strcmp(buf, "3")==0) { 610. UTIL1_strcat(buf, sizeof(buf), " (device+AP)"); 611. } else{ 612. UTIL1_strcat(buf, sizeof(buf), " (ERROR)"); 613. } 614. UTIL1_strcat(buf, sizeof(buf), "\r\n"); 615. } 616. CLS1_SendStatusStr(" AT+CWMODE?", buf, io->stdOut); 617. 618. if(ESP_GetIPAddrString(buf, sizeof(buf)) != ERR_OK) { 619. UTIL1_strcpy(buf, sizeof(buf), "FAILED\r\n"); 620. } else{ 621. UTIL1_strcat(buf, sizeof(buf), "\r\n"); 622. } 623. CLS1_SendStatusStr(" AT+CIFSR", buf, io->stdOut); 624. 625. if(ESP_GetConnectedAPString(buf, sizeof(buf)) != ERR_OK) { 626. UTIL1_strcpy(buf, sizeof(buf), "FAILED\r\n"); 627. } else{ 628. UTIL1_strcat(buf, sizeof(buf), "\r\n"); 629. } 630. CLS1_SendStatusStr(" AT+CWJAP?", buf, io->stdOut); 631. 632. if(ESP_GetCIPMUXString(buf, sizeof(buf)) != ERR_OK) { 633. UTIL1_strcpy(buf, sizeof(buf), "FAILED\r\n"); 634. } else{ 635. if(UTIL1_strcmp(buf, "0")==0) { 636. UTIL1_strcat(buf, sizeof(buf), " (single connection)"); 637. } elseif(UTIL1_strcmp(buf, "1")==0) { 638. UTIL1_strcat(buf, sizeof(buf), " (multiple connections)"); 639. } else{ 640. UTIL1_strcat(buf, sizeof(buf), " (ERROR)"); 641. } 642. UTIL1_strcat(buf, sizeof(buf), "\r\n"); 643. } 644. CLS1_SendStatusStr(" CIPMUX", buf, io->stdOut); 645. returnERR_OK; 646.} 647. 648.uint8_t ESP_ParseCommand(constunsigned char*cmd, bool *handled, constCLS1_StdIOType *io) { 649. uint32_t val; 650. uint8_t res; 651. constunsigned char*p; 652. uint8_t pwd[24], ssid[24]; 653. 654. if(UTIL1_strcmp((char*)cmd, CLS1_CMD_HELP)==0|| UTIL1_strcmp((char*)cmd, "ESP help")==0) { 655. *handled = TRUE; 656. res = ESP_PrintHelp(io); 657. } elseif(UTIL1_strcmp((char*)cmd, CLS1_CMD_STATUS)==0|| UTIL1_strcmp((char*)cmd, "ESP status")==0) { 658. *handled = TRUE; 659. res = ESP_PrintStatus(io); 660. } elseif(UTIL1_strncmp((char*)cmd, "ESP send ", sizeof("ESP send ")-1)==0) { 661. *handled = TRUE; 662. p = cmd+sizeof("ESP send ")-1; 663. 664. (void)ESP_SendStr(p, io); 665. } elseif(UTIL1_strcmp((char*)cmd, "ESP test")==0) { 666. *handled = TRUE; 667. if(ESP_TestAT()!=ERR_OK) { 668. CLS1_SendStr("TEST failed!\r\n", io->stdErr); 669. res = ERR_FAILED; 670. } else{ 671. CLS1_SendStr("TEST ok!\r\n", io->stdOut); 672. } 673. } elseif(UTIL1_strcmp((char*)cmd, "ESP listAP")==0) { 674. *handled = TRUE; 675. (void)ESP_SendStr("AT+CWLAP", io); 676. /* AT + CWLAP 677. response 678. + CWLAP: , , [, ] 679. OK Or Fails, the return ERROR 680. 0 OPEN 681. 1 WEP 682. 2 WPA_PSK 683. 3 WPA2_PSK 684. 4 WPA_WPA2_PSK 685. string parameter, the access point name 686. signal strength 687. 0: manually connect 1: An automatic connection 688. */ 689. returnERR_OK; 690. } elseif(UTIL1_strncmp((char*)cmd, "ESP connectAP ", sizeof("ESP connectAP ")-1)==0) { 691. *handled = TRUE; 692. p = cmd+sizeof("ESP connectAP ")-1; 693. ssid[0] = '\0'; pwd[0] = '\0'; 694. res = UTIL1_ScanDoubleQuotedString(&p, ssid, sizeof(ssid)); 695. if(res==ERR_OK && *p!='\0'&& *p==',') { 696. p++; /* skip comma */ 697. res = UTIL1_ScanDoubleQuotedString(&p, pwd, sizeof(pwd)); 698. } else{ 699. CLS1_SendStr("Comma expected between strings!\r\n", io->stdErr); 700. res = ERR_FAILED; 701. } 702. if(res==ERR_OK) { 703. res = ESP_JoinAP(ssid, pwd, 3, io); 704. } else{ 705. CLS1_SendStr("Wrong command format!\r\n", io->stdErr); 706. res = ERR_FAILED; 707. } 708. } elseif(UTIL1_strcmp((char*)cmd, "ESP server start")==0) { 709. *handled = TRUE; 710. res = ESP_StartWebServer(io); 711. ESP_WebServerIsOn = res==ERR_OK; 712. } elseif(UTIL1_strcmp((char*)cmd, "ESP server stop")==0) { 713. *handled = TRUE; 714. ESP_WebServerIsOn = FALSE; 715. } elseif(UTIL1_strcmp((char*)cmd, "ESP restart")==0) { 716. *handled = TRUE; 717. ESP_Restart(io, 2000); 718. } 719. returnres; 720.} 721. 722.voidESP_Deinit(void) { 723. /* nothing to do */ 724.} 725. 726.voidESP_Init(void) { 727. AS2_ClearRxBuf(); /* clear buffer */ 728.} The application interface in Application.h is rather short :-): view source print? 01./* 02. * Application.h 03. * 04. * Author: Erich Styger 05. */ 06. 07.#ifndef APPLICATION_H_ 08.#define APPLICATION_H_ 09. 10./*! 11. * \brief Application main routine 12. */ 13.voidAPP_Run(void); 14. 15.#endif /* APPLICATION_H_ */ The main loop of the application is Application.c, along with the application specific web server code. As the SendWebPage function contains HTML code, I’m posting it here separately: view source print? 01.staticuint8_t SendWebPage(uint8_t ch_id, bool ledIsOn, uint8_t temperature, constCLS1_StdIOType *io) { 02. staticuint8_t http[1024]; 03. uint8_t cmd[24], rxBuf[48], expected[48]; 04. uint8_t buf[16]; 05. uint8_t res = ERR_OK; 06. 07. /* construct web page content */ 08. UTIL1_strcpy(http, sizeof(http), (uint8_t*)"HTTP/1.0 200 OK\r\nContent-Type: text/html\r\nPragma: no-cache\r\n\r\n"); 09. UTIL1_strcat(http, sizeof(http), (uint8_t*)"\r\n\r\n"); 10. UTIL1_strcat(http, sizeof(http), (uint8_t*)"\r\n"); 11. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Web Server using ESP8266\r\n"); 12. UTIL1_strcat(http, sizeof(http), (uint8_t*)" 13.\r\n"); 14. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Temp: OC"); 17. if(ledIsOn) { 18. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Red LED off"); 19. UTIL1_strcat(http, sizeof(http), (uint8_t*)" 20.Red LED on"); 21. } else{ 22. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Red LED off"); 23. UTIL1_strcat(http, sizeof(http), (uint8_t*)" 24.Red LED on"); 25. } 26. UTIL1_strcat(http, sizeof(http), (uint8_t*)""); 27. UTIL1_strcat(http, sizeof(http), (uint8_t*)"\r\n\r\n"); 28. 29. UTIL1_strcpy(cmd, sizeof(cmd), "AT+CIPSEND="); /* parameters are , */ 30. UTIL1_strcatNum8u(cmd, sizeof(cmd), ch_id); 31. UTIL1_chcat(cmd, sizeof(cmd), ','); 32. UTIL1_strcatNum16u(cmd, sizeof(cmd), UTIL1_strlen(http)); 33. UTIL1_strcpy(expected, sizeof(expected), cmd); /* we expect the echo of our command */ 34. UTIL1_strcat(expected, sizeof(expected), "\r\r\n> "); /* expect "> " */ 35. UTIL1_strcat(cmd, sizeof(cmd), "\r\n"); 36. res = ESP_SendATCommand(cmd, rxBuf, sizeof(rxBuf), expected, ESP_DEFAULT_TIMEOUT_MS, io); 37. if(res!=ERR_OK) { 38. if(io!=NULL) { 39. CLS1_SendStr("INFO: TIMEOUT, closing connection!\r\n", io->stdOut); 40. } 41. } else{ 42. if(io!=NULL) { 43. CLS1_SendStr("INFO: Sending http page...\r\n", io->stdOut); 44. } 45. UTIL1_strcat(http, sizeof(http), "\r\n\r\n"); /* need to add this to end the command! */ 46. res = ESP_SendATCommand(http, NULL, 0, NULL, ESP_DEFAULT_TIMEOUT_MS, io); 47. if(res!=ERR_OK) { 48. CLS1_SendStr("Sending page failed!\r\n", io->stdErr); /* copy on console */ 49. } else{ 50. for(;;) { /* breaks */ 51. res = ESP_ReadCharsUntil(buf, sizeof(buf), '\n', 1000); 52. if(res==ERR_OK) { /* line read */ 53. if(io!=NULL) { 54. CLS1_SendStr(buf, io->stdOut); /* copy on console */ 55. } 56. } 57. if(UTIL1_strncmp(buf, "SEND OK\r\n", sizeof("SEND OK\r\n")-1)==0) { /* ok from module */ 58. break; 59. } 60. } 61. } 62. } 63. returnres; 64.} The rest of Application.c is rather simple: view source print? 01./* 02. * Application.c 03. * 04. * Author: Erich Styger 05. */ 06.#include "PE_Types.h" 07.#include "CLS1.h" 08.#include "WAIT1.h" 09.#include "Shell.h" 10.#include "UTIL1.h" 11.#include "ESP8266.h" 12.#include "LEDR.h" 13.#include "LEDG.h" 14.#include "AS2.h" 15. 16.staticuint8_t APP_EspMsgBuf[512]; /* buffer for messages from ESP8266 */ 17. 18.staticvoidWebProcess(void) { 19. uint8_t res=ERR_OK; 20. bool isGet; 21. uint8_t ch_id=0; 22. uint16_t size=0; 23. constuint8_t *p; 24. constCLS1_StdIOType *io; 25. 26. if(ESP_IsServerOn()) { 27. io = CLS1_GetStdio(); 28. res = ESP_GetIPD(APP_EspMsgBuf, sizeof(APP_EspMsgBuf), &ch_id, &size, &isGet, 1000, io); 29. if(res==ERR_OK) { 30. if(isGet) { /* GET: put web page */ 31. res = SendWebPage(ch_id, LEDR_Get()!=FALSE, 21/*dummy temperature*/, io); 32. if(res!=ERR_OK && io!=NULL) { 33. CLS1_SendStr("Sending page failed!\r\n", io->stdErr); /* copy on console */ 34. } 35. } else{ /* POST: received info */ 36. intpos; 37. 38. pos = UTIL1_strFind(APP_EspMsgBuf, "radio="); 39. if(pos!=-1) { /* found */ 40. if(UTIL1_strncmp(&APP_EspMsgBuf[pos], "radio=0", sizeof("radio=0")-1)) { 41. LEDR_On(); 42. } elseif(UTIL1_strncmp(&APP_EspMsgBuf[pos], "radio=1", sizeof("radio=1")-1)) { 43. LEDR_Off(); 44. } 45. } 46. res = SendWebPage(ch_id, LEDR_Get()!=FALSE, 20/*dummy temperature*/, io); 47. if(res!=ERR_OK && io!=NULL) { 48. CLS1_SendStr("Sending page failed!\r\n", io->stdErr); /* copy on console */ 49. } 50. } 51. CLS1_SendStr("INFO: Closing connection...\r\n", io->stdOut); 52. res = ESP_CloseConnection(ch_id, io, ESP_DEFAULT_TIMEOUT_MS); 53. } 54. } else{ /* copy messages we receive to console */ 55. while(AS2_GetCharsInRxBuf()>0) { 56. uint8_t ch; 57. 58. (void)AS2_RecvChar(&ch); 59. CLS1_SendChar(ch); 60. } 61. } 62.} 63. 64.voidAPP_Run(void) { 65. CLS1_ConstStdIOType *io; 66. 67. WAIT1_Waitms(1000); /* wait after power-on */ 68. ESP_Init(); 69. SHELL_Init(); 70. io = CLS1_GetStdio(); 71. CLS1_SendStr("\r\n------------------------------------------\r\n", io->stdOut); 72. CLS1_SendStr("ESP8266 with FRDM-KL25Z\r\n", io->stdOut); 73. CLS1_SendStr("------------------------------------------\r\n", io->stdOut); 74. CLS1_PrintPrompt(io); 75. for(;;) { 76. WebProcess(); 77. SHELL_Parse(); 78. WAIT1_Waitms(10); 79. LEDG_Neg(); 80. } 81.} In main.c I call the application part: view source print? 01./* ################################################################### 02.** Filename : main.c 03.** Project : FRDM-KL25Z_ESP8266 04.** Processor : MKL25Z128VLK4 05.** Version : Driver 01.01 06.** Compiler : GNU C Compiler 07.** Date/Time : 2014-10-15, 14:28, # CodeGen: 0 08.** Abstract : 09.** Main module. 10.** This module contains user's application code. 11.** Settings : 12.** Contents : 13.** No public methods 14.** 15.** ###################################################################*/ 16./*! 17.** @file main.c 18.** @version 01.01 19.** @brief 20.** Main module. 21.** This module contains user's application code. 22.*/ 23./*! 24.** @addtogroup main_module main module documentation 25.** @{ 26.*/ 27./* MODULE main */ 28. 29./* Including needed modules to compile this module/procedure */ 30.#include "Cpu.h" 31.#include "Events.h" 32.#include "WAIT1.h" 33.#include "UTIL1.h" 34.#include "AS1.h" 35.#include "ASerialLdd1.h" 36.#include "CLS1.h" 37.#include "CS1.h" 38.#include "AS2.h" 39.#include "ASerialLdd2.h" 40.#include "LEDR.h" 41.#include "LEDpin1.h" 42.#include "BitIoLdd1.h" 43.#include "LEDG.h" 44.#include "LEDpin2.h" 45.#include "BitIoLdd2.h" 46.#include "LEDB.h" 47.#include "LEDpin3.h" 48.#include "BitIoLdd3.h" 49./* Including shared modules, which are used for whole project */ 50.#include "PE_Types.h" 51.#include "PE_Error.h" 52.#include "PE_Const.h" 53.#include "IO_Map.h" 54./* User includes (#include below this line is not maintained by Processor Expert) */ 55.#include "Application.h" 56. 57./*lint -save -e970 Disable MISRA rule (6.3) checking. */ 58.intmain(void) 59./*lint -restore Enable MISRA rule (6.3) checking. */ 60.{ 61. /* Write your local variable definition here */ 62. 63. /*** Processor Expert internal initialization. DON'T REMOVE THIS CODE!!! ***/ 64. PE_low_level_init(); 65. /*** End of Processor Expert internal initialization. ***/ 66. 67. APP_Run(); 68. 69. /*** Don't write any code pass this line, or it will be deleted during code generation. ***/ 70. /*** RTOS startup code. Macro PEX_RTOS_START is defined by the RTOS component. DON'T MODIFY THIS CODE!!! ***/ 71. #ifdef PEX_RTOS_START 72. PEX_RTOS_START(); /* Startup of the selected RTOS. Macro is defined by the RTOS component. */ 73. #endif 74. /*** End of RTOS startup code. ***/ 75. /*** Processor Expert end of main routine. DON'T MODIFY THIS CODE!!! ***/ 76. for(;;){} 77. /*** Processor Expert end of main routine. DON'T WRITE CODE BELOW!!! ***/ 78.} /*** End of main routine. DO NOT MODIFY THIS TEXT!!! ***/ 79. 80./* END main */ 81./*! 82.** @} 83.*/ 84./* 85.** ################################################################### 86.** 87.** This file was created by Processor Expert 10.4 [05.10] 88.** for the Freescale Kinetis series of microcontrollers. 89.** 90.** ################################################################### 91.*/ Processor Expert Components In addition, I’m using several Processor Expert component which are available fromSourceForge. Processor Expert Components Wait: Busy waiting component, e.g. to wait for a few milliseconds. Utility: string manipulation and utility functions. AsynchroSerial (AS1): serial interface to the host for the shell command line interface Shell: command line shell implementation CriticalSection: for creating critical sections AsynchroSerial (AS2): serial interface to the ESP8266 module LEDR, LEDG and LEDB: Red, Green and Blue LED on the FRDM-KL25Z board AS1 is configured as UART connection (over OpenSDA) for the shell: Shell UART Settings There are no special settings for the Shell component: Shell Settings Important are the correct settings to the ESP8266 UART: 115200 baud and using the correct pins on the board connected to the Rx and Tx lines of the ESP8266. I’m using rather large input and output buffers: UART connection to ESP8266 The LED components are configured for the pins used on the board: PTB18 for red, PTB19 for green and PTD1 for blue LED. Red LED for FRDM-KL25Z Sending Commands The shell implements the command ESP send which I can use to send a string or command to the module: ESP send Note that for every command a trailing “\r\n” will be sent. So instead of using the programmatic way, the shell can be used to ‘manually’ drive a web server, at least most of the part. So I’m using command line commands below to explore how the ESP8266 module works. Using the Shell With the project (link to GitHub below), I have a serial connection and command line shell interface to the module. Compile the project and download it to the FRDM-KL25Z board and use a terminal program (I use Termite) to talk with the module. It power-up, the program shows a greeting message: Greeting Message With ‘help‘ I get a list of the available commands: Help Command The ‘status‘ command gives a system status: Status Command Output With this, I’m ready to send commands to the module :-). Connection Test To test the connection I send a simple ‘AT’ command ESP send AT AT Command Output and the module should respond with AT\r\r\n\r\nOK\r\n Module Restart Sometimes the module gets stuck. What helps is a power-on reset of the module. Another way is to send the AT+RST command to reset the module. The module will boot up and print a ‘ready’ message: Reset of the ESP8266 Access Point or Device First I need to configure if the ESP is either a device or an access point. For this, theCWMODE command is used: AT+CWMODE= where is one of: 1: ‘Sta’, ESP8266 is a device, it connects to an existing access point 2: ‘AP’, ESP8266 is an access point, so other devices can connect to it 3: ‘both’. Not really clear to me, but it seems that in this mode the device is in a hybrid mode? To have the ESP as device so it can connect to an existing access point I use AT+CWMODE=1 and the module should answer with AT+CWMODE=1\r\r\n\r\nOK\r\n or with a ‘no change': AT+CWMODE=1\r\r\nno change\r\n With AT+CWMODE? I can ask for the current mode: Retrieving Current Mode List of Access Points With AT+CWLAP I get a list of access points. It reports a list like this: AT+CWLAP +CWLAP:(0,"",0) +CWLAP:(4,"APforESP",-39) +CWLAP:(4,"iza-97497",-94) OK :!: I experienced problems with that command in an environment with lots of access points visible. In this case it seems the module hands up. Try first in a place with only a few access points. For this tutorial I have configured an access point with SSID “APforESP” which shows up in my list. The list is formatted like this + CWLAP: , , [, ] With following encoding: : 0: OPEN 1: WPA_PSK 2: WPA2_PSK 4: WPA_WPA2_PSK : the SSID (string) of the access point. : Signal strength. : 0: manually connect 1: automatic connect Connecting to Access Point To connect to an access point I use the command AT+CWJAP="","" Of course replace and with your setup. The module should report back an “OK” message, and you are connected :-). :!: The module stores the ssid and password. After power-up, the module will automatically reconnect to the Access Point. IP Address Once connected I can check the IP address I have been assigned to with AT+CIFSR which should give something like AT+CIFSR 192.168.0.111 So now I know my module IP address :-). With this I can ping my module: Pinging my ESP Module Building a Web Server Now as we hav a connection, it is time to use it to run a web server :-).What I want to serve a web page which I can use to turn on or off the LEDs on the board. Number of Connections: CIPMUX Before I start the server I need to make sure it accepts multiple connections. For this I use the following command: AT+CIPMUX=1 The parameter is either 0 (single connection), or 1 (multiple connections). For a web server I need to set it up for multiple connections. The ESP module should respond with AT+CIPMUX=1\r\n\r\nOK\r\n :info: To make it clear, I have included the ‘\r’ and ‘\n’ in the responses. Starting the Server: CIPSERVER I start the server with AT+CIPSERVER=1,80 The first parameter is either 0 (close connection) or 1 (open connection), followed by the port. I use here the standard http port (80). The module should answer with: AT+CIPSERVER=1,80\r\r\n\r\nOK\r\n or if it is already running the server with a ‘no change': AT+CIPSERVER=1,80\r\r\nno change\r\n No I have a connection open on my IP address (see above: 192.168.0.111), listening to the port I have specified (80). Connecting to the Server with Browser I enter the IP address in a web browser: http://192.168.0.111:80 For clarity I have specified the standard HTTP port (80). So if you are using a different port, make sure you specify it in the address line. Connection from FireFox The browser now sends a GET request to the module, and I will see this from the message printed out from the module: First response from Module The ‘Link’ indicates that it has established a link. IPD (IP Data?) is followed by the channel number (this will the one we have to respond to), plus the size of the following data (296 bytes in that case). As I’m not responding (yet), there will be a timeout (after about 1 minute or so), with an ‘Unlink’ message from the module: Link +IPD,0,296:GET / HTTP/1.1 Host: 192.168.0.111 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: de,en-US;q=0.7,en;q=0.3 Accept-Encoding: gzip, deflate Connection: keep-alive OK Unlink Unlink message Sending Data to Server: CIPSEND Now I need to respond and send data to the browser. For this I need to know the channel number, and this is provided in the IPD message from above, right after the comma: +IPD,0 To send data, I use the command AT+CIPSEND=, So I connect again with the browser, and I send 5 bytes (“hello”) with: AT+CIPSEND=0,5 The ESP8266 responds with AT+CIPSEND=0,5\r\n> Notice the ‘>’ at the end: this is my signal to send the actual data (“hello” in my case): hello The ESP8266 now resonds with a SEND OK: Data Sent However, the browser is still busy and spins around. I already thought that I did something wrong, but after the browser run into a timeout (after about one minute), my data is there! :-) Hello in Browser Closing Connection: CIPCLOSE So things *are* working :-). The trick is that I have to close the connection after I have sent the data. There is a CIPCLOSE command I can use: AT+CIPCLOSE= which I can use to close a channel. So I close the connection with AT+CIPCLOSE=0 and now the browser shows the content right away :-). Web Server Implementation So far I have used the module in command line and manual mode. This is great for exploration of the protocol, but for building the web server I need to do this programmatically. For this I run my ‘main’ loop in APP_Run(). After printing a greeting message and initializing the sub modules, it processes the web/module responses, parses the shell command line interfaces and blinks the green (LEDG) LED. view source print? 01.voidAPP_Run(void) { 02. CLS1_ConstStdIOType *io; 03. 04. WAIT1_Waitms(1000); /* wait after power-on */ 05. ESP_Init(); 06. SHELL_Init(); 07. io = CLS1_GetStdio(); 08. CLS1_SendStr("\r\n------------------------------------------\r\n", io->stdOut); 09. CLS1_SendStr("ESP8266 with FRDM-KL25Z\r\n", io->stdOut); 10. CLS1_SendStr("------------------------------------------\r\n", io->stdOut); 11. CLS1_PrintPrompt(io); 12. for(;;) { 13. WebProcess(); 14. SHELL_Parse(); 15. WAIT1_Waitms(10); 16. LEDG_Neg(); 17. } 18.} With ESP server start I start the web server: Starting the Web Server It sends the AT+CIPMUX command followed by the AT+CIPSERVER to start the server, and then listens to the port. Reading and responding messages is done in WebProcess(): view source print? 01.staticvoidWebProcess(void) { 02. uint8_t res=ERR_OK; 03. bool isGet; 04. uint8_t ch_id=0; 05. uint16_t size=0; 06. constuint8_t *p; 07. constCLS1_StdIOType *io; 08. 09. if(ESP_IsServerOn()) { 10. io = CLS1_GetStdio(); 11. res = ESP_GetIPD(APP_EspMsgBuf, sizeof(APP_EspMsgBuf), &ch_id, &size, &isGet, 1000, io); 12. if(res==ERR_OK) { 13. if(isGet) { /* GET: put web page */ 14. res = SendWebPage(ch_id, LEDR_Get()!=FALSE, 21/*dummy temperature*/, io); 15. if(res!=ERR_OK && io!=NULL) { 16. CLS1_SendStr("Sending page failed!\r\n", io->stdErr); /* copy on console */ 17. } 18. } else{ /* POST: received info */ 19. intpos; 20. 21. pos = UTIL1_strFind(APP_EspMsgBuf, "radio="); 22. if(pos!=-1) { /* found */ 23. if(UTIL1_strncmp(&APP_EspMsgBuf[pos], "radio=0", sizeof("radio=0")-1)) { 24. LEDR_On(); 25. } elseif(UTIL1_strncmp(&APP_EspMsgBuf[pos], "radio=1", sizeof("radio=1")-1)) { 26. LEDR_Off(); 27. } 28. } 29. res = SendWebPage(ch_id, LEDR_Get()!=FALSE, 20/*dummy temperature*/, io); 30. if(res!=ERR_OK && io!=NULL) { 31. CLS1_SendStr("Sending page failed!\r\n", io->stdErr); /* copy on console */ 32. } 33. } 34. CLS1_SendStr("INFO: Closing connection...\r\n", io->stdOut); 35. res = ESP_CloseConnection(ch_id, io, ESP_DEFAULT_TIMEOUT_MS); 36. } 37. } else{ /* copy messages we receive to console */ 38. while(AS2_GetCharsInRxBuf()>0) { 39. uint8_t ch; 40. 41. (void)AS2_RecvChar(&ch); 42. CLS1_SendChar(ch); 43. } 44. } 45.} If the server is not enabled, it simply copies the received messages to the console: view source print? 1.} else{ /* copy messages we receive to console */ 2. while(AS2_GetCharsInRxBuf()>0) { 3. uint8_t ch; 4. 5. (void)AS2_RecvChar(&ch); 6. CLS1_SendChar(ch); 7. } 8. } Otherwise it scans for an IPD message (ESP_GetIPD()). This function returns the whole message, the channel, the message size and if it is a GET or POST message: 1 res = ESP_GetIPD(APP_EspMsgBuf, sizeof(APP_EspMsgBuf), &ch_id, &size, &isGet, 1000, io); If it is a GET message, then it sends a HTML page to the module: 1 res = SendWebPage(ch_id, LEDR_Get()!=FALSE, 21 /*dummy temperature*/, io); This web page shows the status of the red LED on the board, a (dummy) temperature value and a button to submit new LED values: WSP8266 Web Server The HTML code for this page is constructed in SendWebPage() and sent withAT+CIPSEND: view source print? 01.staticuint8_t SendWebPage(uint8_t ch_id, bool ledIsOn, uint8_t temperature, constCLS1_StdIOType *io) { 02. staticuint8_t http[1024]; 03. uint8_t cmd[24], rxBuf[48], expected[48]; 04. uint8_t buf[16]; 05. uint8_t res = ERR_OK; 06. 07. /* construct web page content */ 08. UTIL1_strcpy(http, sizeof(http), (uint8_t*)"HTTP/1.0 200 OK\r\nContent-Type: text/html\r\nPragma: no-cache\r\n\r\n"); 09. UTIL1_strcat(http, sizeof(http), (uint8_t*)"\r\n\r\n"); 10. UTIL1_strcat(http, sizeof(http), (uint8_t*)"\r\n"); 11. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Web Server using ESP8266\r\n"); 12. UTIL1_strcat(http, sizeof(http), (uint8_t*)" 13.\r\n"); 14. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Temp: OC"); 17. if(ledIsOn) { 18. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Red LED off"); 19. UTIL1_strcat(http, sizeof(http), (uint8_t*)" 20.Red LED on"); 21. } else{ 22. UTIL1_strcat(http, sizeof(http), (uint8_t*)"Red LED off"); 23. UTIL1_strcat(http, sizeof(http), (uint8_t*)" 24.Red LED on"); 25. } 26. UTIL1_strcat(http, sizeof(http), (uint8_t*)""); 27. UTIL1_strcat(http, sizeof(http), (uint8_t*)"\r\n\r\n"); 28. 29. UTIL1_strcpy(cmd, sizeof(cmd), "AT+CIPSEND="); /* parameters are , */ 30. UTIL1_strcatNum8u(cmd, sizeof(cmd), ch_id); 31. UTIL1_chcat(cmd, sizeof(cmd), ','); 32. UTIL1_strcatNum16u(cmd, sizeof(cmd), UTIL1_strlen(http)); 33. UTIL1_strcpy(expected, sizeof(expected), cmd); /* we expect the echo of our command */ 34. UTIL1_strcat(expected, sizeof(expected), "\r\r\n> "); /* expect "> " */ 35. UTIL1_strcat(cmd, sizeof(cmd), "\r\n"); 36. res = ESP_SendATCommand(cmd, rxBuf, sizeof(rxBuf), expected, ESP_DEFAULT_TIMEOUT_MS, io); 37. if(res!=ERR_OK) { 38. if(io!=NULL) { 39. CLS1_SendStr("INFO: TIMEOUT, closing connection!\r\n", io->stdOut); 40. } 41. } else{ 42. if(io!=NULL) { 43. CLS1_SendStr("INFO: Sending http page...\r\n", io->stdOut); 44. } 45. UTIL1_strcat(http, sizeof(http), "\r\n\r\n"); /* need to add this to end the command! */ 46. res = ESP_SendATCommand(http, NULL, 0, NULL, ESP_DEFAULT_TIMEOUT_MS, io); 47. if(res!=ERR_OK) { 48. CLS1_SendStr("Sending page failed!\r\n", io->stdErr); /* copy on console */ 49. } else{ 50. for(;;) { /* breaks */ 51. res = ESP_ReadCharsUntil(buf, sizeof(buf), '\n', 1000); 52. if(res==ERR_OK) { /* line read */ 53. if(io!=NULL) { 54. CLS1_SendStr(buf, io->stdOut); /* copy on console */ 55. } 56. } 57. if(UTIL1_strncmp(buf, "SEND OK\r\n", sizeof("SEND OK\r\n")-1)==0) { /* ok from module */ 58. break; 59. } 60. } 61. } 62. } 63. returnres; 64.} In case of a POST message (user has pressed the button), I scan for the radio element string and turn on/off the LED accordingly, and re-submit the new web page: view source print? 01.} else{ /* POST: received info */ 02. intpos; 03. 04. pos = UTIL1_strFind(APP_EspMsgBuf, "radio="); 05. if(pos!=-1) { /* found */ 06. if(UTIL1_strncmp(&APP_EspMsgBuf[pos], "radio=0", sizeof("radio=0")-1)) { 07. LEDR_On(); 08. } elseif(UTIL1_strncmp(&APP_EspMsgBuf[pos], "radio=1", sizeof("radio=1")-1)) { 09. LEDR_Off(); 10. } 11. } 12. res = SendWebPage(ch_id, LEDR_Get()!=FALSE, 20/*dummy temperature*/, io); 13. if(res!=ERR_OK && io!=NULL) { 14. CLS1_SendStr("Sending page failed!\r\n", io->stdErr); /* copy on console */ 15. } 16. } Finally, it closes the connection at the end: view source print? 1.CLS1_SendStr("INFO: Closing connection...\r\n", io->stdOut); 2. res = ESP_CloseConnection(ch_id, io, ESP_DEFAULT_TIMEOUT_MS); With this, I handle GET and POST messages and can toggle the LED on my board :-) :-). Summary It is amazing what is possible with this tiny and inexpensive ($4.50) WiFi module. The simple AT interface allows small and tiny microprocesors to connect to the internet or the local network. With all the hype around ‘Internet of Things’ this is where things very likely will end up: small nodes connecting in an easy way to the network. The processor on that ESP8266 is probably more powerful than the KL25Z (the specs and data sheets of that ESP8266 are still evolving). Or it is possible to run that module in standalone mode too which is a very interesting approach too, see the links at the end of this article. But still having an UART way to connect to the network is very useful and powerful. Other modules costs multiple times more. I expect that many vendors will come up with similar integrated modules e.g. to combine an ARM processor with the WiFi radio, similar that ESP8266 module. For sure that ESP8266 has a head start and paved the way how WiFi connectivity should work. We all will see what the future brings. Until then, that ESP8266 module is something I can use in many projects :-). The sources and project files can be found on GitHub: https://github.com/ErichStyger/mcuoneclipse/tree/master/Examples/KDS/FRDM-KL25Z/FRDM-KL25Z_ESP8266 Happy Web-Serving :-) Useful Links: http://www.electrodragon.com/w/Wi07c http://scargill.wordpress.com/category/esp8266/ https://github.com/esp8266/esp8266-webserver http://www.cse.dmu.ac.uk/~sexton/ESP8266/ http://defcon-cc.dyndns.org/wiki/ESP8266#Update http://www.xess.com/blog/esp8266-resources/

Compared to other features, real-time search capability is undoubtedly one of the most important features in Elasticsearch. Today we’ll look closely how is provided real-time search by Elasticsearch. Real time First of all, if we need to explain the concept of real-time, in general, we can say that the delay between input and out time in the information is small at real-time systems. This means, data is taken without data accumulation, processed in real time. Today, the best solution Elasticsearch known for real-time search, when a record is added to it for storage makes it searchable in 1 second. How? As is known, the disks are able to create a risk of bottleneck for I/O operations at the data persistence step. Also some mechanisms used for prevent any loss of data increases cost of time. At this point Elasticsearch uses the file-system cache that sitting between itself and the disk for overcome the risk of bottleneck and ensure the a new document can be searched in real time. A new segment is written to the file-system cache first and only later it flushed to disk by Elasticsearch. This lightweight process of writing and opening a new segment is called a refresh in Elasticsearch. By default, all shards is refreshed automatically once every second. In this way, Elasticsearch support real-time search. Test time Above digression about the time of refresh of the shards you can bring to mind the following questions: What happens, when a new document is requested in less than 1 second time? Can be documents requested, without having to depend of the refresh period shards of managed by Elasticsearch? Short answers. Elasticsearch does not return the document. Yes. Now let’s get clarity on this issue is a simple example. hakdogan$ curl -XPUT localhost:9200/kodcucom/document/1 -d'{ > "title": "Document A" > }' We sent a document to Elasticsearch. The index name is kodcucom, type document, id value 1. The title field is only field in the document and the value of "Document A". Let’s take this document from Elasticsearch. hakdogan$ curl -XGET localhost:9200/kodcucom/document/1?pretty { "_index" : "kodcucom", "_type" : "document", "_id" : "1", "_version" : 1, "found" : true, "_source":{ "title": "Document A" } } As expected, the document was returned to us. Well, if we keep short the time between document recording and get request than default shard refresh time what will happen? Let’s see. hakdogan$ curl -XPUT localhost:9200/kodcucom/document/2 -d'{"title": "Document B"}'; curl -XGET localhost:9200/kodcucom/_search?pretty {"_index":"kodcucom","_type":"document","_id":"2","_version":1,"created":true}{ "took" : 38, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 1.0, "hits" : [ { "_index" : "kodcucom", "_type" : "document", "_id" : "1", "_score" : 1.0, "_source":{ "title": "Document A" } } ] } } As can be seen, only the previous document was returned to us by Elasticsearch when we do concurrently create and get request. Well, how can I get the document concurrently? Let’s see. hakdogan$ curl -XPUT localhost:9200/kodcucom/document/3 -d'{"title": "Document C"}'; curl -XGET localhost:9200/kodcucom/_refresh; curl -XGET localhost:9200/kodcucom/_search?pretty {"_index":"kodcucom","_type":"document","_id":"3","_version":1,"created":true}{"_shards":{"total":10,"successful":5,"failed":0}{ "took" : 3, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 3, "max_score" : 1.0, "hits" : [ { "_index" : "kodcucom", "_type" : "document", "_id" : "1", "_score" : 1.0, "_source":{ "title": "Document A" } }, { "_index" : "kodcucom", "_type" : "document", "_id" : "2", "_score" : 1.0, "_source":{"title": "Document B"} }, { "_index" : "kodcucom", "_type" : "document", "_id" : "3", "_score" : 1.0, "_source":{"title": "Document C"} } ] } } In this command, we perform to refresh operation on kodcucom index before the search request. In this way, the document was returned to us. Auto refresh time can be changed. By setting the index.refresh_interval parameter in the configuration file. Applies to all indices in the cluster. A per-index basis by updated index setting. In addition to these, you can turn off automatic refresh. An important point to keep in mind about the refresh time of the shards, the refresh operation is costly in terms of system resources. If you wished to make changes to the auto-refresh time, this situation should be taken into account. Extension of the automatic refresh time, enables faster indexing but new documents and changes made to the existing documents will not appear in searches during specified period of time.

Nowadays it has become mandatory to Gzipping the APIs response due to huge amount of data we are sending in response. It saves network bandwidth and delivery time and off course space over the internet. While using CXF; it provides an option to use the Gzip Compression in no of ways. Blueprint Annotation Blueprint: Annotation: First you need to register the GZIPOutInterceptor in out interceptors list. For that you need to hook into CXF initialization classes. public class InterceptorManager extends AbstractFeature { private static final Logger LOGGER = Logger.getLogger( "simcore" ); private static final Interceptor< Message > GZIP = new GZIPOutInterceptor(); //private static final Interceptor< Message > GZIP = new GZIPOutInterceptor(512); /* (non-Javadoc) * @see org.apache.cxf.feature.AbstractFeature#initializeProvider(org.apache.cxf.interceptor.InterceptorProvider, org.apache.cxf.Bus) */ @Override protected void initializeProvider( InterceptorProvider provider, Bus bus ) { /** * Adding Gzip interceptor to all outbound requests/responses */ LOGGER.debug( " ############## Adding Gzip as OUT Interceptor ##############" ); provider.getOutInterceptors().add( GZIP ); } } GZIPOutInterceptor comes with an option to set the Threshold value as no of Bytes. If response size will be below this threshold value then it will not be compressed. It is extremely useful when we will be sending empty lists and status messages/codes only. Because compressing those small responses will be overhead at server side. But there is another factor which is no of users requesting the response. So set this value by thinking over all the cases in mind. @GZIP Now we can use this annotation on any of our web-services controller to implement compression on all the APIs provided in that class. @WebService @Consumes ( { MediaType.TEXT_PLAIN, MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON } ) @Produces ( MediaType.APPLICATION_JSON ) @GZIP public interface WebServicesController { @GET @Path ( "/myGzipData" ) @Produces ( { MediaType.APPLICATION_JSON } ) Response getZipData( ); } Moreover we can set different parameters in Gzip annotation. @GZIP ( force = true, threshold = 512 )

there is currently a strong trend for microservice based architectures and frequent discussions comparing them to monoliths. there is much advice about breaking-up monoliths into microservices and also some amusing fights between proponents of the two paradigms - see the great microservices vs monolithic melee . the term 'monolith' is increasingly being used as a generic insult in the same way that 'legacy' is! however, i believe that there is a great deal of misunderstanding about exactly what a 'monolith' is and those discussing it are often talking about completely different things. a monolith can be considered an architectural style or a software development pattern (or anti-pattern if you view it negatively). styles and patterns usually fit into different viewtypes (a viewtype is a set, or category, of views that can be easily reconciled with each other [clements et al., 2010]) and some basic viewtypes we can discuss are: module - the code units and their relation to each other at compile time. allocation - the mapping of the software onto its environment. runtime - the static structure of the software elements and how they interact at runtime. a monolith could refer to any of the basic viewtypes above. module monolith if you have a module monolith then all of the code for a system is in a single codebase that is compiled together and produces a single artifact. the code may still be well structured (classes and packages that are coherent and decoupled at a source level rather than a big-ball-of-mud) but it is not split into separate modules for compilation. conversely a non-monolithic module design may have code split into multiple modules or libraries that can be compiled separately, stored in repositories and referenced when required. there are advantages and disadvantages to both but this tells you very little about how the code is used - it is primarily done for development management. allocation monolith for an allocation monolith, all of the code is shipped/deployed at the same time. in other words once the compiled code is 'ready for release' then a single version is shipped to all nodes. all running components have the same version of the software running at any point in time. this is independent of whether the module structure is a monolith. you may have compiled the entire codebase at once before deployment or you may have created a set of deployment artifacts from multiple sources and versions. either way this version for the system is deployed everywhere at once (often by stopping the entire system, rolling out the software and then restarting). a non-monolithic allocation would involve deploying different versions to individual nodes at different times. this is again independent of the module structure as different versions of a module monolith could be deployed individually. runtime monolith a runtime monolith will have a single application or process performing the work for the system (although the system may have multiple, external dependencies). many systems have traditionally been written like this (especially line-of-business systems such as payroll, accounts payable, cms etc). whether the runtime is a monolith is independent of whether the system code is a module monolith or not. a runtime monolith often implies an allocation monolith if there is only one main node/component to be deployed (although this is not the case if a new version of software is rolled out across regions, with separate users, over a period of time). note that my examples above are slightly forced for the viewtypes and it won't be as hard-and-fast in the real world. conclusion be very carefully when arguing about 'microservices vs monoliths'. a direct comparison is only possible when discussing the runtime viewtype and properties. you should also not assume that moving away from a module or allocation monolith will magically enable a microservice architecture (although it will probably help). if you are moving to a microservice architecture then i'd advise you to consider all these viewtypes and align your boundaries across them i.e. don't just code, build and distribute a monolith that exposes subsets of itself on different nodes.

My last post talked about what’s wrong with type classes (in general, but also specifically in Haskell). This post generated some great feedback on Reddit, including some valid criticism that I didn’t explain why I hated on newtypes so much. I took some of that feedback and incorporated it into a revised version of the post, but I have even more to say about “newtypes," so I decided to write another blog post. What’s in a Newtype Newtypes are a feature of Haskell that let you define a new type in terms of an existing type. In the following example, I create a newtype for Email, which “holds” a String. newtype Email = Email String They are similar to type synonyms (type Email = String), except that type synonyms don’t create new types, they just allow you to refer to existing types by other names. Every newtype can be easily translated into a data declaration. In fact, only the keyword changes: data Email = Email String There’s a slight semantic difference between the two, but for purposes of this blog post, any criticism I have against newtypes apply equally to similar constructs modeled with type or data. The Promise of Newtypes Newtypes are used to provide and select between alternate implementations of type classes for some base types. I think that’s a hack (albeit a necessary one), but I’ve already talked about this so I won’t belabor it here. The other promise of newtypes is that we can use them to make our code more type safe. Instead of passing around String as an email, for example, we can create a super lightweight “wrapper” around String called Email, and make it an error to use a String wherever an Email is expected. This practice isn’t restricted to Haskell. Even in Java, it’s considered good coding practice to wrap primitives with classes whose names denote the meaning of the wrapper (Email, SSN, Address, etc.). There’s a part of this promise that’s certainly true. If I have to define a function accepting four parameters, and three of them are strings, but one of those strings denotes an email, then I have two choices: Model the email parameter with a String. In this case, I may accidentally use the email where I intended to use the other two string parameters, or I may use one of the other two string parameters where I intended to use the email. Considering just these choices, there are five ways my program may go wrong if I use the wrong name in the wrong position. Model the email parameter with a newtype. In this case, I cannot use the email where I intended to use the other two string parameters, because the compiler may stop me. Similarly, I cannot use the other two string parameters where I intended to use the email, for the same reason. Looking at just these choices, there are 0 ways my program may go wrong. Thus, newtypes, like all good FP practices, reduce the number of ways my program can go wrong. Unfortunately, in my opinion, they don’t go nearly far enough. False Security For most intent and purposes, newtypes are isomorphic to the single value they hold. In my preceding example, given a String, I can get an email (Email "foo"). Given an Email, I can also get a String, e.g. by pattern matching on the Email constructor. Stated differently, and also approximately because I’m ignoring bottom: the String and Email types are isomorphic; they contain the same inhabitants, for any useful definition of “same”. The only substantive difference between the preceding String and Email is the name of the data constructor (call Email an AbergrackleFoozyWatzit, and what has changed?). Hence, my previous criticism of newtypes as “programming by name”. By themselves, newtypes don’t really reduce the number of ways my program can go wrong. They just make it a bit harder to go wrong. But any newtype is isomorphic to the value it holds, and it’s trivial to convert between the two. In fact, if my code doesn’t need to convert between the two (either directly or indirectly), then it’s better off generic. That is, if I never need to convert an Email to a String, or a String to an Email, then I should really write the code generically to work with any value (even if that means making data structures or functions more polymorphic). Parametricity provides a massive reduction in the number of ways my program can go wrong. Newtypes, on the other hand, just make it a bit harder to go wrong, by adding one layer of indirection. In this example, as with many newtypes, I’ve created a bad isomorphism. The domain model of an email is not isomorphic to the data model of a string. But by using a newtype, I have implicitly declared that they are isomorphic. Calling a string an email may make me feel better, because of the different name, but fundamentally, with a newtype, it’s still a string, and I’m only ever one more step away from going wrong. In my experience, too many newtypes create an isomorphism between things that, properly modeled, are not isomorphic. Fortunately, there’s a well-worn workaround that lets us get more mileage out of newtypes. Smart Constructors If I define Email in a module, I can make its data constructor private, and export a helper function to construct an Email. Such helper functions are called smart constructors. They can be used to break the natural isomorphisms created by newtyping. An example is shown below: newtype Email = MkEmail String mkEmail :: String -> Maybe Email mkEmail s = ... In this example, I create a smart constructor which does not promise that it can turn every string into an email. It promises only that it might be able to turn a string into an email, by returning a Maybe Email. With the smart constructor approach, I’ve modeled the fact that while every email has a string representation, not every string has an email representation. Going back to my earlier example of passing same-typed parameters to a function, if I use a smart constructor, then while I can still use an email anywhere a string is expected (by converting), I can’t use a string anywhere an email is expected. (Well, ignoring the fromJust abomination!) Smart Constructors, Dumb Data Smart constructors take us one step closer toward modeling data in a type safe fashion. Unfortunately, I still don’t think it’s far enough. With smart constructors, our data model is fundamentally underconstrained, so we patch that up by restricting who can create the data. That’s putting a band-aid on the real problem. Why not just solve the root issue — viz., that our data model is underconstrained? Dumb Constructors, Smart Data The best solution to a great many newtype problems, I believe, is creating a data model where there is a true isomorphism between the entity modeled by our data and the values passed to the data constructor. That is, creating a data model such that there exists no regions in our data’s state space which correspond to invalid states. Email is a simple example, because there are well-defined models for what constitutes a data model, which can be translated into data declarations in straightforward, if tedious fashion. (To some extent, it’s a failure of most languages I know that such specifications cannot be easily translated into code without tedious boilerplate!) When our data declaration precisely fits our data model specification, there’s no need for smart constructors, and no need for newtypes. There’s far fewer ways that code can go wrong, and because our domain model is captured precisely by our data model, we can transform that data model in ways that make semantic sense (e.g. transforming just the name part of an email, since we’re now in the realm of structured data). Summary As I’ve explained in this blog post, I don’t really hate newtypes. I think they’re very useful, and I do use them, because they make it more difficult for my programs to go wrong. Ultimately, however, I think a lot of problems solved with newtypes (modeling coordinates, positions, emails, etc.) are better solved by more precise data modeling. That is, by making our programs stop lying about isomorphisms. Precision may be tedious due to limitations of the languages we work in, but honestly, what’s more tedious than debugging broken code?

I don’t do a lot of plotting in my job, but I recently heard about a website called Plotly that provides a plotting service for anyone’s data. They even have a plotly package for Python (among others)! So in this article we will be learning how to plot with their package. Let’s have some fun making graphs! Getting Started You will need the plotly package to follow along with this article. You can use pip to get the package and install it: pip install plotly Now that you have it installed, you’ll need to go to the Plotly website and create a free account. Once that’s done, you will get an API key. To make things super simple, you can use your username and API key to create a credentials file. Here’s how to do that: import plotly.tools as tls tls.set_credentials_file( username="your_username", api_key="your_api_key") # to get your credentials credentials = tls.get_credentials_file() If you don’t want to save your credentials, then you can also sign in to their service by doing the following: import plotly.plotly as py py.sign_in('your_username','your_api_key') For the purposes of this article, I’m assuming you have created the credentials file. I found that makes interacting with their service a bit easier to use. Creating a Graph Plotly seems to default to a Scatter Plot, so we’ll start with that. I decided to grab some data from a census website. You can download any US state’s population data, along with other pieces of data. In this case, I downloaded a CSV file that contained the population of each county in the state of Iowa. Let’s take a look: import csv import plotly.plotly as py #---------------------------------------------------------------------- def plot_counties(csv_path): """ http://census.ire.org/data/bulkdata.html """ counties = {} county = [] pop = [] counter = 0 with open(csv_path) as csv_handler: reader = csv.reader(csv_handler) for row in reader: if counter == 0: counter += 1 continue county.append(row[8]) pop.append(row[9]) trace = dict(x=county, y=pop) data = [trace] py.plot(data, filename='ia_county_populations') if __name__ == '__main__': csv_path = 'ia_county_pop.csv' plot_counties(csv_path) If you run this code, you should see a graph that looks like this: <br> You can also view the graph here. Anyway, as you can see in the code above, all I did was read the CSV file and extract out the county name and the population. Then I put that data into two different Python lists. Finally I created a dictionary of those lists and then wrapped that dictionary in a list. So you end up with a list that contains a dictionary that contains two lists! To make the Scatter Plot, I passed the data to plotly’s plot method. Converting to a Bar Chart Now let’s see if we can change the ScatterPlot to a Bar Chart. First off, we’ll play around with the plot data. The following was done via the Python interpreter: >>> scatter = py.get_figure('driscollis', '0') >>> print scatter.to_string() Figure( data=Data([ Scatter( x=[u'Adair County', u'Adams County', u'Allamakee County', u'..', ], y=[u'7682', u'4029', u'14330', u'12887', u'6119', u'26076', '..' ] ) ]) ) This shows how we can grab the figure using the username and the plot’s unique number. Then we printed out the data structure. You will note that it doesn’t print out the entire data structure. Now let’s do the actual conversion to a Bar Chart: from plotly.graph_objs import Data, Figure, Layout scatter_data = scatter.get_data() trace_bar = Bar(scatter_data[0]) data = Data([trace_bar]) layout = Layout(title="IA County Populations") fig = Figure(data=data, layout=layout) py.plot(fig, filename='bar_ia_county_pop') This will create a bar chart at the following URL: https://plot.ly/~driscollis/1. Here’s the image of the graph: <br> This code is slightly different than the code we used originally. In this case, we explicitly created a Bar object and passed it the scatter plot’s data. Then we put that data into a Data object. Next we created a Layout object and gave our chart a title. Then we created a Figure object using the data and layout objects. Finally we plotted the bar chart. Saving the Graph to Disk Plotly also allows you to save your graph to your hard drive. You can save it in the following formats: png, svg, jpeg, and pdf. Assuming you still have the Figure object from the previous example handy, you can do the following: py.image.save_as(fig, filename='graph.png') If you want to save using one of the other formats, then just use that format’s extension in the filename. Wrapping Up At this point you should be able to use the plotly package pretty well. There are many other graph types available, so be sure to read Plotly’s documentation thoroughly. They also support streaming graphs. As I understand it, Plotly allows you to create 10 graphs for free. After that you would either have to delete some of your graphs or pay a monthly fee. Additional Reading Plotly Python documentation Plotly User Guide

In the modern world of microservices it's important to provide strict and polyglot clients for your service. It's better if your API is self-documented. One of the best tools for it is Apache Thrift. I want to explain how to use it with my favorite platform for microservices - Spring Boot. All project source code is available on GitHub: https://github.com/bsideup/spring-boot-thrift Project skeleton I will use Gradle to build our application. First, we need our main build.gradle file: buildscript { repositories { jcenter() } dependencies { classpath("org.springframework.boot:spring-boot-gradle-plugin:1.1.8.RELEASE") } } allprojects { repositories { jcenter() } apply plugin:'base' apply plugin: 'idea' } subprojects { apply plugin: 'java' } Nothing special for a Spring Boot project. Then we need a gradle file for thrift protocol modules (we will reuse it in next part): import org.gradle.internal.os.OperatingSystem repositories { ivy { artifactPattern "http://dl.bintray.com/bsideup/thirdparty/[artifact]-[revision](-[classifier]).[ext]" } } buildscript { repositories { jcenter() } dependencies { classpath "ru.trylogic.gradle.plugins:gradle-thrift-plugin:0.1.1" } } apply plugin: ru.trylogic.gradle.thrift.plugins.ThriftPlugin task generateThrift(type : ru.trylogic.gradle.thrift.tasks.ThriftCompileTask) { generator = 'java:beans,hashcode' destinationDir = file("generated-src/main/java") } sourceSets { main { java { srcDir generateThrift.destinationDir } } } clean { delete generateThrift.destinationDir } idea { module { sourceDirs += [file('src/main/thrift'), generateThrift.destinationDir] } } compileJava.dependsOn generateThrift dependencies { def thriftVersion = '0.9.1'; Map platformMapping = [ (OperatingSystem.WINDOWS) : 'win', (OperatingSystem.MAC_OS) : 'osx' ].withDefault { 'nix' } thrift "org.apache.thrift:thrift:$thriftVersion:${platformMapping.get(OperatingSystem.current())}@bin" compile "org.apache.thrift:libthrift:$thriftVersion" compile 'org.slf4j:slf4j-api:1.7.7' } We're using my Thrift plugin for Gradle. Thrift will generate source to the "generated-src/main/java" directory. By default, Thrift uses slf4j v1.5.8, while Spring Boot uses v1.7.7. It will cause an error in runtime when you will run your application, that's why we have to force a slf4j api dependency. Calculator service Let's start with a simple calculator service. It will have 2 modules: protocol and app.We will start with protocol. Your project should look as follows: calculator/ protocol/ src/ main/ thrift/ calculator.thrift build.gradle build.gradle settings.gradle thrift.gradle Where calculator/protocol/build.gradle contains only one line: apply from: rootProject.file('thrift.gradle') Don't forget to put these lines to settings.gradle, otherwise your modules will not be visible to Gradle: include 'calculator:protocol' include 'calculator:app' Calculator protocol Even if you're not familiar with Thrift, its protocol description file (calculator/protocol/src/main/thrift/calculator.thrift) should be very clear to you: namespace cpp com.example.calculator namespace d com.example.calculator namespace java com.example.calculator namespace php com.example.calculator namespace perl com.example.calculator namespace as3 com.example.calculator enum TOperation { ADD = 1, SUBTRACT = 2, MULTIPLY = 3, DIVIDE = 4 } exception TDivisionByZeroException { } service TCalculatorService { i32 calculate(1:i32 num1, 2:i32 num2, 3:TOperation op) throws (1:TDivisionByZeroException divisionByZero); } Here we define TCalculatorService with only one method - calculate. It can throw an exception of type TDivisionByZeroException. Note how many languages we're supporting out of the box (in this example we will use only Java as a target, though) Now run ./gradlew generateThrift, you will get generated Java protocol source in the calculator/protocol/generated-src/main/java/ folder. Calculator application Next, we need to create the service application itself. Just create calculator/app/ folder with the following structure: src/ main/ java/ com/ example/ calculator/ handler/ CalculatorServiceHandler.java service/ CalculatorService.java CalculatorApplication.java build.gradle Our build.gradle file for app module should look like this: apply plugin: 'spring-boot' dependencies { compile project(':calculator:protocol') compile 'org.springframework.boot:spring-boot-starter-web' testCompile 'org.springframework.boot:spring-boot-starter-test' } Here we have a dependency on protocol and typical starters for Spring Boot web app. CalculatorApplication is our main class. In this example I will configure Spring in the same file, but in your apps you should use another config class instead. package com.example.calculator; import com.example.calculator.handler.CalculatorServiceHandler; import org.apache.thrift.protocol.*; import org.apache.thrift.server.TServlet; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.EnableAutoConfiguration; import org.springframework.context.annotation.*; import javax.servlet.Servlet; @Configuration @EnableAutoConfiguration @ComponentScan public class CalculatorApplication { public static void main(String[] args) { SpringApplication.run(CalculatorApplication.class, args); } @Bean public TProtocolFactory tProtocolFactory() { //We will use binary protocol, but it's possible to use JSON and few others as well return new TBinaryProtocol.Factory(); } @Bean public Servlet calculator(TProtocolFactory protocolFactory, CalculatorServiceHandler handler) { return new TServlet(new TCalculatorService.Processor(handler), protocolFactory); } } You may ask why Thrift servlet bean is called "calculator". In Spring Boot, it will register your servlet bean in context of the bean name and our servlet will be available at /calculator/. After that we need a Thrift handler class: package com.example.calculator.handler; import com.example.calculator.*; import com.example.calculator.service.CalculatorService; import org.apache.thrift.TException; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Component; @Component public class CalculatorServiceHandler implements TCalculatorService.Iface { @Autowired CalculatorService calculatorService; @Override public int calculate(int num1, int num2, TOperation op) throws TException { switch(op) { case ADD: return calculatorService.add(num1, num2); case SUBTRACT: return calculatorService.subtract(num1, num2); case MULTIPLY: return calculatorService.multiply(num1, num2); case DIVIDE: try { return calculatorService.divide(num1, num2); } catch(IllegalArgumentException e) { throw new TDivisionByZeroException(); } default: throw new TException("Unknown operation " + op); } } } In this example I want to show you that Thrift handler can be a normal Spring bean and you can inject dependencies in it. Now we need to implement CalculatorService itself: package com.example.calculator.service; import org.springframework.stereotype.Component; @Component public class CalculatorService { public int add(int num1, int num2) { return num1 + num2; } public int subtract(int num1, int num2) { return num1 - num2; } public int multiply(int num1, int num2) { return num1 * num2; } public int divide(int num1, int num2) { if(num2 == 0) { throw new IllegalArgumentException("num2 must not be zero"); } return num1 / num2; } } That's it. Well... almost. We still need to test our service somehow. And it should be an integration test. Usually, even if your application is providing JSON REST API, you still have to implement a client for it. Thrift will do it for you. We don't have to care about it. Also, it will support different protocols. Let's use a generated client in our test: package com.example.calculator; import org.apache.thrift.protocol.*; import org.apache.thrift.transport.THttpClient; import org.apache.thrift.transport.TTransport; import org.junit.*; import org.junit.runner.RunWith; import org.springframework.beans.factory.annotation.*; import org.springframework.boot.test.IntegrationTest; import org.springframework.boot.test.SpringApplicationConfiguration; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; import org.springframework.test.context.web.WebAppConfiguration; import static org.junit.Assert.*; @RunWith(SpringJUnit4ClassRunner.class) @SpringApplicationConfiguration(classes = CalculatorApplication.class) @WebAppConfiguration @IntegrationTest("server.port:0") public class CalculatorApplicationTest { @Autowired protected TProtocolFactory protocolFactory; @Value("${local.server.port}") protected int port; protected TCalculatorService.Client client; @Before public void setUp() throws Exception { TTransport transport = new THttpClient("http://localhost:" + port + "/calculator/"); TProtocol protocol = protocolFactory.getProtocol(transport); client = new TCalculatorService.Client(protocol); } @Test public void testAdd() throws Exception { assertEquals(5, client.calculate(2, 3, TOperation.ADD)); } @Test public void testSubtract() throws Exception { assertEquals(3, client.calculate(5, 2, TOperation.SUBTRACT)); } @Test public void testMultiply() throws Exception { assertEquals(10, client.calculate(5, 2, TOperation.MULTIPLY)); } @Test public void testDivide() throws Exception { assertEquals(2, client.calculate(10, 5, TOperation.DIVIDE)); } @Test(expected = TDivisionByZeroException.class) public void testDivisionByZero() throws Exception { client.calculate(10, 0, TOperation.DIVIDE); } } This test will run your Spring Boot application, bind it to a random port and test it. All client-server communications will be performed in the same way real world clients are. Note how easy to use our service is from the client side. We're just calling methods and catching exceptions.

A hashmap which maintains keys as regular expressions. Any pattern matching the expression will be able to retrieve the same value. Internally it maintains two maps, one containing the regex to value, and another containing matched pattern to regex. Whenever there is a new pattern to 'get', there will be a O(n) search through the compiled regex(s) (which have been 'put' as keys) to find a match. Existing patterns will have constant time lookup through two maps. import java.util.ArrayList; import java.util.Collection; import java.util.HashMap; import java.util.Iterator; import java.util.List; import java.util.Map; import java.util.Set; import java.util.WeakHashMap; import java.util.regex.Pattern; public class RegexHashMap implements Map { private class PatternMatcher { private final String regex; private final Pattern compiled; PatternMatcher(String name) { regex = name; compiled = Pattern.compile(regex); } boolean matched(String string) { if(compiled.matcher(string).matches()) { ref.put(string, regex); return true; } return false; } } /** * Map of input to pattern */ private final Map ref; /** * Map of pattern to value */ private final Map map; /** * Compiled patterns */ private final List matchers; @Override public String toString() { return "RegexHashMap [ref=" + ref + ", map=" + map + "]"; } /** * */ public RegexHashMap() { ref = new WeakHashMap(); map = new HashMap(); matchers = new ArrayList(); } /** * Returns the value to which the specified key pattern is mapped, or null if this map contains no mapping for the key pattern */ @Override public V get(Object weakKey) { if(!ref.containsKey(weakKey)) { for(PatternMatcher matcher : matchers) { if(matcher.matched((String) weakKey)) { break; } } } if(ref.containsKey(weakKey)) { return map.get(ref.get(weakKey)); } return null; } /** * Associates a specified regular expression to a particular value */ @Override public V put(String key, V value) { V v = map.put(key, value); if (v == null) { matchers.add(new PatternMatcher(key)); } return v; } /** * Removes the regular expression key */ @Override public V remove(Object key) { V v = map.remove(key); if(v != null) { for(Iterator iter = matchers.iterator(); iter.hasNext();) { PatternMatcher matcher = iter.next(); if(matcher.regex.equals(key)) { iter.remove(); break; } } for(Iterator> iter = ref.entrySet().iterator(); iter.hasNext();) { Entry entry = iter.next(); if(entry.getValue().equals(key)) { iter.remove(); } } } return v; } /** * Set of view on the regular expression keys */ @Override public Set> entrySet() { return map.entrySet(); } @Override public void putAll(Map m) { for(Entry entry : m.entrySet()) { put(entry.getKey(), entry.getValue()); } } @Override public int size() { return map.size(); } @Override public boolean isEmpty() { return map.isEmpty(); } /** * Returns true if this map contains a mapping for the specified regular expression key. */ @Override public boolean containsKey(Object key) { return map.containsKey(key); } /** * Returns true if this map contains a mapping for the specified regular expression matched pattern. * @param key * @return */ public boolean containsKeyPattern(Object key) { return ref.containsKey(key); } @Override public boolean containsValue(Object value) { return map.containsValue(value); } @Override public void clear() { map.clear(); matchers.clear(); ref.clear(); } /** * Returns a Set view of the regular expression keys contained in this map. */ @Override public Set keySet() { return map.keySet(); } /** * Returns a Set view of the regex matched patterns contained in this map. The set is backed by the map, so changes to the map are reflected in the set, and vice-versa. * @return */ public Set keySetPattern() { return ref.keySet(); } @Override public Collection values() { return map.values(); } /** * Produces a map of patterns to values, based on the regex put in this map * @param patterns * @return */ public Map transform(List patterns) { for(String pattern : patterns) { get(pattern); } Map transformed = new HashMap(); for(Entry entry : ref.entrySet()) { transformed.put(entry.getKey(), map.get(entry.getValue())); } return transformed; } public static void main(String...strings) { RegexHashMap rh = new RegexHashMap(); rh.put("[o|O][s|S][e|E].?[1|2]", "This is a regex match"); rh.put("account", "This is a direct match"); System.out.println(rh); System.out.println("get:ose-1 -> "+rh.get("ose-1")); System.out.println("get:OSE2 -> "+rh.get("OSE2")); System.out.println("get:OSE112 -> "+rh.get("OSE112")); System.out.println("get:ose-2 -> "+rh.get("ose-2")); System.out.println("get:account -> "+rh.get("account")); System.out.println(rh); } }

daniel bryant , simon and i recently had a discussion about how to represent system communication with external apis. the requirement for integration with external apis is now extremely common but it's not immediately obvious how to clearly show them in architectural diagrams. how to represent an external system? the first thing we discussed was what symbol to use for a system supplying an api. traditionally, uml has used the actor (stick man) symbol to represent a "user or any other system that interacts with the subject" (uml superstructure specification, v2.1.2). therefore a system providing an api may look like this: i've found that this symbol tends to confuse those who aren't well versed in uml as most people assume that the actor symbol always represents a *person* rather than a system. sometimes this is stereotyped to make it more obvious e.g. however the symbol is very powerful and tends to overpower the stereotype. therefore i prefer to use a stereotyped box for an external system supplying an api. let's compare two context diagrams using boxes vs stick actors. in which diagram is it more obvious what are systems or people? note that archimate has a specific symbol for application service that can be used to represent an api: (application service notation from the open group's archimate 2.1 specification) an api or the system that supplies it? whatever symbol we choose, what we've done is to show the *system* rather than the actual api. the api is a definition of a service provided by the system in question. how should we provide more details about the api? there are a number of ways we could do this but my preference is to give details of the api on the connector (line connecting two elements/boxes). in c4 the guidelines for a container diagram includes listing protocol information on the connector and an api can be viewed as the layer above the protocol. for example: multiple apis per external system many api providers supply multiple services/apis (i'm not referring to different operations within an api but multiple sets of operations in different apis, which may even use different underlying protocols.) for example a financial marketplace may have apis that do the following: allow a bulk, batch download of static data (such as details of companies listed on a stock market) via xml over http. supply real time, low latency updates of market prices via bespoke messages over udp. allow entry of trades via industry standard fpml over a queuing system. supply a bulk, batch download of trades for end-of-day reconciliation via fpml over http. two of the services use the same protocol (xml over http) but have very different content and use. one of the apis is used to constantly supply information after user subscription (market data) and the last service involves the user supplying all the information with no acknowledgment (although it should reconcile at eod). there are multiple ways of showing this. we could: have a single service element, list the apis on it and have all components linking to it. show each service/api as a separate box and connect the components that use the individual service to the relevant box. show a single service element with multiple connections. each connection is labeled and represents an api. use a port and connector style notation to represent each api from the service provider. provide a key for the ports. use a uml style 'cup and ball' notation to define interfaces and their usage. some examples are below: a single service element and simple description in the above diagram the containers are stating what they are using but contain no information about how to use the apis. we don't know if it is a single api (with different operations) or anything about the mechanisms used to transport the data. this isn't very useful for anyone implementing a solution or resolving operational issues. single, service box with descriptive connectors in this diagram there is a single, service box with descriptive connectors. the above diagram shows all the information so is much more useful as a diagnostic or implementation tool. however it does look quite crowded. services/apis shown as separate boxes here the external system has its services/apis shown as separate boxes. this contains all the information but might be mistaken as defining the internal structure of the external system. we want to show the services it provides but we know nothing about the internal structure. using ports to represent apis in the above diagram the services/apis are shown as 'ports' on the external system and the details have been moved into a separate key/table. this is less likely to be mistaken as showing any internal structure of the external service. (note that i could have also shown outgoing rports from the brokerage system.) uml interfaces this final diagram is using a uml style interface provider and requirer. this is a clean diagram but requires the user to be aware of what the cup and ball means (although i could have explained this in the key). conclusion any of these solutions could be appropriate depending on the complexity of the api set you are trying to represent. i'd suggest starting with a simple representation (i.e. fully labeled connections) and moving to a more complex one if needed but remember to use a key to explain any elements you use!

REST Easy with SQL/NoSQL Integration and CQRS Pattern implementation New demands are being put on IT organizations everyday to deliver agile, high-performance, integrated mobile and web applications. In the meantime, the technology landscape is getting complex everyday with the advent of new technologies like REST, NoSQL, Cloud while existing technologies like SOAP and SQL still rule everyday work. Rather than taking religious side of the debate, NoSQL can successfully co-exist with SQL in this ‘polyglot’ of data storage and formats. However, this integration also adds another layer of complexity both in architecture and implementation. This document offers a guide on how some of the relatively newer technologies like REST can help bridge the gap between SQL and NoSQL with an example of a well known pattern called CQRS. This document is organized as follows: Introduction to SQL development process NoSQL Do I have to choose between SQL and NoSQL? CQRS Pattern How to implement CQRS pattern using REST services Introduction to SQL development process Developers have been using SQL Databases for decades to build and deliver enterprise business applications. The process of creating tables, attributes,and relationships is second nature for most developers. Data architects think in terms of tables and columns and navigate relationships for data. The basic concepts of delivery and transformation takes place at the web server level which means the server developer is reading and ‘binding’ to the tables and mapping attributes to a REST response. Application development lifecycle meant changes to the database schema first, followed by the bindings, then internal schema mapping, and finally the SOAP or JSON services, and eventually the client code. This all costs the project time and money. It also means that the ‘code’ (pick your language here) and the business logic would also need to be modified to handle the changes to the model. NoSQL NoSQL is gaining supporters among many SQL shops for various reasons including: Low cost Ability to handle unstructured dataa Scalability Performance The first thing database folks notice is that there is no schema. These document style storage engines can handle huge volumes of structured, semi-structured, and unstructured data. The very nature of schema-less documents allows change to a document structure without having to go through the formal change management process (or data architect). The other major difference is that NoSQL (no-schema) also means no joins or relationships. The document itself contains the embedded information by design. So an order entry would contain the customer with all the orders and line items for each order in a single document. There are many different NoSQL vendors (popular NoSQL databases include MongoDB, Casandra) that are being used for BI and Analytics (read-only) purposes. We are also seeing many customers starting to use NoSQL for auditing, logging, and archival transactions. Do I have to choose between SQL and NoSQL? The purpose of this article is to not get into the religious debate about whether to use SQL or NoSQL. Bottom line is both have their place and are suited for certain type of data – SQL for structured data and NoSQL for unstructured data. So why not have the capability to mix and match this data depending on the application. This can be done by creating a single REST API across both SQL and NoSQL databases. Why a single REST API? The answer is simple – the new agile and mobile world demands this ‘mashup’ of data into a document style JSON response. CQRS (Command Query Responsibility Segmentation) Pattern There are many design patterns for delivery of high performance RESTful services but the one that stands out was described in an article written by Martin Fowler, one of the software industry veterans. He described the pattern called CQRS that is more relevant today in a ‘polyglot’ of servers, data, services, and connections. “We may want to look at the information in a different way to the record store, perhaps collapsing multiple records into one, or forming virtual records by combining information for different places. On the update side we may find validation rules that only allow certain combinations of data to be stored, or may even infer data to be stored that’s different from that we provide.” – Martin Fowler 2011 In this design pattern, the REST API requests (GET) return documents from multiple sources (e.g. mashups). In the update process, the data is subject to business logic derivations, validations, event processing, and database transactions. This data may then be pushed back into the NoSQL using asynchronous events. With the wide-spread adoption of NoSQL databases like MongoDB and schema-less, high capacity data store; most developers are challenged with providing security, business logic, event handling, and integration to other systems. MongoDB; one the popular NoSQL databases and SQL databases share many similar concepts. However the MongoDB programming language itself is very different from the SQL we all know. How to implement CQRS pattern using a RESTFul Architecture A REST server should meet certain requirements to support the CQRS pattern. The server should run on-premise or in the cloud and appears to the mobile and web developer as an HTTP endpoint. The server architecture should implement the following: Connections and Mapping necessary for SQL and NoSQL connectivity and API services needed to create and return GET, PUT, POST, and DELETE REST responses Security Business Logic Connections and Mapping There are two main approaches to creating REST Servers and APIs for SQL and NoSQL databases: Open source frameworks like Apache Tomcat, Spring/Hibernate Commercial framework like Espresso Logic Open source Frameworks Using various open source frameworks like Tomcat, Spring/Hibernate, Node.js, JDBC and MongoDB drivers, a REST server can be created, but we would still be left with the following tasks: Creation and mapping of the necessary SQL objects Create a REST server container and configurations Create Jersey/Jackson classes and annotations Create and define REST API for tables, views, and procedures Hand write validation, event and business logic Handle persistence, optimistic locking, transaction paging Adding identity management and security by roles Now we can start down the same path to connect to MongoDB and write code to connect, select, and return data in JSON and then create the REST calls to merge these two different document styles into a single RESTful endpoint. This is a lot of work for a development team to manage and control and frankly pretty boring and repetitive and is better done by a well designed framework Commercial Frameworks Many commercial frameworks may take care of this complexity without the need to do extensive programming. Here is an example from Espresso Logic and how it handles this complexity with a point and click interface: Running REST server in the cloud or on-premise Connections to external SQL databases Object mapping to tables, views, and procedures Automatic creation of RESTful endpoints from model Reactive business rules and rich event model Integrated role-based security and authentication services. Point-and-click document API creation for SQL and MongoDB endpoints In the example below, the editor shows an SQL (customersTransactions) joined with archived details from MongoDB (archivedTransactions). The MongoDB document for each customer may include transaction details, check images, customer service notes and other relevant account information. This new mashup becomes a single REST call that can be published to mobile and web application developer. Security Security is an important part of building and delivery of RESTful services which can be broken down into two parts; authentication and access control. Authentication Before allowing anyone access to corporate data you want to use the existing corporate identity management (some call this authentication services) to capture and validate the user. This identity management service is based on using existing corporate standards such as LDAP, Windows AD, SQL Database. Role-based Access Control Each user may be assigned one or more corporate roles and these roles are then assigned specific access privileges to each resource (e.g. READ, INSERT, UPDATE, and DELETE). Role-based access should also be able to restrict permissions to specific rows and columns of the API (e.g. only sales reps can see their own orders or a manager can see and change his department salaries but cannot change his own). This restriction should be applied regardless of how or where the API is used or called. Remember, the SQL database already provides some level of security and access which must be considered when designing and delivering new front-end services to internal and external users. Business Logic for REST When data is updated to a REST Server several things need to happen. First, the authentication and access control should determine if this is a valid request and if the user has rights to the endpoint. In addition, the server may need to de-alias REST attributes back to the actual SQL column names. In a full featured business logic server, there should be a series of events and business rules to perform various calculations, validations, and fire other events on dependent tables. Finally, the entire multi-table transaction is written back to the SQL database in a single transaction. Updates are then sent asynchronously to MongoDB as part of the commit event (after the SQL transaction has completed). Conclusion In the real-world of API services, the demand for more complex document style RESTful services is a requirement. That is, the ability to create ‘mashups’ of data from multiple tables, NoSQL collections, and other external systems is a large part of this new design pattern. In addition, the ability to alias attribute names and formats from these source fields has become critical for partners and customers systems. Using REST with the CQRS pattern to blend MongoDB and SQL seamlessly to your existing data will become a major part of your future mobile strategy. To implement these REST services, one can use open source tools and spend a lot of time or select a right commercial framework. This framework should support cloud or on-premise connectivity, security, API integration, as well as business logic. This will make the design and delivery of new application services more rapid and agile in the heterogeneous world of information.

Spring provides a great out of the box support for caching expensive method calls. The caching abstraction is covered in a great detail here. My objective here is to cover one of the newer cache implementations that Spring now provides with 4.0+ version of the framework - using Google Guava Cache In brief, consider a service which has a few slow methods: public class DummyBookService implements BookService { @Override public Book loadBook(String isbn) { // Slow method 1. } @Override public List loadBookByAuthor(String author) { // Slow method 2 } } With Spring Caching abstraction, repeated calls with the same parameter can be sped up by an annotation on the method along these lines - here the result of loadBook is being cached in to a "book" cache and listing of books cached into another "books" cache: public class DummyBookService implements BookService { @Override @Cacheable("book") public Book loadBook(String isbn) { // slow response time.. } @Override @Cacheable("books") public List loadBookByAuthor(String author) { // Slow listing } } Now, Caching abstraction support requires a CacheManager to be available which is responsible for managing the underlying caches to store the cached results, with the new Guava Cache support the CacheManager is along these lines: @Bean public CacheManager cacheManager() { return new GuavaCacheManager("books", "book"); } Google Guava Cache provides a rich API to be able to pre-load the cache, set eviction duration based on last access or created time, set the size of the cache etc, if the cache is to be customized then a guava CacheBuilder can be passed to the CacheManager for this customization: @Bean public CacheManager cacheManager() { GuavaCacheManager guavaCacheManager = new GuavaCacheManager(); guavaCacheManager.setCacheBuilder(CacheBuilder.newBuilder().expireAfterAccess(30, TimeUnit.MINUTES)); return guavaCacheManager; } This works well if all the caches have a similar configuration, what if the caches need to be configured differently - for eg. in the sample above, I may want the "book" cache to never expire but the "books" cache to have an expiration of 30 mins, then the GuavaCacheManager abstraction does not work well, instead a better solution is actually to use a SimpleCacheManager which provides a more direct way to get to the cache and can be configured this way: @Bean public CacheManager cacheManager() { SimpleCacheManager simpleCacheManager = new SimpleCacheManager(); GuavaCache cache1 = new GuavaCache("book", CacheBuilder.newBuilder().build()); GuavaCache cache2 = new GuavaCache("books", CacheBuilder.newBuilder() .expireAfterAccess(30, TimeUnit.MINUTES) .build()); simpleCacheManager.setCaches(Arrays.asList(cache1, cache2)); return simpleCacheManager; } This approach works very nicely, if required certain caches can be configured to be backed by a different caching engines itself, say a simple hashmap, some by Guava or EhCache some by distributed caches like Gemfire.

As memory gets cheaper and cheaper, our applications can keep more data readily available in main memory, or even all as in case of in-memory databases. To make real use of the growing heap memory, appropriate data structures must be used. Interesting enough, there seem to be no specialized implementations for lists - by far the most used collection. This article introduces BigList, a list designed for handling large collections where large means that all data still fit completely in the heap memory. The article will show the special requirements for handling large collections, how BigList is implemented and how it compares to other list implementations. 1. Requirements What are the special requirements we need to handle large collections efficiently? Memory: Sparing use of memory: The list should need little memory for its own implementation so memory can be used for storing application data. Specialized versions for primitives: It must be possible to store common primitives like ints in a memory saving way. Avoid copying large data blocks: If the list grows or shrinks, only a small part of the data must be copied around, as this operation becomes expensive and needs the same amount of memory again. Data sharing: copying collections is a frequent operation which should be efficiently possible even if the collection is large. An efficient implementation requires some sort of data sharing as copying all elements is per se a costly operation. Performance: Good performance for normal operations like reading, storing, adding or removing single elements. Great performance for bulk operations like adding or removing multiple elements. Predictable overhead of operations, so similar operations should need a similar amount of time without excessive worst case scenarios. If an implementation does not offer these features, some operations will not only be slow for really large collections, but will becomse just not feasible because memory or CPU usage will be too exhaustive. Introduction to BigList BigList is a member of the Brownies Collections library which also includes GapList, the fastest list implementation known. GapList is a drop-in replacement for ArrayList, LinkedList, or ArrayDequeue and offers fast access by index and fast insertion/removal at the beginning and at the end at the same time. GapList however has not been designed to cope with large collections, so adding or removing elements can make it necessary to copy a lot of elements around which will lead to performance problems. Also copying a large collection becomes an expensive operation, both in term of time and memory consumption. It will simply not be possible to make a copy of a large collections if not the same amount of memory is available a second time. And this is a common operation as you often want to return a copy of an internal list through your API which has no reference the original list. BigList addresses both problems. The first problem is solved by storing the collection elements in fixed size blocks. Add or remove operations are then implemented to use only data from one block. The copying problem is solved by maintaining a reference count on the fixed size blocks which allows to implement a copy-on-write approach. For efficient access to the fixed size blocks, they are maintained in a specialized tree structure. 2. BigList Details Each BigList instance stores the following information: Elements are stored in in fixed-size blocks A single block is implemented as GapList with a reference count for sharing All blocks are maintained in a tree for fast access Access information for the current block is cached for better performance The following illustration shows these details for two instances of BigList which share one block. 2.1 Use of Blocks Elements are stored in in fixed-size blocks with a default block size of 1000. Where this default may look pretty small, it is most of the time a good choice because it guarantees that write operation only need to move few elements. Read operations will profit from locality of reference by using the currently cached block to be fast. It is however possible to specify the block size for each created BigList instance. All blocks except the first are allocated with this fixed size and will not grow or shrink. The first block will grow to the specified block size to save memory for small lists. If a block has reached its maximum size and more data must be stored within, the block needs to be split up in two blocks before more elements can be stored. If elements are added to the head or tail of the list, the block will only be filled up to a threshold of 95%. This allows inserts into the block without the immediate need for split operations. To save memory, blocks are also merged. This happens automatically if two adjacent blocks are both filled less than 35% after a remove operation. 2.2 Locality of Reference For each operation on BigList, the affected block must be determined first. As predicted by locality of reference, most of the time the affected block will be the same as for the last operation. The implementation of BigList has therefore been designed to profit from locality of reference which makes common operations like iterating over a list very efficient. Instead of always traversing the block tree to determine the block needed for an operation, lower and upper index of the last used block are cached. So if the next operation happens near to the previous one, the same block can be used again without need to traverse the tree. 2.3 Reference Counting To support a copy-on-write approach, BigList stores a reference count for each fixed size blocks indicating whether this block is private or shared. Initially all lists are private having a reference count of 0, so that modification are allowed. If a list is copied, the reference count is incremented which prohibits further modifications. Before a modification then can be made, the block must be copied decrementing the block's reference count and setting the reference count of the copy to 0. The reference count of a block is then decremented by the finalizer of BigList. 3. Benchmarks To prove the excellence of BigList in time and memory consumption, we compare it with some other List implementations. And here are the nominees: Type Library Description BigList brownie-collections List optimized for storing large number of elements. Elements stored in fixed size blocks which are maintained in a tree. GapList brownie-collections Fastest list implementation known. Fast access by index and fast insertion/removal at end and at beginning. ArrayList JDK Maintains elements in a single array. Fast access by index, fast insertion/removal at end, but slow at beginning. LinkedList JDK Elements stored using a linked list. Slow access by index. Memory overhead for each element stored. TreeList commons-collections Elements stored in a tree. All operations are not really fast, but there are no very slow operations. Memory overhead for each element stored. FastTable javolution Elements stored in a "fractal"-like data structure. Good performance and use of memory. However no bulk operations and collection does not shrink. 3.1 Handling Objects In the first part of the benchmark, we compare memory consumption and performance of the different list implementations. Let's first have a look at the memory consumption. The following table shows the bytes used to hold a list with 1'000'000 null elements: BigList GapList ArrayList LinkedList TreeList FastTable 32 bit 4'298'466 4'021'296 4'861'992 8'000'028 18'000'028 4'142'892 64 bit 8'544'254 8'042'552 9'723'964 16'000'044 26'000'044 8'222'988 We can see that BigList, GapList, ArrayList, and FastTable only add small overhead to the stored elements, where as Linkedlist needs twice the memory and TreeList even more. Now to the performance. Here are the results of 9 benchmarks which have been run for each of the 6 candidates with JDK 8 in a 32 bit Windows environment and a list of 1'000'000 elements: The result table can be read as follows: the fastest candidate for each test has a relative performance indicator of 1 the value for the other candidates indicate how many times they have been slower, so a factor of 3 means that this implementation was 3 times slower than the best one The different factor are colored like this: 
factor 1: green (best)
 factor <5: blue (good) 
factor <25: yellow (moderate) 
factor >25: red (poor) If we look the benchmark result, we can see that the performance of BigList is best for all expect two benchmarks. The only moderate result is produces in getting elements in a totally random order. This could be expected as there is no locality of reference which can be exploited, so for each access, the block tree must be traversed to find the correct block. Luckily this is a rare use case in real applications. And the benchmark "Get local" shows that performance is back to good as soon as elements next to each other must be retrieved - as it is the case if we iterate over a range. 3.2 Handling Primitives In the second part of the benchmark, we want see how big the savings are if we use a data structure specialized for storing primitives compared to strong wrapped objects. For this reason, we compare IntBigList and BigList. The following table shows memory needed to store 1'000'000 integer values: BigList IntBigList 32 bit 16'298'454 4'534'840 64 bit 28'544'234 4'570'432 Obviously it is easy to save a lot of memory. In a 32 bit environment, IntBigList just needs 25% percent of memory, in a 64 bit environment only 14%! These figures become plausible if you recall that a simple object needs 8 bytes in a 32 bit, but already 16 bytes in a 64 bit environment, where as a primitive integer value always only needs 4 bytes. The measurable performance gain is not so impressive, it is something below 10% for simple get operations and something above 10% for add and remove operations. These numbers show that the JVM is impressively fast in creating wrapper objects and boxing and unboxing primitive values. We must however also consider that each created object will need to be garbage collected once and therefore adds to the total load of the JVM. 4. Summary BigList is a scalable high-performance list for storing large collections. Its design guarantees that all operations will be predictable and efficient both in term of performance and memory consumption, even copying large collections is tremendous fast. Benchmarks haven proven this and shown that BigList outperform other known list implementations. The library also offers specialized implementations for primitive types like IntBigList which save much memory and provide superior performance. BigList for handling objects and the specializations for handling primitives are part of the Brownies Collections library and can be downloaded from http://www.magicwerk.org/collections.

if someone asked you to develop a rest api on the jvm, which frameworks would you use? i was recently tasked with such a project. my client asked me to implement a rest api to ingest requests from a 3rd party. the project entailed consuming xml requests, storing the data in a database, then exposing the data to internal application with a json endpoint. finally, it would allow taking in a json request and turning it into an xml request back to the 3rd party. with the recent release of apache camel 2.14 and my success using it , i started by copying my apache camel / cxf / spring boot project and trimming it down to the bare essentials. i whipped together a simple hello world service using camel and spring mvc. i also integrated swagger into both. both implementations were pretty easy to create ( sample code ), but i decided to use spring mvc. my reasons were simple: its rest support was more mature, i knew it well, and spring mvc test makes it easy to test apis. camel's swagger support without web.xml as part of the aforementioned spike, i learned out how to configure camel's rest and swagger support using spring's javaconfig and no web.xml. i made this into a sample project and put it on github as camel-rest-swagger . this article shows how i built a rest api with java 8, spring boot/mvc, jaxb and spring data (jpa and rest components). i stumbled a few times while developing this project, but figured out how to get over all the hurdles. i hope this helps the team that's now maintaining this project (my last day was friday) and those that are trying to do something similar. xml to java with jaxb the data we needed to ingest from a 3rd party was based on the ncpdp standards. as a member, we were able to download a number of xsd files, put them in our project and generate java classes to handle the incoming/outgoing requests. i used the maven-jaxb2-plugin to generate the java classes. org.jvnet.jaxb2.maven2 maven-jaxb2-plugin 0.8.3 generate -xtostring -xequals -xhashcode -xcopyable org.jvnet.jaxb2_commons jaxb2-basics 0.6.4 src/main/resources/schemas/ncpdp the first error i ran into was about a property already being defined. [info] --- maven-jaxb2-plugin:0.8.3:generate (default) @ spring-app --- [error] error while parsing schema(s).location [ file:/users/mraible/dev/spring-app/src/main/resources/schemas/ncpdp/structures.xsd{1811,48}]. com.sun.istack.saxparseexception2; systemid: file:/users/mraible/dev/spring-app/src/main/resources/schemas/ncpdp/structures.xsd; linenumber: 1811; columnnumber: 48; property "multipletimingmodifierandtimingandduration" is already defined. use to resolve this conflict. at com.sun.tools.xjc.errorreceiver.error(errorreceiver.java:86) i was able to workaround this by upgrading to maven-jaxb2-plugin version 0.9.1. i created a controller and stubbed out a response with hard-coded data. i confirmed the incoming xml-to-java marshalling worked by testing with a sample request provided by our 3rd party customer. i started with a curl command, because it was easy to use and could be run by anyone with the file and curl installed. curl -x post -h 'accept: application/xml' -h 'content-type: application/xml' \ --data-binary @sample-request.xml http://localhost:8080/api/message -v this is when i ran into another stumbling block: the response wasn't getting marshalled back to xml correctly. after some research, i found out this was caused by the lack of @xmlrootelement annotations on my generated classes. i posted a question to stack overflow titled returning jaxb-generated elements from spring boot controller . after banging my head against the wall for a couple days, i figured out the solution . i created a bindings.xjb file in the same directory as my schemas. this causes jaxb to generate @xmlrootelement on classes. to add namespaces prefixes to the returned xml, i had to modify the maven-jaxb2-plugin to add a couple arguments. -extension -xnamespace-prefix and add a dependency: org.jvnet.jaxb2_commons jaxb2-namespace-prefix 1.1 then i modified bindings.xjb to include the package and prefix settings. i also moved into a global setting. i eventually had to add prefixes for all schemas and their packages. i learned how to add prefixes from the namespace-prefix plugins page . finally, i customized the code-generation process to generate joda-time's datetime instead of the default xmlgregoriancalendar . this involved a couple custom xmladapters and a couple additional lines in bindings.xjb . you can see the adapters and bindings.xjb with all necessary prefixes in this gist . nicolas fränkel's customize your jaxb bindings was a great resource for making all this work. i wrote a test to prove that the ingest api worked as desired. @runwith(springjunit4classrunner.class) @springapplicationconfiguration(classes = application.class) @webappconfiguration @dirtiescontext(classmode = dirtiescontext.classmode.after_class) public class initiaterequestcontrollertest { @inject private initiaterequestcontroller controller; private mockmvc mockmvc; @before public void setup() { mockitoannotations.initmocks(this); this.mockmvc = mockmvcbuilders.standalonesetup(controller).build(); } @test public void testgetnotallowedonmessagesapi() throws exception { mockmvc.perform(get("/api/initiate") .accept(mediatype.application_xml)) .andexpect(status().ismethodnotallowed()); } @test public void testpostpainitiationrequest() throws exception { string request = new scanner(new classpathresource("sample-request.xml").getfile()).usedelimiter("\\z").next(); mockmvc.perform(post("/api/initiate") .accept(mediatype.application_xml) .contenttype(mediatype.application_xml) .content(request)) .andexpect(status().isok()) .andexpect(content().contenttype(mediatype.application_xml)) .andexpect(xpath("/message/header/to").string("3rdparty")) .andexpect(xpath("/message/header/sendersoftware/sendersoftwaredeveloper").string("hid")) .andexpect(xpath("/message/body/status/code").string("010")); } } spring data for jpa and rest with jaxb out of the way, i turned to creating an internal api that could be used by another application. spring data was fresh in my mind after reading about it last summer. i created classes for entities i wanted to persist, using lombok's @data to reduce boilerplate. i read the accessing data with jpa guide, created a couple repositories and wrote some tests to prove they worked. i ran into an issue trying to persist joda's datetime and found jadira provided a solution. i added its usertype.core as a dependency to my pom.xml: org.jadira.usertype usertype.core 3.2.0.ga ... and annotated datetime variables accordingly. @column(name = "last_modified", nullable = false) @type(type="org.jadira.usertype.dateandtime.joda.persistentdatetime") private datetime lastmodified; with jpa working, i turned to exposing rest endpoints. i used accessing jpa data with rest as a guide and was looking at json in my browser in a matter of minutes. i was surprised to see a "profile" service listed next to mine, and posted a question to the spring boot team. oliver gierke provided an excellent answer . swagger spring mvc's integration for swagger has greatly improved since i last wrote about it . now you can enable it with a @enableswagger annotation. below is the swaggerconfig class i used to configure swagger and read properties from application.yml . @configuration @enableswagger public class swaggerconfig implements environmentaware { public static final string default_include_pattern = "/api/.*"; private relaxedpropertyresolver propertyresolver; @override public void setenvironment(environment environment) { this.propertyresolver = new relaxedpropertyresolver(environment, "swagger."); } /** * swagger spring mvc configuration */ @bean public swaggerspringmvcplugin swaggerspringmvcplugin(springswaggerconfig springswaggerconfig) { return new swaggerspringmvcplugin(springswaggerconfig) .apiinfo(apiinfo()) .genericmodelsubstitutes(responseentity.class) .includepatterns(default_include_pattern); } /** * api info as it appears on the swagger-ui page */ private apiinfo apiinfo() { return new apiinfo( propertyresolver.getproperty("title"), propertyresolver.getproperty("description"), propertyresolver.getproperty("termsofserviceurl"), propertyresolver.getproperty("contact"), propertyresolver.getproperty("license"), propertyresolver.getproperty("licenseurl")); } } after getting swagger to work, i discovered that endpoints published with @repositoryrestresource aren't picked up by swagger. there is an open issue for spring data support in the swagger-springmvc project. liquibase integration i configured this project to use h2 in development and postgresql in production. i used spring profiles to do this and copied xml/yaml (for maven and application*.yml files) from a previously created jhipster project. next, i needed to create a database. i decided to use liquibase to create tables, rather than hibernate's schema-export. i chose liquibase over flyway based of discussions in the jhipster project . to use liquibase with spring boot is dead simple: add the following dependency to pom.xml, then place changelog files in src/main/resources/db/changelog . org.liquibase liquibase-core i started by using hibernate's schema-export and changing hibernate.ddl-auto to "create-drop" in application-dev.yml . i also commented out the liquibase-core dependency. then i setup a postgresql database and started the app with "mvn spring-boot:run -pprod". i generated the liquibase changelog from an existing schema using the following command (after downloading and installing liquibase). liquibase --driver=org.postgresql.driver --classpath="/users/mraible/.m2/repository/org/postgresql/postgresql/9.3-1102-jdbc41/postgresql-9.3-1102-jdbc41.jar:/users/mraible/snakeyaml-1.11.jar" --changelogfile=/users/mraible/dev/spring-app/src/main/resources/db/changelog/db.changelog-02.yaml --url="jdbc:postgresql://localhost:5432/mydb" --username=user --password=pass generatechangelog i did find one bug - the generatechangelog command generates too many constraints in version 3.2.2 . i was able to fix this by manually editing the generated yaml file. tip: if you want to drop all tables in your database to verify liquibase creation is working in postgesql, run the following commands: psql -d mydb drop schema public cascade; create schema public; after writing minimal code for spring data and configuring liquibase to create tables/relationships, i relaxed a bit, documented how everything worked and added a loggingfilter . the loggingfilter was handy for viewing api requests and responses. @bean public filterregistrationbean loggingfilter() { loggingfilter filter = new loggingfilter(); filterregistrationbean registrationbean = new filterregistrationbean(); registrationbean.setfilter(filter); registrationbean.seturlpatterns(arrays.aslist("/api/*")); return registrationbean; } accessing api with resttemplate the final step i needed to do was figure out how to access my new and fancy api with resttemplate . at first, i thought it would be easy. then i realized that spring data produces a hal -compliant api, so its content is embedded inside an "_embedded" json key. after much trial and error, i discovered i needed to create a resttemplate with hal and joda-time awareness. @bean public resttemplate resttemplate() { objectmapper mapper = new objectmapper(); mapper.configure(deserializationfeature.fail_on_unknown_properties, false); mapper.registermodule(new jackson2halmodule()); mapper.registermodule(new jodamodule()); mappingjackson2httpmessageconverter converter = new mappingjackson2httpmessageconverter(); converter.setsupportedmediatypes(mediatype.parsemediatypes("application/hal+json")); converter.setobjectmapper(mapper); stringhttpmessageconverter stringconverter = new stringhttpmessageconverter(); stringconverter.setsupportedmediatypes(mediatype.parsemediatypes("application/xml")); list> converters = new arraylist<>(); converters.add(converter); converters.add(stringconverter); return new resttemplate(converters); } the jodamodule was provided by the following dependency: com.fasterxml.jackson.datatype jackson-datatype-joda with the configuration complete, i was able to write a messagesapiitest integration test that posts a request and retrieves it using the api. the api was secured using basic authentication, so it took me a bit to figure out how to make that work with resttemplate. willie wheeler's basic authentication with spring resttemplate was a big help. @runwith(springjunit4classrunner.class) @contextconfiguration(classes = integrationtestconfig.class) public class messagesapiitest { private final static log log = logfactory.getlog(messagesapiitest.class); @value("http://${app.host}/api/initiate") private string initiateapi; @value("http://${app.host}/api/messages") private string messagesapi; @value("${app.host}") private string host; @inject private resttemplate resttemplate; @before public void setup() throws exception { string request = new scanner(new classpathresource("sample-request.xml").getfile()).usedelimiter("\\z").next(); responseentity response = resttemplate.exchange(gettesturl(initiateapi), httpmethod.post, getbasicauthheaders(request), org.ncpdp.schema.transport.message.class, collections.emptymap()); assertequals(httpstatus.ok, response.getstatuscode()); } @test public void testgetmessages() { httpentity request = getbasicauthheaders(null); responseentity> result = resttemplate.exchange(gettesturl(messagesapi), httpmethod.get, request, new parameterizedtypereference>() {}); httpstatus status = result.getstatuscode(); collection messages = result.getbody().getcontent(); log.debug("messages found: " + messages.size()); assertequals(httpstatus.ok, status); for (message message : messages) { log.debug("message.id: " + message.getid()); log.debug("message.datecreated: " + message.getdatecreated()); } } private httpentity getbasicauthheaders(string body) { string plaincreds = "user:pass"; byte[] plaincredsbytes = plaincreds.getbytes(); byte[] base64credsbytes = base64.encodebase64(plaincredsbytes); string base64creds = new string(base64credsbytes); httpheaders headers = new httpheaders(); headers.add("authorization", "basic " + base64creds); headers.add("content-type", "application/xml"); if (body == null) { return new httpentity<>(headers); } else { return new httpentity<>(body, headers); } } } to get spring data to populate the message id, i created a custom restconfig class to expose it. i learned how to do this from tommy ziegler . /** * used to expose ids for resources. */ @configuration public class restconfig extends repositoryrestmvcconfiguration { @override protected void configurerepositoryrestconfiguration(repositoryrestconfiguration config) { config.exposeidsfor(message.class); config.setbaseuri("/api"); } } summary this article explains how i built a rest api using jaxb, spring boot, spring data and liquibase. it was relatively easy to build, but required some tricks to access it with spring's resttemplate. figuring out how to customize jaxb's code generation was also essential to make things work. i started developing the project with spring boot 1.1.7, but upgraded to 1.2.0.m2 after i found it supported log4j2 and configuring spring data rest's base uri in application.yml. when i handed the project off to my client last week, it was using 1.2.0.build-snapshot because of a bug when running in tomcat . this was an enjoyable project to work on. i especially liked how easy spring data makes it to expose jpa entities in an api. spring boot made things easy to configure once again and liquibase seems like a nice tool for database migrations. if someone asked me to develop a rest api on the jvm, which frameworks would i use? spring boot, spring data, jackson, joda-time, lombok and liquibase. these frameworks worked really well for me on this particular project.

In Parts 1 and 2 we have covered a number of common issues people run into when managing a sharded MongoDB cluster. In this final post of the series we will cover a subtle, but important distinction in terms of balancing a sharded cluster as well as an interesting limitation that can be worked around relatively easily, but is nonetheless surprising when it comes up. 6. Chunk balancing != data balancing != traffic balancing The balancer in a sharded cluster cares about just one thing: Are chunks for a given collection evenly balanced across all shards? If they are not, then it will take steps to rectify that imbalance. This all sounds perfectly logical, and even with extra complexity like tagging involved the logic is pretty straight forward. If we assume that all chunks are equal, then we can rest assured that our data is being evenly balanced across all the shards in our cluster and rest easy at night. Although that is sometimes, perhaps even frequently, the case it is not always true - chunks are not always equal. There can be massive “jumbo” chunks that exceed the maximum chunk size (64MiB), completely empty chunks and everything in between. Let’s use an example from our first pitfall, the monotonically increasing shard key. For our example, we have picked just such a key to shard on (date), and up until this point we have had just one shard and had not sharded the collection. We are about to add a second shard to our cluster and so we enable sharding on the collection and do the necessary admin work to add the new shard into the cluster. Once the collection is enabled for sharding, the first shard contains all the newly minted chunks. Let’s represent them in a simplified table of 10 chunks. This is not representative of a real data set, but it will do for illustrative purposes: Table 1 - Initial Chunk Layout Now we add our second shard. The balancer will kick in and attempt to distribute the chunks evenly. It will do this by moving the lowest range chunks to the new shard until the counts are identical. Once it is finished balancing, our table now looks like this: Table 2 - Balanced Chunk Layout That looks pretty good at the moment, but lets imagine that more recent chunks are more likely to have more activity (updates say) than older chunks. Adding the traffic share estimates for each chunk shows that shard1 is taking far more traffic (72%) than shard2 (28%) despite the chunks seeming balanced overall based on the approximate size. Hence, chunk balancing is not equal to traffic balancing. Using that same example, let’s add another wrinkle - periodic deletion of old data. Every 3 months we run a job to delete any data older than 12 months. Let’s look at the impact of that on our table after we run it for the first time (assuming the first run happens on July 1st 2015). Table 3 - Post-Delete Chunk Layout The distribution of data is now completely skewed toward shard1 - shard2 is in fact empty! However, the balancer is completely unaware of this imbalance - the chunk count has remained the same the entire time, and as far as it is concerned the system is in a steady state. With no data on shard2, our traffic imbalance as seen above will be even worse, and we have essentially negated the benefit of having a second shard for this collection. Possible Mitigation Strategies If data and traffic balance are important, select an appropriate shard key Move chunks manually to address the imbalances - swap “hot” chunks for “cool” chunks, empty chunks for larger chunks 7. Waiting too long to shard a collection (collection too large) This is not very common, but when it falls on your shoulders, it can be quite challenging to solve. There is a maximum data size for a collection when when it is initially split which is a function of the chunk size and data size as noted on the limits page. If your collection contains less than 256GiB of data, then there will be no issue. If the collection size exceeds 256GiB but is less than 400GiB, then MongoDB may be able to do an initial split without any special measures being taken. Otherwise, with larger initial data sizes and the default settings, the initial split will fail. It is worth noting that once split the collection may grow as needed and without any real limitations as long as you can continue to add shards as data size grows. Possible Mitigation Strategies Since the limit is dictated by the chunk size and the data size, and assuming there is not much to be done about the data size, then the remaining variable is the chunk size. This is adjustable (default is 64MiB) and can be raised in order to let a large collection split initially and then reduced once that has been completed. The required chunk size increase will depend on the actual data size. However, this is relatively easy to work out - simply divide your data size by 256GB and then multiply that figure by 64MiB (and round up if it is not a nice even number). As an example, let’s consider a 4TiB collection: 4TiB divided by 256GiB = 16 64MiB x 16 = 1024MiB Hence, set the max chunk size to 1024MiB, then perform the initial sharding of the collection, and then finally reduce the chunk size back to 64MiB using the same procedure. . Thanks for reading through the Sharding Pitfall series! If you want to learn more about managing MongoDB deployments at scale, sign up for my online education course, MongoDB Advanced Deployment and Operations. Planning for scale? No problem: MongoDB is here to help. Get a preview of what it’s like to work with MongoDB’s Technical Services Team. Give us some details on your deployment and we can set you up with an expert who can provide detailed guidance on all aspects of scaling with MongoDB, based on our experience with hundreds of deployments.

In this article, we learn how to utilize AngularJS's array filter features in a variety of use cases. Read on to get started.

introduction in this tutorial, the apache lucene and apache tika frameworks will be explained through their core concepts (e.g. parsing, mime detection, content analysis, indexing, scoring, boosting) via illustrative examples that should be applicable to not only seasoned software developers but to beginners to content analysis and programming as well. we assume you have a working knowledge of the java™ programming language and plenty of content to analyze. throughout this tutorial, you will learn: how to use apache tika's api and its most relevant functions how to develop code with apache lucene api and its most important modules how to integrate apache lucene and apache tika in order to build your own piece of software that stores and retrieves information efficiently. (project code is available for download) what are lucene and tika? according to apache lucene's site, apache lucene represents an open source java library for indexing and searching from within large collections of documents. the index size represents roughly 20-30% the size of text indexed and the search algorithms provide features like: ranked searching - best results returned first many powerful query types: phrase queries, wildcard queries, proximity queries, range queries and more. in this tutorial we will demonstrate only phrase queries. fielded search (e.g. title, author, contents) sorting by any field flexible faceting, highlighting, joins and result grouping pluggable ranking models, including the vector space model and okapi bm25 but lucene's main purpose is to deal directly with text and we want to manipulate documents, who have various formats and encoding. for parsing document content and their properties the apache tika library it is necessary. apache tika is a library that provides a flexible and robust set of interfaces that can be used in any context where metadata analyzis and structured text extraction is needed. the key component of apache tika is the parser (org.apache.tika.parser.parser ) interface because it hides the complexity of different file formats while providing a simple and powerful mechanism to extract structured text content and metadata from all sorts of documents. criterias for tika parsing design streamed parsing the interface should require neither the client application nor the parser implementation to keep the full document content in memory or spooled to disk. this allows even huge documents to be parsed without excessive resource requirements. structured content a parser implementation should be able to include structural information (headings, links, etc.) in the extracted content. a client application can use this information for example to better judge the relevance of different parts of the parsed document. input metadata a client application should be able to include metadata like the file name or declared content type with the document to be parsed. the parser implementation can use this information to better guide the parsing process. output metadata a parser implementation should be able to return document metadata in addition to document content. many document formats contain metadata like the name of the author that may be useful to client applications. context sensitivity while the default settings and behaviour of tika parsers should work well for most use cases, there are still situations where more fine-grained control over the parsing process is desirable. it should be easy to inject such context-specific information to the parsing process without breaking the layers of abstraction. requirements maven 2.0 or higher java 1.6 se or higher lesson 1: automate metadata extraction from any file type our premisses are the following: we have a collection of documents stored on disk/database and we would like to index them; these documents can be word documents, pdfs, htmls, plain text files etc. as we are developers, we would like to write reusable code that extracts file properties regarding format (metadata) and file content. apache tika has a mimetype repository and a set of schemes (any combination of mime magic, url patterns, xml root characters, or file extensions) to determine if a particular file, url, or piece of content matches one of its known types. if the content does match, tika has detected its mimetype and can proceed to select the appropriate parser. in the sample code, the file type detection and its parsing is being covered inside the class com.retriever.lucene.index.indexcreator , method indexfile. listing 1.1 analyzing a file with tika public static documentwithabstract indexfile(analyzer analyzer, file file) throws ioexception { metadata metadata = new metadata(); contenthandler handler = new bodycontenthandler(10 * 1024 * 1024); parsecontext context = new parsecontext(); parser parser = new autodetectparser(); inputstream stream = new fileinputstream(file); //open stream try { parser.parse(stream, handler, metadata, context); //parse the stream } catch (tikaexception e) { e.printstacktrace(); } catch (saxexception e) { e.printstacktrace(); } finally { stream.close(); //close the stream } //more code here } the above code displays how a file it is being parsed using org.apache.tika.parser.autodetectparser; this kind of implementation was chosen because we would like to achieve parsing documents disregarding their format. also, for handling the content the org.apache.tika.sax.bodycontenthandler wasconstructed with a writelimit given as parameter ( 10*1024*1024); this type of constructor creates a content handler that writes xhtml body character events to an internal string buffer and in case of documents with large content is less likely to throw a saxexception (thrown when the default write limit is reached). as a result of our parsing we have obtained a metadata object that we can now use to detect file properties (title or any other header specific to a document format). metadata processing can be done as described below ( com.retriever.lucene.index.indexcreator , method indexfiledescriptors) : listing 1.2 processing metadata private static document indexfiledescriptors(string filename, metadata metadata) { document doc = new document(); //store file name in a separate textfield doc.add(new textfield(isearchconstants.field_file, filename, store.yes)); for (string key : metadata.names()) { string name = key.tolowercase(); string value = metadata.get(key); if (stringutils.isblank(value)) { continue; } if ("keywords".equalsignorecase(key)) { for (string keyword : value.split(",?(\\s+)")) { doc.add(new textfield(name, keyword, store.yes)); } } else if (isearchconstants.field_title.equalsignorecase(key)) { doc.add(new textfield(name, value, store.yes)); } else { doc.add(new textfield(name, filename, store.no)); } } in the method presented above we store the file name in a separate field and also the document's title ( a document can have a title different from its file name); we are not interested in storing other informations.

introduction i had an interesting conversation today about the cost of using string concatenation in log statements. in particular, debug log statements often need to print out parameters or the current value of variables, so they need to build the log message dynamically. so you wind up with something like this: logger.debug("parameters: " + param1 + "; " + param2); the issue arises when debug logging is turned off. inside the logger.debug() statement a flag is checked and the method returns immediately; this is generally pretty fast. but the string concatenation had to occur to build the parameter prior to calling the method, so you still pay its cost. since debug tends to be turned off in production, this is the time when this difference matters. for this reason, we have pretty much all been trained to do this: if (logger.isdebugenabled()) { logger.debug("parameters: " + param1 + "; " + param2); } the discussion was about how much difference this “good practice” makes. caliper this kind of question is perfect for a micro benchmark. my own favorite tool for this purpose is caliper . caliper runs small snippets of code enough times to average out variations. it passes in a number of repetitions, which it calculates in order to make sure that the whole method takes long enough to measure given the resolution of the system clock. caliper also detects garbage collection and hotspot compiling that might impact the accuracy of the tests. caliper uploads results to a google app engine application. its sign-in supports google logins and issues an api key that can be used to organize results and list them. a typical timing methods looks like this: public string timemultstringnocheck(long reps) { for (int i = 0; i < reps; i++) { logger.debug(strings[0] + " " + strings[1] + " " + strings[2] + " " + strings[3] + " " + strings[4]); } return strings[0]; } the return string is not used; it is included in the method solely to ensure that java does not optimize away the method. similarly, the content of the variables used should be randomly generated to avoid compile-time optimization. the full example is available in one of my github repositories , in the org.anvard.introtojava.log package. results the outcome is pretty interesting. string concatenation creates a pretty significant penalty, around two orders of magnitude for our example that concatenates five strings. interesting is that even in the case where we do not use string concatenation (i.e. the simplestring methods), the penalty is around 4x. this is probably the time spent pushing the string parameter onto the stack. the examples with doubles, using string.format() , is even more extreme, four orders of magnitude. the elapsed time here about 4ms, large enough that if the log statement were in a commonly used method, the performance hit would be noticeable. the final method, multstringparams , uses a feature that is available in the slf4j api. it works similarly to string.format() , but in a simple token replace fashion. most importantly, it does not perform the token replace unless the logging level is enabled. this makes this form just as fast as the “check” forms, but in a more compact form. of course, this only works if no special formatting is needed of the log string, or if the formatting can be shifted to a method such as tostring() . what is especially surprising is that this method did not show a penalty in building the object array necessary to pass the parameters into the method. this may have been optimized out by the java runtime since there was no chance of the parameters being used. conclusion the practice of checking whether a logging level is enabled before building the log statement is certainly worthwhile and should be something teams check during peer review.

Learn how to use lambda expressions and anonymous functions in Java 8.

when using a bootloader (see “ serial bootloader for the freedom board with processor expert “), then i usually protect the bootloader flash areas, so it does not get accidentally erased by the application ;-). when programming my boards with the p&e multilink, then the p&e firmware will automatically unlock and erase the chip. that’s not the same if working with the segger j-link, as it requires extra steps. protected flash pages with processor expert failed programming with protected flash if i try to re-program the protected bootloader with segger j-link (both in codewarrior and eclipse/kds with gdb), then the download silently fails. the effect is that somehow the application on the board does not match what it should be. looking at the console view, it shows that erase has failed (but no real error reported) :-(: jlink: failed to erase sectors 0 @ address 0x00000000 (algo135: flash protection violation. flash is write-protected.) j-link failed to erase in codewarrior the gnu arm eclipse segger integration with gdb (e.g. kinetis design studio) is not better: no error sign, the only thing is a hidden error in the console log of the jlinkgdbservercl: error: failed to erase sectors 0 @ address 0x00000000 (algo135: flash protection violation. flash is write-protected.) error algo135 flash protection violation about failed flash programming what i need is to unprotect the memory and then erase it. erasing the segger j-link features a very fast programming. part of that speed is that the segger firmware checks each flash page if it really needs to be programmed, and only then it erases and reprogrammed that page. so downloading twice the same application actually will not touch the flash memory at all. additionally, it does not do a complete erase of the device: it only programs the pages i’m using in my application. the advantage of that is first speed. and it does not erase the application data i’m using in non-volatile memory (see “ configuration data: using the internal flash instead of an external eeprom “). however, sometimes i really need to clear all my data in flash too, and then i need to erase all my flash pages on the device. segger has product named ‘j-flash’ which is used to flash and erase devices outside of an ide. there is a free-of-charge ‘lite’ version available for download from segger. this utility is not intended to be used for production. with this utility i have a gui to erase and program my device. j-flash lite but j-flash lite cannot unlock my locked flash pages :-(. if my device is not locked, i can use the codewarrior ‘flash file to target’ (see “ flashing with a button (and a magic wand) “) to erase the device: erasing device with flash file to target again, this does not work if the device is locked. codewarrior has another feature called ‘target task’ which can be used to erase/unsecure (if your device is supported), see “ device is secure? “. so i need to use a different tool to unlock and unprotect my device: the j-link commander . unlocking and erasing with j-link commander to unlock the device, segger has a utility named ‘j-link commander’, available from http://www.segger.com/jlink-software.html . the binary is ‘jlink.exe’ on windows and is a command line utility. to unlock the device use unlock kinetis unlocking device but it seems that i need to do an unlock, followed by an erase to make it permanent. to erase the device, i can use the same command line utility. but i need to specify the device name first, and then i can erase it (example for the kl25z): device mkl25z128xxx4 unlock kinetis erase :!: i need to do the erase operation right after the unlock. a) set device b) unlock and c) erase, otherwise it will fail? unlocking and erasing with j-link commander summary in order to re-program the protected flash sectors with segger j-link, i need first to unlock and mass erase the device. for this, there is the j-link commander utility which has a command line interface to unprotect and erase the device. for erasing only, the j-flash (and lite) is a very useful tool, especially to get a ‘clean’ device memory. to me, the segger way and tools are very powerful. in this case, things are very flexible, but not that obvious. so i hope this post can help others to get his device unlocked and erased. happy erasing :-)

Originally Written by Muhammad Irfan MySQL replication is a core process for maintaining multiple copies of data – and replication is a very important aspect in database administration. In order to synchronize data between master and slaves you need to make sure that data transfers smoothly, and to do so you need to act promptly regarding replication errors to continue data synchronization. Here on the Percona Support team, we often help customers with replication broken-related issues. In this post I’ll highlight the top most critical replication error code 1236 along with the causes and cure. MySQL replication error “Got fatal error 1236” can be triggered by multiple reasons and I will try to cover all of them. Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: ‘log event entry exceeded max_allowed_packet; Increase max_allowed_packet on master; the first event ‘binlog.000201′ at 5480571 This is a typical error on the slave(s) server. It reflects the problem around max_allowed_packet size. max_allowed_packet refers to single SQL statement sent to the MySQL server as binary log event from master to slave. This error usually occurs when you have a different size of max_allowed_packet on the master and slave (i.e. master max_allowed_packet size is greater then slave server). When the MySQL master server tries to send a bigger packet than defined on the slave server, the slave server then fails to accept it and hence the error. In order to alleviate this issue please make sure to have the same value for max_allowed_packet on both slave and master. You can read more about max_allowed_packet here. This error usually occurs when updating a huge number of rows on the master and it doesn’t fit into the value of slave max_allowed_packet size because slave max_allowed_packet size is lower then the master. This usually happens with queries “LOAD DATA INFILE” or “INSERT .. SELECT” queries. As per my experience, this can also be caused by application logic that can generate a huge INSERT with junk data. Take into account, that one new variable introduced in MySQL 5.6.6 and later slave_max_allowed_packet_size which controls the maximum packet size for the replication threads. It overrides the max_allowed_packet variable on slave and it’s default value is 1 GB. In this post, “max_allowed_packet and binary log corruption in MySQL,”my colleague Miguel Angel Nieto explains this error in detail. Got fatal error 1236 from master when reading data from binary log: ‘Could not find first log file name in binary log index file’ This error occurs when the slave server required binary log for replication no longer exists on the master database server. In one of the scenarios for this, your slave server is stopped for some reason for a few hours/days and when you resume replication on the slave it fails with above error. When you investigate you will find that the master server is no longer requesting binary logs which the slave server needs to pull in order to synchronize data. Possible reasons for this include the master server expired binary logs via system variable expire_logs_days – or someone manually deleted binary logs from master via PURGE BINARY LOGS command or via ‘rm -f’ command or may be you have some cronjob which archives older binary logs to claim disk space, etc. So, make sure you always have the required binary logs exists on the master server and you can update your procedures to keep binary logs that the slave server requires by monitoring the “Relay_master_log_file” variable from SHOW SLAVE STATUS output. Moreover, if you have set expire_log_days in my.cnf old binlogs expire automatically and are removed. This means when MySQL opens a new binlog file, it checks the older binlogs, and purges any that are older than the value of expire_logs_days (in days). Percona Server added a feature to expire logs based on total number of files used instead of the age of the binlog files. So in that configuration, if you get a spike of traffic, it could cause binlogs to disappear sooner than you expect. For more information check Restricting the number of binlog files. In order to resolve this problem, the only clean solution I can think of is to re-create the slave server from a master server backup or from other slave in replication topology. – Got fatal error 1236 from master when reading data from binary log: ‘binlog truncated in the middle of event; consider out of disk space on master; the first event ‘mysql-bin.000525′ at 175770780, the last event read from ‘/data/mysql/repl/mysql-bin.000525′ at 175770780, the last byte read from ‘/data/mysql/repl/mysql-bin.000525′ at 175771648.’ Usually, this caused by sync_binlog <>1 on the master server which means binary log events may not be synchronized on the disk. There might be a committed SQL statement or row change (depending on your replication format) on the master that did not make it to the slave because the event is truncated. The solution would be to move the slave thread to the next available binary log and initialize slave thread with the first available position on binary log as below: mysql>CHANGE MASTERTOMASTER_LOG_FILE='mysql-bin.000526',MASTER_LOG_POS=4; – [ERROR] Slave I/O: Got fatal error 1236 from master when reading data from binary log: ‘Client requested master to start replication from impossible position; the first event ‘mysql-bin.010711′ at 55212580, the last event read from ‘/var/lib/mysql/log/mysql-bin.000711′ at 4, the last byte read from ‘/var/lib/mysql/log/mysql-bin.010711′ at 4.’, Error_code: 1236 I foresee master server crashed or rebooted and hence binary log events not synchronized on disk. This usually happens when sync_binlog != 1 on the master. You can investigate it as inspecting binary log contents as below: $mysqlbinlog--base64-output=decode-rows--verbose--verbose--start-position=55212580mysql-bin.010711 You will find this is the last position of binary log and end of binary log file. This issue can usually be fixed by moving the slave to the next binary log. In this case it would be: mysql>CHANGE MASTER TOMASTER_LOG_FILE='mysql-bin.000712',MASTER_LOG_POS=4; This will resume replication. To avoid corrupted binlogs on the master, enabling sync_binlog=1 on master helps in most cases. sync_binlog=1 will synchronize the binary log to disk after every commit. sync_binlog makes MySQL perform on fsync on the binary log in addition to the fsync by InnoDB. As a reminder, it has some cost impact as it will synchronize the write-to-binary log on disk after every commit. On the other hand, sync_binlog=1 overhead can be very minimal or negligible if the disk subsystem is SSD along with battery-backed cache (BBU). You can read more about this here in the manual. sync_binlog is a dynamic option that you can enable on the fly. Here’s how: mysql-master>SET GLOBAL sync_binlog=1; To make the change persistent across reboot, you can add this parameter in my.cnf. As a side note, along with replication fixes, it is always a better option to make sure your replica is in the master and to validate data between master/slaves. Fortunately, Percona Toolkit has tools for this purpose: pt-table-checksum & pt-table-sync. Before checking for replication consistency, be sure to check the replication environment and then, later, to sync any differences.