The great thing with open source tools like Eclipse and GNU (gcc, gdb) is that there is a wealth of excellent tools: one thing I had in mind to explore for a while is how to generate code coverage of my embedded application. Yes, GNU and Eclipse come with code profiling and code coverage tools, all for free! The only downside seems to be that these tools seem to be rarely used for embedded targets. Maybe that knowledge is not widely available? So here is my attempt to change this :-). Or: How cool is it to see in Eclipse how many times a line in my sources has been executed? Line Coverage in Eclipse And best of all, it does not stop here…. Coverage with Eclipse To see how much percentage of my files and functions are covered? gcov in Eclipse Or even to show the data with charts? Coverage Bar Graph View Outline In this tutorial I’m using a Freescale FRDM-K64F board: this board has ARM Cortex-M4F on it, with 1 MByte FLASH and 256 KByte of RAM. The approach used in this tutorial can be used with any embedded target, as long there is enough RAM to store the coverage data on the target. I’m using Eclipse Kepler with the ARM Launchpad GNU tools (q3 2014 release), but with small modifications any Eclipse version or GNU toolchain could be used. To generate the Code Coverage information, I’m using gcov. Freescale FRDM-K64F Board Generating Code Coverage Information with gcov gcov is an open source program which can generate code coverage information. It tells me how often each line of a program is executed. This is important for testing, as that way I can know which parts of my application actually has been executed by the testing procedures. Gcov can be used as well for profiling, but in this post I will use it to generate coverage information only. The general flow to generate code coverage is: Instrument code: Compile the application files with a special option. This will add (hidden) code and hooks which records how many times a piece of code is executed. Generate Instrumentation Information: as part of the previous steps, the compiler generates basic block and line information. This information is stored on the host as *.gcno (Gnu Coverage Notes Object?) files. Run the application: While the application is running on the target, the instrumented code will record how many the lines or blocks in the application are executed. This information is stored on the target (in RAM). Dump the recorded information: At application exit (or at any time), the recorded information needs to be stored and sent to the host. By default gcov stores information in files. As a file system might not be alway available, other methods can be used (serial connection, USB, ftp, …) to send and store the information. In this tutorial I show how the debugger can be used for this. The information is stored as *.gcda (Gnu Coverage Data Analysis?) files. Generate the reports and visualize them with gcov. General gcov Flow gcc does the instrumentation and provides the library for code coverage, while gcov is the utility to analyze the generated data. Coverage: Compiler and Linker Options To generate the *.gcno files, the following option has to be added for each file which should generate coverage information: -fprofile-arcs -ftest-coverage :idea: There is as well the ‘–coverage’ option (which is a shortcut option) which can be used both for the compiler and linker. But I prefer the ‘full’ options so I know what is behind the options. -fprofile-arcs Compiler Option The option -fprofile-arcs adds code to the program flow to so execution of source code lines are counted. It does with instrumenting the program flow arcs. From https://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html: -fprofile-arcs Add code so that program flow arcs are instrumented. During execution the program records how many times each branch and call is executed and how many times it is taken or returns. When the compiled program exits it saves this data to a file called auxname.gcda for each source file. The data may be used for profile-directed optimizations (-fbranch-probabilities), or for test coverage analysis (-ftest-coverage). Each object file’s auxname is generated from the name of the output file, if explicitly specified and it is not the final executable, otherwise it is the basename of the source file. In both cases any suffix is removed (e.g. foo.gcda for input file dir/foo.c, or dir/foo.gcda for output file specified as -o dir/foo.o). See Cross-profiling. If you are not familiar with compiler technology or graph theory: An ‘Arc‘ (alternatively ‘edge’ or ‘branch’) is a directed link between a pair ‘Basic Blocks‘. A Basic is a sequence of code which has no branching in it (it is executed in a single sequence). For example if you have the following code: k = 0; if (i==10) { i += j; j++; } else { foo(); } bar(); Then this consists of the following four basic blocks: Basic Blocks The ‘Arcs’ are the directed edges (arrows) of the control flow. It is important to understand that not every line of the source gets instrumented, but only the arcs: This means that the instrumentation overhead (code size and data) depends how ‘complicated’ the program flow is, and not how many lines the source file has. However, there is an important aspect to know about gcov: it provides ‘condition coverage‘ if a full expression evaluates to TRUE or FALSE. Consider the following case: if (i==0 || j>=20) { In other words: I get coverage how many times the ‘if’ has been executed, but *not* how many times ‘i==0′ or ‘j>=20′ (which would be ‘decision coverage‘, which is not provided here). See http://www.bullseye.com/coverage.html for all the details. -ftest-coverage Compiler Option The second option for the compiler is -ftest-coverage (from https://gcc.gnu.org/onlinedocs/gcc-3.4.5/gcc/Debugging-Options.html): -ftest-coverage Produce a notes file that the gcov code-coverage utility (see gcov—a Test Coverage Program) can use to show program coverage. Each source file’s note file is called auxname.gcno. Refer to the -fprofile-arcs option above for a description of auxname and instructions on how to generate test coverage data. Coverage data will match the source files more closely, if you do not optimize. So this option generates the *.gcno file for each source file I decided to instrument: gcno file generated This file is needed later to visualize the data with gcov. More about this later. Adding Compiler Options So with this knowledge, I need to add -fprofile-arcs -ftest-coverage as compiler option to every file I want to profile. It is not necessary profile the full application: to save ROM and RAM and resources, I can add this option only to the files needed. Actually as a starter, I recommend to instrument a single source file only at the beginning. For this I select the properties (context menu) of my file Test.c I add the options in ‘other compiler flags': Coverage Added to Compilation File -fprofile-arcs Linker Option Profiling not only needs a compiler option: I need to tell the linker that it needs to link with the profiler library. For this I add -fprofile-arcs to the linker options: -fprofile-arcs Linker Option Coverage Stubs Depending on your library settings, you might now get a lot of unresolved symbol linker errors. This is because by default the profiling library assumes to write the profiling information to a file system. However, most file systems do *not* have a file system. To overcome this, I add a stubs for all the needed functions. I have them added with a file to my project (see latest version of that file on GitHub): /* * coverage_stubs.c * * These stubs are needed to generate coverage from an embedded target. */ #include #include #include #include #include #include #include "UTIL1.h" #include "coverage_stubs.h" /* prototype */ void gcov_exit(void); /* call the coverage initializers if not done by startup code */ void static_init(void) { void (**p)(void); extern uint32_t __init_array_start, __init_array_end; /* linker defined symbols, array of function pointers */ uint32_t beg = (uint32_t)&__init_array_start; uint32_t end = (uint32_t)&__init_array_end; while(begst_mode = S_IFCHR; return 0; } int _getpid(void) { return 1; } int _isatty(int file) { switch (file) { case STDOUT_FILENO: case STDERR_FILENO: case STDIN_FILENO: return 1; default: errno = EBADF; return 0; } } int _kill(int pid, int sig) { (void)pid; (void)sig; errno = EINVAL; return (-1); } int _lseek(int file, int ptr, int dir) { (void)file; (void)ptr; (void)dir; return 0; /* return offset in file */ } #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wreturn-type" __attribute__((naked)) static unsigned int get_stackpointer(void) { __asm volatile ( "mrs r0, msp \r\n" "bx lr \r\n" ); } #pragma GCC diagnostic pop void *_sbrk(int incr) { extern char __HeapLimit; /* Defined by the linker */ static char *heap_end = 0; char *prev_heap_end; char *stack; if (heap_end==0) { heap_end = &__HeapLimit; } prev_heap_end = heap_end; stack = (char*)get_stackpointer(); if (heap_end+incr > stack) { _write (STDERR_FILENO, "Heap and stack collision\n", 25); errno = ENOMEM; return (void *)-1; } heap_end += incr; return (void *)prev_heap_end; } int _read(int file, char *ptr, int len) { (void)file; (void)ptr; (void)len; return 0; /* zero means end of file */ } :idea: In this code I’m using the UTIL1 (Utility) Processor Expert component, available on SourceForge. If you do not want/need this, you can remove the lines with UTIL1. - Coverage Stubs File in Project Coverage Constructors There is one important thing to mention: the coverage data structures need to be initialized, similar to constructors for C++. Depending on your startup code, this might *not* be done automatically. Check your linker .map file for some _GLOBAL__ symbols: .text._GLOBAL__sub_I_65535_0_TEST_Test 0x0000395c 0x10 ./Sources/Test.o Such a symbol should exist for every source file which has been instrumented with coverage information. These are functions which need to be called as part of the startup code. Set a breakpoint in your code at the given address to check if it gets called. If not, you need to call it yourself. :!: Typically I use the linker option ‘-nostartfiles’), and I have my startup code. In that case, these constructors are not called by default, so I need to do myself. See http://stackoverflow.com/questions/6343348/global-constructor-call-not-in-init-array-section In my linker file I have this: .init_array : { PROVIDE_HIDDEN (__init_array_start = .); KEEP (*(SORT(.init_array.*))) KEEP (*(.init_array*)) PROVIDE_HIDDEN (__init_array_end = .); } > m_text This means that there is a list of constructor function pointers put together between __init_array_start and __init_array_end. So all what I need is to iterate through this array and call the function pointers: /* call the coverage initializers if not done by startup code */ void static_init(void) { void (**p)(void); extern uint32_t __init_array_start, __init_array_end; /* linker defined symbols, array of function pointers */ uint32_t beg = (uint32_t)&__init_array_start; uint32_t end = (uint32_t)&__init_array_end; while(beg stack) { _write (STDERR_FILENO, "Heap and stack collision\n", 25); errno = ENOMEM; return (void *)-1; } heap_end += incr; return (void *)prev_heap_end; } :!: It might be that several kBytes of heap are needed. So if you are running in a memory constraint system, be sure that you have enough RAM available. The above implementation assumes that I have space between my heap end and the stack area. :!: If your memory mapping/linker file is different, of course you will need to change that _sbrk() implementation. Compiling and Building Now the application should compile and link without errors.Check that the .gcno files are generated: :idea: You might need to refresh the folder in Eclipse. - .gcno files generated In the next steps I’m showing how to get the coverage data as *.gcda files to the host using gdb. Using Debugger to get the Coverage Data The coverage data gets dumped when _exit() gets called by the application. Alternatively I could call gcov_exit() or __gcov_flush() any time. What it then does is Open the *.gcda file with _open() for every instrumented source file. Write the data to the file with _write(). So I can set a breakpoint in the debugger to both _open() and _write() and have all the data I need :-) With _open() I get the file name, and I store it in a global pointer so I can reference it in _write(): static const unsigned char *fileName; /* file name used for _open() */ int _open (const char *ptr, int mode) { (void)mode; fileName = (const unsigned char*)ptr; /* store file name for _write() */ return 0; } In _write() I get a pointer to the data and the length of the data. Here I can dump the data to a file using the gdb command: dump binary memory I could use a calculator to calculate the memory dump range, but it is much easier if I let the program generate the command line for gdb :-): int _write(int file, char *ptr, int len) { static unsigned char gdb_cmd[128]; /* command line which can be used for gdb */ (void)file; /* construct gdb command string */ UTIL1_strcpy(gdb_cmd, sizeof(gdb_cmd), (unsigned char*)"dump binary memory "); UTIL1_strcat(gdb_cmd, sizeof(gdb_cmd), fileName); UTIL1_strcat(gdb_cmd, sizeof(gdb_cmd), (unsigned char*)" 0x"); UTIL1_strcatNum32Hex(gdb_cmd, sizeof(gdb_cmd), (uint32_t)ptr); UTIL1_strcat(gdb_cmd, sizeof(gdb_cmd), (unsigned char*)" 0x"); UTIL1_strcatNum32Hex(gdb_cmd, sizeof(gdb_cmd), (uint32_t)(ptr+len)); return 0; } That way I can copy the string in the gdb debugger: Generated GDB Memory Dump Command That command gets pasted and executed in the gdb console: gdb command line After execution of the program, the *.gcda file gets created (refresh might be necessary to show it up): gcda file created Repeat this for all instrumented files as necessary. Showing Coverage Information To show the coverage information, I need the *.gcda, the *.gcno plus the .elf file. :idea: Use Refresh if not all files are shown in the Project Explorer view Files Ready to Show Coverage Information Then double-click on the gcda file to show coverage results: Double Click on gcda File Press OK, and it opens the gcov view. Double click on file in that view to show the details: gcov Views Use the chart icon to create a chart view: Chart view Bar Graph View Video of Steps to Create and Use Coverage The following video summarizes the steps needed: Data and Code Overhead Instrumenting code to generate coverage information means that it is an intrusive method: it impacts the application execution speed, and needs extra RAM and ROM. How much heavily depends on the complexity of the control flow and on the number of arcs. Higher compiler optimizations would reduce the code size footprint, however optimizations are not recommended for coverage sessions, as this might make the job of the coverage much harder. I made a quick comparison using my test application. I used the ‘size’ GNU command (see “Printing Code Size Information in Eclipse”). Without coverage enabled, the application footprint is: arm-none-eabi-size --format=berkeley "FRDM-K64F_Coverage.elf" text data bss dec hex filename 6360 1112 5248 12720 31b0 FRDM-K64F_Coverage.elf With coverage enabled only for Test.c gave: arm-none-eabi-size --format=berkeley "FRDM-K64F_Coverage.elf" text data bss dec hex filename 39564 2376 9640 51580 c97c FRDM-K64F_Coverage.elf Adding main.c to generate coverage gives: arm-none-eabi-size --format=berkeley "FRDM-K64F_Coverage.elf" text data bss dec hex filename 39772 2468 9700 51940 cae4 FRDM-K64F_Coverage.elf So indeed there is some initial add-up because of the coverage library, but afterwards adding more source files does not add up much. Summary It took me a while and reading many articles and papers to get code coverage implemented for an embedded target. Clearly, code coverage is easier if I have a file system and plenty of resources available. But I’m now able to retrieve coverage information from a rather small embedded system using the debugger to dump the data to the host. It is not practical for large sets of files, but at least a starting point :-). I have committed my Eclipse Kepler/Launchpad project I used in this tutorial on GitHub. Ideas I have in my mind: Instead using the debugger/gdb, use FatFS and SD card to store the data Exploring how to use profiling Combining multiple coverage runs Happy Covering :-) Links: Blog article who helped me to explore gcov for embedded targets: http://simply-embedded.blogspot.ch/2013/08/code-coverage-introduction.html Paper about using gcov for Embedded Systems: http://sysrun.haifa.il.ibm.com/hrl/greps2007/papers/gcov-on-an-embedded-system.pdf Article about coverage options for GNU compiler and linker: http://bobah.net/d4d/tools/code-coverage-with-gcov How to call static constructor methods manually: http://stackoverflow.com/questions/6343348/global-constructor-call-not-in-init-array-section Article about using gcov with lcov: https://qiaomuf.wordpress.com/2011/05/26/use-gcov-and-lcov-to-know-your-test-coverage/ Explanation of different coverage methods and terminology: http://www.bullseye.com/coverage.html

Recently I was asked about diagnosing deadlocks in SQL Server – I’ve done a lot of work in this area way back in 2008, so I figure it’s time for a refresher. If there’s a lot of interest in exploring SQL Server and deadlocks further, I’m happy to write an extended article going into far more detail. Just let me know. Before we get into diagnosis and investigation, it’s a good time to pose the question: “what is a deadlock?”: From TechNet: A deadlock occurs when two or more tasks permanently block each other by each task having a lock on a resource which the other tasks are trying to lock. The following graph presents a high level view of a deadlock state where: Task T1 has a lock on resource R1 (indicated by the arrow from R1 to T1) and has requested a lock on resource R2 (indicated by the arrow from T1 to R2). Task T2 has a lock on resource R2 (indicated by the arrow from R2 to T2) and has requested a lock on resource R1 (indicated by the arrow from T2 to R1). Because neither task can continue until a resource is available and neither resource can be released until a task continues, a deadlock state exists. The SQL Server Database Engine automatically detects deadlock cycles within SQL Server. The Database Engine chooses one of the sessions as a deadlock victim and the current transaction is terminated with an error to break the deadlock. Basically, it’s a resource contention issue which blocks one process or transaction from performing actions on resources within SQL Server. This can be a serious condition, not just for SQL Server as processes become suspended, but for the applications which rely on SQL Server as well. The T-SQL Approach A fast way to respond is to execute a bit of T-SQL on SQL Server, making use of System Views. The following T-SQL will show you the “victim” processes, much like activity monitor does: select * from sys.sysprocesses where blocked > 0 Which is not particularly useful (but good to know, so you can see the blocked count). To get to the heart of the deadlock, this is what you want (courtesy of this SO question/answer): SELECT Blocker.text –, Blocker.*, * FROM sys.dm_exec_connections AS Conns INNER JOIN sys.dm_exec_requests AS BlockedReqs ON Conns.session_id = BlockedReqs.blocking_session_id INNER JOIN sys.dm_os_waiting_tasks AS w ON BlockedReqs.session_id = w.session_id CROSS APPLY sys.dm_exec_sql_text(Conns.most_recent_sql_handle) AS Blocker This will show you line and verse (the actual statement causing the resource block) – see the attached screenshot for an example. However, the generally accepted way to determine and diagnose deadlocks is through the use of SQL Server trace flags. SQL Trace Flags They are (usually) set temporarily, and they cause deadlocking information to be dumped to the SQL management logs. The flags that are useful are flags 1204 and 1222. From TechNet: https://technet.microsoft.com/en-us/library/ms178104%28v=sql.105%29.aspx Trace flags are set on or off by using either of the following methods: · Using the DBCC TRACEON and DBCC TRACEOFF commands. For example, DBCC TRACEON 2528: To enable the trace flag globally, use DBCC TRACEON with the -1 argument: DBCC TRACEON (2528, -1). To turn off a global trace flag, use DBCC TRACEOFF with the -1 argument. · Using the -T startup option to specify that the trace flag be set on during startup. The -T startup option enables a trace flag globally. You cannot enable a session-level trace flag by using a startup option. So to enable or disable deadlock trace flags globally, you’d use the following T-SQL: DBCC TRACEON (1204, -1) DBCC TRACEON (1222, -1) DBCC TRACEOFF (1204, -1) DBCC TRACEOFF (1222, -1) Due to the overhead, it’s best to enable the flag at runtime rather than on start up. Note that the scope of a non-startup trace flag can be global or session-level. Basic Deadlock Simulation By way of a very simple scenario, you can make use of SQL Management Studio (and breakpoints) to roughly simulate a deadlock scenario. Given the following basic table schema: CREATE TABLE [dbo].[UploadedFile]( [Id] [int] NOT NULL, [Filename] [nvarchar](50) NOT NULL, [DateCreated] [datetime] NOT NULL, [DateModified] [datetime] NULL, CONSTRAINT [PK_UploadedFile] PRIMARY KEY CLUSTERED ( [Id] ASC )WITH (STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF) ) With some basic test data in it: If you create two separate queries in SQL Management Studio, use the following transaction (Query #1) to lock rows in the table: SET TRANSACTION ISOLATION LEVEL SERIALIZABLE BEGIN TRANSACTION SELECT [Id],[Filename],[DateCreated],[DateModified] FROM [dbo].[UploadedFile] WHERE DateCreated > ‘2015-01-01′ ROLLBACK TRANSACTION Now add a “victim” script (Query #2) in a separate query session: UPDATE [dbo].[UploadedFile] SET [DateModified] = ‘2014-12-31′ WHERE DateCreated > ‘2015-01-01′ As long as you set a breakpoint on the ROLLBACK TRANSACTION statement, you’ll block the second query due to the isolation level of the transaction which wraps query #1. Now you can use the diagnostic T-SQL to examine the victim and the blocking transaction. Enjoy!

Recently I had a dispute with my colleagues regarding performance penalty of lazy vals in Scala. It resulted in a set of microbenchmarks which compare lazy and non-lazy vals performance. All the sources can be found at http://git.io/g3WMzA. But before going to the benchmark results let's try to understand what can cause the performance penalty. For my JMH benchmark I created a very simple Scala class with lazy val in it: @State(Scope.Benchmark) class LazyValCounterProvider { lazy val counter = SlowInitializer.createCounter() } Now let's take a look at what is hidden under the hood of lazy keyword. At first, we need to compile given code with scalac, and then it can be decompiled to correspondent Java code. For this sake I used JD decompiler. It produced the following code: @State(Scope.Benchmark) @ScalaSignature(bytes="...") public class LazyValCounterProvider { private SlowInitializer.Counter counter; private volatile boolean bitmap$0; private SlowInitializer.Counter counter$lzycompute() { synchronized (this) { if (!this.bitmap$0) { this.counter = SlowInitializer.createCounter(); this.bitmap$0 = true; } return this.counter; } } public SlowInitializer.Counter counter() { return this.bitmap$0 ? this.counter : counter$lzycompute(); } } As it's seen, the lazy keyword is translated to a classical double-checked locking idiom for delayed initialization. Thus, most of the time the only performance penalty may come from a single volatile read per lazy val read (except for the time it takes to initialize lazy val instance since its very first usage). Let's finally measure its impact in numbers. My JMH-based microbenchmark is as simple as: public class LazyValsBenchmarks { @Benchmark public long baseline(ValCounterProvider eagerProvider) { return eagerProvider.counter().incrementAndGet(); } @Benchmark public long lazyValCounter(LazyValCounterProvider provider) { return provider.counter().incrementAndGet(); } } A baseline method access a final counter object and increments an integer value by 1 by calling incrementAndGet . And as we've just found out, the main benchmark method - lazyValCounter - in addition to what baseline method does also does one volatile read. Note: all measurements are performed on MBA with Core i5 1.7GHz CPU. All results were obtained by running JMH in a throughput mode. Both score and score error columns show operations/second. Each JMH run made 10 iterations and took 50 seconds. I performed 6 measurements with the different JVM and JMH options: client VM, 1 thread Benchmark Score Score error baseline 412277751.619 8116731.382 lazyValCounter 352209296.485 6695318.185 client VM, 2 threads Benchmark Score Score error baseline 542605885.932 15340285.497 lazyValCounter 383013643.710 53639006.105 client VM, 4 threads Benchmark Score Score error baseline 551105008.767 5085834.663 lazyValCounter 394175424.898 3890422.327 server VM, 1 thread Benchmark Score Score error baseline 407010942.139 9004641.910 lazyValCounter 341478430.115 18183144.277 server VM, 2 threads Benchmark Score Score error baseline 531472448.578 22779859.685 lazyValCounter 428898429.124 24720626.198 server VM, 4 threads Benchmark Score Score error baseline 549568334.970 12690164.639 lazyValCounter 374460712.017 17742852.788 The numbers show that lazy vals performance penalty is quite small and can be ignored in practice. For further reading about the subject I would recommend SIP 20 - Improved Lazy Vals Initialization, which contains very interesting in-depth analysis of existing issues with lazy initialization implementation in Scala.

In this article I will attempt to run the Mule ESB community edition in Docker in order to see whether it is feasible without any greater inconvenience. My goal is to be able to use Docker both when testing as well as in a production environment in order to gain better control over the environment and to separate different types of environments. I imagine that most of the Docker-related information can be applied to other applications – I have used Mule since it is what I usually work with. The conclusion I have made after having completed my experiments is that it is possible to run Mule ESB in Docker without any inconvenience. In addition, Docker will indeed allow me to have better control over the different environments and also allow me to separate them as I find appropriate. Finally, I just want to mention that I have used Docker in an Ubuntu environment. I have not attempted any of the exercises in Docker running on Windows or Mac OS X. Docker Briefly In short, Docker allows for creating of images that serve as blueprints for containers. A Docker container is an instance of a Docker image in the same way a Java object is an instance of a Java class. FROM codingtony/java MAINTAINER tony(dot)bussieres(at)ticksmith(dot)com RUN wget https://repository.mulesoft.org/nexus/content/repositories/releases/org/mule/distributions/mule-standalone/3.5.0/mule-standalone-3.5.0.tar.gz RUN cd /opt && tar xvzf ~/mule-standalone-3.5.0.tar.gz RUN echo "4a94356f7401ac8be30a992a414ca9b9 /mule-standalone-3.5.0.tar.gz" | md5sum -c RUN rm ~/mule-standalone-3.5.0.tar.gz RUN ln -s /opt/mule-standalone-3.5.0 /opt/mule CMD [ "/opt/mule/bin/mule" ] The resource isolation features of Linux are used to create Docker containers, which are more lightweight than virtual machines and are separated from the environment in which Docker runs, the host. Using Docker an image can be created that, every time it is started has a known state. In order to remove any doubts about whether the environment has been altered in any way, the container can be stopped and a new container started. I can even run multiple Docker containers on one and the same computer to simulate a multi-server production environment. Applications can also be run in their own Docker containers, as shown in this figure. Three Docker containers, each containing a specific application, running in one host. A more detailed introduction to Docker is available here. The main entry point to the Docker documentation can be found here. Motivation Some of the motivations I have for using Docker in both testing and production environments are: The environment in which I test my application should be as similar as the final deployment environment as possible, if not identical. Making the deployment environment easy to scale up and down. If it is easy to start a new processing node when need arise and stop it if it is no longer used, I will be able to adapt to changes rather quickly and thus reduce errors caused by, for instance, load peaks. Maintain an increased number of nodes to which applications can be deployed. Instead of running one instance of some kind of application server, Mule ESB in my case, on a computer, I want multiple instances that are partitioned, for instance, according to importance. High-priority applications run on one separate instance, which have higher priority both as far as resources (CPU, memory, disk etc) are concerned but also as far as support is concerned. Applications which are less critical run on another instance. Enable quick replacement of instances in the deployment environment. Reasons for having to replace instances may be hardware failure etc. Better control over the contents of the different environments. The concept of an environment that, at any time, may be disposed (and restarted) discourages hacks in the environment, which are usually poorly documented and sometimes difficult to trace. Using Docker, I need to change the appropriate Docker image if I want to make changes to some application environment. The Docker image file, commonly known as Dockerfile, can be checked into any ordinary revision control system, such as Git, Subversion etc, making changes reversible and traceable. Automate the creation of a testing environment. An example could be a nightly job that runs on my build server which creates a test environment, deploys one or more applications to it and then performs tests, such as load-testing. Prerequisites To get the best possible experience when running Docker, I run it under Ubuntu. According to the current documentation, Docker is supported under the following versions of Ubuntu: 12.04 LTS (64-bit) 13.04 (64-bit) 13.10 (64-bit) 14.04 (64-bit) Against my usual conservative self, I chose Ubuntu 14.10, which at the time of writing this article is the latest version. While I haven’t run into any issues, I cannot promise anything regarding compatibility with Docker as far as this version of Ubuntu is concerned. Installing Docker Before we install anything, those who have the Docker version from the Ubuntu repository should remove this version before installing a newer version of Docker, since the Ubuntu repository does not contain the most recent version and the package does not have the same name as the Docker package we will install: sudo apt-get remove docker.io The simplest way to install Docker is to use an installation script made available at the Docker website: curl -sSL https://get.docker.com/ubuntu/ | sudo sh If you are not running Ubuntu or if you do not want to use the above way of installing Docker, please refer to this page containing instructions on how to install Docker on various platforms. To verify the Docker installation, open a terminal window and enter: sudo docker version Output similar to the following should appear: Client version: 1.4.1 Client API version: 1.16 Go version (client): go1.3.3 Git commit (client): 5bc2ff8 OS/Arch (client): linux/amd64 Server version: 1.4.1 Server API version: 1.16 Go version (server): go1.3.3 Git commit (server): 5bc2ff8 We are now ready to start a Mule instance in Docker. Running Mule in Docker One of the advantages with Docker is that there is a large repository of Docker images that are ready to be used, and even extended if one so wishes. ThisDocker image is the one that I will use in this article. It is well documented, there is a source repository and it contains a recent version of the Mule ESB Community Edition. Some additional details on the Docker image: Ubuntu 14.04. Oracle JavaSE 1.7.0_65. This version will change as the PPA containing the package is updated. Mule ESB CE 3.5.0 Note that the image may change at any time and the specifications above may have changed. If you intend to use Docker in your organization, I would suspect that the best alternative is to create your own Docker images that are totally under your control. The Docker image repository is an excellent source of inspiration and aid even in this case. Starting a Docker Container To start a Docker container using this image, open a terminal window and write: sudo docker run codingtony/mule The first time an image is used it needs to be downloaded and created. This usually takes quite some time, so I suggest a short break here – perhaps for a cup of coffee or tea. If you just want to download an image without starting it, exchange the Docker command “run” with “pull”. Once the container is started, you will see some output to the console. If you are familiar with Mule, you will recognize the log output: MULE_HOME is set to /opt/mule-standalone-3.5.0 Running in console (foreground) mode by default, use Ctrl-C to exit... MULE_HOME is set to /opt/mule-standalone-3.5.0 Running Mule... --> Wrapper Started as Console Launching a JVM... Starting the Mule Container... Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org Copyright 1999-2006 Tanuki Software, Inc. All Rights Reserved. INFO 2015-01-05 04:41:42,302 [WrapperListener_start_runner] org.mule.module.launcher.MuleContainer: ********************************************************************** * Mule ESB and Integration Platform * * Version: 3.5.0 Build: ff1df1f3 * * MuleSoft, Inc. * * For more information go to http://www.mulesoft.org * * * * Server started: 1/5/15 4:41 AM * * JDK: 1.7.0_65 (mixed mode) * * OS: Linux (3.16.0-28-generic, amd64) * * Host: f95698cfb796 (172.17.0.2) * ********************************************************************** Note that: In the text-box containing information about the Mule ESB and Integration Platform, there is a row which starts with “Host:”. The hexadecimal digit that follows is the Docker container id and the IP-address is the external IP-address of the Docker container in which Mule is running. Before we do anything with the Mule instance running in Docker, let’s take a look at Docker containers. Docker Containers We can verify that there is a Docker container running by opening another terminal window, or a tab in the first terminal window, and running the command: sudo docker ps As a result, you will see output similar to the following (I have edited the output in order for the columns to be aligned with the column titles): CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f95698cfb796 codingtony/mule:latest "/opt/mule/bin/mule" 7 min ago Up 7 min jolly_hopper From this output we can see that: The ID of the container is f95698cfb796. This ID can be used when performing operations on the container, such as stopping it, restarting it etc. The name of the image used to created the container. The command that is currently executing. If we look at the Dockerfile for the image, we can see that the last line in this file is: CMD [ “/opt/mule/bin/mule” ] This is the command that is executed whenever an instance of the Docker image is launched and it matches what we see in the COMMAND column for the Docker container. The CREATED column shows how much time has passed since the container was created. The STATUS column shows the current status of the image. When you have used Docker for a while, you can view all the containers using: sudo docker ps -a This will show you containers that are not running, in addition to the running ones. Containers that are not running can be restarted. The PORTS column shows any port mappings for the container. More about port mappings later. Finally, the NAMES column contain a more human-friendly container name. This container name can be used in the same way as the container id. Docker containers will consume disk-space and if you want to determine how much disk-space each of the containers on your computer use, issue the following command: sudo docker ps -a -s An additional column, SIZE, will be shown and in this column I see that my Mule container consumes 41,76kB. Note that this is in addition to the disk-space consumed by the Docker image. This number will grow if you use the container under a longer period of time, as the container retains any files written to disk. To completely remove a stopped Docker container, find the id or name of the container and use the command: sudo docker rm [container id or name here] Before going further, let’s stop the running container and remove it: sudo docker stop [container id or name here] sudo docker rm [container id or name here] Files and Docker Containers So far we have managed to start a Mule instance running inside a Docker container, but there were no Mule applications deployed to it and the logs that were generated were only visible in the terminal window. I want to be able to deploy my applications to the Mule instance and examine the logs in a convenient way. In this section I will show how to: Share one or more directories in the host file-system with a Docker container. Access the files in a Docker container from the host. As the first step in looking at sharing directories between the host operating system and a Docker container, we are going to look at Mule logs. As part of this exercise we also set up the directories in the host operating system that are going to be shared with the Docker container. In your home directory, create a directory named “mule-root”. In the “mule-root” directory, create three directories named “apps”, “conf” and “logs”. Download the Mule CE 3.5.0 standalone distribution from this link. From the Mule CE 3.5.0 distribution, copy the files in the “apps” directory to the “mule-root/apps” directory you just created. From the Mule CE 3.5.0 distribution, copy the files in the “conf” directory to the “mule-root/conf” directory you created. The resulting file- and directory-structure should look like this (shown using the tree command): ~/mule-root/ ├── apps │ └── default │ └── mule-config.xml ├── conf │ ├── log4j.properties │ ├── tls-default.conf │ ├── tls-fips140-2.conf │ ├── wrapper-additional.conf │ └── wrapper.conf └── logs Edit the log4j.properties file in the “mule-root/conf” directory and set the log-level on the last line in the file to “DEBUG”. This modification has nothing to do with sharing directories, but is in order for us to be able to see some more output from Mule when we run it later. The last two lines should now look like this: # Mule classes log4j.logger.org.mule=DEBUG Binding Volumes We are now ready to launch a new Docker container and when we do, we will tell Docker to map three directories in the Docker container to three directories in the host operating system. Three directories in a Docker container bound to three directories in the host. Launch the Docker container with the command below. The -v option tells Docker that we want to make the contents of a directory in the host available at a certain path in the Docker container file-system. The -d option runs the container in the background and the terminal prompt will be available as soon as the id of the newly launched Docker container has been printed. sudo docker run -d -v ~/mule-root/apps:/opt/mule/apps -v ~/mule-root/conf:/opt/mule/conf -v ~/mule-root/logs:/opt/mule/logs codingtony/mule Examine the “mule-root” directory and its subdirectories in the host, which should now look like below. The files on the highlighted rows have been created by Mule. mule-root/ ├── apps │ ├── default │ │ └── mule-config.xml │ └── default-anchor.txt ├── conf │ ├── log4j.properties │ ├── tls-default.conf │ ├── tls-fips140-2.conf │ ├── wrapper-additional.conf │ └── wrapper.conf └── logs ├── mule-app-default.log ├── mule-domain-default.log └── mule.log Examine the “mule.log” file using the command “tail -f ~/mule-root/logs/mule.log”. There should be periodic output written to the log file similar to the following: DEBUG 2015-01-05 12:05:37,216 [Mule.app.deployer.monitor.1.thread.1] org.mule.module.launcher.DeploymentDirectoryWatcher: Checking for changes... DEBUG 2015-01-05 12:05:37,216 [Mule.app.deployer.monitor.1.thread.1] org.mule.module.launcher.DeploymentDirectoryWatcher: Current anchors: default-anchor.txt DEBUG 2015-01-05 12:05:37,216 [Mule.app.deployer.monitor.1.thread.1] org.mule.module.launcher.DeploymentDirectoryWatcher: Deleted anchors: Stop and remove the container: sudo docker stop [container id or name here] sudo docker rm [container id or name here] Direct Access to Docker Container Files When running Docker under the Ubuntu OS it is also possible to access the file-system of a Docker container from the host file-system. It may be possible to do this under other operating systems too, but I haven’t had the opportunity to test this. This technique may come in handy during development or testing with Docker containers for which you haven’t bound any volumes. Note! If given the choice to use either volume binding, as seen above, or direct access to container files as we will look at in this section for something more than a temporary file access, I would chose to use volume binding. Direct access to Docker container files relies on implementation details that I suspect may change in future versions of Docker if the developers find it suitable. With all that said, lets get the action started: Start a new Docker container: sudo docker run -d codingtony/mule Find the id of the newly launched Docker container: sudo docker ps Examine low-level information about the newly launched Docker container: sudo docker inspect [container id or name here] Output similar to this will be printed to the console (portions removed to conserve space): [{ "AppArmorProfile": "", "Args": [], "Config": { ... }, "Created": "2015-01-12T07:58:47.913905369Z", "Driver": "aufs", "ExecDriver": "native-0.2", "HostConfig": { ... }, "HostnamePath": "/var/lib/docker/containers/68b40def7ad6a7f819bd654d5627ad1c3a0f40c84e0fb0f875760f1bd6790eef/hostname", "HostsPath": "/var/lib/docker/containers/68b40def7ad6a7f819bd654d5627ad1c3a0f40c84e0fb0f875760f1bd6790eef/hosts", "Id": "68b40def7ad6a7f819bd654d5627ad1c3a0f40c84e0fb0f875760f1bd6790eef", "Image": "bcd0f37d48d4501ad64bae941d95446b157a6f15e31251e26918dbac542d731f", "MountLabel": "", "Name": "/thirsty_darwin", "NetworkSettings": { ... }, "Path": "/opt/mule/bin/mule", "ProcessLabel": "", "ResolvConfPath": "/var/lib/docker/containers/68b40def7ad6a7f819bd654d5627ad1c3a0f40c84e0fb0f875760f1bd6790eef/resolv.conf", "State": { ... }, "Volumes": {}, "VolumesRW": {} }] Locate the “Driver” node (highlighted in the above output) and ensure that its value is “aufs”. If it is not, you may need to modify the directory paths below replacing “aufs” with the value of this node. Personally I have only seen the “aufs” value at this node so anything else is uncharted territory to me. Copy the long hexadecimal value that can be found at the “Id” node (also highlighted in the above output). This is the long id of the Docker container. In a terminal window, issue the following command, inserting the long id of your container where noted: sudo ls -al /var/lib/docker/aufs/mnt/[long container id here] You are now looking at the root of the volume used by the Docker container you just launched. In the same terminal window, issue the following command: sudo ls -al /var/lib/docker/aufs/mnt/[long container id here]/opt The output from this command should look like this: total 12 drwxr-xr-x 4 root root 4096 jan 12 15:58 . drwxr-xr-x 75 root root 4096 jan 12 15:58 .. lrwxrwxrwx 1 root root 26 aug 10 04:19 mule -> /opt/mule-standalone-3.5.0 drwxr-xr-x 17 409 409 4096 jan 12 15:58 mule-standalone-3.5.0 Examine this line in the Dockerfile:RUN ln -s /opt/mule-standalone-3.5.0 /opt/muleWe see that a symbolic link is created and that the directory name and the name of the symbolic link matches the output we saw earlier. This matches the directory output in the previous step. To examine the Mule log file that we looked at when binding volumes earlier, use the following command: sudo cat /var/lib/docker/aufs/mnt/[long container id here]/opt/mule-standalone-3.5.0/logs/mule.log Next we create a new file in the Docker container using vi: sudo vi /var/lib/docker/aufs/mnt/[long container id here]/opt/mule-standalone-3.5.0/test.txt Enter some text into the new file by first pressing i and the type the text. When you are finished entering the text, press the Escape key and write the file to disk by typing the characters “:wq” without quotes. This writes the new contents of the file to disk and quits the editor. Leave the Docker container running after you are finished. In the next section, we are going to look at the file we just created from inside the Docker container. We have seen that we can examine the file system of a Docker container without binding volumes. It is also possible to copy or move files from the host file-system to the container’s file system using the regular commands. Root privileges are required both when examining and writing to the Docker container’s file system. Entering a Docker Container In order to verify that the file we just created in the host was indeed written to the Docker container, we are going to start a bash shell in the running Docker container and examine the location where the new file is expected to be located and the contents of the file. In the process we will see how we can execute commands in a Docker container from the host. Issue the command below in a terminal window. The exec Docker command is used to run a command, bash in this case, in a running Docker container. The -i flags tell Docker to keep the input stream open while the command is being executed. In this example, it allows us to enter commands into the bash shell running inside the Docker container. The -t flag cause Docker to allocate a text terminal to which the output from the command execution is printed. sudo docker exec -i -t [container id or name here] bash Note the prompt, which should change to [user]@[Docker container id]. In my case it looks like this: root@3ea374a280da:/# Go to the Mule installation directory using this command: cd /opt/mule-standalone-3.5.0/ Examine the contents of the directory: ls -al Among the other files, you should see the “test.txt” file: -rw-r--r-- 1 root root 53 Jan 14 03:19 test.txt Examine the contents of the “text.txt” file. The contents of the file should match what you entered earlier. cat text.txt Exit to the host OS: exit Stop and remove the container: sudo docker stop [container id or name here] sudo docker rm [container id or name here] We have seen that we can execute commands in a running Docker container. In this particular example, we used it to execute the bash shell and examine a file. I draw the conclusion that I should be able to set up a Docker image that contains a very controlled environment for some type of test and then create a container from that image and start the test from the host. Deploying a Mule Application In this section we will look at deploying a Mule application to an instance of the Mule ESB running in a Docker container. We will use volume binding, that we looked at in the section on files and Docker containers, to share directories in the host with the Docker container in order to make it easy to deploy applications, modify running applications, examine logs etc. Preparations Before deploying the application, we need to make some preparations: First of all, we restore the original log-level that we changed earlier. In this example, there will be log output when the applications we will deploy is run and we can limit the log generated by Mule. Edit the log4j.properties file in the “mule-root/conf” directory in the host and set the log-level on the last line in the file back to “INFO” and add one line, as in the listing below. The last three lines should now look like this: # Mule classes log4j.logger.org.mule=INFO log4j.logger.org.mule.tck.functional=DEBUG Next, we create the Mule application which we will deploy to the Mule ESB running in Docker: In some directory, create a file named “mule-deploy.properties” with the following contents: redeployment.enabled=true encoding=UTF-8 domain=default config.resources=HelloWorld.xml In the same directory create a file named “HelloWorld.xml”. This file contains the Mule configuration for our example application: Create a zip-archive named “mule-hello.zip” containing the two files created above: zip mule-hello.zip mule-deploy.properties HelloWorld.xml Deploy the Mule Application Before you start the Docker container in which the Mule EBS will run, make sure that you have created and prepared the directories in the host as described in the section Files and Docker Containers above. Start a new Mule Docker container using the command that we used when binding volumes: sudo docker run -d -v ~/mule-root/apps:/opt/mule/apps -v ~/mule-root/conf:/opt/mule/conf -v ~/mule-root/logs:/opt/mule/logs codingtony/mule As before, the -v option tells Docker to bind three directories in the host to three locations in the Docker container’s file system. Find the IP-address of the Docker container: sudo docker inspect [container id or name here] | grep IPAddress In my case, I see the following line which reveals the IP-address of the Docker container: “IPAddress”: “172.0.17.2”, Open a terminal window or tab and examine the Mule log. Leave this window or tab open during the exercise, in order to be able to verify the output from Mule. tail -f ~/mule-root/logs/mule.log Copy the zip-archive “mule-hello.zip” created earlier to the host directory ~/mule-root/apps/. Verify that the application has been deployed without errors in the Mule log: ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + Started app 'mule-hello' + ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Leave the Docker container running after you are finished. In the next section we will look at how to access endpoints exposed by applications running in Docker containers. By binding directories in the host thus making them available in the Docker container, it becomes very simple to deploy Mule applications to an instance of Mule ESB running in a Docker container. I am considering this setup for a production environment as well, since it will enable me to perform backups of the directories containing Mule applications and configuration without having to access the Docker container’s file system. It is also in accord with the idea that a Docker container should be able to be quickly and easily restarted, which I feel it would not be if I had to deploy a number of Mule applications to it in order to recreate its previous state. Accessing Endpoints We now know that we can run the Mule ESB in a Docker container, we can deploy applications and examine the logs quite easily but one final, very important question remains to be answered; how to access endpoints exposed by applications running in a Docker container. This section assumes that the Mule application we deployed to Mule in the previous section is still running. In the host, open a web-browser and issue a request to the Docker container’s IP-address at port 8181. In my case, the URL is http://172.17.0.2:8181 Alternatively use the curl command in a terminal window. In my case I would write: curl 172.17.0.2:8181 The result should be a greeting in the following format: Hello World! It is now: 2015-01-14T07:39:03.942Z In addition, you should be able to see that a message was received in the Mule log. Now try the URL http://localhost:8181 You will get a message saying that the connection was refused, provided that you do not already have a service listening at that port. If you have another computer available that is connected to the same network as the host computer running Ubuntu, do the following: – Find the IP-address of the Ubuntu host computer using the ifconfigcommand. – In a web-browser on the other computer, try accessing port 8181 at the IP-address of the Ubuntu host computer. Again you will get a message saying that the connection was refused. Stop and remove the container: sudo docker stop [container id or name here] sudo docker rm [container id or name here] Without any particular measures taken, we see that we can access a service exposed in a Docker container from the Docker host but we did not succeed in accessing the service from another computer. To make a service exposed in a Docker container reachable from outside of the host, we need to tell Docker to publish a port from the Docker container to a port in the host using the -p flag: Launch a new Docker container using the following command: sudo docker run -d -p 8181:8181 -v ~/mule-root/apps:/opt/mule/apps -v ~/mule-root/conf:/opt/mule/conf -v ~/mule-root/logs:/opt/mule/logs codingtony/mule The added flag -p 8181:8181 makes the service exposed at port 8181 in the Docker container available at port 8181 in the host. Try accessing the URL http://localhost:8181 from a web-browser on the host computer.The result should be a greeting of the form we have seen earlier. Try accessing port 8181 at the IP-address of the Ubuntu host computer from another computer.This should also result in a greeting message. Stop and remove the container: sudo docker stop [container id or name here] sudo docker rm [container id or name here] Using the -p flag, we have seen that we can expose a service in a Docker container so that it becomes accessible from outside of the host computer. However, we also see that this information need to be supplied at the time of launching the Docker container. The conclusions that I draw from this is that: I can test and develop against a Mule ESB instance running in a Docker container without having to publish any ports, provided that my development computer is the Docker host computer. In a production environment or any other environment that need to expose services running in a Docker container to “the outside world” and where services will be added over time, I would consider deploying an Apache HTTP Server or NGINX on the Docker host computer and use it to proxy the services that are to be exposed. This way I can avoid re-launching the Docker container each time a new service is added and I can even (temporarily) redirect the proxy to some other computer if I need to perform some maintenance. Is There More? Of course! This article should only be considered an introduction and I am just a beginner with Docker. I hope I will have the time and inspiration to write more about Docker as I learn more.

An increasing number of systems are being built to handle the Volume, Velocity and Variety of Big Data, and hopefully help gain new insights and make better business decisions. Here, we will look at ways to deal with Big Data’s Volume and Velocity simultaneously, within a single architecture solution. Volume + Velocity Apache Hadoop provides both reliable storage (HDFS) and a processing system (MapReduce) for large data sets across clusters of computers. MapReduce is a batch query processor that is targeted at long-running background processes. Hadoop can handle Volume. But to handle Velocity, we need real-time processing tools that can compensate for the high-latency of batch systems, and serve the most recent data continuously, as new data arrives and older data is progressively integrated into the batch framework. Therefore we need both batch and real-time to run in parallel, and add a real-time computational system (e.g. Apache Storm) to our batch framework. This architectural combination of batch and real-time computation is referred to as a Lambda Architecture (λ). Generic Lambda λ has three layers: The Batch Layer manages the master data and precomputes the batch views The Speed Layer serves recent data only and increments the real-time views The Serving Layer is responsible for indexing and exposing the views so that they can be queried. The three layers are outlined in the below diagram along with a sample choice of technology stacks: Incoming data is dispatched to both Batch and Speed layers for processing. At the other end, queries are answered by merging both batch and real-time views. Note that real-time views are transient by nature and their data is discarded (making room for newer data) once propagated through the Batch and Serving layers. Most of the complexity is pushed onto the much smaller Speed layer where the results are only temporary, a process known as “complexity isolation“. We are indeed isolating the complexity of concurrent data updates in a layer that is regularly purged and kept small in size. λ is technology agnostic. The data pipeline is broken down into layers with clear demarcation of responsibilities, and at each layer, we can choose from a number of technologies. The Speed layer for instance could use either Apache Storm, or Apache Spark Streaming, or Spring “XD” ( eXtreme Data) etc. How do we recover from mistakes in λ ? Basically, we recompute the views. If that takes too long, we just revert to the previous, non-corrupted versions of our data. We can do that because of data immutability in the master dataset: data is never updated, only appended to (time-based ordering). The system is therefore Human Fault-Tolerant: if we write bad data, we can just remove that data altogether and recompute. Unified Lambda The downside of λ is its inherent complexity. Keeping in sync two already complex distributed systems is quite an implementation and maintenance challenge. People have started to look for simpler alternatives that would bring just about the same benefits and handle the full problem set. There are basically three approaches: 1) Adopt a pure streaming approach, and use a flexible framework such as Apache Samza to provide some type of batch processing. Although its distributed streaming layer is pluggable, Samza typically relies on Apache Kafka. Samza’s streams are replayable, ordered partitions. Samza can be configured for batching, i.e. consume several messages from the same stream partition in sequence. 2) Take the opposite approach, and choose a flexible Batch framework that would also allow micro-batches, small enough to be close to real-time, with Apache Spark/Spark Streaming or Storm’s Trident. Spark streaming is essentially a sequence of small batch processes that can reach latency as low as one second.Trident is a high-level abstraction on top of Storm that can process streams as small batches as well as do batch aggregation. 3) Use a technology stack already combining batch and real-time, such as Spring “XD”, Summingbird or Lambdoop. Summingbird (“Streaming MapReduce”) is a hybrid system where both batch/real-time workflows can be run at the same time and the results merged automatically.The Speed layer runs on Storm and the Batch layer on Hadoop, Lambdoop (Lambda-Hadoop, with HBase, Storm and Redis) also combines batch/real-time by offering a single API for both processing paradigms: The integrated approach (unified λ) seeks to handle Big Data’s Volume and Velocity by featuring a hybrid computation model, where both batch and real-time data processing are combined transparently. And with a unified framework, there would be only one system to learn, and one system to maintain.

as i said in resolving the conflict between software architecture and code , my focus for this year is representing a software architecture model as code. in simple sketches for diagramming your software architecture , i showed an example system context diagram for my techtribes.je website. it's a simple diagram that shows techtribes.je in the middle, surrounded by the key types of users and system dependencies. it's your typical "big picture" view. this diagram was created using omnigraffle (think microsoft visio for mac os x) and it's exactly that - a static diagram that needs to be manually kept up to date. instead, wouldn't it be great if this diagram was based upon a model that we could better version control, collaborate on and visualize? if you're not sure what i mean by a "model", take a look at models, sketches and everything in between . this is basically what the aim of structurizr is. it's a way to describe a software architecture model as code, and then visualize it in a simple way. the structurizr java library is available on github and you can download a prebuilt binary . just as a warning, this is very much a work in progress and so don't be surprised if things change! here's some java code to recreate the techtribes.je system context diagram. package com.structurizr.example; import com.structurizr.io.json.jsonwriter; import com.structurizr.model.location; import com.structurizr.model.model; import com.structurizr.model.person; import com.structurizr.model.softwaresystem; import com.structurizr.view.systemcontextview; import com.structurizr.view.viewset; import java.io.stringwriter; /** * this is a model of the system context for the techtribes.je system, * the code for which can be found at https://github.com/techtribesje/techtribesje */ public class techtribessystemcontext { public static void main(string[] args) throws exception { // create a model and the software system we want to describe model model = new model("techtribes.je", "this is a model of the system context for the techtribes.je system, the code for which can be found at https://github.com/techtribesje/techtribesje"); softwaresystem techtribes = model.addsoftwaresystem(location.internal, "techtribes.je", "techtribes.je is the only way to keep up to date with the it, tech and digital sector in jersey and guernsey, channel islands"); // create the various types of people (roles) that use the software system person anonymoususer = model.addperson(location.external, "anonymous user", "anybody on the web."); anonymoususer.uses(techtribes, "view people, tribes (businesses, communities and interest groups), content, events, jobs, etc from the local tech, digital and it sector."); person authenticateduser = model.addperson(location.external, "aggregated user", "a user or business with content that is aggregated into the website."); authenticateduser.uses(techtribes, "manage user profile and tribe membership."); person adminuser = model.addperson(location.external, "administration user", "a system administration user."); adminuser.uses(techtribes, "add people, add tribes and manage tribe membership."); // create the various software systems that techtribes.je has a dependency on softwaresystem twitter = model.addsoftwaresystem(location.external, "twitter", "twitter.com"); techtribes.uses(twitter, "gets profile information and tweets from."); softwaresystem github = model.addsoftwaresystem(location.external, "github", "github.com"); techtribes.uses(github, "gets information about public code repositories from."); softwaresystem blogs = model.addsoftwaresystem(location.external, "blogs", "rss and atom feeds"); techtribes.uses(blogs, "gets content using rss and atom feeds from."); // now create the system context view based upon the model viewset viewset = new viewset(model); systemcontextview contextview = viewset.createcontextview(techtribes); contextview.addallsoftwaresystems(); contextview.addallpeople(); // and output the model and view to json (so that we can render it using structurizr.com) jsonwriter jsonwriter = new jsonwriter(true); stringwriter stringwriter = new stringwriter(); jsonwriter.write(viewset, stringwriter); system.out.println(stringwriter.tostring()); } } executing this code creates this json , which you can then copy and paste into the try it page of structurizr. the result (if you move the boxes around) is something like this. don't worry, there will eventually be an api for uploading software architecture models and the diagrams will get some styling, but it proves the concept. what we have then is an api that implements the various levels in my c4 software architecture model, with a simple browser-based rendering tool. hopefully that's a nice simple introduction of how to represent a software architecture model as code, and gives you a flavour for the sort of direction i'm taking it. having the software architecture as code provides some interesting opportunities that you don't get with static diagrams from visio, etc and the ability to keep the models up to date automatically by scanning the codebase is what i find particularly exciting. if you have any thoughts on this, please do drop me a note.

My objective here is to recreate a similar set-up in a smaller unit test mode.

API Management? Did you say “API Management?” Software application development models are evolutionary things. New technologies are always being created and require new approaches. It’s frequently the case today, that a service oriented architecture (SOA) model is used and that the end product is a software service that can be used by applications. The explosion in growth of mobile devices has only accelerated this trend. Every new mobile phone sold is another platform onto which applications are deployed. These applications are often built from services provided from multiple sources. The applications often consume these services through their APIs. OK, that’s all interesting, but why does this matter? Here’s why: If you are providing a service, you’d probably like to receive payment when it’s used by an application. For example, let’s say that you’ve spent months creating a new service that provides incredibly accurate and timely driving directions. You can imagine every mobile phone GPS app making use of your service someday. That is, however, assuming that you can find a way to enforce a contract on consumers of the API and provide them with a service level agreement (SLA). Also, you have to find a way to actually track consumers’ use of the API so that you can actually enforce that SLA. Finally, you have to have the means to update a service and publish new versions of services. Likewise, if you are consuming a service, for example, if you want to build the killer app that will use that cool new mapping service, you have to have the means to find the API, identify the API’s endpoint, and register your usage of the API with its provider. The approach that is followed to fulfill both service providers’ and consumers’ needs is...API Management. JBoss apiman 1.0 apiman is JBoss’ open source API Management system. apiman fulfills service API providers’ and consumers’ needs by implementing: API Manager - The API Manager provides an easy way for API/service providers to use a web UI to define service contracts for their APIs, apply these contracts across multiple APIs, and control role-based user access and API versioning. These contracts can govern access to services and limits on the rate at which consumers can access services. The same UI enables API consumers to easily locate and access APIs. API Gateway - The gateway applies the service contract policies of API Management by enforcing at runtime the rules defined in the contracts and tracking the service API consumers’ use of the APIs for every request made to the services. The way that the API Gateway works is that the consumer of the service accesses the service through a URL that designates the API Gateway as a proxy for the service. If the policies defined to govern access to the service (see a later section in this post for a discussion of apiman polices), the API Gateway then proxies requests to the service’s backend API implementation. The best way to understand API Management with apiman is to see it in action. In this post, we’ll install apiman 1.0, configure an API with contracts through the API Manager, and watch the API Gateway control access to the API and track its use. Prerequisites We don’t need very much to run apiman out of the box. Before we install apiman, you’ll have to have Java (version 1.7 or newer) installed on your system. You’ll also need to git and maven installed to be able to build the example service that we’ll use. A note on software versions: In this post we’ll use the latest available version of apiman as of December 2014. As if this writing, version 1.0 of apiman was just released (December 2014). Depending on the versions of software that you use, some screen displays may look a bit different. Getting apiman Like all JBoss software, installation of apiman is simple. First, you will need an application server on which to install and run apiman. We’ll use the open source JBoss WildFly server release 8.2 (http://www.wildfly.org/). To make things easier, apiman includes a pointer to JBoss WildFly on its download page here: http://www.apiman.io/latest/download.html To install WildFly, simply download http://download.jboss.org/wildfly/8.2.0.Final/wildfly-8.2.0.Final.zip and unzip the file into the directory in which you want to run the sever. Then, download the apiman 1.0 WildFly overlay zip file inside the directory that was created when you un-zipped the WildFly download. The apiman 1.0 WildFly overlay zip file is available here: http://downloads.jboss.org/overlord/apiman/1.0.0.Final/apiman-distro-wildfly8-1.0.0.Final-overlay.zip The commands that you will execute will look something like this: mkdir apiman cd apiman unzip wildfly-8.2.0.Final.zip unzip -o apiman-distro-wildfly8-1.0.0.Final-overlay.zip -d wildfly-8.2.0.Final Then, to start the server, execute these commands: cd wildfly-8.2.0.Final ./bin/standalone.sh -c standalone-apiman.xml The server will write logging messages to the screen. When you see some messages that look like this, you’ll know that the server is up and running with apiman installed: 13:57:03,229 INFO [org.jboss.as.server] (ServerService Thread Pool -- 29) JBAS018559: Deployed "apiman-ds.xml" (runtime-name : "apiman-ds.xml") 13:57:03,261 INFO [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://127.0.0.1:9990/management 13:57:03,262 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:9990 13:57:03,262 INFO [org.jboss.as] (Controller Boot Thread) JBAS015874: WildFly 8.2.0.Final "Tweek" started in 5518ms - Started 754 of 858 services (171 services are lazy, passive or on-demand) If this were a production server, the first thing that we’d do is to change the OOTB default admin username and/or password. apiman is configured by default to use JBoss KeyCloak (http://keycloak.jboss.org/) for password security. Also, the default database used by apiman to store contract and service information is the H2 database. For a production server, you’d want to reconfigure this to use a production database. Note: apiman includes DDLs for both MySQL and PostgreSQL. For the purposes of our demo, we’ll keep things simple and use the default configuration. To access apiman’s API Manager UI, go to: http://localhost:8080/apiman-manager, and log in. The admin user account that we’ll use has a username of “admin” and a password of “admin123!” You should see a screen that looks like this: Before we start using apiman, let’s take a look at how apiman defines how services and the meta data on which they depend are organized. Policies, Plans, and Organizations apiman uses a hierarchical data model that consists of these elements: Polices, Plans, and Organizations: Policies Policies are at the lowest level of the data model, and they are the basis on which the higher level elements of the data model are built. A policy defines an action that is performed by the API Gateway at runtime. Everything defined in the API Manager UI is there to enable apiman to apply policies to requests made to services. When a request to a service is made, apiman creates a chain of policies to be applied to that request. apiman policy chains define a specific sequence order in which the policies defined in the API Manager UI are applied to service requests. The sequence in which incoming service requests have policies applied is: First, at the application level. In apiman, an application is contracted to use one or more services. Second, at the plan level. In apiman, policies are organized into groups called plans. (We’ll discuss plans in the next section of this post.) Third, at the individual service level. What happens is that when a service request is received by the API Gateway at runtime, the policy chain is applied in the order of application, plan, and service. If no failures, such as a rate counter being exceeded, occur, the API Gateway sends the request to the service’s backend API implementation. As we mentioned earlier in this post, the API Gateway acts as a proxy for the service: Next, when the API Gateway receives a response from the service’s backend implementation, the policy chain is applied again, but this time in the reverse order. The service policies are applied first, then the plan policies, and finally the application policies. If no failures occur, then the service response is sent back to the consumer of the service. By applying the policy chain twice, both for the originating incoming request and the resulting response, apiman allows policy implementations two opportunities to provide management functionality during the lifecycle. The following diagram illustrates this two-way approach to applying policies: Plans In apiman, a “plan” is a set policies that together define the level of service that apiman provides for service. Plans enable apiman users to define multiple different levels of service for their APIs, based on policies. It’s common to define different plans for the same service, where the differences depend on configuration options. For example, a group or company may offer both a “gold” and “silver” plan for the same service. The gold plan may be more expensive than the silver plan, but it may offer a higher level of service requests in a given (and configurable) time period. Organizations The “organization” is at top level of the apiman data model. An organization contains and manages all elements used by a company, university, group inside a company, etc. for API management with apiman. All plans, services, applications, and users for a group are defined in an apiman organization. In this way, an organization acts as a container of other elements. Users must be associated with an organization before they can use apiman to create or consume services. apiman implements role-based access controls for users. The role assigned to a user defines the actions that a user can perform and the elements that a user can manage. Before we can define a service, the policies that govern how it is accessed, the users who will be able to access, and the organizations that will create and consume it, we need a service and a client to access that service. Luckily, creating the service and deploying it to our WildFly server, and accessing it through a client are easy. Getting and Building and Deploying the Example Service The source code for the example service is contained in a git repo (http://git-scm.com) hosted at github (https://github.com/apiman). To download a copy of the example service, navigate to the directory in which you want to build the service and execute this git command: git clone [email protected]:apiman/apiman-quickstarts.git As the source code is downloading, you'll see output that looks like this: git clone [email protected]:apiman/apiman-quickstarts.git Initialized empty Git repository in /tmp/tmp/apiman-quickstarts/.git/ remote: Counting objects: 104, done. remote: Total 104 (delta 0), reused 0 (delta 0) Receiving objects: 100% (104/104), 18.16 KiB, done. Resolving deltas: 100% (40/40), done. And, after the download is complete, you'll see a populated directory tree that looks like this: └── apiman-quickstarts ├── echo-service │ ├── pom.xml │ ├── README.md │ └── src │ └── main │ ├── java │ │ └── io │ │ └── apiman │ │ └── quickstarts │ │ └── echo │ │ ├── EchoResponse.java │ │ └── EchoServlet.java │ └── webapp │ └── WEB-INF │ ├── jboss-web.xml │ └── web.xml ├── LICENSE ├── pom.xml ├── README.md ├── release.sh └── src └── main └── assembly └── dist.xml As we mentioned earlier in the post, the example service is very simple. The only action that the service performs is to echo back in responses the meta data in the REST (http://en.wikipedia.org/wiki/Representational_state_transfer) requests that it receives. Maven is used to build the service. To build the service into a deployable .war file, navigate to the directory into which you downloaded the service example: cd apiman-quickstarts/echo-service And then execute this maven command: mvn package As the service is being built into a .war file, you'll see output that looks like this: [INFO] Scanning for projects... [INFO] [INFO] Using the builder org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder with a thread count of 1 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building apiman-quickstarts-echo-service 1.0.1-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ apiman-quickstarts-echo-service --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /jboss/local/redhat_git/apiman-quickstarts/echo-service/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ apiman-quickstarts-echo-service --- [INFO] Compiling 2 source files to /jboss/local/redhat_git/apiman-quickstarts/echo-service/target/classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ apiman-quickstarts-echo-service --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /jboss/local/redhat_git/apiman-quickstarts/echo-service/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ apiman-quickstarts-echo-service --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ apiman-quickstarts-echo-service --- [INFO] No tests to run. [INFO] [INFO] --- maven-war-plugin:2.2:war (default-war) @ apiman-quickstarts-echo-service --- [INFO] Packaging webapp [INFO] Assembling webapp in [/jboss/local/redhat_git/apiman-quickstarts/echo-service/target/apiman-quickstarts-echo-service-1.0.1-SNAPSHOT] [INFO] Processing war project [INFO] Copying webapp resources [/jboss/local/redhat_git/apiman-quickstarts/echo-service/src/main/webapp] [INFO] Webapp assembled in [23 msecs] [INFO] Building war: /jboss/local/redhat_git/apiman-quickstarts/echo-service/target/apiman-quickstarts-echo-service-1.0.1-SNAPSHOT.war [INFO] WEB-INF/web.xml already added, skipping [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1.184 s [INFO] Finished at: 2014-12-26T16:11:19-05:00 [INFO] Final Memory: 14M/295M [INFO] ------------------------------------------------------------------------ If you look closely, near the end of the output, you'll see the location of the .war file: /jboss/local/redhat_git/apiman-quickstarts/echo-service/target/apiman-quickstarts-echo-service-1.0.1-SNAPSHOT.war To deploy the service, we can copy the .war file to our WildFly server's "deployments" directory. After you copy the service's .war file to the deployments directory, you'll see output like this generated by the WildFly server: 16:54:44,313 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) JBAS015876: Starting deployment of "apiman-quickstarts-echo-service-1.0.1-SNAPSHOT.war" (runtime-name: "apiman-quickstarts-echo-service-1.0.1-SNAPSHOT.war") 16:54:44,397 INFO [org.wildfly.extension.undertow] (MSC service thread 1-16) JBAS017534: Registered web context: /apiman-echo 16:54:44,455 INFO [org.jboss.as.server] (DeploymentScanner-threads - 1) JBAS018559: Deployed "apiman-quickstarts-echo-service-1.0.1-SNAPSHOT.war" (runtime-name : "apiman-quickstarts-echo-service-1.0.1-SNAPSHOT.war") Make special note of this line of output: 16:54:44,397 INFO [org.wildfly.extension.undertow] (MSC service thread 1-16) JBAS017534: Registered web context: /apiman-echo This output indicates that the URL of the deployed example service is: [a href="http://localhost:8080/apiman-echo" style="text-decoration: none;"]http://localhost:8080/apiman-echo Remember, however, that this is the URL of the deployed example service if we access it directly. We'll refer to this as the "unmanaged service" as we are able to connect to the service directly, without going through the API Gateway. The URL to access the service through the API Gateway ("the managed service") at runtime will be different. Now that our example service is installed, it’s time to install and configure our client to access the server. Accessing the Example Service Through a Client There are a lot of options available when it comes to what we can use for a client to access our service. We’ll keep the client simple so that we can keep our focus on apiman and simply install a REST client into the FireFox browser. The REST Client FireFox add-on (http://restclient.net/) is available here: https://addons.mozilla.org/en-US/firefox/addon/restclient/versions/2.0.3 After you install the client into FireFox, you can access the deployed service using the URL that we just defined. If you execute a GET command, you’ll see output that looks like this: Now that our example service is built, deployed and running, it’s time to create the organizations for the service provider and the service consumer. The differences between the requirements of the two organizations will be evident in their apiman configuration properties. Creating Users for the Service Provider and Consumer Before we create the organizations, we have to create a user for each organization. We'll start by creating the service provider user. To do this, logout from the admin account in the API Manager UI. The login dialog will then be displayed. Select the "New user" Option and register the service provider user: Then, logout and repeat the process to register a new application developer user too: Now that the new users are registered we can create the organizations. Creating the Service Producer Organization To create the service producer organization, log back into the API Manager UI as the servprov user and select “Create a new Organization”: Select a name and description for the organization, and press “Create Organization”: And, here’s our organization: Note that in a production environment, users would request membership in an organization. The approval process for accepting new members into an organization would follow the organization's workflow, but this would be handled outside of the API Manager. For the purposes of our demonstration, we'll keep things simple. Configuring the Service, its Policies, and Plans To configure the service, we’ll first create a plan to contain the policies that we want applied by the API Gateway at runtime when requests to the service are made. To create a new plan, select the “Plans” tab. We’ll create a “gold” plan: Once the plan is created, we will add policies to it: apiman provides several OOTB policies. Since we want to be able to demonstrate a policy being applied, we’ll select a Rate Limiting Policy, and set its limit to a very low level. If our service receives more than 10 requests in a day, the policy should block all subsequent requests. So much for a “gold” level of service! After we create the policy and add it to the plan, we have to lock the plan: And, here is the finished, and locked plan: At this point, additional plans can be defined for the service. We’ll also create a “silver” plan, that will offer a lower level of service (i.e., a request rate limit lower than 10 per day) than the gold plan. Since the process to create this silver plan is identical to that of the gold plan, we’ll skip the screenshots. Now that the two plans are complete and locked, it’s time to define the service. We’ll give the service an appropriate name, so that providers and consumers alike will be able to run a query in the API Manager to find it. After the service is defined, we have to define its implementation. In the context of the API Manager, the API Endpoint is the service’s direct URL. Remember that the API Gateway will act as a proxy for the service, so it must know the service’s actual URL. In the case of our example service, the URL is: http://localhost:8080/apiman-echo The plans tab shows which plans are available to be applied to the service: Let’s make our service more secure by adding an authentication policy that will require users to login before they can access the service. Select the Policies tab, and then define a simple authentication policy. Remember the user name and password that you define here as we’ll need them later on when send requests to the service. After the authentication policy is added, we can publish the service to the API Gateway: And, here it is, the published service: OK, that finishes the definition of the service provider organization and the publication of the service. Next, we'll switch over to the service consumer side and create the service consumer organization and register an application to connect to the managed service through the proxy of the API Gateway. The Service Consumer Organization We'll repeat the process that we used to create the application development organization. Log in to the API Manager UI as the “appdev” user and create the organization: Unlike the process we used when we created the elements used by the service provider, the first step that we’ll take is to create a new application and then search for the service to be used by the application: Searching for the service is easy, as we were careful to set the service name to something memorable: Select the service name, and then specify the plan to be used. We’ll splurge and use the gold plan: Next, select “create contract” for the plan: Then, agree to the contract terms (which seem to be written in a strange form of Latin in the apiman 1.0 release): The last step is to register the application with the API Gateway so that the gateway can act as a proxy for the service: Congratulations! All the steps necessary to provide and consume the service are complete! There’s just one more step that we have to take in order for clients to be able access the service through the API Gateway. Remember the URL that we used to access the unmanaged service directly? Well, forget it. In order to access the managed service through the API Gateway acting as a proxy for other service we have to obtain the managed service's URL. In the API Manager UI, head on over to the "APIs" tab for the application, click on the the “>” character to the left of the service name. This will expose the API Key and the service’s HTTP endpoint in the API Gateway: In order to be able access the service through the API Gateway, we have to provide the API Key with each request. The API Key can be provided either through an HTTP Header (X-API-Key) or a URL query parameter. Luckily, the API Manager UI does the latter for us. Select the icon to the right of the HTTP Endpoint and this dialog is displayed: Copy the URL into the clipboard. We’ll need to enter this into the client in a bit. The combined API Key and HTTP endpoint should look something like this: http://localhost:8080/apiman-gateway/ACMEServices/echo/1.0?apikey=c374c202-d4b3-4442-b9e4-c6654f406e3d Accessing the Managed Service Through the apiman API Gateway, Watching the Policies at Runtime Thanks for hanging in there! The set up is done. Now, we can fire up the client and watch the policies in action as they are applied at runtime by the API Gateway, for example: Open the client, and enter the URL for the managed service (http://localhost:8080/apiman-gateway/ACMEServices/echo/1.0?apikey=c374c202-d4b3-4442-b9e4-c6654f406e3d) What happens first is that the authentication policy is applied and a login dialog is then displayed: Enter the username and password (user1/password) that we defined when we created the authentication policy to access the service. The fact that you are seeing this dialog confirms that you are accessing the managed service and are not accessing the service directly. When you send a GET request to the service, you should see a successful response: So far so good. Now, send 10 more requests and you will see a response that looks like this as the gold plan rate limit is exceeded: And there it is. Your gold plan has been exceeded. Maybe next time you’ll spend a little more and get the platinum plan! ;-) Wrap-up Let’s recap what we just accomplished in this demo: We installed apiman 1.0 onto a WildFly server instance. We used git to download and maven to build a sample REST client. As a service provider, we created an organization, defined policies based on service use limit rates and user authentication, and a plan, and assigned them to a service. As a service consumer, we searched for and found that service, and assigned it to an application. As a client, we accessed the service and observed how the API Gateway managed the service. And, if you note, in the process of doing all this, the only code that we had to write or build was for the client. We were able to fully configure the service, policies, plans, and the application in the API Manager UI. What’s Next? In this post, we’ve only scratched the surface of API Management with apiman. To learn more about apiman, you can explore its website here: http://www.apiman.io/ Join the project mailing list here: https://lists.jboss.org/mailman/listinfo/apiman-user And, better still, get involved! Contribute bug reports or feature requests. Write about your own experiences with apiman. Download the apiman source code, take a look around, and contribute your own additions. apiman 1.0 was just released, there’s no better time to join in and contribute! Acknowledgements The author would like to acknowledge Eric Wittmann for his (never impatient) review comments and suggestions on writing this post! Downloads Used in this Article REST Client (http://restclient.net/) FireFox Add-On - https://addons.mozilla.org/en-US/firefox/addon/restclient/versions/2.0.3 Echo service source code - https://github.com/EricWittmann/apiman-quickstarts apiman 1.0 - http://downloads.jboss.org/overlord/apiman/1.0.0.Final/apiman-distro-wildfly8-1.0.0.Final-overlay.zip WildFly 8.2.0 - http://download.jboss.org/wildfly/8.2.0.Final/wildfly-8.2.0.Final.zip Git - http://git-scm.com Maven - http://maven.apache.org References http://www.apiman.io/ apiman tutorial videos - https://vimeo.com/user34396826 http://www.softwareag.com/blog/reality_check/index.php/soa-what/what-is-api-management/ http://keycloak.jboss.org/

spring boot already comes with great, pre-configured logging system inside, but in real projects it's important to have an ability to search logs, aggregate them and access easy. one of the easiest option for it is http://papertrailapp.com/ . they provide logging service with syslog protocol and 100mb/mo free plan. lets prepare papertrail for our example: create logging group in papetrail dashboard ("create group" button). create log destination in papertrail dashboard. go to "account -> log destinations" click "create log destination" button. make sure your group is selected in " new systems will join" field. you can leave all others fields with their default values, just click "create". remember your log destination (will looks like logs2.papertrailapp:12345), we will use it later spring boot uses logback as default logging system. it's powerful tool for logging with many logging options. for our purposes we will use ch.qos.logback.classic.net.syslogappender . add "logback.xml" file to your "resources" folder with following content: ${papertrail_host} ${papertrail_port} user ${papertrail_app:-app} %highlight([%.-1level]) %35.35logger{35}:\t%m\t%cyan%ex{5} true i'm using environment variables ( (1), (2), and (3) ) for papertrail's credentials, it will allow you to configure different log destinations in different application environments. note excluded throwable at (4). we already have pattern for throwable in suffixpattern ( %cyan%ex{5} ). why we do this? because otherwise exception stacktraces will be printed after main log message line-by-line, it will increase traffic and also you will not be able to see stacktrace in search. i will demonstrate how it works with really simple spring boot application: package com.github.bsideup.spring.boot.example.papertrail; import org.slf4j.logger; import org.slf4j.loggerfactory; import org.springframework.boot.springapplication; import org.springframework.boot.autoconfigure.springbootapplication; import org.springframework.web.bind.annotation.requestmapping; import org.springframework.web.bind.annotation.restcontroller; @springbootapplication @restcontroller public class application { private static final logger log = loggerfactory.getlogger(application.class); public static void main(string[] args) { springapplication.run(application.class, args); } @requestmapping("/") public string index() { log.warn("i'm so tired to welcome everyone", new exception(new exception())); return "hello world!"; } } now, run your application with following environment variables: papertrail_host - host from your log destination (i.e. logs2.papertrailapp.com) papertrail_port - port from your log destination (i.e. 12345) [optional] papertrail_app - application name (default: app) your papertrails output will looks like mine: you can find sample project at github: https://github.com/bsideup/spring-boot-sample-papertrail

This article represents facts on what would it take to build one or more jar files for a given framework/library using Maven, provided the framework’s downloadable files consisted of pom.xml. Please feel free to comment/suggest if I missed to mention one or more important points. Also, sorry for the typos. So far, whenever I came across pom.xml file in the framework that I downloaded in order to get the jar file, I hated it. I used to, then, go to internet and get the compiled jar file(s) for the framework/library. And, good thing is that I have been able to get my work done. This was purely out of my laziness that I did not use to build using maven.Then, I got a chance to work with Twitter HBC library (Java) for integrating with Twitter. And, I downloaded it and wanted to get one or more jar files. And, once again, I came across apom.xml in root folder and unique pom.xml files in hbc-core, hbc-twitter4j and hbc-examples folder. This time, I decided to build the hbc jar files on my system.Following are some of the steps I took to build hbc jar files and get dependencies to run the program using hbc jar files. Download and install Maven. Anyone wanting to install/configure Maven, go to this Maven in 5 Minutes page. It clearly states what needs to be done to install/configure Maven. Once configured, open a command prompt and execute command “mvn -version”. If the version information of Maven is displayed, you are all set. Once determined, go to the folder which consists of pom.xml file. In present case, go to hbc root folder. Go to hbc root folder, hbc-master. Execute following command to build the hbc jar files and also obtain the dependencies (jar files) required to run the library. Command is “mvn clean install -U dependency:copy-dependencies“. This command built the source file and created two different jar files in hbc-twitter4j/target (hbc-twitter4j-2.2.1-SNAPSHOT.jar) and hbc-core/target (hbc-core-2.2.1-SNAPSHOT.jar). Further to that, it downloaded all the dependent jar files in repective target/dependency folder.

I have recently started to build a public REST API with Java for Podcastpedia.org and for the JAX-RS implementation I have chosen Jersey, as I find it “natural” and powerful – you can find out more about it by following the Tutorial – REST API design and implementation in Java with Jersey and Spring. Because Podcastpedia.org is a web application powered by Spring MVC, I wanted to integrate both frameworks in podcastpedia-web, to take advantage of the backend service functionality already present in the project. Anyway this short post will present the steps I had to take to make the integration between the two frameworks work. Framework versions Current versions used: 4.1.0.RELEASE 2.14 Project dependencies The Jersey Spring extension must be present in your project’s classpath. If you are using Maven add it to the pom.xml file of your project: org.glassfish.jersey.ext jersey-spring3 ${jersey.version} org.springframework spring-core org.springframework spring-web org.springframework spring-beans org.glassfish.jersey.media jersey-media-json-jackson ${jersey.version} com.fasterxml.jackson.jaxrs jackson-jaxrs-base com.fasterxml.jackson.core jackson-annotations com.fasterxml.jackson.jaxrs jackson-jaxrs-json-provider Note: I have explicitly excluded the Spring core and the Jackson implementation libraries as they have been already imported in the project with preferred versions. Web.xml configuration In the web.xml, in addition to the Spring MVC servlet configuration I added the jersey-servlet configuration, that will map all requests starting with/api/: Spring MVC Dispatcher Servlet org.springframework.web.servlet.DispatcherServlet contextConfigLocation classpath:spring/application-context.xml 1 Spring MVC Dispatcher Servlet / jersey-serlvet org.glassfish.jersey.servlet.ServletContainer javax.ws.rs.Application org.podcastpedia.web.api.JaxRsApplication 2 jersey-serlvet /api/* Well, that’s pretty much it… If you have any questions drop me a line or comment in the discussion below. In the coming post I will present some of the results of this integration, by showing how to call one method of the REST public API with jQuery, to dynamically load recent episodes of a podcast, so stay tuned.

Recently at DevSKiller.com we've decided to move majority of our stuff to simple containers. It was pretty easy due to use of Spring Boot uber-jars, but the problem was in NewRelic agents which should have to be included separately. That caused uncomfortable situation so we decided to solve it by including NewRelic agent into our uber-jar applications. If you also want to simplify your life please follow provided instructions :) At first we have to add proper dependency into our pom.xml descriptor: com.newrelic.agent.java< newrelic-agent 3.12.1 provided Now since we have proper jar included into our project it's time to unpack the dependency to have all necessary classes in our application jar file: org.apache.maven.plugins maven-dependency-plugin 2.9 prepare-package unpack-dependencies newrelic-agent ${project.build.outputDirectory} After this step we've all agent related classes accessible directly from our jar. But still the file cannot be used as an agent jar. There are some important manifest entries that have to be present in every agent jar. The most important is the Premain-Class attribute specifying main agent class including premain() method. In case of NewRelic it's also important to include Can-Redefine-Classes and Can-Retransform-Classes attributes. The easiest way to do that is to extend maven-jar-plugin configuration: org.apache.maven.plugins maven-jar-plugin 2.5 com.newrelic.bootstrap.BootstrapAgent true true Now is coming the tricky part :) NewRelic agent also contains class with main() method which causes that Spring Boot repackager plugin is unable to find single main() method so build fails. It's not a problem but we have to remember to specify proper main class in spring-boot-maven-plugin (or in gradle plugin): my.custom.Application That's all! You can execute your application with following command: java -javaagent:myapp.jar -jar myapp.jar Last but not least: don't forget to include NewRelic configuration file (newrelic.yml) in the same directory as your application jar. The other solution is to set newrelic.config.file system property to point the fully qualified file name.

Caching solves wide sort of performance problems. There are many ways to integrate caching into our applications. For example when we use Spring there is easy to use @Cacheable support. Quite easy but we still have to configure cache manager, cache regions, etc. Sometimes it's unfortunately like taking a sledgehammer to crack a nut. So what can we do to "go lighter"? There is a technique called memoization. Technically it's as easy as pie but true genius lies in simplicity. Model solution looks as follows: public Foo getValue() { if (storedValue == null) { storedValue = retrieveFoo(); } return storedValue; } As you can see there is no problem in implementing it manually, but as long as we remember about DRY rule we can use already implemented solutions which additionally provides thread safety. Pretty good idea is to use Guava library. // create Supplier memoizer = Suppliers.memoize(this::retrieveFoo); // and use Foo variable = memoizer.get(); Sometimes it's enough but what can we do if we need to specify TTL for our value? We have to store (cache) retrieved value only for few seconds and after exceeding defined duration get this value one more time? One more time we can use functionality provided by Guava. Supplier memoizer = Suppliers.memoizeWithExpiration(this::retrieveFoo, 5, TimeUnit.SECONDS); The above line builds memoizer with TTL = 5 seconds. As you can see - simple... but powerful :)

One of the goals of JBoss AS7 was to make it much more secure by default, when compared to previous versions. One of the areas which was directly impacted by this goal was that you could no longer expect the server to expose some service on a port and get access to it without any authentication/authorization. Remember that in previous versions of JBoss AS you could access the JNDI port, the JMX port without any authentication/authorization, as long as those ports were opened for communication remotely. Finer grained authorizations on such ports for communications, in JBoss AS7, allows the server to control who gets to invoke operations over that port. Of course, this is not just limited to JBoss AS7 but continues to be the goal in WildFly (which is the rename of JBoss Application Server). In fact, WildFly has gone one step further and now has the feature of "one single port" for all communication. JMX communication in JBoss AS7 and WildFly With that background, we'll now focus on JMX communication in JBoss AS7 and WildFly. I'll use WildFly (8.2.0 Final) as a reference for the rest of this article, but the same details apply (with minor changes) to other major versions of JBoss AS7 and WildFly, that have been released till date. WildFly server is composed of "subsystems", each of which expose a particular set of functionality. For example, there's the EE subsystem which supports the Java EE feature set. Then there's the Undertow subsystem which supports web/HTTP server functionality. Similarly, there's a JMX subsystem which exposes the JMX feature set on the server. As you all are aware, I'm sure, JMX service is standardly used for monitoring and even managing Java servers and this includes managing the servers remotely. The JMX subsystem in WildFly allows remote access to the JMX service and port 9990 is what is used for that remote JMX communication. JConsole for remote JMX access against JBoss AS7 and WildFly Java (JDK) comes bundled with the JConsole tool which allows connecting to local or remote Java runtimes which expose the JMX service. The tool is easy to use, all you have to do is run the jconsole command it will show up a graphical menu listing any local Java processes and also an option to specify a remote URL to connect to a remote process: # Start the JConsole $JAVA_HOME/bin/jconsole Let's assume that you have started WildFly standalone server, locally. Now when you start the jconsole, you'll notice that the WildFly Java process is listed in the local running processes to which you can connect to. When you select the WildFly Java instance, you'll be auto connected to it and you'll notice MBeans that are exposed by the server. However, in the context of this article, this "local process" mode in JConsole isn't what we are interested in. Let's use the "Remote process" option in that JConsole menu which allows you to specify the remote URL to connect to the Java runtime and username and password to use to connect to that instance. Even though our WildFly server is running locally, we can use this "Remote process" option to try and connect to it. So let's try it out. Before that though, let's consider a the following few points: Remember that the JMX subsystem in WildFly allows remote access on port 9990 For remote access to JMX, the URL is of the format - service:jmx:[vendor-specific-protocol]://[host]:[port]. The vendor specific protocol is the interesting bit here. In the case of WildFly that vendor-specific-protocol is http-remoting-jmx. Remember that WildFly is secure by default which means that just because the JMX subsystem exposes 9990 port for remote communication, it doesn't mean it's open for communication to anyone. In order to be allowed to communicate over this port, the caller client is expected to be authenticated and authorized. This is backed by the "ManagementRealm" in WildFly. Users authenticated and authorized against this realm are allowed access to that port. Keeping those points in mind, let's first create a user in the Management Realm. This can be done using the add-user command line script (which is present in JBOSS_HOME/bin folder). I won't go into the details of that since there's enough documentation for that. Let's just assume that I created a user named "wflyadmin" with an appropriate password in the Management Realm. To verify that the user has been properly created, in the right realm, let's access the WildFly admin console at the URL http://localhost:9990/console. You'll be asked for username and password for access. Use the same username and password of the newly created user. If the login works, then you are good. If not, then make sure you have done things right while adding the new user (as I said I won't go into the details of adding a new user since it's going to just stretch this article unnecessarily long). So at this point we have created a user named "wflyadmin" belonging to ManagementRealm. We'll be using this same user account for accessing the JMX service on WildFly, through JConsole. So let's now bring up the jconsole as usual: $JAVA_HOME/bin/jconsole On the JConsole menu let's again select the "Remote process" option and use the following URL in the URL text box: service:jmx:http-remoting-jmx://localhost:9990 Note: For JBoss AS 7.x and JBoss EAP 6.x, the vendor specific protocol is remoting-jmx and the port for communication is 9999. So the URL will be service:jmx:remoting-jmx://localhost:9999 In the username and password textboxes, use the same user/pass that you newly created. Finally, click on Connect. What do you see? It doesn't work! The connection fails. So what went wrong? Why isn't the JConsole remote access to WildFly not working? You did all the obvious things necessary to access the WildFly JMX service remotely but you keep seeing that JConsole can't connect to it. What could be the reason? Remember, in one of those points earlier, I noted that the "vendor specific protocol" is an interesting bit? We use http-remoting-jmx and that protocol internally relies on certain WildFly/JBoss specific libraries, primarily for remote communication and authentication and authorization. These libraries are WildFly server specific and hence aren't part of the standard Java runtime environment. When you start jconsole, it uses a standard classpath which just has the relevant libraries that are part of the JDK/JRE. To solve this problem, what you need to do is bring in the WildFly server specific libraries into the classpath of JConsole. Before looking into how to do that, let's see which are the WildFly specific libraries that are needed. All the necessary classes for this to work are part of the jboss-cli-client.jar which is present in JBOSS_HOME/bin/client/ folder. So all we need to do in include this jar in the classpath of the jconsole tool. To do that we use the -J option of jconsole tool which allows passing parameters to the Java runtime of jconsole. The command to do that is: $JAVA_HOME/bin/jconsole -J-Djava.class.path=$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/jconsole.jar:/opt/wildfly-8.2.0.Final/bin/client/jboss-cli-client.jar (Note that for Windows the classpath separator is the semi-colon character instead of the colon) Note, the server specific jar for JBoss AS 7.x and JBoss EAP 6.x is named jboss-client.jar and is present at the same JBOSS_HOME/bin/client directory location. So we are passing -Djava.class.path as the parameter to the jconsole Java runtime, using the -J option. Notice that we have specified more than just our server specific jar in that classpath. That's because, using the -Djava.class.path is expected to contain the complete classpath. We are including the jars from the Java JDK lib folder that are necessary for JConsole and also our server specific jar in that classpath. Running that command should bring up JConsole as usual and let's go ahead and select the "Remote process" option and specify the same URL as before: service:jmx:http-remoting-jmx://localhost:9990 and the same username and password as before and click Connect. This time you should be able to connect and should start seeing the MBeans and others services exposed over JMX. How about providing a script which does this necessary classpath setup? Since it's a common thing to try and use JConsole for remote access against WildFly, it's reasonable to expect to have a script which sets up the classpath (as above) and you could then just use that script. That's why WildFly ships such a script. It's in the JBOSS_HOME/bin folder and is called jconsole.sh (and jconsole.bat for Windows). This is just a wrapper script which internally invokes the jconsole tool present in Java JDK, after setting up the classpath appropriately. All you have to do is run: $JBOSS_HOME/bin/jconsole.sh What about using JConsole from a really remote machine, against WildFly? So far we were using the jconsole tool that was present on the same machine as the WildFly instance, which meant that we have filesystem access to the WildFly server specific jars present in the WildFly installation directory on the filesystem. This allowed us to setup the classpath for jconsole to point to the jar on the local filesystem? What if you wanted to run jconsole from a remote machine against a WildFly server which is installed and running on a different machine. In that case, your remote client machine won't be having filesystem access to the WildFly installation directory. So to get jconsole running in such a scenario, you will have to copy over the JBOSS_HOME/bin/jboss-cli-client.jar to your remote client machine, to a directory of your choice and then setup the classpath for jconsole tool as explained earlier and point it to that jar location. That should get you access to JMX services of WildFly from jconsole on a remote machine. More questions? If you still have problems getting this to work or have other questions, please start a discussion in the JBoss community forums here https://developer.jboss.org/en/wildfly/content.

NEW EDIT: As of Spring Boot 1.4.0, faking of Spring Beans is supported natively via annotation @MockBean. Read Spring Boot docs for more info. OLD EDIT: Here is better example how to mock Spring bean. I've worked with Spring for several years. But I was always frustrated with how messy can XML configuration become. As various annotations and possibilities of Java configuration were popping up, I started to enjoy programming with Spring. That is why I strongly entourage using Java configuration. In my opinion, XML configuration is suitable only when you need to have visualized Spring Integration or Spring Batch flow. Hopefully Spring Tool Suite will be able to visualize Java configurations for these frameworks also. One of the nasty aspects of XML configuration is that it often leads to huge XML configuration files. Developers therefore often create test context configuration for integration testing. But what is the purpose of integration testing, when there isn’t production wiring tested? Such integration test has very little value. So I was always trying to design my production contexts in testable fashion. I except that when you are creating new project / module you would avoid XML configuration as much as possible. So with Java configuration you can create Spring configuration per module / package and scan them in main context (@Configuration is also candidate for component scanning). This way you can naturally create islands Spring beans. These islands can be easily tested in isolation. But I have to admit that it’s not always possible to test production Java configuration as is. Rarely you need to amend behavior or spy on certain beans. There is library for it called Springockito. To be honest I didn’t use it so far, because I always try to design Spring configuration to avoid need for mocking. Looking at Springockito pace of development and number of open issues, I would be little bit worried to introduce it into my test suite stack. Fact that last release was done before Spring 4 release brings up questions like “Is it possible to easily integrate it with Spring 4?”. I don’t know, because I didn’t try it. I prefer pure Spring approach if I need to mock Spring bean in integration test. Spring provides @Primary annotation for specifying which bean should be preferred in the case when two beans with same type are registered. This is handy because you can override production bean with fake bean in integration test. Let’s explore this approach and some pitfalls on examples. I chose this simplistic / dummy production code structure for demonstration: @Repository public class AddressDao { public String readAddress(String userName) { return "3 Dark Corner"; } } @Service public class AddressService { private AddressDao addressDao; @Autowired public AddressService(AddressDao addressDao) { this.addressDao = addressDao; } public String getAddressForUser(String userName){ return addressDao.readAddress(userName); } } @Service public class UserService { private AddressService addressService; @Autowired public UserService(AddressService addressService) { this.addressService = addressService; } public String getUserDetails(String userName){ String address = addressService.getAddressForUser(userName); return String.format("User %s, %s", userName, address); } } AddressDao singleton bean instance is injected into AddressService. AddressService is similarly used in UserService. I have to warn you at this stage. My approach is slightly invasive to production code. To be able to fake existing production beans, we have to register fake beans in integration test. But these fake beans are usually in the same package sub-tree as production beans (assuming you are using standard Maven files structure: “src/main/java” and “src/test/java”). So when they are in the same package sub-tree, they would be scanned during integration tests. But we don’t want to use all bean fakes in all integration tests. Fakes could break unrelated integration tests. So we need to have mechanism, how to tell the test to use only certain fake beans. This is done by excluding fake beans from component scanning completely. Integration test explicitly define which fake/s are being used (will show this later). Now let’s take a look at mechanism of excluding fake beans from component scanning. We define our own marker annotation: public @interface BeanMock { } And exclude @BeanMock annotation from component scanning in main Spring configuration. @Configuration @ComponentScan(excludeFilters = @Filter(BeanMock.class)) @EnableAutoConfiguration public class Application { } Root package of component scan is current package of Application class. So all above production beans needs to be in same package or sub-package. We are now need to create integration test forUserService. Let’s spy on address service bean. Of course such testing doesn’t make practical sense with this production code, but this is just example. So here is our spying bean: @Configuration @BeanMock public class AddressServiceSpy { @Bean @Primary public AddressService registerAddressServiceSpy(AddressService addressService) { return spy(addressService); } } Production AddressService bean is autowired from production context, wrapped into Mockito‘s spy and registered as primary bean for AddressService type. @Primary annotation makes sure that our fake bean will be used in integration test instead of production bean. @BeanMock annotation ensures that this bean can’t be scanned by Application component scanning. Let’s take a look at the integration test now: @RunWith(SpringJUnit4ClassRunner.class) @SpringApplicationConfiguration(classes = { Application.class, AddressServiceSpy.class }) public class UserServiceITest { @Autowired private UserService userService; @Autowired private AddressService addressService; @Test public void testGetUserDetails() { // GIVEN - spring context defined by Application class // WHEN String actualUserDetails = userService.getUserDetails("john"); // THEN Assert.assertEquals("User john, 3 Dark Corner", actualUserDetails); verify(addressService, times(1)).getAddressForUser("john"); } } @SpringApplicationConfigration annotation has two parameters. First (Application.class) declares Spring configuration under test. Second parameter (AddressServiceSpy.class) specifies fake bean that will be loaded for our testing into Spring IoC container. It’s obvious that we can use as many bean fakes as needed, but you don’t want to have many bean fakes. This approach should be used rarely and if you observe yourself using such mocking often, you are probably having serious problem with tight coupling in your application or within your development team in general. TDD methodology should help you target this problem. Bear in mind: “Less mocking is always better!”. So consider production design changes that allow for lower usage of mocks. This applies also for unit testing. Within integration test we can autowire this spy bean and use it for various verifications. In this case we verified if testing method userService.getUserDetails called methodaddressService.getAddressForUser with parameter “john”. I have one more example. In this case we wouldn’t spy on production bean. We will mock it: @Configuration @BeanMock public class AddressDaoMock { @Bean @Primary public AddressDao registerAddressDaoMock() { return mock(AddressDao.class); } } Again we override production bean, but this time we replace it with Mockito’s mock. We can than record behavior for mock in our integration test: @RunWith(SpringJUnit4ClassRunner.class) @SpringApplicationConfiguration(classes = { Application.class, AddressDaoMock.class }) public class AddressServiceITest { @Autowired private AddressService addressService; @Autowired private AddressDao addressDao; @Test public void testGetAddressForUser() { // GIVEN when(addressDao.readAddress("john")).thenReturn("5 Bright Corner"); // WHEN String actualAddress = addressService.getAddressForUser("john"); // THEN Assert.assertEquals("5 Bright Corner", actualAddress); } @After public void resetMock() { reset(addressDao); } } We load mocked bean via @SpringApplicationConfiguration‘s parameter. In test method, we stubaddressDao.readAddress method to return “5 Bright Corner” string when “john” is passed to it as parameter. But bear in mind that recorded behavior can be carried to different integration test via Spring context. We don’t want tests affecting each other. So you can avoid future problems in your test suite by reseting mocks after test. This is done in method resetMock. Source code is on Github.

Learn how to deploy static content on a JBoss server.

This is part of a series of blogs from Espresso Logic’s Application Engineering team on common coding tasks and how they can be accomplished more efficiently using Espresso Logic. The purpose of these blogs is two fold: Provide code examples of how Espresso system can be used to solve certain use cases Provide code samples that you can use within your programs while developing apps This blog deals with a design pattern all web and mobile transaction applications have to deal with – how to create a multiple table self registration using a single POST. This is both a performance issue, reducing the number of calls to and from the server, as well as a demonstration of how Espresso Logic advanced design handles complex transactions without coding. In the prior blog we dealt with rules used to validate credit cards. Self Registration Design Pattern User self registration is a standard design pattern for some web and mobile applications. This becomes a bit more challenging when the database model is using multiple tables. Using Espresso Logic , you can create a multiple table insert using a single POST. The back-end server can be run in the cloud or on-premise to connect to your database and quickly expose RESTful endpoints for each of your SQL tables, views, and stored procedures. Creating a compound nested document (called a Resource) will allow us to design an API that will join related tables into a single REST endpoint. Developers can design a Mobile front-end using a drag-and-drop tools that bind each of the text fields with the JSON nested document fields. Using a single POST, this data is sent to the Espresso Logic REST server to handle the details. Self Registration Account Setup Data Model The model below shows tables that hold each of the relevant parts of the self-registration entry. Each of these tables has an auto increment primary key and a foreign key relationship to the Person table (e.g. Password, PersonPhone, Address, EmailAddress) to support the one-to-many relationship cardinality. So how can we create a single POST to insert into multiple tables and propagate the primary key? Connecting to any SQL database, Espresso Logic will instantly create REST API definitions for every table, view, and stored procedure. Multiple table data model Create new Resource Next, we us the Espresso Logic Design Studio to select one or more tables from a point-and-click interface to create this new compound document endpoint (see Resource documentation), which is a combination of each of the child tables. The ‘Join’ for each child is be automatically completed using the existing relationships defined in the schema. The unused Attributes for the POST can be excluded including the primary key (auto increment) and the foreign parent key PersonID (these are handled by the server) in each child table. Multiple Table Resource Setup POST JSON Sample Data The REST Lab in Espresso Logic Design Studio can be used to test this new resource. We enter all the values of the JSON self registration (shown below) and use the POST command. { "Title": "Mr", "FirstName": "Test", "MiddleName": "M", "LastName": "Record5", "Phone": [ { "PhoneNumber": "(407) 555-1216", "PhoneNumberTypeID": 1 } ], "Address": [ { "AddressTypeID": 2, "AddressLine1": "555 Main St", "AddressLine2": "Apt 6", "City": "Maitland", "StateProvinceID": 15, "PostalCode": "32751" }, { "AddressTypeID": 1, "AddressLine1": "555 Main St", "AddressLine2": "6", "City": "Maitland", "StateProvinceID": 15, "PostalCode": "32751" } ], "Email": [ { "EmailAddress": "[email protected]" } ], "Password": [ { "PasswordHash": " password6", "Username": "user" } ] } How it works The Espresso Logic REST Server will take this JSON and perform the necessary validations, derivations, and event processing (using Reactive Logic Programming) and then insert the Person first (returning the primary key) and then propagate this down to each of the related children in a single transaction. All the logic, validations, and events must succeed or the entire transaction is rolled back. This is not an easy trick to perform for any REST API – so do not try this at home without a deep understanding of SQL, transactions, the data model and relationships between parent and child. Espresso Logic does this out-of-the-box with no code required. Live Browser Espresso Logic offers Live Browser which is an Instant HTML5/Angular view of your data using the active schema. This will lets us view the one-to-many relationships of the self registration process. Try it yourself and see the power of Espresso Logic. View the samples on GitHub. Espresso Logic ‘Mutliple Table insert using a Single POST’ examples are part of the extended demo. Summary The mobile and web developer want a simple and single REST endpoint to populate and update user information. The server should be able to handle the complexity and hide the underlying data model. Using the Espresso Logic back-end server eliminates all the code required to connect to the database, create a nested document endpoint, and propagate the primary key to all the related child tables. Live Browser gives you an instant view to see and test your results – again, no code. This is the fastest way to build and deliver mobile solutions on the market today.

JSR 354 defines a new Java API for working with Money and Currencies, which is planned to be included in Java 9. In this post we will look at the current state of the reference implementation: JavaMoney. Like my post about the Java 8 Date/Time API this post will be mainly driven by code that shows the new API. But before we start, I want to quote a short section from the specification that pretty much sums up the motivation for this new API: Monetary values are a key feature of many applications, yet the JDK provides little or no support. The existing java.util.Currency class is strictly a structure used for representing current ISO 4217 currencies, but not associated values or custom currencies. The JDK also provides no support for monetary arithmetic or currency conversion, nor for a standard value type to represent a monetary amount. If you use Maven, you can easily try the current state of the reference implementation by adding the following dependency to your project: org.javamoney moneta 0.9 All specification classes and interfaces are located in the javax.money.* package. We will start with the two core interfaces CurrencyUnit and MonetaryAmount. After that, we will look into exchange rates, currency conversion and formatting. CurrencyUnit and MonetaryAmount CurrencyUnit models a currency. CurrencyUnit is very similar to the existing java.util.Currency class, except it allows custom implementations. According to the specification it should be possible that java.util.Currency implements CurrencyUnit. CurrencyUnit instances can be obtained using the MonetaryCurrencies factory: // getting CurrencyUnits by currency code CurrencyUnit euro = MonetaryCurrencies.getCurrency("EUR"); CurrencyUnit usDollar = MonetaryCurrencies.getCurrency("USD"); // getting CurrencyUnits by locale CurrencyUnit yen = MonetaryCurrencies.getCurrency(Locale.JAPAN); CurrencyUnit canadianDollar = MonetaryCurrencies.getCurrency(Locale.CANADA); MontetaryAmount represents a concrete numeric representation of a monetary amount. A MonetaryAmount is always bound to a CurrencyUnit. Like CurrencyUnit, MonetaryAmount is an interface that supports different implementations. CurrencyUnit and MonetaryAmount implementations must be immutable, thread safe, serializable and comparable. // get MonetaryAmount from CurrencyUnit CurrencyUnit euro = MonetaryCurrencies.getCurrency("EUR"); MonetaryAmount fiveEuro = Money.of(5, euro); // get MonetaryAmount from currency code MonetaryAmount tenUsDollar = Money.of(10, "USD"); // FastMoney is an alternative MonetaryAmount factory that focuses on performance MonetaryAmount sevenEuro = FastMoney.of(7, euro); Money and FastMoney are two MonetaryAmount implementations of JavaMoney. Money is the default implementation that stores number values using BigDecimal. FastMoney is an alternative implementation which stores amounts in long fields. According to the documentation operations on FastMoney are 10-15 times faster compared to Money. However, FastMoney is limited by the size and precision of the long type. Please note that Money and FastMoney are implementation specific classes (located in org.javamoney.moneta.* instead of javax.money.*). If you want to avoid implementation specific classes, you have to obtain a MonetaryAmountFactory to create a MonetaryAmount instance: MonetaryAmount specAmount = MonetaryAmounts.getDefaultAmountFactory() .setNumber(123.45) .setCurrency("USD") .create(); Two MontetaryAmount instances are considered equal if the implementation classes, the currency units and the numeric values are equal: MonetaryAmount oneEuro = Money.of(1, MonetaryCurrencies.getCurrency("EUR")); boolean isEqual = oneEuro.equals(Money.of(1, "EUR")); // true boolean isEqualFast = oneEuro.equals(FastMoney.of(1, "EUR")); // false MonetaryAmount has various methods that allow accessing the assigned currency, the numeric amount, its precision and more: MonetaryAmount monetaryAmount = Money.of(123.45, euro); CurrencyUnit currency = monetaryAmount.getCurrency(); NumberValue numberValue = monetaryAmount.getNumber(); int intValue = numberValue.intValue(); // 123 double doubleValue = numberValue.doubleValue(); // 123.45 long fractionDenominator = numberValue.getAmountFractionDenominator(); // 100 long fractionNumerator = numberValue.getAmountFractionNumerator(); // 45 int precision = numberValue.getPrecision(); // 5 // NumberValue extends java.lang.Number. // So we assign numberValue to a variable of type Number Number number = numberValue; Working with MonetaryAmounts Mathematical operations can be performed with MonetaryAmount: MonetaryAmount twelveEuro = fiveEuro.add(sevenEuro); // "EUR 12" MonetaryAmount twoEuro = sevenEuro.subtract(fiveEuro); // "EUR 2" MonetaryAmount sevenPointFiveEuro = fiveEuro.multiply(1.5); // "EUR 7.5" // MonetaryAmount can have a negative NumberValue MonetaryAmount minusTwoEuro = fiveEuro.subtract(sevenEuro); // "EUR -2" // some useful utility methods boolean greaterThan = sevenEuro.isGreaterThan(fiveEuro); // true boolean positive = sevenEuro.isPositive(); // true boolean zero = sevenEuro.isZero(); // false // Note that MonetaryAmounts need to have the same CurrencyUnit to do mathematical operations // this fails with: javax.money.MonetaryException: Currency mismatch: EUR/USD fiveEuro.add(tenUsDollar); Rounding is another important part when working with money. MonetaryAmounts can be rounded using a rounding operator: CurrencyUnit usd = MonetaryCurrencies.getCurrency("USD"); MonetaryAmount dollars = Money.of(12.34567, usd); MonetaryOperator roundingOperator = MonetaryRoundings.getRounding(usd); MonetaryAmount roundedDollars = dollars.with(roundingOperator); // USD 12.35 Here 12.3456 US Dollars are rounded with the default rounding for this currency. When working with collections of MonetaryAmounts, some nice utility methods for filtering, sorting and grouping are available. These methods can be used together with the Java 8 Stream API. Consider the following collection: List amounts = new ArrayList<>(); amounts.add(Money.of(2, "EUR")); amounts.add(Money.of(42, "USD")); amounts.add(Money.of(7, "USD")); amounts.add(Money.of(13.37, "JPY")); amounts.add(Money.of(18, "USD")); We can now filter amounts by CurrencyUnit: CurrencyUnit yen = MonetaryCurrencies.getCurrency("JPY"); CurrencyUnit dollar = MonetaryCurrencies.getCurrency("USD"); // filter by currency, get only dollars // result is [USD 18, USD 7, USD 42] List onlyDollar = amounts.stream() .filter(MonetaryFunctions.isCurrency(dollar)) .collect(Collectors.toList()); // filter by currency, get only dollars and yen // [USD 18, USD 7, JPY 13.37, USD 42] List onlyDollarAndYen = amounts.stream() .filter(MonetaryFunctions.isCurrency(dollar, yen)) .collect(Collectors.toList()); We can also filter out MonetaryAmounts smaller or greater than a specific threshold: MonetaryAmount tenDollar = Money.of(10, dollar); // [USD 42, USD 18] List greaterThanTenDollar = amounts.stream() .filter(MonetaryFunctions.isCurrency(dollar)) .filter(MonetaryFunctions.isGreaterThan(tenDollar)) .collect(Collectors.toList()); Sorting works in a similar way: // Sorting dollar values by number value // [USD 7, USD 18, USD 42] List sortedByAmount = onlyDollar.stream() .sorted(MonetaryFunctions.sortNumber()) .collect(Collectors.toList()); // Sorting by CurrencyUnit // [EUR 2, JPY 13.37, USD 42, USD 7, USD 18] List sortedByCurrencyUnit = amounts.stream() .sorted(MonetaryFunctions.sortCurrencyUnit()) .collect(Collectors.toList()); Grouping functions: // Grouping by CurrencyUnit // {USD=[USD 42, USD 7, USD 18], EUR=[EUR 2], JPY=[JPY 13.37]} Map> groupedByCurrency = amounts.stream() .collect(MonetaryFunctions.groupByCurrencyUnit()); // Grouping by summarizing MonetaryAmounts Map summary = amounts.stream() .collect(MonetaryFunctions.groupBySummarizingMonetary()).get(); // get summary for CurrencyUnit USD MonetarySummaryStatistics dollarSummary = summary.get(dollar); MonetaryAmount average = dollarSummary.getAverage(); // "USD 22.333333333333333333.." MonetaryAmount min = dollarSummary.getMin(); // "USD 7" MonetaryAmount max = dollarSummary.getMax(); // "USD 42" MonetaryAmount sum = dollarSummary.getSum(); // "USD 67" long count = dollarSummary.getCount(); // 3 MonetaryFunctions also provides reduction function that can be used to obtain the max, min and sum of a MonetaryAmount collection: List amounts = new ArrayList<>(); amounts.add(Money.of(10, "EUR")); amounts.add(Money.of(7.5, "EUR")); amounts.add(Money.of(12, "EUR")); Optional max = amounts.stream().reduce(MonetaryFunctions.max()); // "EUR 7.5" Optional min = amounts.stream().reduce(MonetaryFunctions.min()); // "EUR 12" Optional sum = amounts.stream().reduce(MonetaryFunctions.sum()); // "EUR 29.5" Custom MonetaryAmount operations MonetaryAmount provides a nice extension point called MonetaryOperator. MonetaryOperator is a functional interface that takes a MonetaryAmount as input and creates a new MonetaryAmount based on the input. // A monetary operator that returns 10% of the input MonetaryAmount // Implemented using Java 8 Lambdas MonetaryOperator tenPercentOperator = (MonetaryAmount amount) -> { BigDecimal baseAmount = amount.getNumber().numberValue(BigDecimal.class); BigDecimal tenPercent = baseAmount.multiply(new BigDecimal("0.1")); return Money.of(tenPercent, amount.getCurrency()); }; MonetaryAmount dollars = Money.of(12.34567, "USD"); // apply tenPercentOperator to MonetaryAmount MonetaryAmount tenPercentDollars = dollars.with(tenPercentOperator); // USD 1.234567 Some standard API features are implemented as MonetaryOperator. For example, the rounding features we saw above are implemented as MonetaryOperator. Exchange rates Currency exchange rates can be obtained using an ExchangeRateProvider. JavaMoney comes with multiple different ExchangeRateProvider implementations. The two most important implementations are ECBCurrentRateProvider and IMFRateProvider. ECBCurrentRateProvider queries the European Central Bank (ECB) data feed for getting current exchange rates while IMFRateProvider uses International Monetary Fund (IMF) conversion rates. // get the default ExchangeRateProvider (CompoundRateProvider) ExchangeRateProvider exchangeRateProvider = MonetaryConversions.getExchangeRateProvider(); // get the names of the default provider chain // [IDENT, ECB, IMF, ECB-HIST] List defaultProviderChain = MonetaryConversions.getDefaultProviderChain(); // get a specific ExchangeRateProvider (here ECB) ExchangeRateProvider ecbExchangeRateProvider = MonetaryConversions.getExchangeRateProvider("ECB"); If no specific ExchangeRateProvider is requested a CompoundRateProvider will be returned. CompoundRateProvider delegates exchange rate requests to a chain of ExchangeRateProviders and returns the result from the first provider that returns an adequate result. // get the exchange rate from euro to us dollar ExchangeRate rate = exchangeRateProvider.getExchangeRate("EUR", "USD"); NumberValue factor = rate.getFactor(); // 1.2537 (at time writing) CurrencyUnit baseCurrency = rate.getBaseCurrency(); // EUR CurrencyUnit targetCurrency = rate.getCurrency(); // USD Currency conversion Conversion between currencies is be done with CurrencyConversions that can be obtained from ExchangeRateProviders: // get the CurrencyConversion from the default provider chain CurrencyConversion dollarConversion = MonetaryConversions.getConversion("USD"); // get the CurrencyConversion from a specific provider CurrencyConversion ecbDollarConversion = ecbExchangeRateProvider.getCurrencyConversion("USD"); MonetaryAmount tenEuro = Money.of(10, "EUR"); // convert 10 euro to us dollar MonetaryAmount inDollar = tenEuro.with(dollarConversion); "USD 12.537" (at the time writing) Note that CurrencyConversion implements MonetaryOperator. Like other operators it can be applied using MonetaryAmount.with(). Formatting and parsing MonetaryAmounts can be parsed/formatted from/to string using a MonetaryAmountFormat: // formatting by locale specific formats MonetaryAmountFormat germanFormat = MonetaryFormats.getAmountFormat(Locale.GERMANY); MonetaryAmountFormat usFormat = MonetaryFormats.getAmountFormat(Locale.CANADA); MonetaryAmount amount = Money.of(12345.67, "USD"); String usFormatted = usFormat.format(amount); // "USD12,345.67" String germanFormatted = germanFormat.format(amount); // 12.345,67 USD // A MonetaryAmountFormat can also be used to parse MonetaryAmounts from strings MonetaryAmount parsed = germanFormat.parse("12,4 USD"); With AmountFormatQueryBuilder custom formats can be created: // Creating a custom MonetaryAmountFormat MonetaryAmountFormat customFormat = MonetaryFormats.getAmountFormat( AmountFormatQueryBuilder.of(Locale.US) .set(CurrencyStyle.NAME) .set("pattern", "00,00,00,00.00 ¤") .build()); // results in "00,01,23,45.67 US Dollar" String formatted = customFormat.format(amount); Note that the ¤ symbol (\u00A) is used as currency placeholder inside the pattern string. Summary We looked at many parts of the new Money and Currency API. The implementation already looks quite solid (but definitely needs some more documentation). I am looking forward to see this API in Java 9 :-) You can find all the examples shown here on GitHub.

Learn all about creating a microservices architecture on Java in this great tutorial.

In Eclipse and CDT, I need to tell the compiler where it has to search for the header files. The normal way is to go to the compiler settings (menu Project > Properties > C/C++ Build > Settings) and then add the include paths, one by one, using the ‘+’ icon: Adding Include Path (shown using the GNU ARM Eclipse plugin) But for many include paths, this is a time-consuming process. But there is another way. Copy-Paste The trick is that this ‘list of items’ control in the compiler settings work with copy and past shortcuts (CTRL-C and CTRL-V on Windows). This is especially useful if I have an existing project with all the paths setup: I can select (use CTRL to select multiple items individually, or CTRL-A (for all, on Windows) to select all items in the list: Selected all Items Then press the host operating system shortcut for copy (CTRL-C on Windows), go to your destination panel and use the paste (CTRL-V on Windows) shortcut, and all the paths get copied. This approach works for all ‘list’ setting items, e.g. linker library settings. :idea: Unfortunately Eclipse does not show a context menu for the copy/paste operation in the settings panel. But you can use copy-paste for pretty much every setting, as long as you copy normal text. Another trick is to use a clipboard viewer or a text file. As the format used is simple text list of items, it is possible to create a file or edit the items in a text editor: clipboard items in text editor That way I can use a script or anything I want to create that list of items, then copy-paste it into the settings. Summary Copy-Paste is a fast way to apply large sets of path (or list of items). That way I can easily copy settings from an existing project. Or I can create a list of folders in text file and then apply them all in one step. That way I can easily assign multiple items in a single step. Happy Including :-)