Idempotency in HTTP means that the same request can be performed multiple times with the same effect as if it was executed just once. If you replace current state of some resource with new one, no matter how many times you do so, in the end state will be the same as if you did it just once. To give more concrete example: deleting a user is idempotent because no matter how many times you delete given user by unique identifier, in the end this user will be deleted. On the other hand creating new user is not idempotent because requesting such operation twice will create two users. In HTTP terms here is what RFC 2616: 9.1.2 Idempotent Methods has to say: 9.1.2 Idempotent Methods Methods can also have the property of "idempotence" in that [...] the side-effects of N > 0 identical requests is the same as for a single request. The methods GET, HEAD, PUT and DELETE share this property. Also, the methods OPTIONS and TRACE SHOULD NOT have side effects, and so are inherently idempotent. Temporal coupling is an undesirable property of a system where the correct behaviour is implicitly dependent on time dimension. In plain English, it might mean that for example system only works when all components are present at the same time. Blocking request-response communication (ReST, SOAP or any other form of RPC) require both client and server to be available at the same time, which is an example of this effect. Having basic understanding what these concepts mean, let's go through a simple case study - massively multiplayer online role-playing game. Our artificial use case is as follows: a player sends premium-rated SMS to purchase virtual sword inside game. Our HTTP gateway is called when SMS is delivered and we need to inform InventoryService, deployed on a different machine. Current API involves ReST and looks as follows: @Slf4j @RestController class SmsController { private final RestOperations restOperations; @Autowired public SmsController(RestOperations restOperations) { this.restOperations = restOperations; } @RequestMapping(value = "/sms/{phoneNumber}", method = POST) public void handleSms(@PathVariable String phoneNumber) { Optional maybePlayer = phoneNumberToPlayer(phoneNumber); maybePlayer .map(Player::getId) .map(this::purchaseSword) .orElseThrow(() -> new IllegalArgumentException("Unknown player for phone number " + phoneNumber)); } private long purchaseSword(long playerId) { Sword sword = new Sword(); HttpEntity entity = new HttpEntity<>(sword.toJson(), jsonHeaders()); restOperations.postForObject( "http://inventory:8080/player/{playerId}/inventory", entity, Object.class, playerId); return playerId; } private HttpHeaders jsonHeaders() { HttpHeaders headers = new HttpHeaders(); headers.setContentType(MediaType.APPLICATION_JSON); return headers; } private Optional phoneNumberToPlayer(String phoneNumber) { //... } } Which in turns generates request similar to this: > POST /player/123123/inventory HTTP/1.1 > Host: inventory:8080 > Content-type: application/json > > {"type": "sword", "strength": 100, ...} < HTTP/1.1 201 Created < Content-Length: 75 < Content-Type: application/json;charset=UTF-8 < Location: http://inventory:8080/player/123123/inventory/1 This is fairly straightforward. SmsController simply forwards appropriate data to inventory:8080 service by POSTing sword that was purchased. This service, immediately or after a while, returns 201 Created HTTP response confirming the operation was successful. Additionally link to resource is created and returned, so you can query it. One might say: ReST state of the art. However if you care at least a little about money of your customers and understand what ACID is (something that Bitcoin exchanges still have to learn: see [1], [2], [3] and [4]) - this API is too fragile and prone to errors. Imagine all these types of errors: your request never reached inventory server your request reached server but it refused it server accepted connection but failed to read request server read request but hanged server processed request but failed to send response server sent 200 OK response but it was lost and you never received it server's response was received but client failed to process it server's response was sent but client timed-out earlier In all these cases you simply get an exception on the client side and you have no idea what's the server's state. Technically you should retry failed requests, but since POST is not idempotent, you might end up rewarding gamer with more than one sword (in cases 5-8). But without retry you might loose gamer's money without giving him his precious artifact. There must be a better way. Turning POST to idempotent PUT In some cases it's surprisingly simple to convert from POST to idempotent PUT by basically moving ID generation from server to client. With POST it was the server that generated sword's ID and sent it back to the client in Location header. Turns out eagerly generating UUID on the client side and changing the semantics a bit plus enforcing some constraints on the server side is enough: private long purchaseSword(long playerId) { Sword sword = new Sword(); UUID uuid = sword.getUuid(); HttpEntity entity = new HttpEntity<>(sword.toJson(), jsonHeaders()); asyncRetryExecutor .withMaxRetries(10) .withExponentialBackoff(100, 2.0) .doWithRetry(ctx -> restOperations.put( "http://inventory:8080/player/{playerId}/inventory/{uuid}", entity, playerId, uuid)); return playerId; } The API looks as follows: > PUT /player/123123/inventory/45e74f80-b2fb-11e4-ab27-0800200c9a66 HTTP/1.1 > Host: inventory:8080 > Content-type: application/json;charset=UTF-8 > > {"type": "sword", "strength": 100, ...} < HTTP/1.1 201 Created < Content-Length: 75 < Content-Type: application/json;charset=UTF-8 < Location: http://inventory:8080/player/123123/inventory/45e74f80-b2fb-11e4-ab27-0800200c9a66 Why it's such a big deal? Simply put (no pun intended) client can now retry PUT request as many times as he wants. When server receives PUT for the first time, it persists sword in the database with client-generated UUID (45e74f80-b2fb-11e4-ab27-0800200c9a66) as primary key. In case of second PUT attempt we can either update or reject such request. It wasn't possible with POST because every request was treated as a new sword purchase - now we can track whether such PUT came before or not. We just have to remember to subsequent PUT is not a bug, it's an update request: @RestController @Slf4j public class InventoryController { private final PlayerRepository playerRepository; @Autowired public InventoryController(PlayerRepository playerRepository) { this.playerRepository = playerRepository; } @RequestMapping(value = "/player/{playerId}/inventory/{invId}", method = PUT) @Transactional public void addSword(@PathVariable UUID playerId, @PathVariable UUID invId) { playerRepository.findOne(playerId).addSwordWithId(invId); } } interface PlayerRepository extends JpaRepository {} @lombok.Data @lombok.AllArgsConstructor @lombok.NoArgsConstructor @Entity class Sword { @Id @Convert(converter = UuidConverter.class) UUID id; int strength; @Override public boolean equals(Object o) { if (this == o) return true; if (!(o instanceof Sword)) return false; Sword sword = (Sword) o; return id.equals(sword.id); } @Override public int hashCode() { return id.hashCode(); } } @Data @Entity class Player { @Id @Convert(converter = UuidConverter.class) UUID id = UUID.randomUUID(); @OneToMany(cascade = ALL, fetch = EAGER) @JoinColumn(name="player_id") Set swords = new HashSet<>(); public Player addSwordWithId(UUID id) { swords.add(new Sword(id, 100)); return this; } } Few shortcuts were made in code snippet above, like injecting repository directly to controller, as well as annotating is with @Transactional. But you get the idea. Also notice that this code is quite optimistic, assuming two swords with same UUID aren't inserted at exactly the same time. Otherwise constraint violation exception will occur. Side note 1: I use UUID type in both controller and JPA models. They aren't supported out of the box, for JPA you need custom converter: public class UuidConverter implements AttributeConverter { @Override public String convertToDatabaseColumn(UUID attribute) { return attribute.toString(); } @Override public UUID convertToEntityAttribute(String dbData) { return UUID.fromString(dbData); } } Similarly for Spring MVC (one-way only): @Bean GenericConverter uuidConverter() { return new GenericConverter() { @Override public Set getConvertibleTypes() { return Collections.singleton(new ConvertiblePair(String.class, UUID.class)); } @Override public Object convert(Object source, TypeDescriptor sourceType, TypeDescriptor targetType) { return UUID.fromString(source.toString()); } }; } Side note 2: if you can't change client, you can track duplicates by storing each requests' hash on the server side. This way when the same request is sent multiple times (retried by the client), it will be ignored. However sometimes we might have a legitimate use case for sending the exact same request twice (e.g. purchasing two swords within short period of time). Temporal coupling - client unavailability You think you're smart but PUT with retries is not enough. First of all a client can die while re-attempting failed requests. If server is severely damaged or down, retrying might take minutes or even hours. You can't simply block your incoming HTTP request just because one of your downstream dependencies is down - you must handle such requests asynchronously in background - if possible. But extending retry time increases probability of client dying or being restarted, which would loose our request. Imagine we received premium SMS but InventoryService is down at the moment. We can retry after second, two, four, etc., but what if InventoryService was down for couple of hours and it so happened that our service was restarted as well? We just lost that SMS and sword was never given to the gamer. An answer to such issue is to persist pending request first and handle it later in background. Upon SMS receive we barely store player ID in database table called pending_purchases. A background scheduler or an event wakes up asynchronous thread that will collect all pending purchases and try to send them to InventoryService (maybe even in batch?) Periodic batch threads running every minute or even second and collecting all pending requests will unavoidably introduce latency and unneeded database traffic. Thus I'm going for a Quartz scheduler instead that will schedule retry job for each pending request: @Slf4j @RestController class SmsController { private Scheduler scheduler; @Autowired public SmsController(Scheduler scheduler) { this.scheduler = scheduler; } @RequestMapping(value = "/sms/{phoneNumber}", method = POST) public void handleSms(@PathVariable String phoneNumber) { phoneNumberToPlayer(phoneNumber) .map(Player::getId) .map(this::purchaseSword) .orElseThrow(() -> new IllegalArgumentException("Unknown player for phone number " + phoneNumber)); } private UUID purchaseSword(UUID playerId) { UUID swordId = UUID.randomUUID(); InventoryAddJob.scheduleOn(scheduler, Duration.ZERO, playerId, swordId); return swordId; } //... } And job itself: @Slf4j public class InventoryAddJob implements Job { @Autowired private RestOperations restOperations; @lombok.Setter private UUID invId; @lombok.Setter private UUID playerId; @Override public void execute(JobExecutionContext context) throws JobExecutionException { try { tryPurchase(); } catch (Exception e) { Duration delay = Duration.ofSeconds(5); log.error("Can't add to inventory, will retry in {}", delay, e); scheduleOn(context.getScheduler(), delay, playerId, invId); } } private void tryPurchase() { restOperations.put(/*...*/); } public static void scheduleOn(Scheduler scheduler, Duration delay, UUID playerId, UUID invId) { try { JobDetail job = newJob() .ofType(InventoryAddJob.class) .usingJobData("playerId", playerId.toString()) .usingJobData("invId", invId.toString()) .build(); Date runTimestamp = Date.from(Instant.now().plus(delay)); Trigger trigger = newTrigger().startAt(runTimestamp).build(); scheduler.scheduleJob(job, trigger); } catch (SchedulerException e) { throw new RuntimeException(e); } } } Every time we receive premium SMS we schedule asynchronous job to be executed immediately. Quartz will take care of persistence (if application goes down, job will be executed as soon as possible after restart). Moreover if this particular instance goes down, another one can pick up this job - or we can form a cluster and load-balance requests between them: one instance receives SMS, another one requests sword in InventoryService. Obviously if HTTP call fails, retry is re-scheduled later, everything is transactional and fail-safe. In real code you would probably add max retry limit as well as exponential delay, but you get the idea. Temporal coupling - client and server can't meet Our struggle to implement retries correctly is a sign of obscure temporal coupling between client and server - they must live together at the same time. Technically this isn't necessary. Imagine gamer sending an e-mail with order to customer service which they handle within 48 hours, changing his inventory manually. The same can be applied to our case, but replacing e-mail server with some sort of message broker, e.g. JMS: @Bean ActiveMQConnectionFactory activeMQConnectionFactory() { return new ActiveMQConnectionFactory("tcp://localhost:61616"); } @Bean JmsTemplate jmsTemplate(ConnectionFactory connectionFactory) { return new JmsTemplate(connectionFactory); } Having ActiveMQ connection set up we can simply send purchase request to broker: private UUID purchaseSword(UUID playerId) { final Sword sword = new Sword(playerId); jmsTemplate.send("purchases", session -> { TextMessage textMessage = session.createTextMessage(); textMessage.setText(sword.toJson()); return textMessage; }); return sword.getUuid(); } By entirely replacing synchronous request-response protocol with messaging over JMS topic we temporally decouple client from server. They no longer need to live at the same time. Moreover more than one producer and consumer can interact with each other. E.g. you can have multiple purchase channels and more importantly: multiple interested parties, not only InventoryService. Even better, if you use specialized messaging system like Kafka you can technically keep days (months?) worth of messages without loosing performance. The benefit is that if you add another consumer of purchase events to the system next to InventoryService it will receive lots of historical data immediately. Moreover now your application is temporally coupled with broker so since Kafka is distributed and replicated, it works better in that case. Disadvantages of asynchronous messaging Synchronous data exchange, as used in ReST, SOAP or any form of RPC is easy to understand and implement. Who cares this abstraction insanely leaks from latency perspective (local method call is typically orders of magnitude faster compared to remote, not to mention it can fail for numerous reasons unknown locally), it's quick to develop. One true caveat of messaging is feedback channel. You can longer just "send" ("return") message back, as there is no response pipe. You either need response queue with some correlation ID or temporary one-off response queues per request. Also we lied a little bit claiming that putting a message broker between two systems fixes temporal coupling. It does, but now we are coupled to messaging bus - which can just as well go down, especially since it's often under high load and sometimes not replicated properly. This article shows some challenges and partial solutions to provide guarantees in distributed systems. But in the end of the day, remember that "exactly once" semantics are nearly impossible to implement easily, so double check you really need them.

In the traditional 3-tier architecture, data processing is performed by the application server where the data itself is stored in the database server. Application server and database server are typically two different machine. Therefore, the processing cycle proceeds as follows Application server send a query to the database server to retrieve the necessary data Application server perform processing on the received data Application server will save the changed data to the database server In the traditional data processing paradigm, we move data to the code. It can be depicted as follows ... Then big data phenomenon arrives. Because the data volume is huge, it cannot be hold by a single database server. Big data is typically partitioned and stored across many physical DB server machines. On the other hand, application servers need to be added to increase the processing power of big data. However, as we increase the number of App servers and DB servers for storing and processing the big data, more data need to be transfer back and forth across the network during the processing cycle, up to a point where the network becomes a major bottleneck. Moving Code to Data To overcome the network bottleneck, we need a new computing paradigm. Instead of moving data to the code, we move the code to the data and perform the processing at where the data is stored. Notice the change of the program structure The program execution starts at a driver, which orchestrate the execution happening remotely across many worker servers within a cluster. Data is no longer transferred to the driver program, the driver program holds a data reference in its variable rather than the data itself. The data reference is basically an id to locate the corresponding data residing in the database server Code is shipped from the program to the database server, where the execution is happening, and data is modified at the database server without leaving the server machine. Finally the program request a save of the modified data. Since the modified data resides in the database server, no data transfer happens over the network. By moving the code to the data, the volume of data transfer over network is significantly reduced. This is an important paradigm shift for big data processing. In the following session, I will use Apache Spark to illustrate how this big data processing paradigm is implemented. RDD Resilient Distributed Dataset (RDD) is how Spark implements the data reference concept. RDD is a logical reference of a dataset which is partitioned across many server machines in the cluster. To make a clear distinction between data reference and data itself, a Spark program is organized as a sequence of execution steps, which can either be a "transformation" or an "action". Programming Model A typical program is organized as follows From an environment variable "context", create some initial data reference RDD objects Transform initial RDD objects to create more RDD objects. Transformation is expressed in terms of functional programming where a code block is shipped from the driver program to multiple remote worker server, which hold a partition of the RDD. Variable appears inside the code block can either be an item of the RDD or a local variable inside the driver program which get serialized over to the worker machine. After the code (and the copy of the serialized variables) is received by the remote worker server, it will be executed there by feeding the items of RDD residing in that partition. Notice that the result of a transformation is a brand new RDD (the original RDD is not mutated) Finally, the RDD object (the data reference) will need to be materialized. This is achieved through an "action", which will dump the RDD into a storage, or return its value data to the driver program. Here is a word count example # Get initial RDD from the context file = spark.textFile("hdfs://...") # Three consecutive transformation of the RDD counts = file.flatMap(lambda line: line.split(" ")) .map(lambda word: (word, 1)) .reduceByKey(lambda a, b: a + b) # Materialize the RDD using an action counts.saveAsTextFile("hdfs://...") When the driver program starts its execution, it builds up a graph where nodes are RDD and edges are transformation steps. However, no execution is happening at the cluster until an action is encountered. At that point, the driver program will ship the execution graph as well as the code block to the cluster, where every worker server will get a copy. The execution graph is a DAG. Each DAG is a atomic unit of execution. Each source node (no incoming edge) is an external data source or driver memory Each intermediate node is a RDD Each sink node (no outgoing edge) is an external data source or driver memory Green edge connecting to RDD represents a transformation. Red edge connecting to a sink node represents an action Data Shuffling Although we ship the code to worker server where the data processing happens, data movement cannot be completely eliminated. For example, if the processing requires data residing in different partitions to be grouped first, then we need to shuffle data among worker server. Spark carefully distinguish "transformation" operation in two types. "Narrow transformation" refers to the processing where the processing logic depends only on data that is already residing in the partition and data shuffling is unnecessary. Examples of narrow transformation includes filter(), sample(), map(), flatMap() .... etc. "Wide transformation" refers to the processing where the processing logic depends on data residing in multiple partitions and therefore data shuffling is needed to bring them together in one place. Example of wide transformation includes groupByKey(), reduceByKey() ... etc. Joining two RDD can also affect the amount of data being shuffled. Spark provides two ways to join data. In a shuffle join implementation, data of two RDD with the same key will be redistributed to the same partition. In other words, each of the items in each RDD will be shuffled across worker servers. Beside shuffle join, Spark provides another alternative call broadcast join. In this case, one of the RDD will be broadcasted and copied over to every partition. Imagine the situation when one of the RDD is significantly smaller relative to the other, then broadcast join will reduce the network traffic because only the small RDD need to be copied to all worker servers while the large RDD doesn't need to be shuffled at all. In some cases, transformation can be re-ordered to reduce the amount of data shuffling. Below is an example of a JOIN between two huge RDDs followed by a filtering. Plan1 is a naive implementation which follows the given order. It first join the two huge RDD and then apply the filter on the join result. This ends up causing a big data shuffling because the two RDD is huge, even though the result after filtering is small. Plan2 offers a smarter way by using the "push-down-predicate" technique where we first apply the filtering in both RDDs before joining them. Since the filtering will reduce the number of items of each RDD significantly, the join processing will be much cheaper. Execution Planning As explain above, data shuffling incur the most significant cost in the overall data processing flow. Spark provides a mechanism that generate an execute plan from the DAG that minimize the amount of data shuffling. Analyze the DAG to determine the order of transformation. Notice that we starts from the action (terminal node) and trace back to all dependent RDDs. To minimize data shuffling, we group the narrow transformation together in a "stage" where all transformation tasks can be performed within the partition and no data shuffling is needed. The transformations becomes tasks that are chained together within a stage Wide transformation sits at the boundary of two stages, which requires data to be shuffled to a different worker server. When a stage finishes its execution, it persist the data into different files (one per partition) of the local disks. Worker nodes of the subsequent stage will come to pickup these files and this is where data shuffling happens Below is an example how the execution planning turns the DAG into an execution plan involving stages and tasks. Reliability and Fault Resiliency Since the DAG defines a deterministic transformation steps between different partitions of data within each RDD RDD, fault recovery is very straightforward. Whenever a worker server crashes during the execution of a stage, another worker server can simply re-execute the stage from the beginning by pulling the input data from its parent stage that has the output data stored in local files. In case the result of the parent stage is not accessible (e.g. the worker server lost the file), the parent stage need to be re-executed as well. Imagine this is a lineage of transformation steps, and any failure of a step will trigger a restart of execution from its last step. Since the DAG itself is an atomic unit of execution, all the RDD values will be forgotten after the DAG finishes its execution. Therefore, after the driver program finishes an action (which execute a DAG to its completion), all the RDD value will be forgotten and if the program access the RDD again in subsequent statement, the RDD needs to be recomputed again from its dependents. To reduce this repetitive processing, Spark provide a caching mechanism to remember RDDs in worker server memory (or local disk). Once the execution planner finds the RDD is already cache in memory, it will use the RDD right away without tracing back to its parent RDDs. This way, we prune the DAG once we reach an RDD that is in the cache. Overall speaking, Apache Spark provides a powerful framework for big data processing. By the caching mechanism that holds previous computation result in memory, Spark out-performs Hadoop significantly because it doesn't need to persist all the data into disk for each round of parallel processing. Although it is still very new, I think Spark will take off as the main stream approach to process big data.

in my book, unity in action , i mostly cover how to program games. still, it is important to understand how to work on and improve the visuals. visuals are important because you don't want your projects to end up with just blank boxes sliding around. it's fairly easy to create walls in a game--you just use the tiling property to place images of stone and brick. but things get a little more complex with creating a sky visual, one of the most common visuals you’ll need to create in a game. if you want a realistic look for the sky, the most common approach is to use a special kind of texturing using pictures of the sky. in this article, i'll explain how to do this using a skybox, first by using provided sample material and second, by creating from scratch. what is a skybox? by default, the camera’s background color is dark blue. ordinarily that color fills in any empty area of the view (e.g., above the walls of this scene) but it is possible to render pictures of the sky as background. this is where the concept of a “skybox” comes in: definition a skybox is a cube surrounding the camera with pictures of the sky on each side. no matter what direction the camera is facing, it is looking at a picture of the sky. properly implementing a skybox can be tricky; figure 1 shows a diagram of how a skybox works. there are a number of rendering tricks needed so that the skybox will appear as a distant background. fortunately unity already takes care of all that for you. figure 1 diagram of a skybox new scenes actually come with a very simple skybox already assigned. this is why the sky has a gradient from light to dark blue, rather than simply being a flat dark blue. if you open the lighting window (the menu window > lighting) then the very first setting is skybox and the slot for that setting says “default”. this setting is in the environment lighting panel; this window has a number of settings panels related to the advanced lighting system in unity, but for now we only care about the first setting. just like the brick textures earlier, skybox images can be obtained from a variety of websites. search for “skybox textures”; for example, i obtained several great skyboxes from www.93i.de, including their tropicalsunnyday set. once this skybox is applied to the scene, you should see something like figure 2. figure 2 scene with background pictures of the sky as with other textures, skybox images are first assigned to a material, and that gets used in the scene. let’s examine how to create a new skybox material. creating a new skybox material first off, create a new material (as usual either right-click and click create, or choose create from the assets menu) and select it to see settings in the inspector. next you need to change the shader used by this material. the top of the material settings has a menu for shader (see figure 3). definition a shader is a short program that outlines instructions for how to draw a surface, including whether to use any textures. the computer uses these instructions to calculate the pixels when rendering the image. the most common shader takes the color of the material and darkens it according to the light, but shaders can also be used for all sorts of visual effects. every material has a shader that controls it (indeed, you could kind of think of a material as an instance of a shader). new materials are set to the standard shader by default. this shader displays the color of the material (including the texture) while applying basic dark and light across the surface. well, for skyboxes there’s a different shader to use. click the menu in order to see the drop- down list (see figure 3) of all the available shaders. move down to the skybox section and choose 6-sided in that submenu. figure 3 the drop-down menu of available shaders with this shader active, the material now has six large texture slots (instead of just the small albedo texture slot that the default shader had). these six texture slots correspond to the six sides of a cube, so these images should match up at the edges in order to appear seamless. for example, figure 4 shows the images for the sunny skybox. figure 4 six sides of a skybox—images for top, bottom, front, back, left, and right import the skybox images into unity the same way you brought in the brick textures: drag the files into the project view, or right-click in project and select import new asset. there’s just one subtle import setting to change: click the imported texture to see its properties in the inspector, and change the wrap mode setting (shown in figure 5) from repeat to clamp (don’t forget to click apply when you’re done). ordinarily textures can be tiled repeatedly over a surface; for this to appear seamless, opposite edges of the image actually bleed together. however, this blending of edges can create faint lines in the sky where images meet, so the clamp setting will limit the boundaries of the texture and get rid of this blending. figure 5 correct faint edge lines by adjusting the wrap mode now you can drag these images to the texture slots of the skybox material. the names of the images correspond to the texture slot to assign them to (such as left or front). once all six textures are linked up, you can use this new material as the skybox for the scene. open the lighting window again, and set this new material to the skybox slot; either drag the material to that slot, or click the tiny circle icon to bring up a file picker. tip by default, unity will display the skybox (or at least its main color) in the editor’s scene view. you may find this color distracting while editing objects, so you can toggle the skybox on or off. across the top of the scene view’s pane are buttons that control what’s visible; look for the effects button to toggle the skybox on or off. woo-hoo, you’ve learned how to create sky visuals for your scene! a skybox is an elegant way to create the illusion of a vast atmosphere surrounding the player. the next step in polishing the visuals in your level is to create more complex 3d models.

What does implementing microservices mean for a software developer? Especially, for the rookies, greenhorns, and newbs out there? I’m not talking about microservice software architecture here; this is about microservices software development. And not just that, the ultimate implementation goal should be “microservices done right”. For this post, I’ll go with Java. Yes, it’s wordy. Yes, it’s resource intensive (especially when used for the sole purpose of returning a single string). However the concept of classes and objects goes well with my intention of explaining how to do microservices correctly. Plus, it makes sense to use microservices in environments that are heavily biased towards Java. Anyway, please feel free to add your own “Hello, World!” microservice in your favorite language in the comments section below. Hello, monolith! As a prerequisite, you should be familiar with the following piece of code, what it does, and why it has to look the way it does (read this tutorial if you don’t): class Starter { public static void main(String[] args) { System.out.println(“Hello, World!”); } } This is a simple console application that yields the string “Hello, World!” This is not written in the microservice way. This is an example of when not to use the microservices approach: if all you need on your console is a single string, this is all you need. Hello, code duplication! In addition to this console application, I want this string to be available on the web by calling http://localhost:80/helloWorld.servlet from a browser. Here is the required code, implemented as plain HTTP servlet (yes, it’s wordy. Get over it.) class HelloWorldServlet extends HttpServlet { public void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { response.getWriter().println(“Hello, World!”); } } The string “Hello, World!” has to be “implemented” again. Sure, this is no big deal. But this simple string could be so much more. It could be the result of a complex calculation or it could be the result of a time consuming search query. So, just imagine that the string “Hello, World!” is the result of a week’s worth of hard work (If you’re new to programming, it may very well be...). How should you go about making it available to apps and services that you create? Step 1: HelloWorldService.java To save yourself from duplicating a week’s worth of coding, allow me to introduce the HelloWorldService class: class HelloWorldService { public String greet() { return “Hello, World!”; } } You can re-use this fine piece of software craftmanship in all your apps and classes without re-implementing or duplicating code. Here’s our console application again: class Starter { HelloWorldService helloWorldService = new HelloWorldService(); public static void main(String[] args) { String message = helloWorldService.greet(); System.out.println(message); } } The same goes for servlets: class HelloWorldServlet extends HttpServlet { HelloWorldService helloWorldService = new HelloWorldService(); public void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { String message = helloWorldService.greet(); response.getWriter().println(message); } } It also works great for Spring MVC controllers: @Controller class HelloWorldController { HelloWorldService helloWorldService = new HelloWorldService(); @RequestMapping("/helloWorld") public String greet() { String message = helloWorldService.greet(); return message; } } I could go on and show more examples, but I think you get the point (spoiler: it’s the bold lines that matter). Those of you who are familiar with microservices could point out that this may be fine for getting rid of code duplication, but this is no microservice. You’re right, but to get to “microservices done right,” you have to be able to separate you app’s concerns, which is what I did here in the most possible basic way: I separated the app’s frontend concerns from its backend concerns. The frontend is either a console app or a servlet, the backend is HelloWorldService. Serviceward, ho! To go down microservice lane from here, all we have to do is wrap HelloWorldService into some kind of web component that makes it accessible via HTTP, right? Let’s see… First, we could just use our servlet code from above, as it conveniently returns the string as a response to any HTTP request. But we won’t. Why? Because there’s something missing: fault tolerance. What could possibly fail when returning a simple string? That’s not the point. What matters is that the client side (the code that calls HelloWorldService) should be given enough information to effectively react to failures. We face two possible problems: The service as a whole may be unavailable The service may be unable to return a proper response The service is unavailable If a service is unavailable, it’s the client that is responsible for dealing with the situation. Frameworks like unirest.io save you the effort of writing many lines of code when dealing with HTTP requests. Future> future = Unirest.post("HTTP://helloworld.myservices.local/greet") .header("accept", "application/json") .asJsonAsync(new Callback() { public void failed(UnirestException e) { //tell them UI folks that the request went south } public void completed(HttpResponse response) { //extract data from response and fulfill it’s destiny } public void cancelled() { //shot a note to UI dept that the request got cancelled } } ); With this code, the client now knows when the service is not available or has timed out following no response. Wee can easily have an error message displayed in place of the string we expected to receive. Try/catch is probably the right solution here. Invalid responses however pose more of a challenge. The service fails If the service fails, we can just return a string with an appropriate error message. But how can you know if a message is an error message or a correct response? Yes, you can start every error message with [ERROR] or invent another “smart” (read: not-so-smart) workaround, but this won’t be a solution you’ll be proud of. And, there’s always the possibility that even valid responses may begin with ERROR because it’s simply part of the message. I’d go with JSON or XML for wrapping the answer. I prefer JSON because it’s a little less wordy than XML. And I really like using the JSON-HTML tool over at json.bloople.net for visualizing results during development. Of course, you might go for any of the numerous alternatives, like protobuf or a proprietary solution of your own. The main point is that you need to be able to apply structure to responses: { “status”:”ok”, ”message”:”Hello, World!” } By checking the status attribute, you can easily decide whether to handle an error or to display an appropriate message. { “status”:”error”, ”message”:”Invalid input parameter” } The possibilities are endless here. You can add an error code or additional properties. This all boils down to a single important point: apply structure to your responses. Structure, why? Because structure not only helps you keep your code maintainable, it also serves as the foundation of the API of your service. An API definition consists of more than a URL like this: GET HTTP://helloworld.myservices.local/greet API definitions also consist of the response structures that can be expected as a response (you know this already from a few lines back): { “status”:”ok”, ”message”:”Hello, World!” } Most important takeaway Keeping the API specifications of a service’s request and response stable is a key requirement for succeeding with microservices. Conclusion Are you (and your project) ready for microservices? If you read this and kept asking yourself, what good is all the overhead of microservices, then either your project won’t benefit from microservices or you’re just not there yet (for mindset perspective see my previous post about the value of microservices ). If you can’t stop thinking about microservices, then you probably are ready.

RESTEasy (and Jersey as well) contain a minimal web server within their libraries which enables their users to start up a tiny web server.

In this post, let’s look at the HTTP request to resource method matching in JAX-RS. It is one of the most fundamental features of JAX-RS. Generally, the developers using the JAX-RS API are not exposed to (or do not really need to know) the nitty gritty of the matching process, rest assured that the JAX-RS runtime churns out its algorithms quietly in the background as our RESTful clients keep those HTTP requests coming! Just in case the term request to resource method matching is new to you – it’s nothing but the process via which the JAX-RS provider dispatches a HTTP request to a particular method of your one of your resource classes (decorated with @Path). Hats off to the JAX-RS spec doc for explaining this in great detail (we’ll just cover the tip of the iceberg in this post though!) Primary criteria What are the factors taken into consideration during the request matching process ? HTTP request URI HTTP request method (GET, PUT, POST, DELETE etc) Media type of the HTTP request Media type of requested response High level steps A rough diagram should help. Before we look at that, here is the example scenario Two resource classes – Books.java, Movies.java Resource methods paths in Books.java – /books/, /books/{id} (URI path parameter), /books?{isbn} (URI query parameter) HTTP request URI – /books?isbn=xyz Who will win ? @Path("books") public class Books{ @Produces("application/json") @GET public List findAll(){ //find all books } @Produces("application/json") @GET @Path("{id}") public Book findById(@PathParam("id") String bookId){ //find book by id e.g. /books/123 } @Produces("application/json") @GET public Book findByISBN(@QueryParam("isbn") String bookISBN){ //find book by ISBN e.g. /books?isbn=xyz } } @Path("movies") public class Books{ @Produces("application/json") @GET public List findAll(){ //find all movies e.g. /movies/ } @Produces("application/json") @GET @Path("{name}") public Movie findById(@PathParam("name") String name){ //find movie by name e.g. /movies/SourceCode } } JAX-RS request to method matching process Break down of what’s going on Narrow down the possible matching candidates to a set of resource classes This is done by matching the HTTP request URI with the value of the @Path annotation on the resource classes From the set of resource classes in previous step, find a set of methods which are possible matching candidates (algorithm is applied to the filtered set of resource classes) Boil down to the exact method which can server the HTTP request The HTTP request verb is compared against the HTTP method specific annotations (@GET, @POST etc), the request media type specified by the Content-Type header is compared against the media type specified in the @Consumes annotation and the response media type specified by the Accept header is compared against the media type specified in the @Produces annotation I would highly recommend looking at the Jersey server side logic for implementation classes in the org.glassfish.jersey.server.internal.routing package to get a deeper understanding. Some of the classes/implementation which you can look at are MatchResultInitializerRouter SubResourceLocatorRouter MethodSelectingRouter PathMatchingRouter Time to dig in….? Happy hacking !

Many organizations in which I've coached are concerned when the people on their teams aren't being "utilized" at 100% of their capacity. If someone is at work for 8 hours per day, minus an hour for lunch and breaks, that's 7 hours of potential capacity. Some organizations are progressive enough to see that the organization's overhead of administrative activities lowers that value to 6 hours per day. By extension, a team's capacity is simply a multiple of the number of team members and the number of hours available to be utilized, i.e. a team of 5 has 30 person-hours per day. So, anything less than 30 person-hours per day spent on tasks means that a team isn't being utilized 100%. Which is bad, apparently. A Thought Exercise If you're reading this, you have a computer and it's connected to a network. Your computer might be a small one that makes phone calls or a large one that can also handle the complex scenery of games at 60 frames per second without breaking a sweat. Regardless, it's a computer. Thanks to John von Neumann, almost all computers today have the same basic architecture, the heart of which is the Central Processing Unit or CPU. When the CPU on your computer, regardless of its size, reaches 100% utilization, what happens? Your computer slows down. Significantly. The CPU is saturated with instructions and does its best to process them, but it has a finite limit on how quickly it can do that. There's a finite limit to the speed at which the instructions and the data they use can be passed around among the internal components of the CPU. That processor just can't go any faster. To a person using the computer, it appears to be locked up. To see this in action, try running a modern game like Faster Than Light on a 1st generation iPad. The older processor can't keep up with the computing demand from a game targeted towards more a powerful CPU. The communications network to which your computer is connected might be wireless or have a "hard" connection, but behind the scenes the public internet is dominated by TCP/IP and Ethernet. What happens when the network to which your computer is connected is running at 100% capacity? The network slows down. Significantly. The equipment moving the packets of data about will experience numerous collisions and will have to send requests back to your computer to resend data. The equipment will also begin to simply "drop" packets because it can't process them quickly enough. As far as you're concerned the network has become either very slow or has simply stopped. To see this in real life, look at what happens to a web site when it's the subject of a Denial of Service (DoS) attack, which effectively saturates that site's network with data requests. The site either crashes completely or appears to be unresponsive because it can't handle the traffic. Knowledge work such as software development requires a huge amount of thinking (processing) and a huge amount of collaboration (communication). The human brain is a processor, and like the von Neumann architecture has subcomponents (lobes), working memory, etc. What happens when our processor hits 100% utilization? We simply can't process all the inputs fast enough and our ability to process information slows down dramatically. We even reach a state where the decision-making centre in our brain shuts itself down. Coupled with a decreasing ability to process information and make decisions, we experience increasing levels of anxiety. Now, let's look at teams. They represent a communications network, with the people being the nodes in the network. When the team reaches 100% utilization, the individual processors (the brains of the people!) begin to slow as the ability to process information diminishes and anxiety increases. This has the effect of slowing the team's ability to communicate and operate effectively. When one or multiple people reach the point of shutting down, the network collapses. Why does anyone think that 100% utilization is a good thing? In manufacturing, 100% utilization of the capacity of your workers is actually desirable. The thinking aspects have already been done in the design and engineering of the item being created, and the assembly process is infinitely repeatable until a new item comes along. This approach to the process was taken from the manufacturing world and applied to the development of software, with the thinking that assembly was done by the programmers after all the design and engineering had taken place. Well, that simply isn't true. The "assembly" of software isn't programming, it's the compilation and packing into something deployable. Writing software is a confluence of design and engineering that has creative and technical aspects. It's not an assembly line, and probably won't ever be within my lifetime. As long ago as the late 1950's, management guru Peter Drucker realized that there were people who "think for a living". He coined the term knowledge worker to describe that type of work. Developing software is a classic knowledge worker role and therefore has different rules for productivity. As Drucker said in his 1999 book Management Challenges of the 21st Century, Productivity of the knowledge worker is not — at least not primarily — a matter of the quantity of output. Quality is at least as important. If we're pushing individuals and teams to 100% capacity, the quality of work and therefore the productivity of the team will be reduced. In 2001, Tom deMarco released Slack: Getting Past Burnout, Busywork and the Myth of Total Efficiency, an entire book describing the problems created by trying to achieve 100% utilization. I highly recommend the book, and also recommend you give your manager a copy. :) Here are some selected quotes that are pertinent to reducing the load on our processor and communication network: Very successful companies have never struck me as particularly busy; in fact, they are, as a group, rather laid back. Energy is evident in the workplace, but it's not the energy tinged with fear that comes from being slightly behind on everything. The principal resource needed for invention is slack. When companies can't invent, it's usually because their people are too damn busy. People under time pressure don't think faster. - Tim Lister How Can We Deal With This? First and foremost, stop trying to do so damned much! This is truly a slow down to go faster type of solution. For a Team If you're an agile team using iterations or sprints, pull less work into a sprint even if your velocity says you should be able to pull more. The extra time can be used to increase the quality of the work that you do complete, which is in itself a motivating factor for knowledge workers. Increased quality means decreased rework, which means the team as a whole delivers faster over the long term. Similarly, the reduced anxiety and stress on the team members increases their ability to think about what they're doing, meaning better and more innovative solutions to problems. Also, ensure that teams are focusing on activities that contribute directly to their work. Meetings tend to be the worst offenders of this, including the meetings defined within agile processes. Some suggestions: No meeting can be scheduled unless there is a specific question to be answered or specific information to be passed along. Reduce the default meeting length from 1 hour to 30 minutes in whatever calendar tool you use. Even better, reduce it to 20 minutes. Choose blocks of time that are deemed meeting free zones. No one, regardless of their position in the org chart can schedule a meeting with the team or the team members during that time. Minimize meetings where people are on a phone. My observation is that people who call into meetings tend to speak longer than when they are in the room with everyone else. I know that I do this, and I see it often in others. For an Individual Learn to say, "No." At the very least, learn to say, "Not yet." It's really difficult to do this, and I speak from personal experience. We want to help others, we want to be team players. However, if we don't say no, we take on too much, our processing and communication ability suffers and we end up disappointing those we were trying to help! Also, learn to take breaks. Yes, take breaks. The same concepts from Slack apply here, with brief diversions clearing our mind and allowing us to work better afterwards. At the extreme, if you feel like you can't keep your eyes open, take a nap! I highly recommend the Pomodoro Technique as at least a starting point for giving yourself some slack. Step away from your work for just a few minutes and return recharged. Conclusion We don't want the CPU in our computer to be working at 100% and we don't want the communications network to which it's connected to be at 100% capacity either. Our brains are processors and teams are a communications network, so why would we want those to be 100% busy all the time? In fact, ensuring that teams and the people on them are always busy is a provably wrong approach for software development. It has it's roots in a manufacturing mindset and developing software is knowledge work. Software development requires substantially different ways to make teams and the individual people on them as productive as possible. Finally, there is no magic number for what percentage of utilization is best. People are variable and 90% utilization for a person might be fine on a given day while %30 is all that the same person can handle on the next. Don't aim for a number, aim for an environment where people know that they don't have to appear busy. That will leave plenty of capacity for each person's processor and their team's communications network to run smoothly and deliver real value more effectively.

I was recently looking at a way to convert a Java 8 Stream to Rx-JavaObservable. There is one api in Observable that appears to do this : public static final Observable from(java.lang.Iterable iterable) So now the question is how do we transform a Stream to an Iterable. Stream does not implement the Iterable interface, and there are good reasons for this. So to return an Iterable from a Stream, you can do the following: Iterable iterable = new Iterable() { @Override public Iterator iterator() { return aStream.iterator(); } }; Observable.from(iterable); Since Iterable is a Java 8 functional interface, this can be simplified to the following using Java 8 Lambda expressions!: Observable.from(aStream::iterator); First look it does appear cryptic, however if it is seen as a way to simplify the expanded form of Iterable then it slowly starts to make sense. Reference: This is entirely based on what I read on this Stackoverflow question.

I’ve been playing around with dplyr over the weekend and wanted to extract the values from a data frame column to use in a later filtering step. I had a data frame: library(dplyr) df = data.frame(userId = c(1,2,3,4,5), score = c(2,3,4,5,5)) And wanted to extract the userIds of those people who have a score greater than 3. I started with: highScoringPeople = df %>% filter(score > 3) %>% select(userId) > highScoringPeople userId 1 3 2 4 3 5 And then filtered the data frame expecting to get back those 3 people: > df %>% filter(userId %in% highScoringPeople) [1] userId score <0 rows> (or 0-length row.names) No rows! I created vector with the numbers 3-5 to make sure that worked: > df %>% filter(userId %in% c(3,4,5)) userId score 1 3 4 2 4 5 3 5 5 That works as expected so highScoringPeople obviously isn’t in the right format to facilitate an ‘in lookup’. Let’s explore: > str(c(3,4,5)) num [1:3] 3 4 5 > str(highScoringPeople) 'data.frame': 3 obs. of 1 variable: $ userId: num 3 4 5 Now it’s even more obvious why it doesn’t work – highScoringPeople is still a data frame when we need it to be a vector/list. One way to fix this is to extract the userIds using the $ syntax instead of the select function: highScoringPeople = (df %>% filter(score > 3))$userId > str(highScoringPeople) num [1:3] 3 4 5 > df %>% filter(userId %in% highScoringPeople) userId score 1 3 4 2 4 5 3 5 5 Or if we want to do the column selection using dplyr we can extract the values for the column like this: highScoringPeople = (df %>% filter(score > 3) %>% select(userId))[[1]] > str(highScoringPeople) num [1:3] 3 4 5 Not so difficult after all.

As a long time Java application developer working in variety of corporate environments one of the common activities I have to perform is to write mappings to translate one Java model object into another. Regardless of the technology or library I use to write the mapper, the same question comes up. What is the best way to unit test it? I've been through various approaches, all with a variety of pros and cons related to the amount of time it takes to write what is essentially a pretty simple test. The tendency (I hate to admit) is to skimp on testing all fields and focus on what I deem to be the key fields in order to concentrate on, dare I say it, more interesting areas of the codebase. As any coder knows, this is the road to bugs and the time spent writing the test is repaid many times over in reduced debugging later. Enter eXpectamundo eXpectamundo is an open source Java library hosted on github that takes a new approach to testing model objects. It allows the Java developer to write a prototype object which has been set up with expectations. This prototype can then be used to test the actual output in a unit test. The snippet below illustrates the setup of the prototype. ... User expected = prototype(User.class); expect(expected.getCreateTs()).isWithin(1, TimeUnit.SECONDS, Moments.today()); expect(expected.getFirstName()).isEqualTo("John"); expect(expected.getUserId()).isNull(); expect(expected.getDateOfBirth()).isComparableTo(AUG(9, 1975)); expectThat(actual).matches(expected); .. For a complete example lets take a simple Data Transfer Object (DTO) which transfers the definition of a new user from a UI. package org.exparity.expectamundo.sample.mapper; import java.util.Date; public class UserDTO { private String username, firstName, surname; private Date dateOfBirth; public UserDTO(String username, String firstName, String surname, Date dateOfBirth) { this.username = username; this.firstName = firstName; this.surname = surname; this.dateOfBirth = dateOfBirth; } public String getUsername() { return username; } public String getFirstName() { return firstName; } public String getSurname() { return surname; } public Date getDateOfBirth() { return dateOfBirth; } } This DTO needs to mapped into the domain model User object which can then be manipulated, stored, etc by the service layer. The domain User object is defined as below: package org.exparity.expectamundo.sample.mapper; import java.util.Date; public class User { private Integer userId; private Date createTs = new Date(); private String username, firstName, surname; private Date dateOfBirth; public User(String username, String firstName, String surname, final Date dateOfBirth) { this.username = username; this.firstName = firstName; this.surname = surname; this.dateOfBirth = dateOfBirth; } public Integer getUserId() { return userId; } public Date getCreateTs() { return createTs; } public String getUsername() { return username; } public String getFirstName() { return firstName; } public String getSurname() { return surname; } public Date getDateOfBirth() { return dateOfBirth; } } The code for the mapper is simple so we'll use a simple hand coded mapping layer however I've introduced a bug into the mapper which we'll detect later with our unit test. package org.exparity.expectamundo.sample.mapper; public class UserDTOToUserMapper { public User map(final UserDTO userDTO) { return new User(userDTO.getUsername(), userDTO.getSurname(), userDTO.getFirstName(), userDTO.getDateOfBirth()); } } We then write a unit test for the mapper using eXpectamundo to test the expectation. package org.exparity.expectamundo.sample.mapper; import java.util.concurrent.TimeUnit; import org.junit.Test; import static org.exparity.dates.en.FluentDate.AUG; import static org.exparity.expectamundo.Expectamundo.*; import static org.exparity.hamcrest.date.Moments.now; public class UserDTOToUserMapperTest { @Test public void canMapUserDTOToUser() { UserDTO dto = new UserDTO("JohnSmith", "John", "Smith", AUG(9, 1975)); User actual = new UserDTOToUserMapper().map(dto); User expected = prototype(User.class); expect(expected.getCreateTs()).isWithin(1, TimeUnit.SECONDS, now()); expect(expected.getFirstName()).isEqualTo("John"); expect(expected.getSurname()).isEqualTo("Smith"); expect(expected.getUsername()).isEqualTo("JohnSmith"); expect(expected.getUserId()).isNull(); expect(expected.getDateOfBirth()).isSameDay(AUG(9, 1975)); expectThat(actual).matches(expected); } } The test shows how simple equality tests can be performed and also introduced some of the specialised tests which can be performed, such as testing for null, or testing the bounds of the create timestamp and performing a comparison check on the dateOfBirth property. Running the unit test reports the failure in the mapper where the firstname and surname properties have been transposed by the mapper. java.lang.AssertionError: Expected a User containing properties : getCreateTs() is expected within 1 seconds of Sun Jan 18 13:00:33 GMT 2015 getFirstName() is equal to John getSurname() is equal to Smith getUsername() is equal to JohnSmith getUserId() is null getDateOfBirth() is comparable to Sat Aug 09 00:00:00 BST 1975 But actual is a User containing properties : getFirstName() is Smith getSurname() is John A simple fix to the mapper resolves the issue: package org.exparity.expectamundo.sample.mapper; public class UserDTOToUserMapper { public User map(final UserDTO userDTO) { return new User(userDTO.getUsername(),userDTO.getFirstName(), userDTO.getSurname(), userDTO.getDateOfBirth()); } } But I can do this with hamcrest! The hamcrest equivalent to this test would follow one of two patterns; a custom implementation of org.hamcrest.Matcher for matching User objects, or a set of inline assertions as per the following example: package org.exparity.expectamundo.sample.mapper; import java.util.concurrent.TimeUnit; import org.junit.Test; import static org.exparity.dates.en.FluentDate.AUG; import static org.exparity.hamcrest.date.DateMatchers.within; import static org.exparity.hamcrest.date.Moments.now; import static org.hamcrest.MatcherAssert.assertThat; import static org.hamcrest.Matchers.*; public class UserDTOToUserMapperHamcrestTest { @Test public void canMapUserDTOToUser() { UserDTO dto = new UserDTO("JohnSmith", "John", "Smith", AUG(9, 1975)); User actual = new UserDTOToUserMapper().map(dto); assertThat(actual.getCreateTs(), within(1, TimeUnit.SECONDS, now())); assertThat(actual.getFirstName(), equalTo("John")); assertThat(actual.getSurname(), equalTo("Smith")); assertThat(actual.getUsername(), equalTo("JohnSmith")); assertThat(actual.getUserId(), nullValue()); assertThat(actual.getDateOfBirth(), comparesEqualTo(AUG(9, 1975))); } } In this example the only difference eXpectamundo offers over hamcrest is a different way of reporting mismatches. eXpectamundo will report all differences between the expected vs the actual whereas the hamcrest test will fail on the first difference. An improvement, but not really a reason to consider alternatives. Where the approach eXpectomundo offers starts to differentiate itself is when testing more complex object collections and graphs. Collection testing with eXpectamundo If we move our code forward and we create a repository to allow us to store and retrieve User instances. For the sake of simplicity I've used a basic HashMap backed repository. The code for the repository is as follows: package org.exparity.expectamundo.sample.mapper; import java.util.*; public class UserRepository { private Map userMap = new HashMap<>(); public List getAll() { return new ArrayList<>(userMap.values()); } public void addUser(final User user) { this.userMap.put(user.getUsername(), user); } public User getUserByUsername(final String username) { return userMap.get(username); } } We then write a unit test to confirm the behaviour of repository package org.exparity.expectamundo.sample.mapper; import java.util.Date; import java.util.concurrent.TimeUnit; import org.junit.Test; import static org.exparity.dates.en.FluentDate.AUG; import static org.exparity.expectamundo.Expectamundo.*; public class UserRepositoryTest { private static String FIRST_NAME = "John"; private static String SURNAME = "Smith"; private static String USERNAME = "JohnSmith"; private static Date DATE_OF_BIRTH = AUG(9, 1975); private static User EXPECTED_USER; static { EXPECTED_USER = prototype(User.class); expect(EXPECTED_USER.getCreateTs()).isWithin(1, TimeUnit.SECONDS, new Date()); expect(EXPECTED_USER.getFirstName()).isEqualTo(FIRST_NAME); expect(EXPECTED_USER.getSurname()).isEqualTo(SURNAME); expect(EXPECTED_USER.getUsername()).isEqualTo(USERNAME); expect(EXPECTED_USER.getUserId()).isNull(); expect(EXPECTED_USER.getDateOfBirth()).isComparableTo(DATE_OF_BIRTH); } @Test public void canGetAll() { User user = new User(USERNAME, FIRST_NAME, SURNAME, DATE_OF_BIRTH); UserRepository repos = new UserRepository(); repos.addUser(user); expectThat(repos.getAll()).contains(EXPECTED_USER); } @Test public void canGetByUsername() { User user = new User(USERNAME, FIRST_NAME, SURNAME, DATE_OF_BIRTH); UserRepository repos = new UserRepository(); repos.addUser(user); expectThat(repos.getUserByUsername(USERNAME)).matches(EXPECTED_USER); } } The test shows how the prototype, once constructed, can be used to perform a deep verification of an object and, if desired, can be re-used in multiple tests. The equivalent matcher in hamcrest is to write a custom matcher for the User object, or as below with flat objects using a multi matcher. (Note there are a number of ways to write the matcher, the one below I felt was the most terse example). package org.exparity.expectamundo.sample.mapper; import java.util.Date; import java.util.concurrent.TimeUnit; import org.hamcrest.*; import org.junit.Test; import static org.exparity.dates.en.FluentDate.AUG; import static org.exparity.hamcrest.BeanMatchers.hasProperty; import static org.exparity.hamcrest.date.DateMatchers.*; import static org.hamcrest.MatcherAssert.assertThat; import static org.hamcrest.Matchers.*; public class UserRepositoryHamcrestTest { private static String FIRST_NAME = "John"; private static String SURNAME = "Smith"; private static String USERNAME = "JohnSmith"; private static Date DATE_OF_BIRTH = AUG(9, 1975); private static final Matcher EXPECTED_USER = Matchers.allOf( hasProperty("CreateTs", within(1, TimeUnit.SECONDS, new Date())), hasProperty("FirstName", equalTo(FIRST_NAME)), hasProperty("Surname", equalTo(SURNAME)), hasProperty("Username", equalTo(USERNAME)), hasProperty("UserId", nullValue()), hasProperty("DateOfBirth", sameDay(DATE_OF_BIRTH))); @Test public void canGetAll() { User user = new User(USERNAME, FIRST_NAME, SURNAME, DATE_OF_BIRTH); UserRepository repos = new UserRepository(); repos.addUser(user); assertThat(repos.getAll(), hasItem(EXPECTED_USER)); } @Test public void canGetByUsername() { User user = new User(USERNAME, FIRST_NAME, SURNAME, DATE_OF_BIRTH); UserRepository repos = new UserRepository(); repos.addUser(user); assertThat(repos.getUserByUsername(USERNAME), is(EXPECTED_USER)); } } In comparison this hamcrest-based test matches the eXpectamundo test in compactness but not in type-safety. A type-safe matcher can be created which checks each property individual which would make considerably more code for no benefit over the eXpectamundo equivalent. The error reporting during failures is also clear and intuitive for the eXpectamundo test, less so for the hamcrest-equivalent. (Again an equivalent descriptive test can be written using hamcrest but will require much more code). An example of the error reporting is below where the surname is returned in place of the firstname: java.lang.AssertionError: Expected a list containing a User with properties: getCreateTs() is a expected within 1 seconds of Fri Mar 06 17:29:52 GMT 2015 getFirstName() is equal to John getSurname() is equal to Smith getUsername() is equal to JohnSmith getUserId() is is null getDateOfBirth() is is comparable to Sat Aug 09 00:00:00 BST 1975 but actual list contains: User containing properties getFirstName() is Smith Summary In summary eXpectamundo offers a new approach to perform verification of models during testing. It provides a type-safe interface to set expectations making creation of deep model tests, especially in an IDE with auto-complete, particularly simple. Failures are also reported with a clear to understand error trace. Full details of eXpectamundo and the other expectations and features it supports are available on the eXpectamundo page on github. The example code is also available on github. Try it out To try eXpectamundo out for yourself include the dependency in your maven pom or other dependency manager org.exparity expectamundo 0.9.15 test

This technical tip explains how .NET developers can fetch email messages from IMAP server and save it to disk in different formats. The first example shows how to fetch email messages from a server and save them, in EML format, to disk. The following steps are required to save the messages to disk: Create an instance of the ImapClient class. Specify hostname, username and password in the ImapClient constructor. Select the folder using SelectFolder() method. Call the ListMessages() method to get the ImapMessageInfoCollection object. Iterate through the ImapMessageInfoCollection collection, call the SaveMessage() method and provide the output path and file name. To save emails in MSG format, the ImapClient.FetchMessage() method needs to be called. It returns the message in an instance of the MailMessage class. The MailMessage.Save() method can then be called to save the message to MSG. Aspose.Email provides a 2-member overloaded variant of ListMessages to retrieve a specified number of messages based on a query. The IMAP protocol supports listing messages recursively from a mailbox folder. This helps list messages from subfolders of a folder as well. //The code samples below show how to fetch email messages from a server and save them, in EML format, to disk. //[C# Code Sample] // Select the inbox folder client.SelectFolder(ImapFolderInfo.InBox); // Get the message info collection ImapMessageInfoCollection list = client.ListMessages(); // Download each message for (int i = 0; i < list.Count; i++) { //Save the EML file locally client.SaveMessage(list[i].UniqueId, "c:\\temp\\" + list[i].UniqueId + ".eml"); } //[VB.NET Code Sample] ' Select the inbox folder client.SelectFolder(ImapFolderInfo.InBox) ' Get the message info collection Dim list As ImapMessageInfoCollection = client.ListMessages() ' Download each message Dim i As Integer = 0 Do While i < list.Count 'Save the EML file locally client.SaveMessage(list(i).UniqueId, "c:\temp\" & list(i).UniqueId & ".eml") i += 1 Loop //Saving Messages from IMAP Server in MSG Format //[C# Code Sample] // Select the inbox folder client.SelectFolder(ImapFolderInfo.InBox); // Get the message info collection ImapMessageInfoCollection list = client.ListMessages(); // Download each message for (int i = 0; i < list.Count; i++) { // Save the message in MSG format MailMessage msg = client.FetchMessage(list[i].UniqueId); Msg.Save(list[i].UniqueId + ".msg", MailMessageSaveType.OutlookMessageFormat); } //[VB.NET Code Sample] ' Select the inbox folder client.SelectFolder(ImapFolderInfo.InBox) ' Get the message info collection Dim list As ImapMessageInfoCollection = client.ListMessages() ' Download each message Dim i As Integer = 0 Do While i < list.Count ' Save the message in MSG format Dim msg As MailMessage = client.FetchMessage(list(i).UniqueId) Msg.Save(list(i).UniqueId & ".msg", MailMessageSaveType.OutlookMessageFormat) i += 1 Loop //List Messages with Maximum Number of Messages //[C# Code Sample] imapClient.SelectFolder("Inbox"); ImapQueryBuilder builder = new ImapQueryBuilder(); MailQuery query = builder.Or( builder.Or( builder.Or( builder.Or( builder.Subject.Contains(" (1) "), builder.Subject.Contains(" (2) ")), builder.Subject.Contains(" (3) ")), builder.Subject.Contains(" (4) ")), builder.Subject.Contains(" (5) ")); ImapMessageInfoCollection messageInfoCol4 = imapClient.ListMessages(query, 4); Console.WriteLine((messageInfoCol4.Count == 4) ? "Success" : "Failure"); //[VB.NET Code Sample] ImapClient client = new ImapClient(); client.Host = "domain.com"; client.Username = "username"; client.Password = "password"; client.SelectFolder("InBox"); ImapMessageInfoCollection msgsColl = client.ListMessages(true); Console.WriteLine("Total Messages: " + msgsColl.Count);

I think nobody declines the usefulness of Lambda expressions, introduced by Java 8. However, many projects are stuck with Java 7 or even older versions. Upgrading can be time consuming and costly. If third party components are incompatible with Java 8 upgrading might not be possible at all. Besides that, the whole Android platform is stuck on Java 6 and 7. Nevertheless, there is still hope for Lambda expressions! Retrolambda provides a backport of Lambda expressions for Java 5, 6 and 7. From the Retrolambda documentation: Retrolambda lets you run Java 8 code with lambda expressions and method references on Java 7 or lower. It does this by transforming your Java 8 compiled bytecode so that it can run on a Java 7 runtime. After the transformation they are just a bunch of normal .class files, without any additional runtime dependencies. To get Retrolambda running, you can use the Maven or Gradle plugin. If you want to use Lambda expressions on Android, you only have to add the following lines to your gradle build files: /build.gradle: buildscript { dependencies { classpath 'me.tatarka:gradle-retrolambda:2.4.0' } } /app/build.gradle: apply plugin: 'com.android.application' // Apply retro lambda plugin after the Android plugin apply plugin: 'retrolambda' android { compileOptions { // change compatibility to Java 8 to get Java 8 IDE support sourceCompatibility JavaVersion.VERSION_1_8 targetCompatibility JavaVersion.VERSION_1_8 } }

The post expects the reader to be familiar with generic garbage collection principles built into the JVM.

Jenkins is one of the most popular build servers and it runs on a wide variety of platforms (Windows, Linux, Mac OS X) and can build software for most programming languages (Java, C#, C++, …). And best of all, it is fully open source and free to use. By default Jenkins runs on the port 8080, which can be troublesome as this not the standard port 80 used by most web applications. But running on port 80 is in most cases not possible as the webserver is already using this port. Luckily IIS has a neat feature that allows it to act as a reverse proxy. The reverse proxy mode allows to forward traffic from IIS to another web server (Jenkins in this example) and send the responses back through IIS. This allows us to assign a regular DNS address to Jenkins and use the standard HTTP port 80. In this guide, I will explain you how you can set this up. What is required? You need an installation of IIS 7 or higher and you need to install the additional modules “URL Rewrite and “Application Request Routing”. The easiest way to install these modules is through the Microsoft Web Platform Installer. Configuring IIS Once the two necessary modules are installed, you have to create a new website in IIS. In my example I bind this website to the DNS alias “Jenkins.test.intranet”. You can bind this of course to the DNS of your choice (or to no specific DNS entry). Next you must copy the following web.config to the root of newly created website. This rule forwards all the traffic to http://localhost:8080/, the address on which Jenkins is running. It is also possible to configure this through the GUI with the URL Rewrite dialog boxes. I you are not forwarding to a localhost address, you need to go into the dialogs of Application Requet Routing and check the “Enable proxy” property.

feature-based vs. goal-oriented product roadmaps product roadmaps come in different shapes and sizes. but the two most popular formats are probably feature-based and goal-oriented roadmaps. the former are built on product features such as registration, search, or reporting, which are mapped onto a timeline. goal-oriented roadmaps focus on goals or benefits instead and show when which goal will be met. sample goals are acquiring customers or users, retaining them, increasing engagement, activating users, and generating revenue. features are viewed as second-class citizen. they are derived from the goals and usually used sparingly. the following picture illustrates the two different roadmap formats showing two major release, m and n . not that independent of its format, your product roadmap should tell a realistic and coherent story about the likely growth of your product. it should execute the product strategy so that each release builds on the previous one moving you closer towards your vision. it should not contain a random sequence of goals or a loose collection of features! a sample goal-oriented roadmap format if you would like to try out a goal-oriented roadmap, then take a look at my go product roadmap template below. you can download it from romanpichler.com/tools/product-roadmap or by clicking on the following picture. choose the right product roadmap format while i am biased towards goal-oriented roadmaps, they are not always the best fit. to select the right roadmap format for you product, consider the maturity of the product and the stability of its market. the younger your product is, the more uncertainty and change are likely to be present. a young product therefore benefits from a roadmap that focuses on product goals. older, mature products lend themselves to feature-based roadmaps that provide more details. the reason for this is simple: as your product reaches maturity, it experiences fewer changes, and you are in a better position to correctly anticipate its growth. but your product’s age is not the only driver that influences your product roadmapping approach. if your product is mature but its market segment is volatile, for instance, as competitors keep adding new features or the technologies change, you will have to update your product to defend its market share. as a consequence, uncertainty and change creep back into your roadmap. this makes it difficult to plan ahead in detail and to correctly forecast when which feature will be released. instead, you are better off creating a roadmap that is built on goals. taking into account product maturity and market stability results in the following matrix. the matrix above captures the maturity of your product on the vertical axis and the stability of its market on the horizontal axis. by distinguishing a young and a mature product and a dynamic and a stable market, four quadrants emerge that help you choose the appropriate format for your roadmap. i recommend using a product roadmap that is based on goals when your product and / or your market are likely to change. employ a feature-based roadmap only if your product has reached maturity and the market is stable. this is in contrast to what i see most organisations do: create feature-based roadmaps even though the product and the market change. don’t make the same mistake. use feature-based product roadmaps only when little change and uncertainty are present. but bear in mind that mature products can change too. you may, for instance, rejuvenate a mature product like apple has done with the ipod nano. to keep it attractive, the company has considerably changed the product since its launch in 2005. it has shrunk its size and shape and it has given it a touch screen. you may therefore have to switch from a feature-based roadmap back to a goal-oriented one! learn more you can learn more about creating an effective product roadmap by attending my agile product strategy and roadmap training course . please contact me if you would like to have the course delivered onsite or in form of a webinar.

Suppose we have list of four tasks: Task A, Task B, Task C and Task D which perform some complex computation and result into an integer value. These tasks may take random time depending upon various parameters. We can submit these tasks to executor as: ExecutorService executorService = Executors.newFixedThreadPool(4); List futures = new ArrayList>(); futures.add(executorService.submit(A)); futures.add(executorService.submit(B)); futures.add(executorService.submit(C)); futures.add(executorService.submit(D)); Then we can iterate over the list to get the computed result of each future: for (Future future:futures) { Integer result = future.get(); // rest of the code here. } Now the similar functionality can also be achieved using ExecutorCompletionService as: ExecutorService executorService = Executors.newFixedThreadPool(4); CompletionService executorCompletionService= new ExecutorCompletionService<>(executorService ); Then again we can submit the tasks and get the result like: List futures = new ArrayList>(); futures.add(executorCompletionService.submit(A)); futures.add(executorCompletionService.submit(B)); futures.add(executorCompletionService.submit(C)); futures.add(executorCompletionService.submit(D)); for (int i=0; i> solvers) throws InterruptedException { CompletionService ecs = new ExecutorCompletionService(e); int n = solvers.size(); List> futures = new ArrayList>(n); Result result = null; try { for (Callable s : solvers) futures.add(ecs.submit(s)); for (int i = 0; i < n; ++i) { try { Result r = ecs.take().get(); if (r != null) { result = r; break; } } catch(ExecutionException ignore) {} } } finally { for (Future f : futures) f.cancel(true); } if (result != null) use(result); } In the above example the moment we get the result we break out of the loop and cancel out all the other futures. One important thing to note is that the implementation ofExecutorCompletionService contains a queue of results. We need to remember the number of tasks we have added to the service and then should use take or poll to drain the queue otherwise a memory leak will occur. Some people use the Future returned by submit to process results and this is NOT correct usage. There is one interesting solution provided by Dr. Heinz M, Kabutz here. Peeking into ExecutorCompletionService When we look inside the code for this we observe that it makes use of Executor,AbstractExecutorService (default implementations of ExecutorService execution methods) and BlockingQueue (actual instance is of class LinkedBlockingQueue>). public class ExecutorCompletionService implements CompletionService { private final Executor executor; private final AbstractExecutorService aes; private final BlockingQueue> completionQueue; // Remaining code.. } Another important thing to observe is class QueueingFuture which extends FutureTaskand in the method done() the result is pushed to queue. private class QueueingFuture extends FutureTask { QueueingFuture(RunnableFuture task) { super(task, null); this.task = task; } protected void done() { completionQueue.add(task); } private final Future task; } For the curios ones, class FutureTask is base implementation of Future interface with methods to start and cancel a computation, query to see if the computation is complete, and retrieve the result of the computation. And constructor takes RunnableFuture as parameter which is again an interface which extends Future interface and adds only one method run as: public interface RunnableFuture extends Runnable, Future { /** * Sets this Future to the result of its computation * unless it has been cancelled. */ void run(); } That is all for now. Enjoy!!

Principles of REST architectural style were put forth by Dr. Roy Fielding in his thesis “Architectural Styles and the Design of Network-based Software Architectures”. One of the main principles of the style is that REST applications should be hypermedia driven, that is the change of an application's state or, in other words, transition from one resource to another, should be done by following the links. The rationale behind this principle is that all possible operations with the resource can be discovered without the need of any out-of-band documentation and if some URI changes, there is no need to change the client as it is server's responsibility to generate URIs and insert them into representations. This principle is also called Hypermedia As The Engine Of An Application state (HATEOAS). While the Thesis gives the prescription to use hyperlinks in the representations of resources, Hypertext Application Language (HAL) is one possible recipe how to do design representations with links. In particular, it describes how to design JSON representations, although there is a XML counterpart too. Our discussion will be limited only to the JSON variety. As to to HAL media type, currently, according to the document, its media type is application/vnd.hal+json. It has something to do with the fact that there are several registration trees, that is buckets for media types, with different requirements. Examples include standards, vendor and personal trees. The standards tree has no prefix for a media type. The latter two do have vnd and prs respectively. When HAL moves to standards tree, its content type name will be application/hal+json. Currently, the example application provided by the author of HAL, Mike Kelly, produces application/json in its Content-Type header, so the exact media type is not that important for our discussion. There is a convenient way to tell whether an API is RESTful or not. A so-called Richardson Maturity Model (RMM) was introduced by Leonard Richardson in his 2008 QCon talk and later popularized by Martin Fowler. The model introduces four levels of API maturity starting from Level 0, so that at level two each resource is not only identified by its own URI, but all operations with resources are done using HTTP methods like GET, PUT etc. If an API is at Level 3 it can be considered as RESTful. From the point of view of the RMM HAL helps to upgrade a Level 2 API to Level 3 whereby hypermedia is used. To make our hands dirty with HAL let's discuss a simple book catalog API where a user can browse books; this API could be a part of a larger application like a bookstore, but its mission only to be a catalog of books. The data for our API could be scraped from amazon.com. Some URIs and methods are listed below. Method URI Description GET /books Show all books GET /books/{id} Show details of the book with identificator id GET /authors/{id} Show details for author with identificator id Let's start with the representation of a single book. A book has certain properties such as the title, the price, number of pages, language and so on. The simplest JSON representation of a book could be like the following.As we deal with representations, the methods to update or modify resources were omitted, although they could be the part of the administrative interface of the service.s { "Title":"RESTful Web APIs", "Price":"$31.92", "Paperback":"408 pages", "Language":"English" } It is high time to add some links. The first candidate is a link to the object itself, as it is recommended by the HAL specification. To add links there is a reserved keyword in HAL, _links. The first character of it is an underscore which was selected to make the reserved keywords different from the names of the properties of objects in representations, although not all names starting with an underscore are reserved. Actually, _links is the name of an object the properties of which describe the meaning of a particular link. The values of the properties should contain the URI as well as may contain some other goodies. Are the names of the properties fixed? Well, some are, and the list is here, but one can add her own as will be described later. Going back to our self link there is an eponymous relation type in the list and our book object can look like the snippet below. { "_links":{ "self":{ "href":"/book/123" } }, "Title":"RESTful Web APIs", "Price":"$31.92", "Paperback":"408 pages", "Language":"English" } The href property contains the URI by following which our resource can be accessed. That is nice, but most books should have authors. While it is possible to add an array of authors' names to our representation, it is more convenient to add links to authors due to the fact that somebody could wish to navigate to the representation of the author's resource and discover what other books a particular author could have written. While there is an author relation type, there is no keyword for the case when there are several authors, so we can add a relation type for this particular case. Although the relation type could be registered in IANA for public usage to be added to the list, it is important to know how to add one's own rel types. A common practice to do the job is to use a URI by navigating which a description of the relation type could be accessed. The description may contain possible actions, supported representation formats and the purpose of the resource. The first stab at adding authors to our book could look like this. { "_links":{ "self":{ "href":"/book/123" }, "http://booklistapi.com/rels/authors":[ { "href":"/author/4554", "title":"Leonard Richardson" }, { "href":"/author/5758", "title":"Mike Amundsen" }, { "href":"/author/6853", "title":"Sam Ruby" } ] }, "Title":"RESTful Web APIs", "Price":"$31.92", "Paperback":"408 pages", "Language":"English" } By navigating to http://booklistapi.com/rels/authors one can read additional information about this relation type. As it is seen from the snippet above, the properties of a link object are not limited to href, which is the only required one, there are several more properties including title, a human-readable identifier. Some other examples are type for media type and hreflang to hint about the language of the representation. One more notice concerning custom relations is that in its form used above it seems a little bit wordy. HAL has a way to cope with this concern and it is called Compact URI or CURIE. HAL adds to relation types its own type called curies, example usage of which is demonstrated by the following snippet. { "_links":{ "self":{ "href":"/book/123" }, "curries":[ { "name":"ns", "href":"http://booklistapi.com/rels/{rel}", "templated":true } ], "ns:authors":[ { "href":"/author/4554", "title":"Leonard Richardson" }, { "href":"/author/5758", "title":"Mike Amundsen" }, { "href":"/author/6853", "title":"Sam Ruby" } ] }, "Title":"RESTful Web APIs", "Price":"$31.92", "Paperback":"408 pages", "Language":"English" } We used the name property of the link object to provide a key to select an object, which is later used in the name of our custom relation type - ns:authors. Then, we used a templated URI which can be expanded to produce the full URI, same as in the previous example, to access documentation. One additional property is templated, which is false by default but should be set to true when using URI templates. Another place where templates can be used are links that enable search whereby search terms are added to the template to produce the full URI. How this representation can be further improved? Links in a representation could not only be used to add related data, but also to show what possible actions could be taken by the client. For example, if it is an administrative interface, links to edit or delete the book using eponymous relation types may be added. Otherwise, one can add links by following which the client may be able to add a review or rate the book. We have spent enough time with our book and now let's turn to authors. The example above shows that HAL representation may contain the properties of an object as well as some links. Links can help navigate to related objects, such as authors in our book example. Another way to add information about related objects to our representation is to embed some objects in our representation. For example, if one navigates to the representation of an author, a list of all books with some detail is shown. { "_links":{ "self":{ "href":"/author/4554" }, "curries":[ { "name":"ns", "href":"http://booklistapi.com/rels/{rel}", "templated":true } ] }, "_embedded":{ "ns:books":[ { "_links":{ "self":{ "href":"/books/123" } }, "Title":"RESTful Web APIs", "Price":"$31.92" }, { "_links":{ "self":{ "href":"/books/366" } }, "Title":"RESTful Web Services", "Price":"$79.78" }, { "_links":{ "self":{ "href":"/books/865" } }, "Title":"Ruby Cookbook", "Price":"$34.35" } ] } } Another keyword starting with an underscore, _embedded, is used to add data from another object to our representation. Embedded resources are the values of properties which are relation types. The difference from links is that instead of link objects resource objects are used as values. Each embedded object can have properties, links and its own embedded objects, although the latter option is not shown by the example above. The idea is that each representation can contain the trio, including embedded objects, so one has a repeating pattern in nested objects at all levels. Up to this moment we have dealt only with small lists which may not overwhelm a client if transmitted. A representation of the resource which is a list of books could be extremely demanding from the point of view of network bandwidth and memory consumption on the client side, so it may require pagination. For example, if we deal with the resource which is the list of all possible books, one can include some predefined amount of books along with links which enable the client to transition to the next and previous pages using next and previous relation types respectively. It should be noted that a representation can contain a link to some resource along with the same resource being embedded and both share the same relation type. This is done to reduce the number of trips to the server and client can extract the necessary information from representation by using relation type. We have just scratched the surface in learning HAL which is an invaluable tool in designing representations. By reading its specification and pocking around in the example application one can gain full grasp of the format. Two final notes: first, representations can be validated against schema using some on-line tool; second, list of libraries for working with HAL using various programming languages is here. References JSON Linking with HAL HAL – Hypertext Application Language JSON Hypertext Application Language An Interview with HAL Creator Mike Kelly

Searching on the Internet about recursive lambdas, I found no relevant information. Only very old post talking about how to create recursive lambdas in ways that don't work with final version of Java8, or more recent (two years old only) post explaining this possibility had been remove from the JSR specification in October 2012, because “it was felt that there was too much work involved to be worth supporting a corner-case with a simple workaround.” Although it might have been removed from the spec, it is perfectly possible to create recursive lambdas. Lambdas are often used to create anonymous functions. This is not related to the fact that lambdas are implemented through anonymous classes. It is perfectly possible to create a named function, such as: UnaryOperator square = x -> x * x; We may then apply this function as: Integer x = square.apply(4); If we want to create a recursive factorial function, we might want to write: UnaryOperator factorial = x -> x == 0 ? 1 : x * factorial.apply(x - 1 ); But this won't compile because we can't use factorial while it is being defined. The compiler complains that “Cannot reference a field before it is defined”. One possible solution is to first declare the field, and then initialize it later, in the constructor, or in an initializer: UnaryOperator factorial; { factorial = i -> i == 0 ? 1 : i * factorial.apply( i - 1 ); } This works, but it is not very nice. There is however a much simpler way. Just add this. before the name of the function, as in: UnaryOperator factorial = x -> x == 0 ? 1 : x * this.factorial.apply(x - 1 ); Not only this works, but it even allows making the reference final: final UnaryOperator factorial = x -> x== 0 ? 1 : x * this.factorial.apply(x - 1 ); If you prefer a static field, just replace this with the name of the class: static final UnaryOperator factorial = x -> x== 0 ? 1 : x * MyClass.factorial.apply(x - 1 ); Interesting things to note: This function will silently produce an arithmetic overflow for factorial(26), producing a negative result. It will produce 0 for factorial(66) and over, until around 3000, where is will overflow the stack since recursion is implemented on the stack. If you try to find the exact limit, you may be surprised to see that it sometimes happens and sometimes not, for the same values. Try the following example: public class Test { static final UnaryOperator count = x -> x == 0 ? 0 : x + Test.count.apply(x - 1); public static void main(String[] args) { for (int i = 0;; i++) { System.out.println(i + ": " + count.apply(i)); } } } Here's the kind of result you might get: ... 18668: 174256446 18669: 174275115 18670: 174293785 18671: 174312456 Exception in thread "main" java.lang.StackOverflowError You could think that Java is able to handle 18671 level of recursion, but it is not. Try to call count(4000) and you will get a stack overflow. Obviously, Java is memoizing results on each loop execution, allowing to go much further than with a single call. Of course, it is possible to push the limits much beyond by implementing recursion on the heap through the use of trampolining. Another interesting point is that the same function implemented as a method uses much less stack space: static int count(int x) { return x == 0 ? 0 : x + count(x - 1); } This method overflows the stack only around count(6200).

Let’s take the customer endpoint in my earlier article. Here I am going to use Apache Camel-CXF component to expose this customer endpoint as a web service. @WebService(serviceName="customerService") public interface CustomerService { public Customer getCustomerById(String customerId); } public class CustomerEndpoint implements CustomerService { private CustomerEndPointService service; @Override public Customer getCustomerById(String customerId) { Customer customer= service.getCustomerById(customerId); return customer; } } Exposing the service using Camel-CXF component Remember to specify the schema Location and namespace in spring context file Consuming SOAP web service using Camel-CXF Say you have a SOAP web service to the address http://localhost:8181/OrderManagement/order Then you can invoke this web service from a camel route. Please the code snippet below. In a camel-cxf component you can also specify the data format for an endpoint like given below. Hope this will help you to create SOAP web service using Camel-CXF component.

We really are an arrogant, opinionated bunch, aren’t we? We work in an industry where there aren’t any right answers. We pretend what we do is computer “science”. When in reality, its more art than science. It certainly isn’t engineering. Engineering suggests an underlying physics, mathematical models of how the world works. Is there a mathematical model of how to build software at scale? No. Do we understand the difference between what makes good software and bad software? No. Are there papers with published proofs of whether this idea or that idea has any observable difference on written software, as practised by companies the world over? No. It turns out this is a difficult field: software is weird stuff. And yet we work in an industry full of close-minded people, convinced that their way is The One True Way. It’s not science, its basically art. Our industry is dominated by fashion. Which language we work in is fashion: should we use Ruby, or Node.js or maybe Clojure. Hey Go seems pretty cool. By which I mean “I read about it on the internet, and I’d quite like to put it on my CV so can I please f*** up your million pound project in a big experiment of whether I can figure out all the nuances of the language faster than the project can de-rail?” If it’s not the language we’re using, its architectural patterns. The dogma attached to REST. Jesus H Christ. It’s just a bunch of HTTP requests, no need to get so picky! For a while it was SOA. Then that became the old legacy thing, so now it’s all micro-services, which are totally different. Definitely. I read it on the internet, it must be true. Everyone has their opinions. Christ, we’ve got our opinions. Thousands of blogs and wankers on twitter telling you what they think about the world (exactly like this one) As if one person’s observations are useful for anything more than being able to replicate their past success, should you ever by mistake find yourself on their timeline from about six weeks ago. For example: I wrote a post recently about pairing, and some fine specimen of internet based humanity felt the need to tell me that people who need to pair are an embarrassment to the profession, that we should find another line of work. Hahaha I know, don’t read the comments. Especially when it’s in reply to something you wrote. But seriously now, is it necessary to share your close minded ignorance with the world? I shouldn’t get worked up about some asshat on the internet. But it’s not just some asshat on the internet. There are hundreds of thousands of these asshats with their closed minds and dogmatic views on the world. And not just asshats spouting off on the internet, but getting paid to build the software that increasingly runs all our lives. When will we admit that we have no idea what we’re doing. The only way to get better is to learn as many tools and techniques as we can and hopefully, along the way, we’ll learn when to apply which techniques and when not to. For example, I’ve worked with some people that don’t get TDD. Ok, fine – some people just aren’t “test infected”. And a couple of guys that really would rather gut me and fry my liver for dinner than pair with me. Do I feel the need to evangelise to them as though I’ve just found God? No. Does it offend me that they don’t follow my religion? No. Do I feel the need to suicide bomb their project? No. Its your call. Its your funeral. When I have proof that my way is The One True Way and yours is a sham, you can damn well bet I’ll be force feeding it to you. But given that ain’t gonna happen: I think we’re all pretty safe. If you don’t wanna pair, you put your headphones on and disappear into your silent reverie. Those of us that like pairing will pair, those of us that don’t, won’t. I’m fine with that. The trouble is, in this farcical echo chamber of an industry, where the lessons of 40 years ago still haven’t been learnt properly. Where we keep repeating the mistakes of 20 years ago. Of 10 years ago. Of 5 years ago. Of 2 years ago. Of last week. For Christ’s sake people, can we not just learn a little of what’s gone before? All we have is mindless opinion, presented as fact. Everyone’s out to flog you their new shiny products, or whatever bullshit service they’re offering this week. No, sorry, it’s all utter bollocks. We know less about building decent software now than we did 40 years ago. It’s just now we build a massive amount more of it. And it’s even more shit than it ever was. Only now, now we have those crazy bastards that otherwise would stand on street corners telling me that Jesus would save me if only I would let him; but now they’re selling me scrum master training or some other snake oil. All of this is unfortunately entirely indistinguishable from reasoned debate, so for youngsters entering the industry they have no way to know that its just a bunch of wankers arguing which colour to paint this new square wheel they invented. Until after a few years they become as jaded and cynical as the rest of us and decide to take advantage of all the other dumb fools out there. They find their little niche, their little way of making the world a little bit worse but themselves a little bit richer. And so the cycle repeats. Fashion begets fashion. Opinion begets opinion. There aren’t any right answers in creating software. I know what I’ve found works some of the time. I get paid to put into practice what I know. I hope you do, too. But we’ve all had a different set of experiences which means we often don’t agree on what works and what doesn’t. But this is all we have. The plural of anecdote is not data. All we have is individual judgement, borne out of individual experience. There is no grand unified theory of Correct Software Development. The best we can hope to do is learn from each other and try as many different approaches as possible. Try and fail safely and often. The more techniques you’ve tried the better the chance you can find the right technique at the right time. Call it craftsmanship if you like. Call it art if you like. But it certainly isn’t science. And I don’t know about you, but it’s a very long time since I saw any engineering round these parts.