Question 1. What is Thread in java? Answer. Threads consumes CPU in best possible manner, hence enables multi processing. Multi threading reduces idle time of CPU which improves performance of application. Thread are light weight process. A thread class belongs to java.lang package. We can create multiple threads in java, even if we don’t create any Thread, one Thread at least do exist i.e. main thread. Multiple threads run parallely in java. Threads have their own stack. Advantage of Thread : Suppose one thread needs 10 minutes to get certain task, 10 threads used at a time could complete that task in 1 minute, because threads can run parallely. Question 2. What is difference between Process and Thread in java? Answer. One process can have multiple Threads, Thread are subdivision of Process. One or more Threads runs in the context of process. Threads can execute any part of process. And same part of process can be executed by multiple Threads. Processes have their own copy of the data segment of the parent process while Threads have direct access to the data segment of its process. Processes have their own address while Threads share the address space of the process that created it. Process creation needs whole lot of stuff to be done, we might need to copy whole parent process, but Thread can be easily created. Processes can easily communicate with child processes but interprocess communication is difficult. While, Threads can easily communicate with other threads of the same process using wait() and notify() methods. In process all threads share system resource like heap Memory etc. while Thread has its own stack. Any change made to process does not affect child processes, but any change made to thread can affect the behavior of the other threads of the process. Example to see where threads on are created on different processes and same process. Question 3. How to implement Threads in java? Answer. This is very basic threading question. Threads can be created in two ways i.e. by implementing java.lang.Runnable interface or extending java.lang.Thread class and then extending run method. Thread has its own variables and methods, it lives and dies on the heap. But a thread of execution is an individual process that has its own call stack. Thread are lightweight process in java. Thread creation by implementingjava.lang.Runnableinterface. We will create object of class which implements Runnable interface : MyRunnable runnable=new MyRunnable(); Thread thread=new Thread(runnable); 2) And then create Thread object by calling constructor and passing reference of Runnable interface i.e. runnable object : Thread thread=new Thread(runnable); Question 4 . Does Thread implements their own Stack, if yes how? (Important) Answer. Yes, Threads have their own stack. This is very interesting question, where interviewer tends to check your basic knowledge about how threads internally maintains their own stacks. I’ll be explaining you the concept by diagram. Question 5. We should implement Runnable interface or extend Thread class. What are differences between implementing Runnable and extending Thread? Answer. Well the answer is you must extend Thread only when you are looking to modify run() and other methods as well. If you are simply looking to modify only the run() method implementing Runnable is the best option (Runnable interface has only one abstract method i.e. run() ). Differences between implementing Runnable interface and extending Thread class - Multiple inheritance in not allowed in java : When we implement Runnable interface we can extend another class as well, but if we extend Thread class we cannot extend any other class because java does not allow multiple inheritance. So, same work is done by implementing Runnable and extending Thread but in case of implementing Runnable we are still left with option of extending some other class. So, it’s better to implement Runnable. Thread safety : When we implement Runnable interface, same object is shared amongst multiple threads, but when we extend Thread class each and every thread gets associated with new object. Inheritance (Implementing Runnable is lightweight operation) : When we extend Thread unnecessary all Thread class features are inherited, but when we implement Runnable interface no extra feature are inherited, as Runnable only consists only of one abstract method i.e. run() method. So, implementing Runnable is lightweight operation. Coding to interface : Even java recommends coding to interface. So, we must implement Runnable rather than extending thread. Also, Thread class implements Runnable interface. Don’t extend unless you wanna modify fundamental behaviour of class, Runnable interface has only one abstract method i.e. run() : We must extend Thread only when you are looking to modify run() and other methods as well. If you are simply looking to modify only the run() method implementing Runnable is the best option (Runnable interface has only one abstract method i.e. run() ). We must not extend Thread class unless we're looking to modify fundamental behaviour of Thread class. Flexibility in code when we implement Runnable : When we extend Thread first a fall all thread features are inherited and our class becomes direct subclass of Thread , so whatever action we are doing is in Thread class. But, when we implement Runnable we create a new thread and pass runnable object as parameter,we could pass runnable object to executorService & much more. So, we have more options when we implement Runnable and our code becomes more flexible. ExecutorService : If we implement Runnable, we can start multiple thread created on runnable object with ExecutorService (because we can start Runnable object with new threads), but not in the case when we extend Thread (because thread can be started only once). Question 6. How can you say Thread behaviour is unpredictable? (Important) Answer. The solution to question is quite simple, Thread behaviour is unpredictable because execution of Threads depends on Thread scheduler, thread scheduler may have different implementation on different platforms like windows, unix etc. Same threading program may produce different output in subsequent executions even on same platform. To achieve we are going to create 2 threads on same Runnable Object, create for loop in run() method and start both threads. There is no surety that which threads will complete first, both threads will enter anonymously in for loop. Question 7 . When threads are not lightweight process in java? Answer. Threads are lightweight process only if threads of same process are executing concurrently. But if threads of different processes are executing concurrently then threads are heavy weight process. Question 8. How can you ensure all threads that started from main must end in order in which they started and also main should end in last? (Important) Answer. Interviewers tend to know interviewees knowledge about Thread methods. So this is time to prove your point by answering correctly. We can use join() methodto ensure all threads that started from main must end in order in which they started and also main should end in last.In other words waits for this thread to die. Calling join() method internally calls join(0); DETAILED DESCRIPTION : Join() method - ensure all threads that started from main must end in order in which they started and also main should end in last. Types of join() method with programs- 10 salient features of join. Question 9.What is difference between starting thread with run() and start() method? (Important) Answer. This is quite interesting question, it might confuse you a bit and at time may make you think is there really any difference between starting thread with run() and start() method. When you call start() method, main thread internally calls run() method to start newly created Thread, so run() method is ultimately called by newly created thread. When you call run() method main thread rather than starting run() method with newly thread it start run() method by itself. Question 10. What is significance of using Volatile keyword? (Important) Answer. Java allows threads to access shared variables. As a rule, to ensure that shared variables are consistently updated, a thread should ensure that it has exclusive use of such variables by obtaining a lock that enforces mutual exclusion for those shared variables. If a field is declared volatile, in that case the Java memory model ensures that all threads see a consistent value for the variable. Few small questions> Q. Can we have volatile methods in java? No, volatile is only a keyword, can be used only with variables. Q. Can we have synchronized variable in java? No, synchronized can be used only with methods, i.e. in method declaration. Question 11. Differences between synchronized and volatile keyword in Java? (Important) Answer.Its very important question from interview perspective. Volatilecan be used as a keyword against the variable, we cannot use volatile against method declaration. volatile void method1(){} //it’s illegal, compilation error. While synchronization can be used in method declaration or we can create synchronization blocks (In both cases thread acquires lock on object’s monitor). Variables cannot be synchronized. Synchronized method: synchronized void method2(){} //legal Synchronized block: void method2(){ synchronized (this) { //code inside synchronized block. } } Synchronized variable (illegal): synchronized int i;//it’s illegal, compilatiomn error. Volatile does not acquire any lock on variable or object, but Synchronization acquires lock on method or block in which it is used. Volatile variables are not cached, but variables used inside synchronized method or block are cached. When volatile is used will never create deadlock in program, as volatile never obtains any kind of lock . But in case if synchronization is not done properly, we might end up creating dedlock in program. Synchronization may cost us performance issues, as one thread might be waiting for another thread to release lock on object. But volatile is never expensive in terms of performance. DETAILED DESCRIPTION : Differences between synchronized and volatile keyword in detail with programs. Question 12. Can you again start Thread? Answer.No, we cannot start Thread again, doing so will throw runtimeException java.lang.IllegalThreadStateException. The reason is once run() method is executed by Thread, it goes into dead state. Let’s take an example- Thinking of starting thread again and calling start() method on it (which internally is going to call run() method) for us is some what like asking dead man to wake up and run. As, after completing his life person goes to dead state. Question 13. What is race condition in multithreading and how can we solve it? (Important) Answer. This is very important question, this forms the core of multi threading, you should be able to explain about race condition in detail. When more than one thread try to access same resource without synchronization causes race condition. So we can solve race condition by using either synchronized block or synchronized method. When no two threads can access same resource at a time phenomenon is also called as mutual exclusion. Few sub questions> What if two threads try to read same resource without synchronization? When two threads try to read on same resource without synchronization, it’s never going to create any problem. What if two threads try to write to same resource without synchronization? When two threads try to write to same resource without synchronization, it’s going to create synchronization problems. Question 14. How threads communicate between each other? Answer. This is very must know question for all the interviewees, you will most probably face this question in almost every time you go for interview. Threads can communicate with each other by using wait(), notify() and notifyAll() methods. Question 15. Why wait(), notify() and notifyAll() are in Object class and not in Thread class? (Important) Answer. Every Object has a monitor, acquiring that monitors allow thread to hold lock on object. But Thread class does not have any monitors. wait(), notify() and notifyAll()are called on objects only >When wait() method is called on object by thread it waits for another thread on that object to release object monitor by calling notify() or notifyAll() method on that object. When notify() method is called on object by thread it notifies all the threads which are waiting for that object monitor that object monitor is available now. So, this shows that wait(), notify() and notifyAll() are called on objects only. Now, Straight forward question that comes to mind is how thread acquires object lock by acquiring object monitor? Let’s try to understand this basic concept in detail? Wait(), notify() and notifyAll() method being in Object class allows all the threads created on that object to communicate with other. . As multiple threads exists on same object. Only one thread can hold object monitor at a time. As a result thread can notify other threads of same object that lock is available now. But, thread having these methods does not make any sense because multiple threads exists on object its not other way around (i.e. multiple objects exists on thread). Now let’s discuss one hypothetical scenario, what will happen if Thread class contains wait(), notify() and notifyAll() methods? Having wait(), notify() and notifyAll() methods means Thread class also must have their monitor. Every thread having their monitor will create few problems - >Thread communication problem. >Synchronization on object won’t be possible- Because object has monitor, one object can have multiple threads and thread hold lock on object by holding object monitor. But if each thread will have monitor, we won’t have any way of achieving synchronization. >Inconsistency in state of object (because synchronization won't be possible). Question 16. Is it important to acquire object lock before calling wait(), notify() and notifyAll()? Answer.Yes, it’s mandatory to acquire object lock before calling these methods on object. As discussed above wait(), notify() and notifyAll() methods are always called from Synchronized block only, and as soon as thread enters synchronized block it acquires object lock (by holding object monitor). If we call these methods without acquiring object lock i.e. from outside synchronize block then java.lang. IllegalMonitorStateException is thrown at runtime. Wait() method needs to enclosed in try-catch block, because it throws compile time exception i.e. InterruptedException. Question 17. How can you solve consumer producer problem by using wait() and notify() method? (Important) Answer. Here come the time to answer very very important question from interview perspective. Interviewers tends to check how sound you are in threads inter communication. Because for solving this problem we got to use synchronization blocks, wait() and notify() method very cautiously. If you misplace synchronization block or any of the method, that may cause your program to go horribly wrong. So, before going into this question first i’ll recommend you to understand how to use synchronized blocks, wait() and notify() methods. Key points we need to ensure before programming : >Producer will produce total of 10 products and cannot produce more than 2 products at a time until products are being consumed by consumer. Example> when sharedQueue’s size is 2, wait for consumer to consume (consumer will consume by calling remove(0) method on sharedQueue and reduce sharedQueue’s size). As soon as size is less than 2, producer will start producing. >Consumer can consume only when there are some products to consume. Example> when sharedQueue’s size is 0, wait for producer to produce (producer will produce by calling add() method on sharedQueue and increase sharedQueue’s size). As soon as size is greater than 0, consumer will start consuming. Explanation of Logic > We will create sharedQueue that will be shared amongst Producer and Consumer. We will now start consumer and producer thread. Note: it does not matter order in which threads are started (because rest of code has taken care of synchronization and key points mentioned above) First we will start consumerThread > consumerThread.start(); consumerThread will enter run method and call consume() method. There it will check for sharedQueue’s size. -if size is equal to 0 that means producer hasn’t produced any product, wait for producer to produce by using below piece of code- synchronized (sharedQueue) { while (sharedQueue.size() == 0) { sharedQueue.wait(); } } -if size is greater than 0, consumer will start consuming by using below piece of code. synchronized (sharedQueue) { Thread.sleep((long)(Math.random() * 2000)); System.out.println("consumed : "+ sharedQueue.remove(0)); sharedQueue.notify(); } Than we will start producerThread > producerThread.start(); producerThread will enter run method and call produce() method. There it will check for sharedQueue’s size. -if size is equal to 2 (i.e. maximum number of products which sharedQueue can hold at a time), wait for consumer to consume by using below piece of code- synchronized (sharedQueue) { while (sharedQueue.size() == maxSize) { //maxsize is 2 sharedQueue.wait(); } } -if size is less than 2, producer will start producing by using below piece of code. synchronized (sharedQueue) { System.out.println("Produced : " + i); sharedQueue.add(i); Thread.sleep((long)(Math.random() * 1000)); sharedQueue.notify(); } DETAILED DESCRIPTION with program : Solve Consumer Producer problem by using wait() and notify() methods in multithreading. Question 18. How to solve Consumer Producer problem without using wait() and notify() methods, where consumer can consume only when production is over.? Answer. In this problem, producer will allow consumer to consume only when 10 products have been produced (i.e. when production is over). We will approach by keeping one boolean variable productionInProcess and initially setting it to true, and later when production will be over we will set it to false. Question 19. How can you solve consumer producer pattern by using BlockingQueue? (Important) Answer. Now it’s time to gear up to face question which is most probably going to be followed up by previous question i.e. after how to solve consumer producer problem using wait() and notify() method. Generally you might wonder why interviewer's are so much interested in asking about solving consumer producer problem using BlockingQueue, answer is they want to know how strong knowledge you have about java concurrent Api’s, this Api use consumer producer pattern in very optimized manner, BlockingQueue is designed is such a manner that it offer us the best performance. BlockingQueue is a interface and we will use its implementation class LinkedBlockingQueue. Key methods for solving consumer producer pattern are > put(i); //used by producer to put/produce in sharedQueue. take();//used by consumer to take/consume from sharedQueue. Question 20. What is deadlock in multithreading? Write a program to form DeadLock in multi threading and also how to solve DeadLock situation. What measures you should take to avoid deadlock? (Important) Answer. This is very important question from interview perspective. But, what makes this question important is it checks interviewees capability of creating and detecting deadlock. If you can write a code to form deadlock, than I am sure you must be well capable in solving that deadlock as well. If not, later on this post we will learn how to solve deadlock as well. First question comes to mind is, what is deadlock in multi threading program? Deadlock is a situation where two threads are waiting for each other to release lock holded by them on resources. But how deadlock could be formed : Thread-1 acquires lock on String.class and then calls sleep() method which gives Thread-2 the chance to execute immediately after Thread-1 has acquired lock on String.class and Thread-2 acquires lock on Object.class then calls sleep() method and now it waits for Thread-1 to release lock on String.class. Conclusion: Now, Thread-1 is waiting for Thread-2 to release lock on Object.class and Thread-2 is waiting for Thread-1 to release lock on String.class and deadlock is formed. //Code called by Thread-1 public void run() { synchronized (String.class) { Thread.sleep(100); synchronized (Object.class) { } } } //Code called by Thread-2 publicvoid run() { synchronized (Object.class) { Thread.sleep(100); synchronized (String.class) { } } } Here comes the important part, how above formed deadlock could be solved : Thread-1 acquires lock on String.class and then calls sleep() method which gives Thread-2 the chance to execute immediately after Thread-1 has acquired lock on String.class and Thread-2 tries to acquire lock on String.class but lock is holded by Thread-1. Meanwhile, Thread-1 completes successfully. As Thread-1 has completed successfully it releases lock on String.class, Thread-2 can now acquire lock on String.class and complete successfully without any deadlock formation. Conclusion: No deadlock is formed. //Code called by Thread-1 publicvoid run() { synchronized (String.class) { Thread.sleep(100); synchronized (Object.class) { } } } //Code called by Thread-2 publicvoid run() { synchronized (String.class) { Thread.sleep(100); synchronized (Object.class) { } } } Few important measures to avoid Deadlock > Lock specific member variables of class rather than locking whole class: We must try to lock specific member variables of class rather than locking whole class. Use join() method: If possible try touse join() method, although it may refrain us from taking full advantage of multithreading environment because threads will start and end sequentially, but it can be handy in avoiding deadlocks. If possible try avoid using nested synchronization blocks. Question 21. Have you ever generated thread dumps or analyzed Thread Dumps? (Important) Answer. Answering this questions will show your in depth knowledge of Threads. Every experienced must know how to generate Thread Dumps. VisualVM is most popular way to generate Thread Dump and is most widely used by developers. It’s important to understand usage of VisualVM for in depth knowledge of VisualVM. I’ll recommend every developer must understand this topic to become master in multi threading. It helps us in analyzing threads performance, thread states, CPU consumed by threads, garbage collection and much more. For detailed information see Generating and analyzing Thread Dumps using VisualVM - step by step detail to setup VisualVM with screenshots jstack is very easy way to generate Thread dump and is widely used by developers. I’ll recommend every developer must understand this topic to become master in multi threading. For creating Thread dumps we need not to download any jar or any extra software. For detailed information see Generating and analyzing Thread Dumps using JSATCK - step by step detail to setup JSTACK with screenshots. Question 22. What is life cycle of Thread, explain thread states? (Important) Answer. Thread states/ Thread life cycle is very basic question, before going deep into concepts we must understand Thread life cycle. Thread have following states > New Runnable Running Waiting/blocked/sleeping Terminated (Dead) Thread states/ Thread life cycle in diagram > Thread states in detail > New : When instance of thread is created using new operator it is in new state, but the start() method has not been invoked on the thread yet, thread is not eligible to run yet. Runnable : When start() method is called on thread it enters runnable state. Running : Thread scheduler selects thread to go fromrunnable to running state. In running state Thread starts executing by entering run() method. Waiting/blocked/sleeping : In this state a thread is not eligible to run. >Thread is still alive, but currently it’s not eligible to run. In other words. > How can Thread go from running to waiting state? By calling wait()method thread go from running to waiting state. In waiting state it will wait for other threads to release object monitor/lock. > How can Thread go from running to sleeping state? By calling sleep() methodthread go from running to sleeping state. In sleeping state it will wait for sleep time to get over. Terminated (Dead) : A thread is considered dead when its run() method completes. Question 23. Are you aware of preemptive scheduling and time slicing? Answer. In preemptive scheduling, the highest priority thread executes until it enters into the waiting or dead state. In time slicing, a thread executes for a certain predefined time and then enters runnable pool. Than thread can enter running state when selected by thread scheduler. Question 24. What are daemon threads? Answer.Daemon threads are low priority threads which runs intermittently in background for doing garbage collection. 12 Few salient features of daemon() threads> Thread scheduler schedules these threads only when CPU is idle. Daemon threads are service oriented threads, they serves all other threads. These threads are created before user threads are created and die after all other user threads dies. Priority of daemon threads is always 1 (i.e. MIN_PRIORITY). User created threads are non daemon threads. JVM can exit when only daemon threads exist in system. we can use isDaemon() method to check whether thread is daemon thread or not. we can use setDaemon(boolean on) method to make any user method a daemon thread. If setDaemon(boolean on) is called on thread after calling start() method than IllegalThreadStateException is thrown. You may like to see how daemon threads work, for that you can use VisualVM or jStack. I have provided Thread dumps over there which shows daemon threads which were intermittently running in background. Some of the daemon threads which intermittently run in background are > "RMI TCP Connection(3)-10.175.2.71" daemon"RMI TCP Connection(idle)" daemon"RMI Scheduler(0)" daemon"C2 CompilerThread1" daemon "GC task thread#0 (ParallelGC)" Question 25. Why suspend() and resume() methods are deprecated? Answer.Suspend() method is deadlock prone. If the target thread holds a lock on object when it is suspended, no thread can lock this object until the target thread is resumed. If the thread that would resume the target thread attempts to lock this monitor prior to calling resume, it results in deadlock formation. These deadlocksare generally called Frozen processes. Suspend() method puts thread from running to waiting state. And thread can go from waiting to runnable state only when resume() method is called on thread. It is deprecated method. Resume() method is only used with suspend() method that’s why it’s also deprecated method. Question 26. Why destroy() methods is deprecated? Answer. This question is again going to check your in depth knowledge of thread methods i.e. destroy() method is deadlock prone. If the target thread holds a lock on object when it is destroyed, no thread can lock this object (Deadlock formed are similar to deadlock formed when suspend() and resume() methods are used improperly). It results in deadlock formation. These deadlocksare generally called Frozen processes. Additionally you must know calling destroy() method on Threads throw runtimeException i.e. NoSuchMethodError. Destroy() method puts thread from running to dead state. Question 27. As stop() method is deprecated, How can we terminate or stop infinitely running thread in java? (Important) Answer. This is very interesting question where interviewees thread basics basic will be tested. Interviewers tend to know user’s knowledge about main thread’s and thread invoked by main thread. We will try to address the problem by creating new thread which will run infinitely until certain condition is satisfied and will be called by main Thread. Infinitely running thread can be stopped using boolean variable. Infinitely running thread can be stopped using interrupt() method. Let’s understand Why stop() method is deprecated : Stopping a thread with Thread.stop() causes it to release all of the monitors that it has locked. If any of the objects previously protected by these monitors were in an inconsistent state, the damaged objects become visible to other threads, which might lead to unpredictable behavior. Question 28. what is significance of yield() method, what state does it put thread in? yield() is a native method it’s implementation in java 6 has been changed as compared to its implementation java 5. As method is native it’s implementation is provided by JVM. In java 5, yield() method internally used to call sleep() method giving all the other threads of same or higher priority to execute before yielded thread by leaving allocated CPU for time gap of 15 millisec. But java 6, calling yield() method gives a hint to the thread scheduler that the current thread is willing to yield its current use of a processor. The thread scheduler is free to ignore this hint. So, sometimes even after using yield() method, you may not notice any difference in output. salient features of yield() method > Definition : yield() method when called on thread gives a hint to the thread scheduler that the current thread is willing to yield its current use of a processor.The thread scheduler is free to ignore this hint. Thread state : when yield() method is called on thread it goes from running to runnable state, not in waiting state. Thread is eligible to run but not running and could be picked by scheduler at anytime. Waiting time : yield() method stops thread for unpredictable time. Static method : yield()is a static method, hence calling Thread.yield() causes currently executing thread to yield. Native method : implementation of yield() method is provided by JVM. Let’s see definition of yield() method as given in java.lang.Thread - public static native void yield(); synchronized block : thread need not to to acquire object lock before calling yield()method i.e. yield() method can be called from outside synchronized block. Question 29.What is significance of sleep() method in detail, what statedoes it put thread in ? sleep() is a native method, it’s implementation is provided by JVM. 10 salient features of sleep() method > Definition : sleep() methods causes current thread to sleep for specified number of milliseconds (i.e. time passed in sleep method as parameter). Ex- Thread.sleep(10) causes currently executing thread to sleep for 10 millisec. Thread state : when sleep() is called on thread it goes from running to waiting state and can return to runnable state when sleep time is up. Exception : sleep() method must catch or throw compile time exception i.e. InterruptedException. Waiting time : sleep() method have got few options. sleep(long millis) - Causes the currently executing thread to sleep for the specified number of milliseconds public static native void sleep(long millis) throws InterruptedException; sleep(long millis, int nanos) - Causes the currently executing thread to sleep for the specified number of milliseconds plus the specified number of nanoseconds. public static native void sleep(long millis,int nanos) throws InterruptedException; static method : sleep()is a static method, causes the currently executing thread to sleep for the specified number of milliseconds. Belongs to which class :sleep() method belongs to java.lang.Thread class. synchronized block : thread need not to to acquire object lock before calling sleep()method i.e. sleep() method can be called from outside synchronized block. Question 30. Difference between wait() and sleep() ? (Important) Answer. Should be called from synchronized block :wait() method is always called from synchronized block i.e. wait() method needs to lock object monitor before object on which it is called. But sleep() method can be called from outside synchronized block i.e. sleep() method doesn’t need any object monitor. IllegalMonitorStateException : if wait() method is called without acquiring object lock than IllegalMonitorStateException is thrown at runtime, but sleep() methodnever throws such exception. Belongs to which class : wait() method belongs to java.lang.Object class but sleep() method belongs to java.lang.Thread class. Called on object or thread : wait() method is called on objects but sleep() method is called on Threads not objects. Thread state : when wait() method is called on object, thread that holded object’s monitor goes from running to waiting state and can return to runnable state only when notify() or notifyAll()method is called on that object. And later thread scheduler schedules that thread to go from from runnable to running state. when sleep() is called on thread it goes from running to waiting state and can return to runnable state when sleep time is up. When called from synchronized block :when wait() method is called thread leaves the object lock. But sleep()method when called from synchronized block or method thread doesn’t leaves object lock. Question 31. Differences and similarities between yield() and sleep()? Answer. Differences yield() and sleep() : Definition : yield() method when called on thread gives a hint to the thread scheduler that the current thread is willing to yield its current use of a processor.The thread scheduler is free to ignore this hint. sleep() methods causes current thread to sleep for specified number of milliseconds (i.e. time passed in sleep method as parameter). Ex- Thread.sleep(10) causes currently executing thread to sleep for 10 millisec. Thread state : when sleep() is called on thread it goes from running to waiting state and can return to runnable state when sleep time is up. when yield() method is called on thread it goes from running to runnable state, not in waiting state. Thread is eligible to run but not running and could be picked by scheduler at anytime. Exception : yield() method need not to catch or throw any exception. But sleep() method must catch or throw compile time exception i.e. InterruptedException. Waiting time : yield() method stops thread for unpredictable time, that depends on thread scheduler. But sleep() method have got few options. sleep(long millis) - Causes the currently executing thread to sleep for the specified number of milliseconds sleep(long millis, int nanos) - Causes the currently executing thread to sleep for the specified number of milliseconds plus the specified number of nanoseconds. similarity between yield() and sleep(): > yield() and sleep() method belongs to java.lang.Thread class. > yield() and sleep() method can be called from outside synchronized block. > yield() and sleep() method are called on Threads not objects. Question 32. Mention some guidelines to write thread safe code, most important point we must take care of in multithreading programs? Answer. In multithreading environment it’s important very important to write thread safe code, thread unsafe code can cause a major threat to your application. I have posted many articles regarding thread safety. So overall this will be revision of what we have learned so far i.e. writing thread safe healthy code and avoiding any kind of deadlocks. If method is exposed in multithreading environment and it’s not synchronized (thread unsafe) than it might lead us to race condition, we must try to use synchronized block and synchronized methods. Multiple threads may exist on same object but only one thread of that object can enter synchronized method at a time, though threads on different object can enter same method at same time. Even static variables are not thread safe, they are used in static methods and if static methods are not synchronized then thread on same or different object can enter method concurrently. Multiple threads may exist on same or different objects of class but only one thread can enter static synchronized method at a time, we must consider making static methods as synchronized. If possible, try to use volatile variables. If a field is declared volatile all threads see a consistent value for the variable. Volatile variables at times can be used as alternate to synchronized methods as well. Final variables are thread safe because once assigned some reference of object they cannot point to reference of other object. s is pointing to String object. public class MyClass { final String s=new String("a"); void method(){ s="b"; //compilation error, s cannot point to new reference. } } If final is holding some primitive value it cannot point to other value. public class MyClass { final inti=0; void method(){ i=0; //compilation error, i cannot point to new value. } } Usage of local variables : If possible try to use local variables, local variables are thread safe, because every thread has its own stack, i.e. every thread has its own local variables and its pushes all the local variables on stack. public class MyClass { void method(){ inti=0; //Local variable, is thread safe. } } Using thread safe collections : Rather than using ArrayList we must Vector and in place of using HashMap we must use ConcurrentHashMap or HashTable. We must use VisualVM or jstack to detect problems such as deadlocks and time taken by threads to complete in multi threading programs. Using ThreadLocal:ThreadLocal is a class which provides thread-local variables. Every thread has its own ThreadLocal value that makes ThreadLocal value threadsafe as well. Rather than StringBuffer try using immutable classes such as String. Any change to String produces new String. Question 33. How thread can enter waiting, sleeping and blocked state and how can they go to runnable state ? Answer. This is very prominently asked question in interview which will test your knowledge about thread states. And it’s very important for developers to have in depth knowledge of this thread state transition. I will try to explain this thread state transition by framing few sub questions. I hope reading sub questions will be quite interesting. > How can Thread go from running to waiting state ? By calling wait()method thread go from running to waiting state. In waiting state it will wait for other threads to release object monitor/lock. > How can Thread return from waiting to runnable state ? Once notify() or notifyAll()method is called object monitor/lock becomes available and thread can again return to runnable state. > How can Thread go from running to sleeping state ? By calling sleep() methodthread go from running to sleeping state. In sleeping state it will wait for sleep time to get over. > How can Thread return from sleeping to runnable state ? Once specified sleep time is up thread can again return to runnable state. Suspend() method can be used to put thread in waiting state and resume() method is the only way which could put thread in runnable state. Thread also may go from running to waiting state if it is waiting for some I/O operation to take place. Once input is available thread may return to running state. >When threads are in running state, yield()method can make thread to go in Runnable state. Question 34. Difference between notify() and notifyAll() methods, can you write a code to prove your point? Answer. Goodness. Theoretically you must have heard or you must be aware of differences between notify() and notifyAll().But have you created program to achieve it? If not let’s do it. First, I will like give you a brief description of what notify() and notifyAll() methods do. notify()- Wakes up a single thread that is waiting on this object's monitor. If any threads are waiting on this object, one of them is chosen to be awakened. The choice is random and occurs at the discretion of the implementation. A thread waits on an object's monitor by calling one of the wait methods. The awakened threads will not be able to proceed until the current thread relinquishes the lock on this object. public final native void notify(); notifyAll()- Wakes up all threads that are waiting on this object's monitor. A thread waits on an object's monitor by calling one of the wait methods. The awakened threads will not be able to proceed until the current thread relinquishes the lock on this object. public final native void notifyAll(); Now it’s time to write down a program to prove the point. Question 35. Does thread leaves object lock when sleep() method is called? Answer. When sleep() method is called Thread does not leaves object lock and goes from running to waiting state. Thread waits for sleep time to over and once sleep time is up it goes from waiting to runnable state. Question 36. Does thread leaves object lock when wait() method is called? Answer. When wait() method is called Thread leaves the object lock and goes from running to waiting state. Thread waits for other threads on same object to call notify() or notifyAll() and once any of notify() or notifyAll() is called it goes from waiting to runnable state and again acquires object lock. Question 37. What will happen if we don’t override run method? Answer. This question will test your basic knowledge how start and run methods work internally in Thread Api. When we call start() method on thread, it internally calls run() method with newly created thread. So, if we don’t override run() method newly created thread won’t be called and nothing will happen. class MyThread extends Thread { //don't override run() method } publicclass DontOverrideRun { publicstaticvoid main(String[] args) { System.out.println("main has started."); MyThread thread1=new MyThread(); thread1.start(); System.out.println("main has ended."); } } /*OUTPUT main has started. main has ended. */ As we saw in output, we didn’t override run() method that’s why on calling start() method nothing happened. Question 38. What will happen if we override start method? Answer. This question will again test your basic core java knowledge how overriding works at runtime, what what will be called at runtime and how start and run methods work internally in Thread Api. When we call start() method on thread, it internally calls run() method with newly created thread. So, if we override start() method, run() method will not be called until we write code for calling run() method. class MyThread extends Thread { @Override publicvoid run() { System.out.println("in run() method"); } @Override publicvoid start(){ System.out.println("In start() method"); } } publicclass OverrideStartMethod { publicstaticvoid main(String[] args) { System.out.println("main has started."); MyThread thread1=new MyThread(); thread1.start(); System.out.println("main has ended."); } } /*OUTPUT main has started. In start() method main has ended. */ If we note output. we have overridden start method and didn’t called run() method from it, so, run() method wasn’t call. Question 39. Can we acquire lock on class? What are ways in which you can acquire lock on class? Answer. Yes, we can acquire lock on class’s class object in 2 ways to acquire lock on class. Thread can acquire lock on class’s class object by- Entering synchronized block or Let’s say there is one class MyClass. Now we can create synchronization block, and parameter passed with synchronization tells which class has to be synchronized. In below code, we have synchronized MyClass synchronized (MyClass.class) { //thread has acquired lock on MyClass’s class object. } by entering static synchronized methods. public staticsynchronizedvoid method1() { //thread has acquired lock on MyRunnable’s class object. } As soon as thread entered Synchronization method, thread acquired lock on class’s class object. Thread will leave lock when it exits static synchronized method. Question 40. Difference between object lock and class lock? Answer. It is very important question from multithreading point of view. We must understand difference between object lock and class lock to answer interview, ocjp answers correctly. Object lock Class lock Thread can acquire object lock by- Entering synchronized block or by entering synchronized methods. Thread can acquire lock on class’s class object by- Entering synchronized block or by entering static synchronized methods. Multiple threads may exist on same object but only one thread of that object can enter synchronized method at a time. Threads on different object can enter same method at same time. Multiple threads may exist on same or different objects of class but only one thread can enter static synchronized method at a time. Multiple objects of class may exist and every object has it’s own lock. Multiple objects of class may exist but there is always one class’s class object lock available. First let’s acquire object lock by entering synchronized block. Example- Let’s say there is one class MyClassand we have created it’s object and reference to that object is myClass. Now we can create synchronization block, and parameter passed with synchronization tells which object has to be synchronized. In below code, we have synchronized object reference by myClass. MyClass myClass=newMyclass(); synchronized (myClass) { } As soon thread entered Synchronization block, thread acquired object lock on object referenced by myClass (by acquiring object’s monitor.) Thread will leave lock when it exits synchronized block. First let’s acquire lock on class’s class object by entering synchronized block. Example- Let’s say there is one class MyClass. Now we can create synchronization block, and parameter passed with synchronization tells which class has to be synchronized. In below code, we have synchronized MyClass synchronized (MyClass.class) { } As soon as thread entered Synchronization block, thread acquired MyClass’s class object. Thread will leave lock when it exits synchronized block. publicsynchronizedvoid method1() { } As soon as thread entered Synchronization method, thread acquired object lock. Thread will leave lock when it exits synchronized method. public staticsynchronizedvoid method1() {} As soon as thread entered static Synchronization method, thread acquired lock on class’s class object. Thread will leave lock when it exits synchronized method. Let’s me give you some tricky situation based question, Question 41. Suppose you have 2 threads (Thread-1 and Thread-2) on same object. Thread-1 is in synchronized method1(), can Thread-2 enter synchronized method2() at same time? Answer.No, here when Thread-1 is in synchronized method1() it must be holding lock on object’s monitor and will release lock on object’s monitor only when it exits synchronized method1(). So, Thread-2 will have to waitfor Thread-1 to release lock on object’s monitor so that it could enter synchronized method2(). Likewise, Thread-2 even cannot enter synchronized method1() which is being executed by Thread-1. Thread-2 will have to wait for Thread-1 to release lock on object’s monitor so that it could enter synchronized method1(). Now, let’s see a program to prove our point. Question 42. Suppose you have 2 threads (Thread-1 and Thread-2) on same object. Thread-1 is in static synchronized method1(), can Thread-2 enter static synchronized method2() at same time? Answer.No, here when Thread-1 is in static synchronized method1() it must be holding lock on class class’s object and will release lock on class’s classobject only when it exits static synchronized method1(). So, Thread-2 will have to wait for Thread-1 to release lock on class’s classobject so that it could enter static synchronized method2(). Likewise, Thread-2 even cannot enter static synchronized method1() which is being executed by Thread-1. Thread-2 will have to wait for Thread-1 to release lock on class’s classobject so that it could enter static synchronized method1(). Now, let’s see a program to prove our point. Question 43. Suppose you have 2 threads (Thread-1 and Thread-2) on same object. Thread-1 is in synchronized method1(), can Thread-2 enter static synchronized method2() at same time? Answer.Yes, here when Thread-1 is in synchronized method1() it must be holding lock on object’s monitor and Thread-2 can enter static synchronized method2() by acquiring lock on class’s class object. Now, let’s see a program to prove our point. Question 44. Suppose you have thread and it is in synchronized method and now can thread enter other synchronized method from that method? Answer.Yes, here when thread is in synchronized method it must be holding lock on object’s monitor and using that lock thread can enter other synchronized method. Now, let’s see a program to prove our point. Question 45. Suppose you have thread and it is in static synchronized method and now can thread enter other static synchronized method from that method? Answer. Yes, here when thread is in static synchronized method it must be holding lock on class’s class object and using that lock thread can enter other static synchronized method. Now, let’s see a program to prove our point. Question 46. Suppose you have thread and it is in static synchronized method and now can thread enter other non static synchronized method from that method? Answer.Yes, here when thread is in static synchronized method it must be holding lock on class’s class object and when it enters synchronized method it will hold lock on object’s monitor as well. So, now thread holds 2 locks (it’s also called nested synchronization)- >first one on class’s class object. >second one on object’s monitor (This lock will be released when thread exits non static method).Now, let’s see a program to prove our point. Question 47. Suppose you have thread and it is in synchronized method and now can thread enter other static synchronized method from that method? Answer.Yes, here when thread is in synchronized method it must be holding lock on object’s monitor and when it enters static synchronized method it will hold lock on class’s class object as well. So, now thread holds 2 locks (it’s also called nested synchronization)- >first one on object’s monitor. >second one on class’s class object.(This lock will be released when thread exits static method).Now, let’s see a program to prove our point. Question 48. Suppose you have 2 threads (Thread-1 on object1 and Thread-2 on object2). Thread-1 is in synchronized method1(), can Thread-2 enter synchronized method2() at same time? Answer.Yes, here when Thread-1 is in synchronized method1() it must be holding lock on object1’s monitor. Thread-2 will acquire lock on object2’s monitor and enter synchronized method2(). Likewise, Thread-2 even enter synchronized method1() as well which is being executed by Thread-1 (because threads are created on different objects). Now, let’s see a program to prove our point. Question 49. Suppose you have 2 threads (Thread-1 on object1 and Thread-2 on object2). Thread-1 is in static synchronized method1(), can Thread-2 enter static synchronized method2() at same time? Answer.No, it might confuse you a bit that threads are created on different objects. But, not to forgot that multiple objects may exist but there is always one class’s class object lock available. Here, when Thread-1 is in static synchronized method1() it must be holding lock on class class’s object and will release lock on class’s classobject only when it exits static synchronized method1(). So, Thread-2 will have to wait for Thread-1 to release lock on class’s classobject so that it could enter static synchronized method2(). Likewise, Thread-2 even cannot enter static synchronized method1() which is being executed by Thread-1. Thread-2 will have to wait for Thread-1 to release lock on class’s classobject so that it could enter static synchronized method1(). Now, let’s see a program to prove our point. Question 50. Difference between wait() and wait(long timeout), What are thread states when these method are called? Answer. wait() wait(long timeout) When wait() method is called on object, it causes causes the current thread to wait until another thread invokes the notify() or notifyAll() method for this object. wait(long timeout) - Causes the current thread to wait until either another thread invokes the notify() or notifyAll() methods for this object, or a specified timeout time has elapsed. When wait() is called on object - Thread enters from running to waiting state. It waits for some other thread to call notify so that it could enter runnable state. When wait(1000) is called on object - Thread enters from running to waiting state. Than even if notify() or notifyAll() is not called after timeout time has elapsed thread will go from waiting to runnable state. Question 51. How can you implement your own Thread Pool in java? Answer. What is ThreadPool? ThreadPool is a pool of threads which reuses a fixed number of threads to execute tasks. At any point, at most nThreads threads will be active processing tasks. If additional tasks are submitted when all threads are active, they will wait in the queue until a thread is available. ThreadPool implementation internally uses LinkedBlockingQueue for adding and removing tasks. In this post i will be using LinkedBlockingQueue provide by java Api, you can refer this post for implementing ThreadPool using custom LinkedBlockingQueue. Need/Advantage of ThreadPool? Instead of creating new thread every time for executing tasks, we can create ThreadPool which reuses a fixed number of threads for executing tasks. As threads are reused, performance of our application improves drastically. How ThreadPool works? We will instantiate ThreadPool, in ThreadPool’s constructor nThreads number of threads are created and started. ThreadPool threadPool=new ThreadPool(2); Here 2 threads will be created and started in ThreadPool. Then, threads will enter run() method of ThreadPoolsThread class and will call take() method on taskQueue. If tasks are available thread will execute task by entering run() method of task (As tasks executed always implements Runnable). publicvoid run() { . . . while (true) { . . . Runnable runnable = taskQueue.take(); runnable.run(); . . . } . . . } Else waits for tasks to become available. When tasks are added? When execute() method of ThreadPool is called, it internally calls put() method on taskQueue to add tasks. taskQueue.put(task); Once tasks are available all waiting threads are notified that task is available. Question 52. What is significance of using ThreadLocal? Answer. This question will test your command in multi threading, can you really create some perfect multithreading application or not. ThreadLocal is a class which provides thread-local variables. What is ThreadLocal ? ThreadLocal is a class which provides thread-local variables. Every thread has its own ThreadLocal value that makes ThreadLocal value threadsafe as well. For how long Thread holds ThreadLocal value? Thread holds ThreadLocal value till it hasn’t entered dead state. Can one thread see other thread’s ThreadLocal value? No, thread can see only it’s ThreadLocal value. Are ThreadLocal variables thread safe. Why? Yes, ThreadLocal variables are thread safe. As every thread has its own ThreadLocal value and one thread can’t see other threads ThreadLocal value. Application of ThreadLocal? ThreadLocal are used by many web frameworks for maintaining some context (may be session or request) related value. In any single threaded application, same thread is assigned for every request made to same action, so ThreadLocal values will be available in next request as well. In multi threaded application, different thread is assigned for every request made to same action, so ThreadLocal values will be different for every request. When threads have started at different time they might like to store time at which they have started. So, thread’s start time can be stored in ThreadLocal. Creating ThreadLocal > private ThreadLocal threadLocal = new ThreadLocal(); We will create instance of ThreadLocal. ThreadLocal is a generic class, i will be using String to demonstrate threadLocal. All threads will see same instance of ThreadLocal, but a thread will be able to see value which was set by it only. How thread set value of ThreadLocal > threadLocal.set( new Date().toString()); Thread set value of ThreadLocal by calling set(“”) method on threadLocal. How thread get value of ThreadLocal > threadLocal.get() Thread get value of ThreadLocal by calling get() method on threadLocal. See here for detailed explanation of threadLocal. Question 53. What is busy spin? Answer. What is busy spin? When one thread loops continuously waiting for another thread to signal. Performance point of view - Busy spin is very bad from performance point of view, because one thread keeps on looping continuously ( and consumes CPU) waiting for another thread to signal. Solution to busy spin - We must use sleep() or wait() and notify() method. Using wait() is better option. Why using wait() and notify() is much better option to solve busy spin? Because in case when we use sleep() method, thread will wake up again and again after specified sleep time until boolean variable is true. But, in case of wait() thread will wake up only when when notified by calling notify() or notifyAll(), hence end up consuming CPU in best possible manner. Program - Consumer Producer problem with busy spin > Consumer thread continuously execute (busy spin) in while loop tillproductionInProcess is true. Once producer thread has ended it will make boolean variable productionInProcess false and busy spin will be over. while(productionInProcess){ System.out.println("BUSY SPIN - Consumer waiting for production to get over"); } Question 54. Can a constructor be synchronized? Answer. No, constructor cannot be synchronized. Because constructor is used for instantiating object, when we are in constructor object is under creation. So, until object is not instantiated it does not need any synchronization. Enclosing constructor in synchronized block will generate compilation error. Using synchronized in constructor definition will also show compilation error. COMPILATION ERROR = Illegal modifier for the constructor in type ConstructorSynchronizeTest; only public, protected & private are permitted Though we can use synchronized block inside constructor. Read More about : Constructor in java cannot be synchronized Question 55. Can you find whether thread holds lock on object or not? Answer. holdsLock(object) method can be used to find out whether current thread holds the lock on monitor of specified object. holdsLock(object) method returns true if the current thread holds the lock on monitor of specified object. Question 56. What do you mean by thread starvation? Answer. When thread does not enough CPU for its execution Thread starvation happens. Thread starvation may happen in following scenarios > Low priority threads gets less CPU (time for execution) as compared to high priority threads. Lower priority thread may starve away waiting to get enough CPU to perform calculations. In deadlock two threads waits for each other to release lock holded by them on resources. There both Threads starves away to get CPU. Thread might be waiting indefinitely for lock on object’s monitor (by calling wait() method), because no other thread is calling notify()/notifAll() method on object. In that case, Thread starves away to get CPU. Thread might be waiting indefinitely for lock on object’s monitor (by calling wait() method), but notify() may be repeatedly awakening some other threads. In that case also Thread starves away to get CPU. Question 57. What is addShutdownHook method in java? Answer. addShutdownHook method in java > addShutdownHook method registers a new virtual-machine shutdown hook. A shutdown hook is a initialized but unstarted thread. When JVM starts its shutdown it will start all registered shutdown hooks in some unspecified order and let them run concurrently. When JVM (Java virtual machine) shuts down > When the last non-daemon thread finishes, or when the System.exit is called. Once JVM’s shutdown has begunnew shutdown hook cannot be registered neither previously-registered hook can be de-registered. Any attempt made to do any of these operations causes an IllegalStateException. For more detail with program read : Threads addShutdownHook method in java Question 58. How you can handle uncaught runtime exception generated in run method? Answer. We can use setDefaultUncaughtExceptionHandler method which can handle uncaught unchecked(runtime) exception generated in run() method. What is setDefaultUncaughtExceptionHandler method? setDefaultUncaughtExceptionHandler method sets the default handler which is called when a thread terminates due to an uncaught unchecked(runtime) exception. setDefaultUncaughtExceptionHandler method features > setDefaultUncaughtExceptionHandler method sets the default handler which is called when a thread terminates due to an uncaught unchecked(runtime) exception. setDefaultUncaughtExceptionHandler is a static method method, so we can directly call Thread.setDefaultUncaughtExceptionHandler to set the default handler to handle uncaught unchecked(runtime) exception. It avoids abrupt termination of thread caused by uncaught runtime exceptions. Defining setDefaultUncaughtExceptionHandler method > Thread.setDefaultUncaughtExceptionHandler(new Thread.UncaughtExceptionHandler(){ publicvoid uncaughtException(Thread thread, Throwable throwable) { System.out.println(thread.getName() + " has thrown " + throwable); } }); Question 59. What is ThreadGroup in java, What is default priority of newly created threadGroup, mention some important ThreadGroup methods ? Answer. When program starts JVM creates a ThreadGroup named main. Unless specified, all newly created threads become members of the main thread group. ThreadGroup is initialized with default priority of 10. ThreadGroup important methods > getName() name of ThreadGroup. activeGroupCount() count of active groups in ThreadGroup. activeCount() count of active threads in ThreadGroup. list() list() method has prints ThreadGroups information getMaxPriority() Method returns the maximum priority of ThreadGroup. setMaxPriority(int pri) Sets the maximum priority of ThreadGroup. Question 60. What are thread priorities? Answer. Thread Priority range is from 1 to 10. Where 1 is minimum priority and 10 is maximum priority. Thread class provides variables of final static int type for setting thread priority. /* The minimum priority that a thread can have. */ publicfinalstaticintMIN_PRIORITY= 1; /* The default priority that is assigned to a thread. */ publicfinalstaticintNORM_PRIORITY= 5; /* The maximum priority that a thread can have. */ publicfinalstaticintMAX_PRIORITY= 10; Thread with MAX_PRIORITY is likely to get more CPU as compared to low priority threads. But occasionally low priority thread might get more CPU. Because thread scheduler schedules thread on discretion of implementation and thread behaviour is totally unpredictable. Thread with MIN_PRIORITY is likely to get less CPU as compared to high priority threads. But occasionally high priority thread might less CPU. Because thread scheduler schedules thread on discretion of implementation and thread behaviour is totally unpredictable. setPriority()method is used for Changing the priority of thread. getPriority()method returns the thread’s priority.

a few days ago, visual studio 2015 rc was released. among the many updates to .net framework 4.6 with this release, we now have some new utility methods allowing conversion to/from unix timestamps. although these were added primarily to enable more cross-platform support in .net core framework , unix timestamps are also sometimes useful in a windows environment. for instance, unix timestamps are often used to facilitate redis sorted sets where the score is a datetime (since the score can only be a double ). unix timestamp conversion before .net 4.6 until now, you had to implement conversions to/from unix time yourself. that actually isn’t hard to do. by definition , unix time is the number of seconds since 1st january 1970, 00:00:00 utc. thus we can convert from a local datetime to unix time as follows: var datetime = new datetime(2015, 05, 24, 10, 2, 0, datetimekind.local); var epoch = new datetime(1970, 1, 1, 0, 0, 0, datetimekind.utc); var unixdatetime = (datetime.touniversaltime() - epoch).totalseconds; we can convert back to a local datetime as follows: var timespan = timespan.fromseconds(unixdatetime); var localdatetime = new datetime(timespan.ticks).tolocaltime(); unix timestamp conversion in .net 4.6 quoting the visual studio 2015 rc release notes : new methods have been added to support converting datetime to or from unix time. the following apis have been added to datetimeoffset: static datetimeoffset fromunixtimeseconds(long seconds) static datetimeoffset fromunixtimemilliseconds(long milliseconds) long tounixtimeseconds() long tounixtimemilliseconds() so .net 4.6 gives us some new methods, but to use them, you’ll first have to convert from datetime to datetimeoffset. first, make sure you’re targeting the right version of the .net framework: you can then use the new methods: var datetime = new datetime(2015, 05, 24, 10, 2, 0, datetimekind.local); var datetimeoffset = new datetimeoffset(datetime); var unixdatetime = datetimeoffset.tounixtimeseconds(); …and to change back… var localdatetimeoffset = datetimeoffset.fromunixtimeseconds(unixdatetime) .datetime.tolocaltime();

A key to improving performance is benchmarking. Read about Microsoft Diskspd's tools for storage and server benchmarking, and boost your I/O performance.

The best way to write to a Cassandra cluster are concurrent asynchronous writes. In cases where data exhibits strong temporal locality, speed can be improved.

In this article I will go through a common set-up for a small production environment. A single tier, load balanced application server cluster. Overview A high level overview of what we will be doing. Downloading and installing Apache HTTP server and mod_jk Downloading Tomcat Downloading Java Configuring two local Tomcat servers Clustering the two Tomcat servers Configuring Apache to use mod_jk to forward request to Tomcat Deploying application to Tomcat server that tests our set-up Introduction What is Apache? Apache is an HTTP server. What is mod_jk? It is an Apache module that allows AJP communication between Apache and a back end application server like Tomcat.I am running this on Ubuntu 14.04LTS installed on a dual boot PC with Windows 7. Download Apache2 We are going to use Ubuntu's APT package maintenance system to obtain and install Apache2. sudo apt-get install apache2 This will install in /etc/apache2 Download and install mod_jk The mod_jk module is not included in the Apache2 download so must be obtained and installed separately. The installation requires that the mod_jk module is visible to Apache and configured to ensure that Apache knows where to look for it and what to do with the requests you want to proxy. sudo apt-get install libapache2-mod-jk This will install in /etc/libapache2-mod-jk also two files have been added to the /etc/apache2/mods-available folder. Downloading and installing Tomcat 8 At the time of writing this Tomcat 8 does not have a package in APT so you must download the binaries from the tomcat website.http://tomcat.apache.org/download-80.cgi select the appropriate binary distribution and extract it as follows. tar xvzf apache-tomcat-8.0.5.tar.gz We need two copies of the Tomcat server to be load balanced. I created two directories in the /opt/ location: /opt/tomcat-server1/ and /opt/tomcat-server2/ and copied tomcat into each one. Download and install Java Download Java from APT as follows: apt-get install openjdk-7-jdk and set JAVA_HOME in .bashrc vim ~/.bashrc export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 Configure two local Tomcat servers We will edit only the server.xml of the server2 installation of tomcat. We need to change port numbers to avoid conflicts.We change the following: and comment out the HTTP Connector as we only want the web application to be accessible through the load balancer.Here is my server2 Tomcat server.xml configuration. Configure mod_jk Load balancing is configured in the workers.properties file, located /etc/libapache2-mod-jk/ where workers represent actual or virtual workers.We will define two actual workers and two virtual workers which map to the Tomcat servers. In the worker.list property I have defined two virtual workers: status and loadbalancer, I will refer to these later in the Apache configuration.Workers for each server have been defined using values for the server.xml configuration files. I used the port values for the AJP connectors and I have included an lbfactor that sets the preference that the load balancer will show for that server.Finally we define the virtual workers. The loadbalancer worker is set to type lb and set the workers that represent the Tomcat servers in the balancer_workers properties. The status only needs to be set to type status. worker.list=loadbalancer,status worker.server1.port=8009worker.server1.host=localhostworker.server1.type=ajp13 worker.server2.port=9009worker.server2.host=localhostworker.server2.type=ajp13 worker.server1.lbfactor=1worker.server2.lbfactor=1 worker.loadbalancer.type=lbworker.loadbalancer.balance_workers=server1,server2 worker.status.type=status Ensure that you remove any other worker configuration that are not being used. Configure Apache Web Server to forward requests You will need to add the following to the Apache configurations located in etc/apache2/sites-enabled/000-default.conf JkMount /status status JkMount /* loadbalancer Verify the installation To test that all has been configured correctly we need to deploy an application. A sample application that has been used for years to test such configurations is called the ClusterJSP sample application. You can find it by googling in or from the JBoss site.Now deploy the war to the webapps folder on both servers and start each server using the start-up script /opt/tomcat-server1/bin/startup.sh.Go to http://localhost/clusterjsp/HaJsp.jsp and you should see the page show HttpSession information. Now lets look at the mod_jk status page: http://localhost/status. You will see that this page shows information about the load balancer workers and the workers it is balancing. If everything is working you will see the worker error state show OK or OK/IDLE if they are not currently balancing load. Things to try out Enable sticky sessions: Configure jvmRoute in the server.xml configuration. Further reading Loadbalancing with mod_jk and ApacheWorking with mod_jk Connecting Apache's Web Server to Multiple Instances of Tomcat

I've been playing around with Docker a lot lately. Many reasons for that, one for sure is, that I love to play around with latest technology and even help out to build a demo or two or a lab. The main difference, between what everybody else of my coworkers is doing is, that I run my setup on Windows. Like most of the middleware developers out there. So, If you followed Arun's blog about "Docker Machine to Setup Docker Host" you might have tried to make this work on windows already. Here is the ultimate short how-to guide on using Docker Machine to administrate and spin up your Docker hosts. Docker Machine Machine lets you create Docker hosts on your computer, on cloud providers, and inside your own data center. It creates servers, installs Docker on them, then configures the Docker client to talk to them. You basically don't have to have anything installed on your machine prior to this. Which is a hell lot easier, than having to manually install boot2docker before. So, let's try this out. You want to have at least one thing in place before starting with anything Docker or Machine. Go and get Git for Windows (aka msysgit). It has all kinds of helpful unix tools in his belly, which you need anyway. Prerequisites - The One For All Solution The first is to install the windows boot2docker distribution which I showed in an earlier blog. It contains the following bits configured and ready for you to use: - VirtualBox - Docker Windows Client Prerequisites- The Bits And Pieces I dislike the boot2docker installer for a variety of reasons. Mostly, because I want to know what exactly is going on on my machine. So I played around a bit and here is the bits and pieces installer if you decide against the one-for-all solution. Start with the virtualization solution. We need something like that on Windows, because it just can't run Linux and this is what Docker is based on. At least for now. So, get VirtualBox and ensure that version 4.3.18 is correctly installed on your system (VirtualBox-4.3.18-96516-Win.exe, 105 MB). WARNING: There is a strange issue, when you run Windows itself in Virtualbox. You might run into an issue with starting the host. And while you're at it, go and get the Docker Windows Client. The other is to grab the final from the test servers as a direct download (docker-1.6.0.exe, x86_64, 7.5MB). Rename to "docker" and put it into a folder of your choice (I assume it will be c:\docker\. Now you also need to download Docker Machine, which is another single executable (docker-machine_windows-amd64.exe, 11.5MB). Rename to "docker-machine" and put it into the same folder. Now add this folder to your PATH: set PATH=%PATH%;C:\docker If you change your standard PATH environment variable, this might safe your from a lot of typing. That's it. Now you're ready to create your first Machine managed Docker Host. Create Your Docker Host With Machine All you need is a simple command: docker-machine create --driver virtualbox dev And the output should state: ←[34mINFO←[0m[0000] Creating SSH key... ←[34mINFO←[0m[0001] Creating VirtualBox VM... ←[34mINFO←[0m[0016] Starting VirtualBox VM... ←[34mINFO←[0m[0022] Waiting for VM to start... ←[34mINFO←[0m[0076] "dev" has been created and is now the active machine. ←[34mINFO←[0m[0076] To point your Docker client at it, run this in your shell: eval "$(docker-machine.exe env dev)" This means, you just created a Docker Host using the VirtualBox provider and the name “dev”. Now you need to find out on which IP address the host is running. docker-machine ip 192.168.99.102 If you want to configure your environment variables, needed by the client more easy, just use the following command: docker-machine env dev export DOCKER_TLS_VERIFY=1 export DOCKER_CERT_PATH="C:\\Users\\markus\\.docker\\machine\\machines\\dev" export DOCKER_HOST=tcp://192.168.99.102:2376 Which outputs the Linux version of environment variable definition. All you have to do is to change the "export" keyword to "set", remove the " and the double back-slashes and you are ready to go. C:\Users\markus\Downloads>set DOCKER_TLS_VERIFY=1 C:\Users\markus\Downloads>set DOCKER_CERT_PATH=C:\Users\markus\.docker\machine\machines\dev C:\Users\markus\Downloads>set DOCKER_HOST=tcp://192.168.99.102:2376 Time to test our Docker Client And here we go now run WildFly on your freshly created host: docker run -it -p 8080:8080 jboss/wildfly Watch the container being downloaded and check, that it is running by redirecting your browser to http://192.168.99.102:8080/. Congratulations on having setup your very first docker host with Maschine on Windows.

In containers and microservices, we’re facing the greatest potential change in how we deliver and run software services since the arrival of virtual machines.

Written by Dave Syer on the Spring blog In this article we look at how to bind a Spring Boot application to data services (JDBC, NoSQL, messaging etc.) and the various sources of default and automatic behaviour in Cloud Foundry, providing some guidance about which ones to use and which ones will be active under what conditions. Spring Boot provides a lot of autoconfiguration and external binding features, some of which are relevant to Cloud Foundry, and many of which are not. Spring Cloud Connectors is a library that you can use in your application if you want to create your own components programmatically, but it doesn’t do anything “magical” by itself. And finally there is the Cloud Foundry java buildpack which has an “auto-reconfiguration” feature that tries to ease the burden of moving simple applications to the cloud. The key to correctly configuring middleware services, like JDBC or AMQP or Mongo, is to understand what each of these tools provides, how they influence each other at runtime, and and to switch parts of them on and off. The goal should be a smooth transition from local execution of an application on a developer’s desktop to a test environment in Cloud Foundry, and ultimately to production in Cloud Foundry (or otherwise) with no changes in source code or packaging, per the twelve-factor application guidelines. There is some simple source code accompanying this article. To use it you can clone the repository and import it into your favourite IDE. You will need to remove two dependencies from the complete project to get to the same point where we start discussing concrete code samples, namely spring-boot-starter-cloud-connectors and auto-reconfiguration. NOTE: The current co-ordinates for all the libraries being discussed are org.springframework.boot:spring-boot-*:1.2.3.RELEASE,org.springframework.boot:spring-cloud-*-connector:1.1.1.RELEASE,org.cloudfoundry:auto-reconfiguration:1.7.0.RELEASE. TIP: The source code in github includes a docker-compose.yml file (docs here). You can use that to create a local MySQL database if you don’t have one running already. You don’t actually need it to run most of the code below, but it might be useful to validate that it will actually work. Punchline for the Impatient If you want to skip the details, and all you need is a recipe for running locally with H2 and in the cloud with MySQL, then start here and read the rest later when you want to understand in more depth. (Similar options exist for other data services, like RabbitMQ, Redis, Mongo etc.) Your first and simplest option is to simply do nothing: do not define a DataSource at all but put H2 on the classpath. Spring Boot will create the H2 embedded DataSource for you when you run locally. The Cloud Foundry buildpack will detect a database service binding and create a DataSource for you when you run in the cloud. If you add Spring Cloud Connectors as well, your app will also work in other cloud platforms, as long as you include a connector. That might be good enough if you just want to get something working. If you want to run a serious application in production you might want to tweak some of the connection pool settings (e.g. the size of the pool, various timeouts, the important test on borrow flag). In that case the buildpack auto-reconfiguration DataSource will not meet your requirements and you need to choose an alternative, and there are a number of more or less sensible choices. The best choice is probably to create a DataSource explicitly using Spring Cloud Connectors, but guarded by the “cloud” profile: @Configuration @Profile("cloud") public class DataSourceConfiguration { @Bean public Cloud cloud() { return new CloudFactory().getCloud(); } @Bean @ConfigurationProperties(DataSourceProperties.PREFIX) public DataSource dataSource() { return cloud().getSingletonServiceConnector(DataSourceclass, null); } } You can use spring.datasource.* properties (e.g. in application.properties or a profile-specific version of that) to set the additional properties at runtime. The “cloud” profile is automatically activated for you by the buildpack. Now for the details. We need to build up a picture of what’s going on in your application at runtime, so we can learn from that how to make a sensible choice for configuring data services. Layers of Autoconfiguration Let’s take a a simple app with DataSource (similar considerations apply to RabbitMQ, Mongo, Redis): @SpringBootApplication public class CloudApplication { @Autowired private DataSource dataSource; public static void main(String[] args) { SpringApplication.run(CloudApplication.class, args); } } This is a complete application: the DataSource can be @Autowired because it is created for us by Spring Boot. The details of the DataSource (concrete class, JDBC driver, connection URL, etc.) depend on what is on the classpath. Let’s assume that the application uses Spring JDBC via the spring-boot-starter-jdbc (or spring-boot-starter-data-jpa), so it has aDataSource implementation available from Tomcat (even if it isn’t a web application), and this is what Spring Boot uses. Consider what happens when: Classpath contains H2 (only) in addition to the starters: the DataSource is the Tomcat high-performance pool from DataSourceAutoConfiguration and it connects to an in memory database “testdb”. Classpath contains H2 and MySQL: DataSource is still H2 (same as before) because we didn’t provide any additional configuration for MySQL and Spring Boot can’t guess the credentials for connecting. Add spring-boot-starter-cloud-connectors to the classpath: no change inDataSource because the Spring Cloud Connectors do not detect that they are running in a Cloud platform. The providers that come with the starter all look for specific environment variables, which they won’t find unless you set them, or run the app in Cloud Foundry, Heroku, etc. Run the application in “cloud” profile with spring.profiles.active=cloud: no change yet in the DataSource, but this is one of the things that the Java buildpack does when your application runs in Cloud Foundry. Run in “cloud” profile and provide some environment variables to simulate running in Cloud Foundry and binding to a MySQL service: VCAP_APPLICATION={"name":"application","instance_id":"FOO"} VCAP_SERVICES={"mysql":[{"name":"mysql","tags":["mysql"],"credentials":{"uri":"mysql://root:root@localhost/test"}]} (the “tags” provides a hint that we want to create a MySQL DataSource, the “uri” provides the location, and the “name” becomes a bean ID). The DataSource is now using MySQL with the credentials supplied by the VCAP_* environment variables. Spring Boot has some autoconfiguration for the Connectors, so if you looked at the beans in your application you would see a CloudFactory bean, and also the DataSource bean (with ID “mysql”). Theautoconfiguration is equivalent to adding @ServiceScan to your application configuration. It is only active if your application runs in the “cloud” profile, and only if there is no existing @Bean of type Cloud, and the configuration flagspring.cloud.enabled is not “false”. Add the “auto-reconfiguration” JAR from the Java buildpack (Maven co-ordinatesorg.cloudfoundry:auto-reconfiguration:1.7.0.RELEASE). You can add it as a local dependency to simulate running an application in Cloud Foundry, but it wouldn’t be normal to do this with a real application (this is just for experimenting with autoconfiguration). The auto-reconfiguration JAR now has everything it needs to create a DataSource, but it doesn’t (yet) because it detects that you already have a bean of type CloudFactory, one that was added by Spring Boot. Remove the explicit “cloud” profile. The profile will still be active when your app starts because the auto-reconfiguration JAR adds it back again. There is still no change to theDataSource because Spring Boot has created it for you via the @ServiceScan. Remove the spring-boot-starter-cloud-connectors dependency, so that Spring Boot backs off creating a CloudFactory. The auto-reconfiguration JAR actually has its own copy of Spring Cloud Connectors (all the classes with different package names) and it now uses them to create a DataSource (in a BeanFactoryPostProcessor). The Spring Boot autoconfigured DataSource is replaced with one that binds to MySQL via theVCAP_SERVICES. There is no control over pool properties, but it does still use the Tomcat pool if available (no support for Hikari or DBCP2). Remove the auto-reconfiguration JAR and the DataSource reverts to H2. TIP: use web and actuator starters with endpoints.health.sensitive=false to inspect the DataSource quickly through “/health”. You can also use the “/beans”, “/env” and “/autoconfig” endpoints to see what is going in in the autoconfigurations and why. NOTE: Running in Cloud Foundry or including auto-reconfiguration JAR in classpath locally both activate the “cloud” profile (for the same reason). The VCAP_* env vars are the thing that makes Spring Cloud and/or the auto-reconfiguration JAR create beans. NOTE: The URL in the VCAP_SERVICES is actually not a “jdbc” scheme, which should be mandatory for JDBC connections. This is, however, the format that Cloud Foundry normally presents it in because it works for nearly every language other than Java. Spring Cloud Connectors or the buildpack auto-reconfiguration, if they are creating a DataSource, will translate it into a jdbc:* URL for you. NOTE: The MySQL URL also contains user credentials and a database name which are valid for the Docker container created by the docker-compose.yml in the sample source code. If you have a local MySQL server with different credentials you could substitute those. TIP: If you use a local MySQL server and want to verify that it is connected, you can use the “/health” endpoint from the Spring Boot Actuator (included in the sample code already). Or you could create a schema-mysql.sql file in the root of the classpath and put a simple keep alive query in it (e.g. SELECT 1). Spring Boot will run that on startupso if the app starts successfully you have configured the database correctly. The auto-reconfiguration JAR is always on the classpath in Cloud Foundry (by default) but it backs off creating any DataSource if it finds a org.springframework.cloud.CloudFactorybean (which is provided by Spring Boot if the CloudAutoConfiguration is active). Thus the net effect of adding it to the classpath, if the Connectors are also present in a Spring Boot application, is only to enable the “cloud” profile. You can see it making the decision to skip auto-reconfiguration in the application logs on startup: 015-04-14 15:11:11.765 INFO 12727 --- [ main] urceCloudServiceBeanFactoryPostProcessor : Skipping auto-reconfiguring beans of type javax.sql.DataSource 2015-04-14 15:11:57.650 INFO 12727 --- [ main] ongoCloudServiceBeanFactoryPostProcessor : Skipping auto-reconfiguring beans of type org.springframework.data.mongodb.MongoDbFactory 2015-04-14 15:11:57.650 INFO 12727 --- [ main] bbitCloudServiceBeanFactoryPostProcessor : Skipping auto-reconfiguring beans of type org.springframework.amqp.rabbit.connection.ConnectionFactory 2015-04-14 15:11:57.651 INFO 12727 --- [ main] edisCloudServiceBeanFactoryPostProcessor : Skipping auto-reconfiguring beans of type org.springframework.data.redis.connection.RedisConnectionFactory ... etc. Create your own DataSource The last section walked through most of the important autoconfiguration features in the various libraries. If you want to take control yourself, one thing you could start with is to create your own instance of DataSource. You could do that, for instance, using aDataSourceBuilder which is a convenience class and comes as part of Spring Boot (it chooses an implementation based on the classpath): @SpringBootApplication public class CloudApplication { @Bean public DataSource dataSource() { return DataSourceBuilder.create().build(); } ... } The DataSource as we’ve defined it is useless because it doesn’t have a connection URL or any credentials, but that can easily be fixed. Let’s run this application as if it was in Cloud Foundry: with the VCAP_* environment variables and the auto-reconfiguration JAR but not Spring Cloud Connectors on the classpath and no explicit “cloud” profile. The buildpack activates the “cloud” profile, creates a DataSource and binds it to the VCAP_SERVICES. As already described briefly, it removes your DataSource completely and replaces it with a manually registered singleton (which doesn’t show up in the “/beans” endpoint in Spring Boot). Now add Spring Cloud Connectors back into the classpath the application and see what happens when you run it again. It actually fails on startup! What has happened? The@ServiceScan (from Connectors) goes and looks for bound services, and creates bean definitions for them. That’s a bit like the buildpack, but different because it doesn’t attempt to replace any existing bean definitions of the same type. So you get an autowiring error because there are 2 DataSources and no way to choose one to inject into your application in various places where one is needed. To fix that we are going to have to take control of the Cloud Connectors (or simply not use them). Using a CloudFactory to create a DataSource You can disable the Spring Boot autoconfiguration and the Java buildpack auto-reconfiguration by creating your own Cloud instance as a @Bean: @Bean public Cloud cloud() { return new CloudFactory().getCloud(); } @Bean @ConfigurationProperties(DataSourceProperties.PREFIX) public DataSource dataSource() { return cloud().getSingletonServiceConnector(DataSource.class, null); } Pros: The Connectors autoconfiguration in Spring Boot backed off so there is only oneDataSource. It can be tweaked using application.properties via spring.datasource.*properties, per the Spring Boot User Guide. Cons: It doesn’t work without VCAP_* environment variables (or some other cloud platform). It also relies on user remembering to ceate the Cloud as a @Bean in order to disable the autoconfiguration. Summary: we are still not in a comfortable place (an app that doesn’t run without some intricate wrangling of environment variables is not much use in practice). Dual Running: Local with H2, in the Cloud with MySQL There is a local configuration file option in Spring Cloud Connectors, so you don’t have to be in a real cloud platform to use them, but it’s awkward to set up despite being boiler plate, and you also have to somehow switch it off when you are in a real cloud platform. The last point there is really the important one because you end up needing a local file to run locally, but only running locally, and it can’t be packaged with the rest of the application code (for instance violates the twelve factor guidelines). So to move forward with our explicit @Bean definition it’s probably better to stick to mainstream Spring and Spring Boot features, e.g. using the “cloud” profile to guard the explicit creation of a DataSource: @Configuration @Profile("cloud") public class DataSourceConfiguration { @Bean public Cloud cloud() { return new CloudFactory().getCloud(); } @Bean @ConfigurationProperties(DataSourceProperties.PREFIX) public DataSource dataSource() { return cloud().getSingletonServiceConnector(DataSource.class, null); } } With this in place we have a solution that works smoothly both locally and in Cloud Foundry. Locally Spring Boot will create a DataSource with an H2 embedded database. In Cloud Foundry it will bind to a singleton service of type DataSource and switch off the autconfigured one from Spring Boot. It also has the benefit of working with any platform supported by Spring Cloud Connectors, so the same code will run on Heroku and Cloud Foundry, for instance. Because of the @ConfigurationProperties you can bind additional configuration to the DataSource to tweak connection pool properties and things like that if you need to in production. NOTE: We have been using MySQL as an example database server, but actually PostgreSQL is at least as compelling a choice if not more. When paired with H2 locally, for instance, you can put H2 into its “Postgres compatibility” mode and use the same SQL in both environments. Manually Creating a Local and a Cloud DataSource If you like creating DataSource beans, and you want to do it both locally and in the cloud, you could use 2 profiles (“cloud” and “local”), for example. But then you would have to find a way to activate the “local” profile by default when not in the cloud. There is already a way to do that built into Spring because there is always a default profile called “default” (by default). So this should work: @Configuration @Profile("default") // or "!cloud" public class LocalDataSourceConfiguration { @Bean @ConfigurationProperties(DataSourceProperties.PREFIX) public DataSource dataSource() { return DataSourceBuilder.create().build(); } } @Configuration @Profile("cloud") public class CloudDataSourceConfiguration { @Bean public Cloud cloud() { return new CloudFactory().getCloud(); } @Bean @ConfigurationProperties(DataSourceProperties.PREFIX) public DataSource dataSource() { return cloud().getSingletonServiceConnector(DataSource.class, null); } } The “default” DataSource is actually identical to the autoconfigured one in this simple example, so you wouldn’t do this unless you needed to, e.g. to create a custom concreteDataSource of a type not supported by Spring Boot. You might think it’s all getting a bit complicated, but in fact Spring Boot is not making it any harder, we are just dealing with the consequences of needing to control the DataSource construction in 2 environments. Using a Non-Embedded Database Locally If you don’t want to use H2 or any in-memory database locally, then you can’t really avoid having to configure it (Spring Boot can guess a lot from the URL, but it will need that at least). So at a minimum you need to set some spring.datasource.* properties (the URL for instance). That that isn’t hard to do, and you can easily set different values in different environments using additional profiles, but as soon as you do that you need to switch off the default values when you go into the cloud. To do that you could define thespring.datasource.* properties in a profile-specific file (or document in YAML) for the “default” profile, e.g. application-default.properties, and these will not be used in the “cloud” profile. A Purely Declarative Approach If you prefer not to write Java code, or don’t want to use Spring Cloud Connectors, you might want to try and use Spring Boot autoconfiguration and external properties (or YAML) files for everything. For example Spring Boot creates a DataSource for you if it finds the right stuff on the classpath, and it can be completely controlled through application.properties, including all the granular features on the DataSource that you need in production (like pool sizes and validation queries). So all you need is a way to discover the location and credentials for the service from the environment. The buildpack translates Cloud Foundry VCAP_*environment variables into usable property sources in the Spring Environment. Thus, for instance, a DataSource configuration might look like this: spring.datasource.url: ${cloud.services.mysql.connection.jdbcurl:jdbc:h2:mem:testdb} spring.datasource.username: ${cloud.services.mysql.connection.username:sa} spring.datasource.password: ${cloud.services.mysql.connection.password:} spring.datasource.testOnBorrow: true The “mysql” part of the property names is the service name in Cloud Foundry (so it is set by the user). And of course the same pattern applies to all kinds of services, not just a JDBCDataSource. Generally speaking it is good practice to use external configuration and in particular @ConfigurationProperties since they allow maximum flexibility, for instance to override using System properties or environment variables at runtime. Note: similar features are provided by Spring Boot, which provides vcap.services.*instead of cloud.services.*, so you actually end up with more than one way to do this. However, the JDBC urls are not available from the vcap.services.* properties (non-JDBC services work fine with tthe corresponding vcap.services.*credentials.url). One limitation of this approach is it doesn’t apply if the application needs to configure beans that are not provided by Spring Boot out of the box (e.g. if you need 2 DataSources), in which case you have to write Java code anyway, and may or may not choose to use properties files to parameterize it. Before you try this yourself, though, beware that actually it doesn’t work unless you also disable the buildpack auto-reconfiguration (and Spring Cloud Connectors if they are on the classpath). If you don’t do that, then they create a new DataSource for you and Spring Boot cannot bind it to your properties file. Thus even for this declarative approach, you end up needing an explicit @Bean definition, and you need this part of your “cloud” profile configuration: @Configuration @Profile("cloud") public class CloudDataSourceConfiguration { @Bean public Cloud cloud() { return new CloudFactory().getCloud(); } } This is purely to switch off the buildpack auto-reconfiguration (and the Spring Boot autoconfiguration, but that could have been disabled with a properties file entry). Mixed Declarative and Explicit Bean Definition You can also mix the two approaches: declare a single @Bean definition so that you control the construction of the object, but bind additional configuration to it using@ConfigurationProperties (and do the same locally and in Cloud Foundry). Example: @Configuration public class LocalDataSourceConfiguration { @Bean @ConfigurationProperties(DataSourceProperties.PREFIX) public DataSource dataSource() { return DataSourceBuilder.create().build(); } } (where the DataSourceBuilder would be replaced with whatever fancy logic you need for your use case). And the application.properties would be the same as above, with whatever additional properties you need for your production settings. A Third Way: Discover the Credentials and Bind Manually Another approach that lends itself to platform and environment independence is to declare explicit bean definitions for the @ConfigurationProperties beans that Spring Boot uses to bind its autoconfigured connectors. For instance, to set the default values for a DataSourceyou can declare a @Bean of type DataSourceProperties: @Bean @Primary public DataSourceProperties dataSourceProperties() { DataSourceProperties properties = new DataSourceProperties(); properties.setInitialize(false); return properties; } This sets a default value for the “initialize” flag, and allows other properties to be bound fromapplication.properties (or other external properties). Combine this with the Spring Cloud Connectors and you can control the binding of the credentials when a cloud service is detected: @Autowired(required="false") Cloud cloud; @Bean @Primary public DataSourceProperties dataSourceProperties() { DataSourceProperties properties = new DataSourceProperties(); properties.setInitialize(false); if (cloud != null) { List infos = cloud.getServiceInfos(RelationalServiceInfo.class); if (infos.size()==1) { RelationalServiceInfo info = (RelationalServiceInfo) infos.get(0); properties.setUrl(info.getJdbcUrl()); properties.setUsername(info.getUserName()); properties.setPassword(info.getPassword()); } } return properties; } and you still need to define the Cloud bean in the “cloud” profile. It ends up being quite a lot of code, and is quite unnecessary in this simple use case, but might be handy if you have more complicated bindings, or need to implement some logic to choose a DataSource at runtime. Spring Boot has similar *Properties beans for the other middleware you might commonly use (e.g. RabbitProperties, RedisProperties, MongoProperties). An instance of such a bean marked as @Primary is enough to reset the defaults for the autoconfigured connector. Deploying to Multiple Cloud Platforms So far, we have concentrated on Cloud Foundry as the only cloud platform in which to deploy the application. One of the nice features of Spring Cloud Connectors is that it supports other platforms, either out of the box or as extension points. Thespring-boot-starter-cloud-connectors even includes Heroku support. If you do nothing at all, and rely on the autoconfiguration (the lazy programmer’s approach), then your application will be deployable in all clouds where you have a connector on the classpath (i.e. Cloud Foundry and Heroku if you use the starter). If you take the explicit @Bean approach then you need to ensure that the “cloud” profile is active in the non-Cloud Foundry platforms, e.g. through an environment variable. And if you use the purely declarative approach (or any combination involving properties files) you need to activate the “cloud” profile and probably also another profile specific to your platform, so that the right properties files end up in theEnvironment at runtime. Summary of Autoconfiguration and Provided Behaviour Spring Boot provides DataSource (also RabbitMQ or Redis ConnectionFactory, Mongo etc.) if it finds all the right stuff on the classpath. Using the “spring-boot-starter-*” dependencies is sufficient to activate the behaviour. Spring Boot also provides an autowirable CloudFactory if it finds Spring Cloud Connectors on the classpath (but switches off only if it finds a @Bean of type Cloud). The CloudAutoConfiguration in Spring Boot also effectively adds a @CloudScan to your application, which you would want to switch off if you ever needed to create your ownDataSource (or similar). The Cloud Foundry Java buildpack detects a Spring Boot application and activates the “cloud” profile, unless it is already active. Adding the buildpack auto-reconfiguration JAR does the same thing if you want to try it locally. Through the auto-reconfiguration JAR, the buildpack also kicks in and creates aDataSource (ditto RabbitMQ, Redis, Mongo etc.) if it does not find a CloudFactory bean or a Cloud bean (amongst others). So including Spring Cloud Connectors in a Spring Boot application switches off this part of the “auto-reconfiguration” behaviour (the bean creation). Switching off the Spring Boot CloudAutoConfiguration is easy, but if you do that, you have to remember to switch off the buildpack auto-reconfiguration as well if you don’t want it. The only way to do that is to define a bean definition (can be of type Cloud orCloudFactory for instance). Spring Boot binds application.properties (and other sources of external properties) to@ConfigurationProperties beans, including but not limited to the ones that it autoconfigures. You can use this feature to tweak pool properties and other settings that need to be different in production environments. General Advice and Conclusion We have seen quite a few options and autoconfigurations in this short article, and we’ve only really used thee libraries (Spring Boot, Spring Cloud Connectors, and the Cloud Foundry buildpack auto-reconfiguration JAR) and one platform (Cloud Foundry), not counting local deployment. The buildpack features are really only useful for very simple applications because there is no flexibility to tune the connections in production. That said it is a nice thing to be able to do when prototyping. There are only three main approaches if you want to achieve the goal of deploying the same code locally and in the cloud, yet still being able to make necessary tweaks in production: Use Spring Cloud Connectors to explicitly create DataSource and other middleware connections and protect those @Beans with @Profile("cloud"). The approach always works, but leads to more code than you might need for many applications. Use the Spring Boot default autoconfiguration and declare the cloud bindings usingapplication.properties (or in YAML). To take full advantage you have to expliccitly switch off the buildpack auto-reconfiguration as well. Use Spring Cloud Connectors to discover the credentials, and bind them to the Spring Boot@ConfigurationProperties as default values if present. The three approaches are actually not incompatible, and can be mixed using@ConfigurationProperties to provide profile-specific overrides of default configuration (e.g. for setting up connection pools in a different way in a production environment). If you have a relatively simple Spring Boot application, the only way to choose between the approaches is probably personal taste. If you have a non-Spring Boot application then the explicit @Bean approach will win, and it may also win if you plan to deploy your application in more than one cloud platform (e.g. Heroku and Cloud Foundry). NOTE: This blog has been a journey of discovery (who knew there was so much to learn?). Thanks go to all those who helped with reviews and comments, in particularScott Frederick, who spotted most of the mistakes in the drafts and always had time to look at a new revision.

[This article by Chris Haddad comes to you from the DZone Guide to Cloud Development - 2015 Edition. For more information—including in-depth articles from industry experts, best solutions for PaaS, iPaaS, IaaS, and MBaaS, and more—click the link below to download your free copy of the guide.] Microservices can be understood from two angles. First, the differential: teams that take a microservice design approach divide business solutions into distinct, full-stack business services owned by autonomous teams. Second, the integral: microservice-based applications weave multiple atomic microservices into holistic user experiences. Unfortunately, traditional application delivery models and traditional middleware infrastructure do not address microservice-specific demands for on-demand provisioning, dynamic composition, and service level management. On the other hand, the Platform-as-a-Service (PaaS) model addresses these demands perfectly. Running microservices on a PaaS fabric decreases solution fragility, reduces operational burden, and enhances developer productivity. To understand why, we’ll first review how microservices separate concerns from both business and object-oriented design perspectives. Second, we’ll consider how microservice-based design can complicate deployment as applications scale dynamically. Third, we’ll focus on how a PaaS environment helps to solve many of the problems both addressed and introduced by microservices-based architectures — in other words, why PaaS and microservices are a match made in heaven. Microservices: Separating Concerns By Business Solution A microservice approach decomposes monolithic applications according to the single responsibility pattern. In a microservice solution, each microservice interface delivers discrete business capabilities (e.g. customer profile, product catalogue, inventory, order, billing, fulfillment) within a well-defined, bounded context. The atomic microservice interfaces reside on separate and distinct full-stack application platforms that contain separate database storage, integration flows, and web application hosting. By separating concerns onto separate full-stack platforms and not sharing database instances or web application hosts across services, every team is free to choose different runtime languages and frameworks for its own microservice. Also, every team is free to evolve its data schemas, application frameworks, and business logic without impacting other teams. Because microservices are a relatively new design approach, many development teams may have the misconception that creating a microservice-based solution requires simply deploying small web services in containers. But this doesn’t cut quite deep enough. The correct approach is to evolve your monolithic design by applying service-oriented principles (i.e. encapsulation, loose coupling, separation of concerns) in conjunction with domain-driven design techniques and dynamic runtime application composition. For example, in a typical ecommerce scenario, a development team applies the bounded context pattern and single responsibility pattern to refactor a monolithic application into units distinguished by business capability (see Figure 2). By creating a user experience from loosely coupled services instead of tightly coupled native-language business objects, teams have more independence to develop, evolve, and deploy each business capability separately. Obviously, the microservice design approach works best for (a) greenfield projects or (b) modernization efforts where teams focus on refactoring monolithic application assets. The Microservice Execution Trap Although a microservice approach decouples development dependencies and speeds up development iterations, microservices also create a challenging environment for high-performance scaling and reliable runtime execution. More complex, loosely coupled, and dynamic environments distribute business capabilities over the entire network. Even a task as simple as responding to a single web application page request may spread out across several microservice instances residing on a distributed network topology. Martin Fowler and Stefan Tilkov (both microservice proponents) warn teams that successfully implementing a microservice approach requires choosing platforms that decrease solution fragility and reduce operational burdens. What Platform-as-a-Service Offers Platform-as-a-Service environments reduce microservice operational burdens when infrastructure-as-code and declarative policies are used to eliminate all manual actions and increase runtime quality of service (i.e. reliability, availability, scalability, and performance). The appropriate PaaS environment will automatically deploy, provision, and link full-stack microservices. In a microservice architecture, teams want to rapidly release new versions and perform A/B testing across versions. When teams define instance dependencies, scaling properties, and security policies as PaaS metadata or code scripts, the runtime fabric can reduce manual effort and increase release confidence. With a DevOps- friendly PaaS, the team can experiment with new service versions and safely rollback to a prior stable release if a problem arises. Because microservices are full-stack silos *1* that can be composed of multiple server instances (e.g. web server, database, load balancer, integration server), a PaaS can reduce deployment complexity by automatically spinning up and linking all instances. Linking may require discovering instance locations, dynamically initializing network routes, and auto-configuring connection strings based on service version or tenant. A traditional application will compose business functions and user experience by statically linking class files and shared object libraries. In contrast, microservice- based applications use service composition to connect available microservices endpoints and realize a fully functional application. While many microservice proponents promote microservice-based interactions by “smart endpoints through dumb pipes, ‘ effective service composition requires smart infrastructure building blocks to bootstrap and maintain connections between services and consumers. The right PaaS solves these problems. Infrastructure building blocks will register service endpoint locations, associate metadata and policies, connect clients, circuit break around failures, correlate inter-service calls, and load balance traffic. A microservice-friendly PaaS will provide service registries, metadata services, discovery services, and service virtualization gateways. In the pipe, circuit breakers will automatically route traffic on failover or overload. Smart endpoint code will dynamically connect with microservices based on discovery service responses and negotiated quality of service parameters. Rather than being hard-coded to a specific service hostname and URI, endpoint code will query for microservice location based on security assurances, performance guarantees, traffic load, service version, client tenancy, or business domain. When services are unavailable or underperform, smart endpoints will follow the tolerant reader pattern and gracefully degrade experience or proactively recover. A few recovery options include reading from local caches or circuit tripping to backup service endpoints. In conjunction with smart endpoint actions, a smart PaaS will spin up new microservice endpoints and full-stack instances based on service level management metrics. By following microservice architecture best practices, teams create anti-fragile applications that not only withstand a shock, but also improve performance and quality of service when stressed or experiencing failures. To drive this non-intuitive behavior, the underlying platform environment must be ready to scale, repair, and reconnect services. PaaS service level management components will create more resilient and anti-fragile microservices by monitoring performance, elastically provisioning instances, and dynamically re-routing traffic. Scaling an anti-fragile microservice is more difficult than scaling a web application. The PaaS should distribute microservice instances across multiple availability zones and dynamically adjust traffic to reduce latency and response time. Because transient microservice instances will rapidly start, stop, and change location, the service management layer must be completely automated and integrated with routing services. A PaaS environment will deliver the service level management, dynamic service composition, circuit breakers, and on-demand provisioning functions required to overcome the complexity inherent within a distributed microservice-based application architecture. Running microservices on a PaaS fabric will decrease solution fragility, reduce operational burden, and enhance developer productivity. If you are pursuing a microservice design approach, make sure you choose a microservice- friendly PaaS. DOWNLOAD YOUR FREE COPY TODAY

Earlier this month, I explored ZeroMQ and how it proves to be a promising solution for building fast, high-throughput, and scalable distributed systems. Despite lending itself quite well to these types of problems, ZeroMQ is not without its flaws. Its creators have attempted to rectify many of these shortcomings through spiritual successors Crossroads I/O and nanomsg. The now-defunct Crossroads I/O is a proper fork of ZeroMQ with the true intention being to build a viable commercial ecosystem around it. Nanomsg, however, is a reimagining of ZeroMQ—a complete rewrite in C1. It builds upon ZeroMQ’s rock-solid performance characteristics while providing several vital improvements, both internal and external. It also attempts to address many of the strange behaviors that ZeroMQ can often exhibit. Today, I’ll take a look at what differentiates nanomsg from its predecessor and implement a use case for it in the form of service discovery. Nanomsg vs. ZeroMQ A common gripe people have with ZeroMQ is that it doesn’t provide an API for new transport protocols, which essentially limits you to TCP, PGM, IPC, and ITC. Nanomsg addresses this problem by providing a pluggable interface for transports and messaging protocols. This means support for new transports (e.g. WebSockets) and new messaging patterns beyond the standard set of PUB/SUB, REQ/REP, etc. Nanomsg is also fully POSIX-compliant, giving it a cleaner API and better compatibility. No longer are sockets represented as void pointers and tied to a context—simply initialize a new socket and begin using it in one step. With ZeroMQ, the context internally acts as a storage mechanism for global state and, to the user, as a pool of I/O threads. This concept has been completely removed from nanomsg. In addition to POSIX compliance, nanomsg is hoping to be interoperable at the API and protocol levels, which would allow it to be a drop-in replacement for, or otherwise interoperate with, ZeroMQ and other libraries which implement ZMTP/1.0 and ZMTP/2.0. It has yet to reach full parity, however. ZeroMQ has a fundamental flaw in its architecture. Its sockets are not thread-safe. In and of itself, this is not problematic and, in fact, is beneficial in some cases. By isolating each object in its own thread, the need for semaphores and mutexes is removed. Threads don’t touch each other and, instead, concurrency is achieved with message passing. This pattern works well for objects managed by worker threads but breaks down when objects are managed in user threads. If the thread is executing another task, the object is blocked. Nanomsg does away with the one-to-one relationship between objects and threads. Rather than relying on message passing, interactions are modeled as sets of state machines. Consequently, nanomsg sockets are thread-safe. Nanomsg has a number of other internal optimizations aimed at improving memory and CPU efficiency. ZeroMQ uses a simple trie structure to store and match PUB/SUB subscriptions, which performs nicely for sub-10,000 subscriptions but quickly becomes unreasonable for anything beyond that number. Nanomsg uses a space-optimized trie called a radix tree to store subscriptions. Unlike its predecessor, the library also offers a true zero-copy API which greatly improves performance by allowing memory to be copied from machine to machine while completely bypassing the CPU. ZeroMQ implements load balancing using a round-robin algorithm. While it provides equal distribution of work, it has its limitations. Suppose you have two datacenters, one in New York and one in London, and each site hosts instances of “foo” services. Ideally, a request made for foo from New York shouldn’t get routed to the London datacenter and vice versa. With ZeroMQ’s round-robin balancing, this is entirely possible unfortunately. One of the new user-facing features that nanomsg offers is priority routing for outbound traffic. We avoid this latency problem by assigning priority one to foo services hosted in New York for applications also hosted there. Priority two is then assigned to foo services hosted in London, giving us a failover in the event that foos in New York are unavailable. Additionally, nanomsg offers a command-line tool for interfacing with the system called nanocat. This tool lets you send and receive data via nanomsg sockets, which is useful for debugging and health checks. Scalability Protocols Perhaps most interesting is nanomsg’s philosophical departure from ZeroMQ. Instead of acting as a generic networking library, nanomsg intends to provide the “Lego bricks” for building scalable and performant distributed systems by implementing what it refers to as “scalability protocols.” These scalability protocols are communication patterns which are an abstraction on top of the network stack’s transport layer. The protocols are fully separated from each other such that each can embody a well-defined distributed algorithm. The intention, as stated by nanomsg’s author Martin Sustrik, is to have the protocol specifications standardized through the IETF. Nanomsg currently defines six different scalability protocols: PAIR, REQREP, PIPELINE, BUS, PUBSUB, and SURVEY. PAIR (Bidirectional Communication) PAIR implements simple one-to-one, bidirectional communication between two endpoints. Two nodes can send messages back and forth to each other. REQREP (Client Requests, Server Replies) The REQREP protocol defines a pattern for building stateless services to process user requests. A client sends a request, the server receives the request, does some processing, and returns a response. PIPELINE (One-Way Dataflow) PIPELINE provides unidirectional dataflow which is useful for creating load-balanced processing pipelines. A producer node submits work that is distributed among consumer nodes. BUS (Many-to-Many Communication) BUS allows messages sent from each peer to be delivered to every other peer in the group. PUBSUB (Topic Broadcasting) PUBSUB allows publishers to multicast messages to zero or more subscribers. Subscribers, which can connect to multiple publishers, can subscribe to specific topics, allowing them to receive only messages that are relevant to them. SURVEY (Ask Group a Question) The last scalability protocol, and the one in which I will further examine by implementing a use case with, is SURVEY. The SURVEY pattern is similar to PUBSUB in that a message from one node is broadcasted to the entire group, but where it differs is that each node in the group responds to the message. This opens up a wide variety of applications because it allows you to quickly and easily query the state of a large number of systems in one go. The survey respondents must respond within a time window configured by the surveyor. Implementing Service Discovery As I pointed out, the SURVEY protocol has a lot of interesting applications. For example: What data do you have for this record? What price will you offer for this item? Who can handle this request? To continue exploring it, I will implement a basic service-discovery pattern. Service discovery is a pretty simple question that’s well-suited for SURVEY: what services are out there? Our solution will work by periodically submitting the question. As services spin up, they will connect with our service discovery system so they can identify themselves. We can tweak parameters like how often we survey the group to ensure we have an accurate list of services and how long services have to respond. This is great because 1) the discovery system doesn’t need to be aware of what services there are—it just blindly submits the survey—and 2) when a service spins up, it will be discovered and if it dies, it will be “undiscovered.” Here is the ServiceDiscovery class: from collections import defaultdict import random from nanomsg import NanoMsgAPIError from nanomsg import Socket from nanomsg import SURVEYOR from nanomsg import SURVEYOR_DEADLINE class ServiceDiscovery(object): def __init__(self, port, deadline=5000): self.socket = Socket(SURVEYOR) self.port = port self.deadline = deadline self.services = defaultdict(set) def bind(self): self.socket.bind('tcp://*:%s' % self.port) self.socket.set_int_option(SURVEYOR, SURVEYOR_DEADLINE, self.deadline) def discover(self): if not self.socket.is_open(): return self.services self.services = defaultdict(set) self.socket.send('service query') while True: try: response = self.socket.recv() except NanoMsgAPIError: break service, address = response.split('|') self.services[service].add(address) return self.services def resolve(self, service): providers = self.services[service] if not providers: return None return random.choice(tuple(providers)) def close(self): self.socket.close() The discover method submits the survey and then collects the responses. Notice we construct a SURVEYOR socket and set the SURVEYOR_DEADLINE option on it. This deadline is the number of milliseconds from when a survey is submitted to when a response must be received—adjust it accordingly based on your network topology. Once the survey deadline has been reached, a NanoMsgAPIError is raised and we break the loop. The resolve method will take the name of a service and randomly select an available provider from our discovered services. We can then wrap ServiceDiscovery with a daemon that will periodically run discover. import os import time from service_discovery import ServiceDiscovery DEFAULT_PORT = 5555 DEFAULT_DEADLINE = 5000 DEFAULT_INTERVAL = 2000 def start_discovery(port, deadline, interval): discovery = ServiceDiscovery(port, deadline=deadline) discovery.bind() print 'Starting service discovery [port: %s, deadline: %s, interval: %s]' \ % (port, deadline, interval) while True: print discovery.discover() time.sleep(interval / 1000) if __name__ == '__main__': port = int(os.environ.get('PORT', DEFAULT_PORT)) deadline = int(os.environ.get('DEADLINE', DEFAULT_DEADLINE)) interval = int(os.environ.get('INTERVAL', DEFAULT_INTERVAL)) start_discovery(port, deadline, interval) The discovery parameters are configured through environment variables which I inject into a Docker container. Services must connect to the discovery system when they start up. When they receive a survey, they should respond by identifying what service they provide and where the service is located. One such service might look like the following: import os from threading import Thread from nanomsg import REP from nanomsg import RESPONDENT from nanomsg import Socket DEFAULT_DISCOVERY_HOST = 'localhost' DEFAULT_DISCOVERY_PORT = 5555 DEFAULT_SERVICE_NAME = 'foo' DEFAULT_SERVICE_PROTOCOL = 'tcp' DEFAULT_SERVICE_HOST = 'localhost' DEFAULT_SERVICE_PORT = 9000 def register_service(service_name, service_address, discovery_host, discovery_port): socket = Socket(RESPONDENT) socket.connect('tcp://%s:%s' % (discovery_host, discovery_port)) print 'Starting service registration [service: %s %s, discovery: %s:%s]' \ % (service_name, service_address, discovery_host, discovery_port) while True: message = socket.recv() if message == 'service query': socket.send('%s|%s' % (service_name, service_address)) def start_service(service_name, service_protocol, service_port): socket = Socket(REP) socket.bind('%s://*:%s' % (service_protocol, service_port)) print 'Starting service %s' % service_name while True: request = socket.recv() print 'Request: %s' % request socket.send('The answer is 42') if __name__ == '__main__': discovery_host = os.environ.get('DISCOVERY_HOST', DEFAULT_DISCOVERY_HOST) discovery_port = os.environ.get('DISCOVERY_PORT', DEFAULT_DISCOVERY_PORT) service_name = os.environ.get('SERVICE_NAME', DEFAULT_SERVICE_NAME) service_host = os.environ.get('SERVICE_HOST', DEFAULT_SERVICE_HOST) service_port = os.environ.get('SERVICE_PORT', DEFAULT_SERVICE_PORT) service_protocol = os.environ.get('SERVICE_PROTOCOL', DEFAULT_SERVICE_PROTOCOL) service_address = '%s://%s:%s' % (service_protocol, service_host, service_port) Thread(target=register_service, args=(service_name, service_address, discovery_host, discovery_port)).start() start_service(service_name, service_protocol, service_port) Once again, we configure parameters through environment variables set on a container. Note that we connect to the discovery system with a RESPONDENT socket which then responds to service queries with the service name and address. The service itself uses a REP socket that simply responds to any requests with “The answer is 42,” but it could take any number of forms such as HTTP, raw socket, etc. The full code for this example, including Dockerfiles, can be found on GitHub. Nanomsg or ZeroMQ? Based on all the improvements that nanomsg makes on top of ZeroMQ, you might be wondering why you would use the latter at all. Nanomsg is still relatively young. Although it has numerous language bindings, it hasn’t reached the maturity of ZeroMQ which has a thriving development community. ZeroMQ has extensive documentation and other resources to help developers make use of the library, while nanomsg has very little. Doing a quick Google search will give you an idea of the difference (about 500,000 results for ZeroMQ to nanomsg’s 13,500). That said, nanomsg’s improvements and, in particular, its scalability protocols make it very appealing. A lot of the strange behaviors that ZeroMQ exposes have been resolved completely or at least mitigated. It’s actively being developed and is quickly gaining more and more traction. Technically, nanomsg has been in beta since March, but it’s starting to look production-ready if it’s not there already.

[This article was written by Stephane Combaudon] State Snapshot Transfer (SST) is used in Percona XtraDB Cluster (PXC) when a new node joins the cluster or to resync a failed node if Incremental State Transfer (IST) is no longer available. SST is triggered automatically but there is no magic: If it is not configured properly, it will not work and new nodes will never be able to join the cluster. Let’s have a look at a few classic issues. Port for SST is not open The donor and the joiner communicate on port 4444, and if the port is closed on one side, SST will always fail. You will see in the error log of the donor that SST is started: [...] 141223 16:08:48 [Note] WSREP: Node 2 (node1) requested state transfer from '*any*'. Selected 0 (node3)(SYNCED) as donor. 141223 16:08:48 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 6) 141223 16:08:48 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 141223 16:08:48 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.234.101:4444/xtrabackup_sst' --auth 'sstuser:s3cret' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --gtid '04c085a1-89ca-11e4-b1b6-6b692803109b:6'' [...] But then nothing happens, and some time later you will see a bunch of errors: [...] 2014/12/23 16:09:52 socat[2965] E connect(3, AF=2 192.168.234.101:4444, 16): Connection timed out WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 0 1 (20141223 16:09:52.057) WSREP_SST: [ERROR] Cleanup after exit with status:32 (20141223 16:09:52.064) WSREP_SST: [INFO] Cleaning up temporary directories (20141223 16:09:52.068) 141223 16:09:52 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.234.101:4444/xtrabackup_sst' --auth 'sstuser:s3cret' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --gtid '04c085a1-89ca-11e4-b1b6-6b692803109b:6' [...] On the joiner side, you will see a similar sequence: SST is started, then hangs and is finally aborted: [...] 141223 16:08:48 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 6) 141223 16:08:48 [Note] WSREP: Requesting state transfer: success, donor: 0 141223 16:08:49 [Note] WSREP: (f9560d0d, 'tcp://0.0.0.0:4567') turning message relay requesting off 141223 16:09:52 [Warning] WSREP: 0 (node3): State transfer to 2 (node1) failed: -32 (Broken pipe) 141223 16:09:52 [ERROR] WSREP: gcs/src/gcs_group.cpp:long int gcs_group_handle_join_msg(gcs_group_t*, const gcs_recv_msg_t*)():717: Will never receive state. Need to abort. The solution is of course to make sure that the ports are open on both sides. SST is not correctly configured Sometimes you will see an error like this on the donor: 141223 21:03:15 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.234.102:4444/xtrabackup_sst' --auth 'sstuser:s3cretzzz' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --gtid 'e63f38f2-8ae6-11e4-a383-46557c71f368:0'' [...] WSREP_SST: [ERROR] innobackupex finished with error: 1. Check /var/lib/mysql//innobackup.backup.log (20141223 21:03:26.973) And if you look at innobackup.backup.log: 41223 21:03:26 innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_file=/etc/my.cnf;mysql_read_default_group=xtrabackup;mysql_socket=/var/lib/mysql/mysql.sock' as 'sstuser' (using password: YES). innobackupex: got a fatal error with the following stacktrace: at /usr//bin/innobackupex line 2995 main::mysql_connect('abort_on_error', 1) called at /usr//bin/innobackupex line 1530 innobackupex: Error: Failed to connect to MySQL server: DBI connect(';mysql_read_default_file=/etc/my.cnf;mysql_read_default_group=xtrabackup;mysql_socket=/var/lib/mysql/mysql.sock','sstuser',...) failed: Access denied for user 'sstuser'@'localhost' (using password: YES) at /usr//bin/innobackupex line 2979 What happened? The default SST method is xtrabackup-v2 and for it to work, you need to specify a username/password in the my.cnf file: [mysqld] wsrep_sst_auth=sstuser:s3cret And you also need to create the corresponding MySQL user: mysql> GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost' IDENTIFIED BY 's3cret'; So you should check that the user has been correctly created in MySQL and that wsrep_sst_auth is correctly set. Galera versions do not match Here is another set of errors you may see in the error log of the donor: 141223 21:14:27 [Warning] WSREP: unserialize error invalid flags 2: 71 (Protocol error) at gcomm/src/gcomm/datagram.hpp:unserialize():101 141223 21:14:30 [Warning] WSREP: unserialize error invalid flags 2: 71 (Protocol error) at gcomm/src/gcomm/datagram.hpp:unserialize():101 141223 21:14:33 [Warning] WSREP: unserialize error invalid flags 2: 71 (Protocol error) at gcomm/src/gcomm/datagram.hpp:unserialize():101 Here the issue is that you try to connect a node using Galera 2.x and a node running Galera 3.x. This can happen if you try to use a PXC 5.5 node and a PXC 5.6 node. The right solution is probably to understand why you ended up with such inconsistent versions and make sure all nodes are using the same Percona XtraDB Cluster version and Galera version. But if you know what you are doing, you can also instruct the node using Galera 3.x that it will communicate with Galera 2.x nodes by specifying in the my.cnf file: [mysqld] wsrep_provider_options="socket.checksum=1" Conclusion SST errors can have multiple reasons for occurring, and the best way to diagnose the issue is to have a look at the error log of the donor and the joiner. Galera is in general quite verbose so you can follow the progress of SST on both nodes and see where it fails. Then it is mostly about being able to interpret the error messages.

Learn more about Agrona's Threadsafe offheap buffers, and how to increase performance.

written by josh long on the spring blog applications generated more and more data than ever before and a huge part of the challenge - before it can even be analyzed - is accommodating the load in the first place. apache’s kafka meets this challenge. it was originally designed by linkedin and subsequently open-sourced in 2011. the project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. the design is heavily influenced by transaction logs. it is a messaging system, similar to traditional messaging systems like rabbitmq, activemq, mqseries, but it’s ideal for log aggregation, persistent messaging, fast (_hundreds_ of megabytes per second!) reads and writes, and can accommodate numerous clients. naturally, this makes it perfect for cloud-scale architectures! kafka powers many large production systems . linkedin uses it for activity data and operational metrics to power the linkedin news feed, and linkedin today, as well as offline analytics going into hadoop. twitter uses it as part of their stream-processing infrastructure. kafka powers online-to-online and online-to-offline messaging at foursquare. it is used to integrate foursquare monitoring and production systems with hadoop-based offline infrastructures. square uses kafka as a bus to move all system events through square’s various data centers. this includes metrics, logs, custom events, and so on. on the consumer side, it outputs into splunk, graphite, or esper-like real-time alerting. netflix uses it for 300-600bn messages per day. it’s also used by airbnb, mozilla, goldman sachs, tumblr, yahoo, paypal, coursera, urban airship, hotels.com, and a seemingly endless list of other big-web stars. clearly, it’s earning its keep in some powerful systems! installing apache kafka there are many different ways to get apache kafka installed. if you’re on osx, and you’re using homebrew, it can be as simple as brew install kafka . you can also download the latest distribution from apache . i downloaded kafka_2.10-0.8.2.1.tgz , unzipped it, and then within you’ll find there’s a distribution of apache zookeeper as well as kafka, so nothing else is required. i installed apache kafka in my $home directory, under another directory, bin , then i created an environment variable, kafka_home , that points to $home/bin/kafka . start apache zookeeper first, specifying where the configuration properties file it requires is: $kafka_home/bin/zookeeper-server-start.sh $kafka_home/config/zookeeper.properties the apache kafka distribution comes with default configuration files for both zookeeper and kafka, which makes getting started easy. you will in more advanced use cases need to customize these files. then start apache kafka. it too requires a configuration file, like this: $kafka_home/bin/kafka-server-start.sh $kafka_home/config/server.properties the server.properties file contains, among other things, default values for where to connect to apache zookeeper ( zookeeper.connect ), how much data should be sent across sockets, how many partitions there are by default, and the broker id ( broker.id - which must be unique across a cluster). there are other scripts in the same directory that can be used to send and receive dummy data, very handy in establishing that everything’s up and running! now that apache kafka is up and running, let’s look at working with apache kafka from our application. some high level concepts.. a kafka broker cluster consists of one or more servers where each may have one or more broker processes running. apache kafka is designed to be highly available; there are no master nodes. all nodes are interchangeable. data is replicated from one node to another to ensure that it is still available in the event of a failure. in kafka, a topic is a category, similar to a jms destination or both an amqp exchange and queue. topics are partitioned, and the choice of which of a topic’s partition a message should be sent to is made by the message producer. each message in the partition is assigned a unique sequenced id, its offset . more partitions allow greater parallelism for consumption, but this will also result in more files across the brokers. producers send messages to apache kafka broker topics and specify the partition to use for every message they produce. message production may be synchronous or asynchronous. producers also specify what sort of replication guarantees they want. consumers listen for messages on topics and process the feed of published messages. as you’d expect if you’ve used other messaging systems, this is usually (and usefully!) asynchronous. like spring xd and numerous other distributed system, apache kafka uses apache zookeeper to coordinate cluster information. apache zookeeper provides a shared hierarchical namespace (called znodes ) that nodes can share to understand cluster topology and availability (yet another reason that spring cloud has forthcoming support for it..). zookeeper is very present in your interactions with apache kafka. apache kafka has, for example, two different apis for acting as a consumer. the higher level api is simpler to get started with and it handles all the nuances of handling partitioning and so on. it will need a reference to a zookeeper instance to keep the coordination state. let’s turn now turn to using apache kafka with spring. using apache kafka with spring integration the recently released apache kafka 1.1 spring integration adapter is very powerful, and provides inbound adapters for working with both the lower level apache kafka api as well as the higher level api. the adapter, currently, is xml-configuration first, though work is already underway on a spring integration java configuration dsl for the adapter and milestones are available. we’ll look at both here, now. to make all these examples work, i added the libs-milestone-local maven repository and used the following dependencies: org.apache.kafka:kafka_2.10:0.8.1.1 org.springframework.boot:spring-boot-starter-integration:1.2.3.release org.springframework.boot:spring-boot-starter:1.2.3.release org.springframework.integration:spring-integration-kafka:1.1.1.release org.springframework.integration:spring-integration-java-dsl:1.1.0.m1 using the spring integration apache kafka with the spring integration xml dsl first, let’s look at how to use the spring integration outbound adapter to send message instances from a spring integration flow to an external apache kafka instance. the example is fairly straightforward: a spring integration channel named inputtokafka acts as a conduit that forwards message messages to the outbound adapter, kafkaoutboundchanneladapter . the adapter itself can take its configuration from the defaults specified in the kafka:producer-context element or it from the adapter-local configuration overrides. there may be one or many configurations in a given kafka:producer-context element. here’s the java code from a spring boot application to trigger message sends using the outbound adapter by sending messages into the incoming inputtokafka messagechannel . package xml; import org.apache.commons.logging.log; import org.apache.commons.logging.logfactory; import org.springframework.beans.factory.annotation.qualifier; import org.springframework.boot.commandlinerunner; import org.springframework.boot.springapplication; import org.springframework.boot.autoconfigure.springbootapplication; import org.springframework.context.annotation.bean; import org.springframework.context.annotation.dependson; import org.springframework.context.annotation.importresource; import org.springframework.integration.config.enableintegration; import org.springframework.messaging.messagechannel; import org.springframework.messaging.support.genericmessage; @springbootapplication @enableintegration @importresource("/xml/outbound-kafka-integration.xml") public class demoapplication { private log log = logfactory.getlog(getclass()); @bean @dependson("kafkaoutboundchanneladapter") commandlinerunner kickoff(@qualifier("inputtokafka") messagechannel in) { return args -> { for (int i = 0; i < 1000; i++) { in.send(new genericmessage<>("#" + i)); log.info("sending message #" + i); } }; } public static void main(string args[]) { springapplication.run(demoapplication.class, args); } } using the new apache kafka spring integration java configuration dsl shortly after the spring integration 1.1 release, spring integration rockstar artem bilan got to work on adding a spring integration java configuration dsl analog and the result is a thing of beauty! it’s not yet ga (you need to add the libs-milestone repository for now), but i encourage you to try it out and kick the tires. it’s working well for me and the spring integration team are always keen on getting early feedback whenever possible! here’s an example that demonstrates both sending messages and consuming them from two different integrationflow s. the producer is similar to the example xml above. new in this example is the polling consumer. it is batch-centric, and will pull down all the messages it sees at a fixed interval. in our code, the message received will be a map that contains as its keys the topic and as its value another map with the partition id and the batch (in this case, of 10 records), of records read. there is a messagelistenercontainer -based alternative that processes messages as they come. package jc; import org.apache.commons.logging.log; import org.apache.commons.logging.logfactory; import org.springframework.beans.factory.annotation.autowired; import org.springframework.beans.factory.annotation.qualifier; import org.springframework.beans.factory.annotation.value; import org.springframework.boot.commandlinerunner; import org.springframework.boot.springapplication; import org.springframework.boot.autoconfigure.springbootapplication; import org.springframework.context.annotation.bean; import org.springframework.context.annotation.configuration; import org.springframework.context.annotation.dependson; import org.springframework.integration.integrationmessageheaderaccessor; import org.springframework.integration.config.enableintegration; import org.springframework.integration.dsl.integrationflow; import org.springframework.integration.dsl.integrationflows; import org.springframework.integration.dsl.sourcepollingchanneladapterspec; import org.springframework.integration.dsl.kafka.kafka; import org.springframework.integration.dsl.kafka.kafkahighlevelconsumermessagesourcespec; import org.springframework.integration.dsl.kafka.kafkaproducermessagehandlerspec; import org.springframework.integration.dsl.support.consumer; import org.springframework.integration.kafka.support.zookeeperconnect; import org.springframework.messaging.messagechannel; import org.springframework.messaging.support.genericmessage; import org.springframework.stereotype.component; import java.util.list; import java.util.map; /** * demonstrates using the spring integration apache kafka java configuration dsl. * thanks to spring integration ninja artem bilan * for getting the java configuration dsl working so quickly! * * @author josh long */ @enableintegration @springbootapplication public class demoapplication { public static final string test_topic_id = "event-stream"; @component public static class kafkaconfig { @value("${kafka.topic:" + test_topic_id + "}") private string topic; @value("${kafka.address:localhost:9092}") private string brokeraddress; @value("${zookeeper.address:localhost:2181}") private string zookeeperaddress; kafkaconfig() { } public kafkaconfig(string t, string b, string zk) { this.topic = t; this.brokeraddress = b; this.zookeeperaddress = zk; } public string gettopic() { return topic; } public string getbrokeraddress() { return brokeraddress; } public string getzookeeperaddress() { return zookeeperaddress; } } @configuration public static class producerconfiguration { @autowired private kafkaconfig kafkaconfig; private static final string outbound_id = "outbound"; private log log = logfactory.getlog(getclass()); @bean @dependson(outbound_id) commandlinerunner kickoff( @qualifier(outbound_id + ".input") messagechannel in) { return args -> { for (int i = 0; i < 1000; i++) { in.send(new genericmessage<>("#" + i)); log.info("sending message #" + i); } }; } @bean(name = outbound_id) integrationflow producer() { log.info("starting producer flow.."); return flowdefinition -> { consumer spec = (kafkaproducermessagehandlerspec.producermetadataspec metadata)-> metadata.async(true) .batchnummessages(10) .valueclasstype(string.class) .valueencoder(string::getbytes); kafkaproducermessagehandlerspec messagehandlerspec = kafka.outboundchanneladapter( props -> props.put("queue.buffering.max.ms", "15000")) .messagekey(m -> m.getheaders().get(integrationmessageheaderaccessor.sequence_number)) .addproducer(this.kafkaconfig.gettopic(), this.kafkaconfig.getbrokeraddress(), spec); flowdefinition .handle(messagehandlerspec); }; } } @configuration public static class consumerconfiguration { @autowired private kafkaconfig kafkaconfig; private log log = logfactory.getlog(getclass()); @bean integrationflow consumer() { log.info("starting consumer.."); kafkahighlevelconsumermessagesourcespec messagesourcespec = kafka.inboundchanneladapter( new zookeeperconnect(this.kafkaconfig.getzookeeperaddress())) .consumerproperties(props -> props.put("auto.offset.reset", "smallest") .put("auto.commit.interval.ms", "100")) .addconsumer("mygroup", metadata -> metadata.consumertimeout(100) .topicstreammap(m -> m.put(this.kafkaconfig.gettopic(), 1)) .maxmessages(10) .valuedecoder(string::new)); consumer endpointconfigurer = e -> e.poller(p -> p.fixeddelay(100)); return integrationflows .from(messagesourcespec, endpointconfigurer) .>>handle((payload, headers) -> { payload.entryset().foreach(e -> log.info(e.getkey() + '=' + e.getvalue())); return null; }) .get(); } } public static void main(string[] args) { springapplication.run(demoapplication.class, args); } } the example makes heavy use of java 8 lambdas. the producer spends a bit of time establishing how many messages will be sent in a single send operation, how keys and values are encoded (kafka only knows about byte[] arrays, after all) and whether messages should be sent synchronously or asynchronously. in the next line, we configure the outbound adapter itself and then define an integrationflow such that all messages get sent out via the kafka outbound adapter. the consumer spends a bit of time establishing which zookeeper instance to connect to, how many messages to receive (10) in a batch, etc. once the message batches are recieved, they’re handed to the handle method where i’ve passed in a lambda that’ll enumerate the payload’s body and print it out. nothing fancy. using apache kafka with spring xd apache kafka is a message bus and it can be very powerful when used as an integration bus. however, it really comes into its own because it’s fast enough and scalable enough that it can be used to route big-data through processing pipelines. and if you’re doing data processing, you really want spring xd ! spring xd makes it dead simple to use apache kafka (as the support is built on the apache kafka spring integration adapter!) in complex stream-processing pipelines. apache kafka is exposed as a spring xd source - where data comes from - and a sink - where data goes to. spring xd exposes a super convenient dsl for creating bash -like pipes-and-filter flows. spring xd is a centralized runtime that manages, scales, and monitors data processing jobs. it builds on top of spring integration, spring batch, spring data and spring for hadoop to be a one-stop data-processing shop. spring xd jobs read data from sources , run them through processing components that may count, filter, enrich or transform the data, and then write them to sinks. spring integration and spring xd ninja marius bogoevici , who did a lot of the recent work in the spring integration and spring xd implementation of apache kafka, put together a really nice example demonstrating how to get a full working spring xd and kafka flow working . the readme walks you through getting apache kafka, spring xd and the requisite topics all setup. the essence, however, is when you use the spring xd shell and the shell dsl to compose a stream. spring xd components are named components that are pre-configured but have lots of parameters that you can override with --.. arguments via the xd shell and dsl. (that dsl, by the way, is written by the amazing andy clement of spring expression language fame!) here’s an example that configures a stream to read data from an apache kafka source and then write the message a component called log , which is a sink. log , in this case, could be syslogd, splunk, hdfs, etc. xd> stream create kafka-source-test --definition "kafka --zkconnect=localhost:2181 --topic=event-stream | log"--deploy and that’s it! naturally, this is just a tase of spring xd, but hopefully you’ll agree the possibilities are tantalizing. deploying a kafka server with lattice and docker it’s easy to get an example kafka installation all setup using lattice , a distributed runtime that supports, among other container formats, the very popular docker image format. there’s a docker image provided by spotify that sets up a collocated zookeeper and kafka image . you can easily deploy this to a lattice cluster, as follows: ltc create --run-as-root m-kafka spotify/kafka from there, you can easily scale the apache kafka instances and even more easily still consume apache kafka from your cloud-based services. next steps you can find the code for this blog on my github account . we’ve only scratched the surface! if you want to learn more (and why wouldn’t you?), then be sure to check out marius bogoevici and dr. mark pollack’s upcoming webinar on reactive data-pipelines using spring xd and apache kafka where they’ll demonstrate how easy it can be to use rxjava, spring xd and apache kafka!

What is Helix? It is used for the automatic management of partitioned, replicated and distributed resources hosted on a cluster of nodes. Helix automates reassignment of resources in the face of node failure and recovery, cluster expansion, and reconfiguration. Modeling a distributed system as a state machine with constraints on states and transitions. Terminologies Node : A single machine Cluster: Set of Nodes Resource : A logical entry (e.g. database, index, task) Partition: Subset of the resource (Each subtask is referred to as a partition) Replica: Copy of a Partition State (e.g Master, Slave). It increase the availability of the system State: Describes the role of a replica (Each node in the cluster has its own Current State) State Machine and Transitions: An action that allows a replica to move from one state to another, thus changing its role. ( e.g Slave --> Master ) spectators: the external clients. Helix provides an External View that is an aggregated view of the current state across all nodes. Current State: represents resource's actual state at a participating node. - INSTANCE_NAME: Unique name representing the process - SESSION_ID: ID that is automatically assigned every time a process joins the cluster Rebalancer: The core component of Helix is the Controller which runs the Rebalance algorithm on every cluster event. Dynamic Ideal State: Helix powerful is that Ideal State can be changed dynamically. It is adjusting the ideal state. Whenever a cluster event occurs, Helix can operate in one of three modes FULL_AUTO SEMI_AUTO CUSTOMIZED Cluster events can be one of the following: Nodes start and/or stop Nodes experience soft and/or hard failures New nodes are added/removed [1] http://helix.apache.org/Concepts.html

[this article was written by tony mauro .] in a previous blog post , we shared best practices for designing a microservices architecture, based on adrian cockcroft’s presentation at nginx.conf2014 about his experience as director of web engineering and then cloud architect at netflix . in this follow-up post, we’ll review his recommendations for retooling your development team and processes for a smooth transition to microservices. optimize for speed, not efficiency source: [email protected] the top lesson that cockcroft learned at netflix is that speed wins in the marketplace. if you ask any developer whether a slower development process is better, no one ever says yes. nor do management or customers ever complain that your development cycle is too fast for them. the need for speed doesn’t just apply to tech companies, either: as software becomes increasingly ubiquitous on the internet of things – in cars, appliances, and sensors as well as mobile devices – companies that didn’t used to do software development at all now find that their success depends on being good at it. netflix made an early decision to optimize for speed. this refers specifically to tooling your software development process so that you can react quickly to what your customers want, or even better, can create innovative web experiences that attract customers. speed means learning about your customers and giving them what they want at a faster pace than your competitors. by the time competitors are ready to challenge you in a specific way, you’ve moved on to the next set of improvements. this approach turns the usual paradigm of optimizing for efficiency on its head. efficiency generally means trying to control the overall flow of the development process to eliminate duplication of effort and avoid mistakes, with an eye to keeping costs down. the common result is that you end up focusing on savings instead of looking for opportunities that increase revenue. in cockcroft’s experience, if you say “i’m doing this because it’s more efficient,” the unintended result is that you’re slowing someone else down. this is not an encouragement to be wasteful, but you should optimize for speed first. efficiency becomes secondary as you satisfy the constraint that you’re not slowing things down. the way you grow the business to be more efficient is to go faster. make sure your assumptions are still true many large companies that have enjoyed success in their market (we can call them incumbents ) are finding themselves overtaken by nimbler, usually smaller, organizations ( disruptors ) that react much more quickly to changing consumer behavior. their large size isn’t necessarily the root of the problem – netflix is no longer a small company, for example. as cockcroft sees it, the main cause of difficulty for industry incumbents is that they’re operating under business assumptions that are no longer true. or, as will rogers put it, it’s not what we don’t know that hurts. it’s what we know that ain’t so.” of course, you have to make assumptions as you formulate a business model, and then it makes sense to optimize your business practices around them. the danger comes from sticking with assumptions after they’re no longer true, which means you’re optimizing on the wrong thing. that’s when you become vulnerable to industry disruptors who are making the right assumptions and optimizations for the current business climate. as examples, consider the following assumptions that hold sway at many incumbents. we’ll examine them further in the indicated sections and describe the approach netflix adopted. computing power is expensive. this was true when increasing your computing capacity required capital expenditure on computer hardware. see put your infrastructure in the cloud . process prevents problems. at many companies, the standard response to something going wrong is to add a preventative step to the relevant procedure. see create a high freedom, high responsibility culture with less process . here are some ways to avoid holding onto assumptions that have passed their expiration date: as obvious as it might seem, you need to make your assumptions explicit, then periodically review them to make sure they still hold true. keep aware of technological trends. as an example, the cost of solid state storage drive (ssds) storage continues to go down. it’s still more expensive than regular disks, but the cost difference is becoming small enough that many companies are deciding the superior performance is worth paying a bit more for. [ed: in this entertaining video , fastly founder and ceo artur bergman explains why he believes ssds are always the right choice.] talk to people who aren’t your customers. this is especially necessary for incumbents, who need to make sure that potential new customers are interested in their product. otherwise, they don’t hear about the fact that they’re not being used. as an example, some vendors in the storage space are building hyper-converged systems even as more and more companies are storing their data in the cloud and using open source storage management software. netflix, for example, stores data on amazon web services (aws) servers with ssds and manages it with apache cassandra . a single specialist in java distributed systems is managing the entire configuration without any commercial storage tools or help from engineers specializing in storage, san, or backup. don’t base your future strategy on current it spending, but instead on level of adoption by developers. suppose that your company accounts for nearly all spending in the market for proprietary virtualization software, but then a competitor starts offering an open source-based product at only 1% the cost of yours. if people start choosing it instead of your product, than at the point that your share of total spending is still 90%, your market share has declined to only 10%. if you’re only attending to your revenue, it seems like you’re still in good shape, but 10% of market share can collapse really quickly. put your infrastructure in the cloud source: [email protected] in make sure your assumptions are still true , we mentioned that in the past it was valid to base your business plan on the assumption that computing power was expensive, because it was: the only way to increase your computing capacity was to buy computer hardware. you could then make money by using this expensive resource in the right way to solve customer problems. the advent of cloud computing has pretty much completely invalidated this assumption. it is now possible to buy the amount of capacity you need when you need it, and to pay for only the time you actually use it. the new assumption you need to make is that (virtual) machines are ephemeral. you can create and destroy them at the touch of a button or a call to an api, without any need to negotiate with other departments in your company. one way to think of this change is that the self-service cloud makes formerly impossible things instantaneous. all of netflix’s engineers are in california, but they manage a worldwide infrastructure. the cloud enables them to experiment and determine whether (for example) adding servers in particular location improves performance. suppose they notice problems with video delivery in brazil. they can easily set up 100 cloud server instances in são paulo within a couple hours. if after a week they determine that the difference in delivery speed and reliability isn’t large enought to justify the cost of the additional server instances, they can shut them down just as quickly and easily as they created them. this kind of experiment would be so expensive with a traditional infrastructure that you would never attempt it. you would have to hire an agent in são paulo to coordinate the project, find a data center, satisfy brazilian government regulations, ship machines to brazil, and so on. it would be six months before you could even run the test and find out that increased local capacity didn’t improve your delivery speed. create a high freedom, high responsibility culture with less process in make sure your assumptions are still true , we observed that many companies create rules and processes to prevent problems. when someone makes a mistake, they add a rule to the hr manual that says “well, don’t do that again.” if you read some hr manuals from this perspective, you can extract a historical record of everything that went wrong at the company. when something goes wrong in the development process, the corresponding reaction is to add a new step to the procedure. the major problem with creating process to prevent problems is that over time you build up complex “scar tissue” processes that slow you down. netflix doesn’t have an hr manual. there is a single guideline: “act in netflix’s best interest.” the idea is that if an employee can’t figure out how to interpret the guideline in a given situation, he or she doesn’t have enough judgment to work there. if you don’t trust the judgment of the people on your team, you have to ask why you’re employing them. it’s true that you’ll have to fire people occasionally for violating the guideline. overall, the high level of mutual trust among members of a team, and across the company as a whole, becomes a strong binding force. the following books outline new ways of thinking about process if you’re looking to transform your organization: the goal: a process of ongoing improvement by eliyahu m. goldratt and jeff cox. this book has become a standard management text at business schools since its original publication in 1984. written as a novel about a manager who has only 90 days to improve performance at his factory or have it closed down, it embodies goldratt’s theory of constraints in the context of process control and automation. the phoenix project: a novel about it, devops, and helping your business win by gene kim and kevin behr. as the title indicates, it’s also a novel, about an it manager who has 90 days to save a project that’s late and over budget, or his entire department will be outsourced. he discovers devops as the solution to his problem. replace silos with microservice teams most software development groups are separated into silos, with no overlap of personnel between them. the standard process for a software development project starts with the product manager meeting with the user experience and development groups to discuss ideas for new features. after the idea is implemented in code, the code is passed to the quality assurance (qa) and database administration teams and discussed in more meetings. communication with the system, network, and san administrators is often via tickets. the whole process tends to be slow and loaded with overhead. source: adrian cockcroft some companies try to speed up by creating small “start-up”-style teams that handle the development process from end to end, or sometimes such teams are the result of acquisitions where the acquired company continues to run independently as a separate division. but if the small teams are still doing monolithic delivery, there are usually still handoffs between individuals or groups with responsibility for different functions. the process suffers from the same problems as monolithic delivery in larger companies – it’s simply not very efficient or agile. source: adrian cockcroft conway’s law says that the interface structure of a software system will reflect the social structure of the organization that produced it. so if you want to switch to a microservices architecture, you need to organize your staff into product teams and use devops methodology. there are no longer distinct product managers, ux managers, development managers, and so on, managing downward in their silos. there is a manager for each product feature (implemented as a microservice), who supervises a team that handles all aspects of software development for the microservice, from conception through deployment. the platform team provides infrastructure support that the product teams access via apis. at netflix, the platform team was mostly aws in seattle, with some netflix-managed infrastructure layers built on top. but it doesn’t matter whether your cloud platform is in-house or public; the important thing is that it’s api-driven, self-service, and automatable. source: adrian cockcroft adopt continuous delivery, guided by the ooda loop a siloed team organization is usually paired with monolithic delivery model, in which an integrated, multi-function application is released as a unit (often version-numbered) on a regular schedule. most software development teams use this model initially because it is relatively simple and works well enough with a small number of developers (say, 50 or fewer). however, as the team grows it becomes a real issue when you discover a bug in one developer’s code during qa or production testing and the work of 99 other developers is blocked from release until the bug is fixed. in 2009 netflix adopted a continuous delivery model, which meshes perfectly with a microservices architecture. each microservice represents a single product feature that can be updated independently of the other microservices and on its own schedule. discovering a bug in a microservice has no effect on the release schedule of any other microservice. continuous delivery relies on packaging microservices in standard containers. netflix initially used aws machine images (amis) and it was possible to deploy an update into a test or production environment in about 10 minutes. with docker, that time is reduced even further, to mere seconds in some cases. at netflix, the conceptual framework for continuous development and delivery is an observe-orient-decide-act (ooda) loop . source: adrian cockcroft (http://www.slideshare.net/adrianco) observe refers to examining your current status to look for places where you can innovate. you want your company culture to implicitly authorize anyone who notices an opportunity to start a project to exploit it. for example, you might notice what the diagram calls a “customer pain point”: a lot of people abandoning the registration process on your website when they reach a certain step. you can undertake a project to investigate why and fix the problem. orient refers to analyzing metrics to understand the reasons for the phenomena you’ve observed at the observe point. often this involves analyzing large amounts of unstructured data, such as log files; this is often referred to as big data analysis. the answers you’re looking for are not already in your business intelligence database. you’re examining data that no one has previously looked at and asking questions that haven’t been asked before. decide refers to developing and executing a project plan. company culture is a big factor at this point. as previously discussed, in a high-freedom, high-responsibility culture you don’t need to get management approval before starting to make changes. you share your plan, but you don’t have to ask for permission. act refers to testing your solution and putting it into production. you deploy a microservice that includes your incremental feature to a cloud environment, where it’s automatically put into an ab test to compare it to the previous solution, side by side, for as long as it takes to collect the data that shows whether your approach is better. cooperating microservices aren’t disrupted, and customers don’t see your changes unless they’re selected for the test. if your solution is better, you deploy it into production. it doesn’t have to be a big improvement, either. if the number of clients for your microservice is large enough, then even a fraction of a percent improvement (in response time, say) can be shown to be statistically valid, and the cumulative effect over time of many small changes can be significant. now you’re back at the observe point. you don’t always have to perform all the steps or do them in strict order, either. the important characteristic of the process is that it enables you quickly to determine what your customers want and to create it for them. cockcroft says “it’s hard not to win” if you’re basing your moves on enough data points and your competitors are making guesses that take months to be proven or disproven. the state of art is to circle the loop every one to two weeks, but every microservice team can do it independently. with microservices you can go much faster because you’re not trying to get entire company going around the loop in lockstep. how nginx plus can help at nginx we believe it’s crucial to your future success that you adopt a 4-tier application architecture in which applications are developed and deployed as sets of microservices . we hope the information we’ve shared in this post and its predecessor, adopting microservices at netflix: lessons for architectural design , are helpful as you plan your transition to today’s state-of-the-art architecture for application development. when it’s time to deliver your apps, nginx plus offers an application delivery platform that provides the superior performance, reliability, and scalability your users expect. fully adopting a microservices-based architecture is easier and more likely to succeed when you move to a single software tool for web serving, load balancing, and content caching. nginx plus combines those functions and more in one easy to deploy and manage package. our approach empowers developers to define and control the flawless delivery of their microservices, while respecting the standards and best practices put into place by a platform team. click here to learn more about how nginx plus can help your applications succeed. video recordings fast delivery nginx.conf2014, october 2014 migrating to microservices, part 1 silicon valley microservices meetup, august 2014 migrating to microservices, part 2 silicon valley microservices meetup, august 2014

[This article was written by Tony Mauro.] In some recent blog posts, we’ve explained why we believe it’s crucial to adopt a four-tier application architecture in which applications are developed and deployed as sets of microservices. It’s becoming increasingly clear that if you keep using development processes and application architectures that worked just fine ten years ago, you simply can’t move fast enough to capture and hold the interest of mobile users who can choose from an ever-growing number of apps. Switching to a microservices architecture creates exciting opportunities in the marketplace for companies. For system architects and developers, it promises an unprecedented level of control and speed as they deliver innovative new web experiences to customers. But at such a breathless pace, it can feel like there’s not a lot of room for error. In the real world, you can’t stop developing and deploying your apps as you retool the processes for doing so. You know that your future success depends on transitioning to a microservices architecture, but how do you actually do it? Fortunately for us, several early adopters of microservices are now generously sharing their expertise in the spirit of open source, not only in the form of published code but in conference presentations and blog posts. Netflix is a leading example. As the Director of Web Engineering and then Cloud Architect, Adrian Cockcroft oversaw the company’s transition from a traditional development model with 100 engineers producing a monolithic DVD-rental application to a microservices architecture with many small teams responsible for the end-to-end development of hundreds of microservices that work together to stream digital entertainment to millions of Netflix customers every day. Now a Technology Fellow at Battery Ventures, Cockcroft is a prominent evangelist for microservices and cloud-native architectures, and serves on the NGINX Technical Advisory Board. In a two-part series of blog posts, we’ll present top takeaways from two talks that Cockcroft delivered last year, at the first annual NGINX conference in October and at a Silicon Valley Microservices Meetup a couple months earlier. (The complete video recordings are also well worth watching.) This post defines microservices architecture and outlines some best practices for designing one. Adopting Microservices at Netflix: Lessons for Team and Process Design discusses why and how to adopt a new mindset for software development and reorganize your teams around it. What is a Microservices Architecture? Cockcroft defines a microservices architecture as a service-oriented architecture composed of loosely coupled elements that have bounded contexts. Loosely coupled means that you can update the services independently; updating one service doesn’t require changing any other services. If you have a bunch of small, specialized services but still have to update them together, they’re not microservices because they’re not loosely coupled. One kind of coupling that people tend to overlook as they transition to a microservices architecture is database coupling, where all services talk to the same database and updating a service means changing the schema. You need to split the database up and denormalize it. The concept of bounded contexts comes from the book Domain Driven Design by Eric Evans. A microservice with correctly bounded context is self-contained for the purposes of software development. You can understand and update the microservice’s code without knowing anything about the internals of its peers, because the microservices and its peers interact strictly through APIs and so don’t share data structures, database schemata, or other internal representations of objects. If you’ve developed applications for the Internet, you’re already familiar with these concepts, in practice if not by name. Most mobile apps talk to quite a few back-end services, to enable its users to do things like share on Facebook, get directions from Google Maps, and find restaurants on Foursquare, all within the context of the app. If your mobile app were tightly coupled with those services, then before you could release an update you would have to talk to all of their development teams to make sure that your changes aren’t going to break anything. When working with a microservices architecture, you think of other internal development teams like those Internet back ends: as external services that your microservice interacts with through APIs. The commonly understood “contract” between microservices is that their APIs are stable and forward compatible. Just as it’s unacceptable for the Google Maps API to change without warning and in such a way that it breaks its users, your API can evolve but must remain compatible with previous versions. Best Practices for Designing a Microservices Architecture Cockcroft describes his role as Cloud Architect at Netflix not in terms of controlling the architecture, but as discovering and formalizing the architecture that emerged as the Netflix engineers built it. The Netflix development team established several best practices for designing and implementing a microservices architecture. Create a Separate Data Store for Each Microservice Do not use the the same back-end data store across microservices. You want the team for each microservice to choose the database that best suits the service. Moreover, with a single data store it’s too easy for microservices written by different teams to share database structures, perhaps in the name of reducing duplication of work. You end up with the situation where if one team updates a database structure, other services that also use that structure have to be changed too. Breaking apart the data can make data management more complicated, because the separate storage systems can more easily get out sync or become inconsistent, and foreign keys can change unexpectedly. You need to add a tool that performs master data management (MDM) by operating in the background to find and fix inconsistencies. For example, it might examine every database that stores subscriber IDs, to verify that the same IDs exist in all of them (there aren’t missing or extra IDs in any one database). You can write your own tool or buy one. Many commercial relational database management systems (RDBMSs) do these kinds of checks, but they usually impose too many requirements for coupling, and so don’t scale. Keep Code at a Similar Level of Maturity Keep all code in a microservice at a similar level of maturity and stability. In other words, if you need to add or rewrite some of the code in a deployed microservice that’s working well, the best approach is usually to create a new microservice for the new or changed code, leaving the existing microservice in place. [Editor’s note: This is sometimes referred to as the immutable infrastructure principle.] This way you can iteratively deploy and test the new code until it is bug free and maximally efficient, without risking failure or performance degradation in the existing microservice. Once the new microservice is as stable as the original, you can merge them back together if they really perform a single function together, or there are other efficiencies from combining them. However, in Cockcroft’s experience it is much more common to realize you should split up a microservice because it’s gotten too big. Do a Separate Build for Each Microservice Do a separate build for each microservice, so that it can pull in component files from the repository at the revision levels appropriate to it. This sometimes leads to the situation where various microservices pull in a similar set of files, but at different revision levels. That can make it more difficult to clean up your codebase by decommissioning old file versions (because you have to verify more carefully that a revision is no longer being used), but that’s an acceptable trade-off for how easy it is to add new files as you build new microservices. The asymmetry is intentional: you want introducing a new microservice, file, or function easy, not dangerous. Deploy in Containers Deploying microservices in containers is important because it means you just need just one tool to deploy everything. As long as the microservice is in a container, the tool knows how to deploy it. It doesn’t matter what the container is. That said, Docker seems very quickly to have become the de facto standard for containers. Treat Servers as Stateless Treat servers, particularly those that run customer-facing code, as interchangeable members of a group. They all perform the same functions, so you don’t need to be concerned about them individually. Your only concern is that there are enough of them to produce the amount of work you need, and you can use auto scaling to adjust the numbers up and down. If one stops working, it’s automatically replaced by another one. Avoid “snowflake” systems in which you depend on individual servers to perform specialized functions. Cockcroft’s analogy is that you want to think of servers like cattle, not pets. If you have a machine in production that performs a specialized function, and you know it by name, and everyone gets sad when it goes down, it’s a pet. Instead you should think of your servers like a herd of cows. What you care about is how many gallons of milk you get. If one day you notice you’re getting less milk than usual, you find out which cows aren’t producing well and replace them. Netflix Delivery Architecture is Built on nginx Netflix is a longtime nginx user and became the first customer of NGINX, Inc. after it incorporated in 2011. Indeed, Netflix chose nginx as the heart of their delivery infrastructure, the Netflix Open Connect Content Delivery Network (CDN), one of the largest CDNs in the world. With the ability to serve thousands, and sometimes millions, of requests per second, nginx is an optimal solution for high-performance HTTP delivery and enables companies like Netflix to offer high-quality digital experiences to millions of customers every day. Video Recordings Fast Delivery nginx.conf2014, October 2014 Migrating to Microservices, Part 1 Silicon Valley Microservices Meetup, August 2014 Migrating to Microservices, Part 2 Silicon Valley Microservices Meetup, August 2014

Some key concepts for Apache's popular Cassandra Architecture include partitioning, replication, consistency, bootstrapping, and write paths.

i've seen and had lots of discussion about "package by layer" vs "package by feature" over the past couple of weeks. they both have their benefits but there's a hybrid approach i now use that i call "package by component". to recap... package by layer let's assume that we're building a web application based upon the web-mvc pattern. packaging code by layer is typically the default approach because, after all, that's what the books, tutorials and framework samples tell us to do. here we're organising code by grouping things of the same type. there's one top-level package for controllers, one for services (e.g. "business logic") and one for data access. layers are the primary organisation mechanism for the code. terms such as "separation of concerns" are thrown around to justify this approach and generally layered architectures are thought of as a "good thing". need to switch out the data access mechanism? no problem, everything is in one place. each layer can also be tested in isolation to the others around it, using appropriate mocking techniques, etc. the problem with layered architectures is that they often turn into a big ball of mud because, in java anyway, you need to mark your classes as public for much of this to work. package by feature instead of organising code by horizontal slice, package by feature seeks to do the opposite by organising code by vertical slice. now everything related to a single feature (or feature set) resides in a single place. you can still have a layered architecture, but the layers reside inside the feature packages. in other words, layering is the secondary organisation mechanism. the often cited benefit is that it's "easier to navigate the codebase when you want to make a change to a feature", but this is a minor thing given the power of modern ides. what you can do now though is hide feature specific classes and keep them out of sight from the rest of the codebase. for example, if you need any feature specific view models, you can create these as package-protected classes. the big question though is what happens when that new feature set c needs to access data from features a and b? again, in java, you'll need to start making classes publicly accessible from outside of the packages and the big ball of mud will again emerge. package by layer and package by feature both have their advantages and disadvantages. to quote jason gorman from schools of package architecture - an illustration , which was written seven years ago. to round off, then, i would urge you to be mindful of leaning to far towards either school of package architecture. don't just mindlessly put socks in the sock draw and pants in the pants draw, but don't be 100% driven by package coupling and cohesion to make those decisions, either. the real skill is finding the right balance, and creating packages that make stuff easier to find but are as cohesive and loosely coupled as you can make them at the same time. package by component this is a hybrid approach with increased modularity and an architecturally-evident coding style as the primary goals. the basic premise here is that i want my codebase to be made up of a number of coarse-grained components, with some sort of presentation layer (web ui, desktop ui, api, standalone app, etc) built on top. a "component" in this sense is a combination of the business and data access logic related to a specific thing (e.g. domain concept, bounded context, etc). as i've described before , i give these components a public interface and package-protected implementation details, which includes the data access code. if that new feature set c needs to access data related to a and b, it is forced to go through the public interface of components a and b. no direct access to the data access layer is allowed, and you can enforce this if you use java's access modifiers properly. again, "architectural layering" is a secondary organisation mechanism. for this to work, you have to stop using the public keyword by default . this structure raises some interesting questions about testing, not least about how we mock-out the data access code to create quick-running "unit tests". architecturally-aligned testing the short answer is don't bother, unless you really need to. i've spoken about and written about this before, but architecture and testing are related. instead of the typical testing triangle (lots of "unit" tests, fewer slower running "integration" tests and even fewer slower ui tests), consider this. i'm trying to make a conscious effort to not use the term "unit testing" because everybody has a different view of how big a "unit" is. instead, i've adopted a strategy where some classes can and should be tested in isolation. this includes things like domain classes, utility classes, web controllers (with mocked components), etc. then there are some things that are easiest to test as components, through the public interface. if i have a component that stores data in a mysql database, i want to test everything from the public interface right back to the mysql database. these are typically called "integration tests", but again, this term means different things to different people. of course, treating the component as a black box is easier if i have control over everything it touches. if you have a component that is sending asynchronous messages or using an external, third-party service, you'll probably still need to consider adding dependency injection points (e.g. ports and adapters) to adequately test the component, but this is the exception not the rule. all of this still applies if you are building a microservices style of architecture. you'll probably have some low-level class tests, hopefully a bunch of service tests where you're testing your microservices though their public interface, and some system tests that run scenarios end-to-end. oh, and you can still write all of this in a test-first, tdd style if that's how you work. i'm using this strategy for some systems that i'm building and it seems to work really well. i have a relatively simple, clean and (to be honest) boring codebase with understandable dependencies, minimal test-induced design damage and a manageable quantity of test code. this strategy also bridges the model-code gap , where the resulting code actually reflects the architectural intent. in other words, we often draw "components" on a whiteboard when having architecture discussions, but those components are hard to find in the resulting codebase. packaging code by layer is a major reason why this mismatch between the diagram and the code exists. those of you who are familiar with my c4 model will probably have noticed the use of the terms "class" and "component". this is no coincidence. architecture and testing are more related than perhaps we've admitted in the past. p.s. i'll be speaking about this topic over the next few months at events across europe, the us and (hopefully) australia

Clustering is a very important thing to master for any serious user of an application server. Clustering allows for high availability by making your application available on secondary servers when the primary instance is down or it lets you scale up or out by increasing the server density on the host, or by adding servers on other hosts. It can even help to increase performance with effective load balancing between servers based on their respective hardware. Andy Overton has already covered how to set up a cluster of servers in standalone mode fronted by mod_cluster for load balancing, so in this post I'll cover clustering in domain mode. I won't rehash mod_cluster settings, so this will just cover the set up of a doman controller on one host, and the host controller and server instances on another host. To follow along with this blog, you'll need to download either JBoss EAP 6.x or WildFly. I'll be using WildFly 8.2 on Xubuntu 14.04. I'll be using $WF_HOME to refer to your WildFly home directory. Configuring the Domain Controller The domain controller needs both the domain.xml and host.xml configured. In the $WF_HOME/domain/configuration directory, you'll see that those two files are joined by a host-master.xml and a host-slave.xml. These are preconfigured host.xml files which you can use to give you a head start in making a host.xml for the domain controller (master) and host controller (slave) to use. You can either change the name of the file to be host.xml, so it will get picked up and used by default, or you can specify the host configuration you want to use on the command line by adding the --host-config argument: domain.sh --host-config=host-master.xml Whether you choose to modify the host.xml or the host-master.xml, you need to make sure that the empty element has been added to the section. This is so that when WildFly looks to see which server is the domain controller, it knows to become the domain controller itself. The other change is optional, but recommended. We need to tell the domain controller to bind its management interface to the correct IP address because, by default, it will bind to localhost, so the management communication it needs to do with the remote hosts won't be able to reach the domain controller at all! We can set this address permanently in the host.xml by making sure the inet-address value is set to the right IP, by changing the 127.0.0.1 in the example below to the correct IP: The result of that is that the default bind IP of the management interface is no longer localhost, although you can still override this value by starting JBoss with the variable left of the colon as a -D argument: domain.sh -Djboss.bind.address.management=10.0.0.1 Next, we need to modify the domain.xml file, where we need to define our server groups; essentially just defining the cluster. Each server group is named, so we can reference it later, and references a particular profile which needs to be one of the profiles named and defined in the same XML file. As I mentioned in my previous blog, domain mode has several profiles in the same file (domain.xml) rather than multiple files for each, like standalone mode (standalone.xml, standalone-ha.xml etc.). In the screenshot, there are two server groups defined - "main-server-group" which references the "full" profile, and "other-server-group" which references the "full-ha" profile. These are just the defaults which come with WildFly, so you're free to use them and modify the settings or create your own from scratch. Whichever you choose, it's a good idea to rename your server group to something meaningful, like a description of the workload, or the application name. Configuring the Host Controllers Every host server which you want to be part of the cluster must have the host.xml file configured. We've already configured the host.xml on the domain controller, so now we'll focus on the host controller. Remember, this process can be repeated on any number of hosts, depending on how many servers you want in your server group and their topology. First, we need to make sure that the domain controller and the host controller can communicate, and to do that we need a valid management user. On the domain controller, run the add-user.sh or add-user.bat script. You will need to make sure to: Choose a management user Make sure the user is different than the one you would use to log in to the web console Confirm that the new user will connect one AS process to another AS process Make a note of the secret value (this is very important!) You will find that you get prompts similar to the following: mike@mike-C2B2:~$ /opt/wildfly/wildfly-8.2.0.Final/bin/add-user.sh What type of user do you wish to add? a) Management User (mgmt-users.properties) b) Application User (application-users.properties) (a): a Enter the details of the new user to add. Using realm 'ManagementRealm' as discovered from the existing property files. Username : mgmt Password recommendations are listed below. To modify these restrictions edit the add-user.properties configuration file. - The password should not be one of the following restricted values {root, admin, administrator} - The password should contain at least 8 characters, 1 alphabetic character(s), 1 digit(s), 1 non-alphanumeric symbol(s) - The password should be different from the username Password : Re-enter Password : What groups do you want this user to belong to? (Please enter a comma separated list, or leave blank for none)[ ]: About to add user 'mgmt' for realm 'ManagementRealm' Is this correct yes/no? yes Added user 'mgmt' to file '/opt/wildfly/wildfly-8.2.0.Final/standalone/configuration/mgmt-users.properties' Added user 'mgmt' to file '/opt/wildfly/wildfly-8.2.0.Final/domain/configuration/mgmt-users.properties' Added user 'mgmt' with groups to file '/opt/wildfly/wildfly-8.2.0.Final/standalone/configuration/mgmt-groups.properties' Added user 'mgmt' with groups to file '/opt/wildfly/wildfly-8.2.0.Final/domain/configuration/mgmt-groups.properties' Is this new user going to be used for one AS process to connect to another AS process? e.g. for a slave host controller connecting to the master or for a Remoting connection for server to server EJB calls. yes/no? yes To represent the user add the following to the server-identities definition Once we have the secret value for our management user, we can add it to the host.xml file. I'm choosing to modify the host-slave.xml file, since much of the configuration I need is done for me: Next, we need to tell the host controller where to look for the domain controller. We set this to for the domain controller's host.xml file, but in the host-slave.xml we have an example tag filled out for us. All we need to do is add the domain controller's IP or hostname exactly as we did for the management bind address earlier. So our host-slave.xml should go from this: to this: This way, like with the management interface on the domain controller, the default address will be 10.0.0.1, but it can also be overridden on the command line if needed. Once we've sorted the communication out, we need to tell the host controller to actually start some server instances! At the bottom of the host-slave.xml file, there are two predefined servers to use: These are already configured to become members of the two server groups configured in the domain.xml. Note that the second server has to have a port offset. Despite it being in a different server group, it's still going to run on the same host and will attempt to bind to the same ports as the first server unless we tell it not to! We would also need to do the same thing if we added other server instances. Optionally, we can make things a little easier for ourselves when managing a lot of servers on a lot of hosts. We can give each server instance its own unique name, but we can also name the host by adding a name attribute to the parent tag, changing it from: to So both in the logs and in the admin console, you should see this host controller referred to as "host1". Now, if you wanted to name your server instances the same across hosts, you'll be able to tell which is which! If all you wanted was to configure a single domain controller and a single host controller, then that's all we need to do to get them speaking to each other. You can then carry on and configure mod_cluster and Apache to forward requests on to the right server, or just deploy your applications and connect to them directly.

Apache Spark, Solr, and Zookeeper work together to create a fault-tolerant, distributed ETL system that converts RDBMS data into Solr documents.