DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Latest Articles - DZone

article thumbnail
Automating the build of MSI setup packages on Jenkins
a short "how-to" based on an issue one of my work mates recently faced when trying to automate the creation of an msi package on jenkins. normally, visual studio solutions can be build on jenkins by using the appropriate msbuild plugin . apparently though, for visual studio setup projects, msbuild cannot be used and one has to switch to using visual studio itself to execute the build. so the first approach was to use devenv.exe as follows devenv.exe visualstudiosolution.sln /build "release" while this works, the problem is that it is an "async call", meaning that the compilation goes on in the background while the console from which the build is executed, immediately returns. obviously this isn't suited for being used on jenkins. searching around for a while, it turned out that you have to use devenv.com instead of devenv.exe : "c:\program files (x86)\microsoft visual studio 10.0\common7\ide\devenv.com"visualstudiosolution.sln /build "release" once you got that, integrating everything into jenkins is quite straightforward: (obviously you may also simply set an environment variable pointing to devenv.com on your build server rather than indicating the entire path)
March 13, 2014
by Juri Strumpflohner
· 13,647 Views
article thumbnail
A Template for Formulating Great Sprint Goals
I find it helpful to consider three questions when choosing a sprint goal: Why do we carry out the sprint? How do we reach its goal? And how do we know that the goal has been met? My sprint goal template therefore consists of three main parts: the actual goal, the method employed to reach the goal, and the metrics to determine if the goal has been met. It additionally provides a header section that allows you to state to which product and sprint the goal belongs, as the picture below shows. You can download the template as a PDF from romanpichler.com/tools/sprint-goal-template/ or by clicking on the image below. The template above has grown out of my experience of working with Scrum for more than ten years, and it is inspired by the scientific method and Lean Startup. Let’s have a look at the template sections in more detail. The Goal Section The goal section states why it is worthwhile to undertake the sprint. Examples are: Test an assumption about the user interaction and learn what works best for the user, for instance: “Will users be willing to register before using the product features?” Address a technical risk such as: “Does the architecture enable the desired performance?” Release a feature, for instance: “Get the reporting feature for general release.” The sprint goal hence differs from listing the user stories that should be implemented. It communicates the reason for carrying out the work, and it provides a motivation for running the sprint. The sprint goal should be shared: The product owner and the development team should believe that working towards the goal is the right thing to do. To choose the right sprint goal I find it helpful to consider the amount of uncertainty present. In the early sprints, addressing risks and testing assumptions allows me to learn about what the product should look like and do and how it is built. Once the key risks and critical assumptions have been dealt with, I like to focus on completing and optimising features, as the following picture shows: The Method Section This section addresses the question of how the goal is met. The default Scrum answer is simple: Create a (potentially shippable) product increment using the high-priority product backlog items, and demo it to the stakeholders in the sprint review meeting. But writing software and employing a product demo are not always the best methods to achieve the goal! A paper prototype can be good enough to test a visual design idea or an assumption about the user interaction, for instance. What’s more, other methods such as carrying out a usability test or releasing software to run an A/B test may well be more effective than a product demo. You should therefore carefully choose the right method and state it in this section. But don’t stop there. Determine the test group, the people who should provide feedback and data. Who these individuals are depends on the sprint goal: If you are validating an assumption about the visual design, the user interaction or the product functionality, then you probably want to collect feedback and data from the users. But if you are addressing a technical risk, then users may not be able to help you. Consider inviting a senior developer or architect from another team instead. Stating the test group clarifies who “the stakeholders” are, who is required to provide feedback so that the right product is developed. The Metrics Section The metrics section communicates how you determine if the goal has been met. Which metrics you use depends on the method chosen. For a product demo, you may state that at least two thirds of the stakeholders present should respond positively to the new feature, for instance; for a usability test, at least three of the five testers are complete the task successfully in less than a minute; and for the release of a new feature, you might say that at least 80% of the users use the new functionality at least once within five days after launching the feature. Whichever metrics you choose, make sure that they allow you to understand if and to which extent you have met the goal. The Header Section The header section consists of the two subsections “Product” and “Sprint”. They simply allow you to state which product and which sprint the goal belongs to. Customise this section according to your needs. If you work for an agencies or an IT solution provider, you could replace “Product” with “Project”, for instance. User Stories and the Sprint Goal You may be wondering how the template relates to the user stories. Let me first reiterate that your sprint goal should differ from your user stories. The goal explains the why it is a good idea to carry out the sprint an implement the stories. The user stories enable you to reach the goal. It’s a common mistake to confuse the two. To connect the template and the stories you have two options: You can state the relevant user stories in the template’s method section, or you can list them separately on the sprint backlog, as the following picture illustrates. In the picture above, the sprint goal is stated on the left to the sprint backlog, which lists the user stories and the tasks required to meet the goal in form of a task board. Learn more You can learn more about choosing effective sprint gaols and applying the sprint goal template by attending my Certified Scrum Product Owner training course. I have written in more detail about sprint planning in my book “Agile Product Management with Scrum”. Please contact me for onsite and virtual product owner training.
March 12, 2014
by Roman Pichler
· 14,230 Views · 1 Like
article thumbnail
ActiveMQ - Network of Brokers Explained
Objective This 7 part blog series is to share about how to create network of ActiveMQ brokers in order to achieve high availability and scalability. Why network of brokers? ActiveMQ message broker is a core component of messaging infrastructure in an enterprise. It needs to be highly available and dynamically scalable to facilitate communication between dynamic heterogeneous distributed applications which have varying capacity needs. Scaling enterprise applications on commodity hardware is a rage nowadays. ActiveMQ caters to that very well by being able to create a network of brokers to share the load. Many times applications running across geographically distributed data centers need to coordinate messages. Running message producers and consumers across geographic regions/data centers can be architected better using network of brokers. ActiveMQ uses transport connectors over which it communicates with message producers and consumers. However, in order to facilitate broker to broker communication, ActiveMQ uses network connectors. A network connector is a bridge between two brokers which allows on-demand message forwarding. In other words, if Broker B1 initiates a network connector to Broker B2 then the messages on a channel (queue/topic) on B1 get forwarded to B2 if there is at least one consumer on B2 for the same channel. If the network connector was configured to be duplex, the messages get forwarded from B2 to B1 on demand. This is very interesting because it is now possible for brokers to communicate with each other dynamically. In this 7 part blog series, we will look into the following topics to gain understanding of this very powerful ActiveMQ feature: Network Connector Basics - Part 1 Duplex network connectors - Part 2 Load balancing consumers on local/remote brokers - Part 3 Load-balance consumers/subscribers on remote brokers Queue: Load balance remote concurrent consumers - Part 4 Topic: Load Balance Durable Subscriptions on Remote Brokers - Part 5 Store/Forward messages and consumer failover - Part 6 How to prevent stuckmessages Virtual Destinations - Part 7 To give credit where it is due, the following URLs have helped me in creating this blog post series. Advanced Messaging with ActiveMQ by Dejan Bosanac [Slides 32-36] Understanding ActiveMQ Broker Networks by Jakub Korab Prerequisites ActiveMQ 5.8.0 – To create broker instances Apache Ant – To run ActiveMQ sample producer and consumers for demo. We will use multiple ActiveMQ broker instances on the same machine for the ease of demonstration. Network Connector Basics - Part 1 The following diagram shows how a network connector functions. It bridges two brokers and is used to forward messages from Broker-1 to Broker-2 on demand if established by Broker-1 to Broker-2. A network connector can be duplex so messages could be forwarded in the opposite direction; from Broker-2 to Broker-1, once there is a consumer on Broker-1 for a channel which exists in Broker-2. More on this in Part 2 Setup network connector between broker-1 and broker-2 Create two broker instances, say broker-1 and broker-2 Ashwinis-MacBook-Pro:bin akuntamukkala$ pwd /Users/akuntamukkala/apache-activemq-5.8.0/bin Ashwinis-MacBook-Pro:bin akuntamukkala$ ./activemq-admin create ../bridge-demo/broker-1 Ashwinis-MacBook-Pro:bin akuntamukkala$ ./activemq-admin create ../bridge-demo/broker-2 Since we will be running both brokers on the same machine, let's configure broker-2 such that there are no port conflicts. Edit /Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-2/conf/activemq.xml Change transport connector to 61626 from 61616 Change AMQP port from 5672 to 6672 (won't be using it for this blog) Edit /Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-2/conf/jetty.xml Change web console port to 9161 from 8161 Configure Network Connector from broker-1 to broker-2 Add the following XML snippet to/Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-1/conf/activemq.xml The above XML snippet configures two network connectors "T:broker1->broker2" (only topics as queues are excluded) and "Q:broker1->broker2" (only queues as topics are excluded). This allows for nice separation between network connectors used for topics and queues. The name can be arbitrary although I prefer to specify the [type]:[source broker]->[destination broker]. The URI attribute specifies how to connect to broker-2 Start broker-2 Ashwinis-MacBook-Pro:bin akuntamukkala$ pwd /Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-2/bin Ashwinis-MacBook-Pro:bin akuntamukkala$ ./broker-2 console Start broker-1 Ashwinis-MacBook-Pro:bin akuntamukkala$ pwd /Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-1/bin Ashwinis-MacBook-Pro:bin akuntamukkala$ ./broker-1 console Logs on broker-1 show 2 network connectors being established with broker-2 INFO | Establishing network connection from vm://broker-1?async=false&network=true to tcp://localhost:61626 INFO | Connector vm://broker-1 Started INFO | Establishing network connection from vm://broker-1?async=false&network=true to tcp://localhost:61626 INFO | Network connection between vm://broker-1#24 and tcp://localhost/127.0.0.1:61626@52132(broker-2) has been established. INFO | Network connection between vm://broker-1#26 and tcp://localhost/127.0.0.1:61626@52133(broker-2) has been established. Web Console on broker-1 @ http://localhost:8161/admin/connections.jsp shows the two network connectors established to broker-2 The same on broker-2 does not show any network connectors since no network connectors were initiated by broker-2 Let's see this in action Let's produce 100 persistent messages on a queue called "foo.bar" on broker-1. Ashwinis-MacBook-Pro:example akuntamukkala$ pwd /Users/akuntamukkala/apache-activemq-5.8.0/example Ashwinis-MacBook-Pro:example akuntamukkala$ ant producer -Durl=tcp://localhost:61616 -Dtopic=false -Ddurable=true -Dsubject=foo.bar -Dmax=100 broker-1 web console shows that 100 messages have been enqueued in queue "foo.bar" http://localhost:8161/admin/queues.jsp Let's start a consumer on a queue called "foo.bar" on broker-2. The important thing to note here is that the destination name "foo.bar" should match exactly. Ashwinis-MacBook-Pro:example akuntamukkala$ ant consumer -Durl=tcp://localhost:61626 -Dtopic=false -Dsubject=foo.bar We find that all the 100 messages from broker-1's foo.bar queue get forwarded to broker-2's foo.bar queue consumer. broker-1 admin console at http://localhost:8161/admin/queues.jsp broker-2 admin console @ http://localhost:9161/admin/queues.jspshows that the consumer we had started has consumed all 100 messages which were forwarded on-demand from broker-1 broker-2 consumer details on foo.bar queue broker-1 admin console shows that all 100 messages have been dequeued [forwarded to broker-2 via the network connector]. broker-1 consumer details on "foo.bar" queue shows that the consumer is created on demand: [name of connector]_[destination broker]_inbound_[source broker] Thus we have seen the basics of network connector in ActiveMQ. As always, please feel to comment about anything that can be improved. Your inputs are welcome! Stay tuned for Part 2.
March 12, 2014
by Ashwini Kuntamukkala
· 40,173 Views · 2 Likes
article thumbnail
3 Reasons to Choose Vert.x
Vert.x is a lightweight, high performance application platform for the JVM Modern web applications and the rise of mobile clients redefined what is expected from a web server. Node.js was the first technology that recognized the paradigm shift and offered a solution. The application platform Vert.x takes some of the innovations from Node.js and makes them available on the JVM, combining fresh ideas with one of the most sophisticated and fastest runtime environments available. Vert.x comes with a set of exciting features that make it interesting for anybody developing web applications. Non-blocking, event driven runtime Vert.x provides a non-blocking, event-driven runtime. If a server has to do a task that requires waiting for a response (e.g. requesting data from a database) there are two possibilities how this can be implemented: blocking and non-blocking. The traditional approach is a synchronous or blocking call. The program flow pauses and waits for the answer to return. To be able to handle more than one request in parallel, the server would execute each request in a different thread. The advantage is a relatively simple programming model, but the downside is a significant amount of overhead if the number of threads becomes large. The second solution is a non-blocking call. Instead of waiting for the answer, the caller continues execution, but provides a callback that will be executed once data arrives. This approach requires a (slightly) more complex programming model, but has a lot less overhead. In general a non-blocking approach results in much better performance when a large number of requests need to be served in parallel. Simple to use concurrency and scalability A Vert.x application consists of loosely coupled components, which can be rearranged to match increasing performance requirements Vert.x applications are written using an Actor-like concurrency model. An application consists of several components, the so-called Verticles, which run independently. A Verticle runs single-threaded and communicates with other Verticles by exchanging messages on the global event-bus. Because they do not share state, Verticles can run in parallel. The result is an easy to use approach for writing multi-threaded applications.You can create several Verticles which are responsible for the same task and the runtime will distribute the workload among them, which means you can take full advantage of all CPU cores without much effort. Verticles can also be distributed between several machines. This will be transparent to the application code. The Verticles use the same mechanisms to communicate as if they would run on the same machine. This makes it extremely easy to scale your application. Vert.x supports the most popular languages on the JVM. Support for Scala and Clojure is on the way. Polyglot Unlike many other application platforms, Vert.x is polyglot. Applications can be written in several languages. It is even possible to use different languages in the same application. At this point Java, Python, Groovy, Ruby, and JavaScript can be used and support for Scala and Clojure is on the way. Conclusion Vert.x is a relatively young platform and subsequently the ecosystem is not as rich as that of the more established platforms. Nevertheless for the most common tasks, there are extensions available.The advantages of Vert.x are astonishing. Its non-blocking, event-driven nature is extremely well-suited for modern web applications. Vert.x makes it easy to write concurrent applications that scale effortless from a single low-end machine to a cluster with several high-end servers. Add the fact that you can use most popular languages for the JVM and you have a web developers dream come true.
March 11, 2014
by Michael Heinrichs
· 29,251 Views · 7 Likes
article thumbnail
Logging or Debugging
debugging is lame. you should debug log. if your code is structured you do not need debug logging. these are two opinions from the two ends of the line. i am, as usually, standing in the middle, and i will tell you why. first of all, there is no principal difference between debugging versus logging. they are just two different implementations of the same thing: observation of your execution engine state in time dimension. issue with debugging when you debug you step your program forward in time and at any point the execution stops you can examine the value of any variable. the shortage is that you can not step back in time. at some points you realize that you would just like to see what the value of a certain variable was just before some method was called, some object was created or whatsoever happened in the system. what you actually do in such a situation is to restart the code and hoping it behaves deterministic try to catch the execution at the earlier stage that you are interested in. and this is another shortage of debugging. you can not effectively debug a code that does not behave deterministic. and trust me: most bugs behave non deterministic. issue with logging with logs the major issue is different. it is not the time but rather the breadth of states, variables that you can look at is the problem. you insert log statements into your code dumping the values of variables into a log file at a certain point of execution. when you examine the log file you can scroll back and forth. however if you did not print out the value of a certain variable at a certain execution point, there is no way to get it from the log file. the solution is the same as with debugging: execute the code again, this time extended with the new log statements. if, however, you have enough information in your log files, then you will just get enough information to track down a bug even if that is not deterministic. only ‘if you have’ … solution: logging all the states all the times? the ideal solution would be to dump all variables into a possibly binary log file at each state of the execution and examine the content of the file afterwards. the examination would essentially look like a debugger, except that the change of the variables comes from the recorded log file instead of from on the fly calculation. it would be like a playback of a recorded execution and as such you could replay it several times. i do not know if there is any tool like that for the jvm. you just can not define what is “each state” effectively in a multi thread execution environment like the jvm is. this is one of the issues. the other thing is that if you’d start dumping the jvm memory after each command (forgetting the issues of multi-thread) it would require enormous amount of bandwidth and disk space. dreaming about the ideal solution not deliverable is sort of no use. what is the solution that can practically be executed? practical approach you can debug when it is appropriate. full stop. you just did that so far, keep doing that. i tend to use log statements even when i debug some code and if the environment allows it i do it on the fly. when i find the root cause of the issue i am hunting i review the log statements and i delete them. they did the job while debugging, they are not needed anymore. at least that was my practice unit i found myself writing log statements that i have already created before. why? because fixing one bug does not mean that i have fixed all of them. there is nothing like all bugs fixed. but the log items littered the log file and that just increased the work to find the needed information hunting the next bug. in other words the log file is full of noise and that is why i deleted these items the first place. but for the same reason i could also delete the unit tests that already pass. it would save a lot of time during compilation, wouldn’t it? we do not do that. summary in one sentence? log and debug the way it fits you and the issue you are hunting.
March 11, 2014
by Peter Verhas DZone Core CORE
· 5,183 Views
article thumbnail
Mock Constructor
Foreword If you already read some other blog post about unusual mocking, you can skip prelude via this link. I was asked to put together examples of how to mock Java constructs well know for their testability issues: Mock private method Mock final method Mock final class Mock constructor Mock static method I am calling these techniques unusual mocking. I was worried that such examples without any guidance can be widely used by teammates not deeply experienced in mocking frameworks. Developers practicing TDD or BDD should be aware of testability problems behind these constructs and try to avoid them when designing their tests and modules. That is the reason why you probably wouldn't be facing such unusual mocking often on project using these great programming methodologies. But sometimes you have to extend or maintain legacy codebase that usually contains low cohesive classes. In most cases there isn't time in current hectic agile world to make such class easy to unit test standard way. When you are trying to unit test such class you often realize that unusual mocking is needed. That is why I decided to create and share refactoring considerations alongside with examples and workarounds for unusual mocking. Examples are using Mockito and PowerMock mocking frameworks and TestNG unit testing framework. Mock constructor Refactoring considerations If your testing method creates instance/s of some type, there are two possibilities what can happen with these instances Created instance/s are returned from testing method. They are in this case part of the testing method API. This can be tested by verifying against created instances rather than constructor method call. If target instances doesn't have hashCode/equals contract implemented, you can still create test specific comparator to verify created data. Created instances are used as parameter/s passed to some dependency object. This dependency object of testing class is most probably mocked. In this case it's better idea to capture arguments of dependency method call and verify them. Mockito offers good support for this. Created instances are temporary objects that support testing method job. In this case you shouldn't care about creation of these instances, because you should treat testing module as black box that doing the job, but you don't know how. Create factory class for constructing instances and mock it standard way. If that fits to requirement -> Abstract factory design pattern Workaround using Mockito This is my preferred technique when I need to mock constructor. I believe that minor exposing of internal implementation in flavor to enhance testability of testing module is much lower risk for project than fall into bytecode manipulation mocking framework like PowerMock or JMockIt. This technique involves: Encapsulating the constructor into method with default access modifier Partial mock (spy) is used to mock this method during testing Mockito example covers: Partial mocking of factory method Verifying of mocked factory method call Class under test: public class CarFactoryMockito { Car carFactoryMethod(String type, String color) { return new Car(type, color); } public Car constructCar(String type, String color) { carFactoryMethod(type, color); // ... other logic needed to be tested ... return carFactoryMethod(type, color); } } Test: public class CarFactoryMockitoTest { private static final String TESTING_TYPE = "Tatra"; private static final String TESTING_COLOR = "Black"; @Test public void testConstructCar() { CarFactoryMockito carFactory = new CarFactoryMockito(); CarFactoryMockito carFactorySpy = Mockito.spy(carFactory); Car mockedInstance = Mockito.mock(Car.class); Mockito.doReturn(mockedInstance).when(carFactorySpy) .carFactoryMethod(TESTING_TYPE, TESTING_COLOR); // invoke testing method Car actualInstance = carFactorySpy.constructCar(TESTING_TYPE, TESTING_COLOR); Assert.assertEquals(actualInstance, mockedInstance); // ... verify other logic in constructCar() method ... Mockito.verify(carFactorySpy, Mockito.times(2)).carFactoryMethod( TESTING_TYPE, TESTING_COLOR); } } Usage of PowerMock Before usage of this example, please carefully consider if it is worth to bring bytecode manipulation risks into your project. They are gathered in this blog post. In my opinion it should be used only in very rare and non-avoidable cases. Test shows how to mock constructor directly by PowerMock. Example covers: Mocking of constructor Verifying of constructor call Class under test: public class CarFactoryPowerMock { public Car constructCar(String type, String color) { new Car(type, color); return new Car(type, color); } } Test: /** * Demonstrates constructor mocking by PowerMock. * * NOTE: Prepared in PowerMock annotation {@link PrepareForTest} should be class * where is constructor called */ @PrepareForTest(CarFactoryPowerMock.class) public class CarFactoryPowerMockTest extends PowerMockTestCase { private static final String TESTING_TYPE = "Tatra"; private static final String TESTING_COLOR = "Black"; @Test public void testConstructCar() throws Exception { Car expectedCar = Mockito.mock(Car.class); PowerMockito.whenNew(Car.class) .withArguments(TESTING_TYPE, TESTING_COLOR) .thenReturn(expectedCar); // invoke testing method CarFactoryPowerMock carFactory = new CarFactoryPowerMock(); Car actualCar = carFactory.constructCar(TESTING_TYPE, TESTING_COLOR); Assert.assertEquals(actualCar, expectedCar); // ... verify other logic in constructCar() method ... PowerMockito.verifyNew(Car.class, Mockito.times(2)).withArguments( TESTING_TYPE, TESTING_COLOR); } } Links Source code can be downloaded from Github. Other unusual mocking examples: Mock private method Mock final method Mock final class Mock static method
March 11, 2014
by Lubos Krnac
· 98,184 Views · 7 Likes
article thumbnail
Spring Boot & JavaConfig integration
Java EE in general and Context and Dependency Injection has been part of the Vaadin ecosystem since ages. Recently, Spring Vaadin is a joint effort of the Vaadin and the Spring teams to bring the Spring framework into the Vaadin ecosystem, lead by Petter Holmström for Vaadin and Josh Long for Pivotal. Integration is based on the Spring Boot project - and its sub-modules, that aims to ease creating new Spring web projects. This article assumes the reader is familiar enough with Spring Boot. If not the case, please take some time to get to understand basic notions about the library. Note that at the time of this writing, there's no release for Spring Vaadin. You'll need to clone the project and build it yourself. The first step is to create the UI. In order to display usage of Spring's Dependency Injection, it should use a service dependency. Let's injection the UI through Constructor Injection to favor immutability. The only addition to a standard UI is to annotate it with org.vaadin.spring.@VaadinUI. @VaadinUI public class VaadinSpringExampleUi extends UI { private HelloService helloService; public VaadinSpringExampleUi(HelloService helloService) { this.helloService = helloService; } @Override protected void init(VaadinRequest vaadinRequest) { String hello = helloService.sayHello(); setContent(new Label(hello)); } } The second step is standard Spring Java configuration. Let's create two configuration classes, one for the main context and the other for the web one. Two thing of note: The method instantiating the previous UI has to be annotated with org.vaadin.spring.@UIScope in addition to standard Spring org.springframework.context.annotation.@Bean to bind the bean lifecycle to the new scope provided by the Spring Vaadin library. At the time of this writing, a RequestContextListener bean must be provided. In order to be compliant with future versions of the library, it's a good practice to annotate the instantiating method with @ConditionalOnMissingBean(RequestContextListener.class). @Configuration public class MainConfig { @Bean public HelloService helloService() { return new HelloService(); } } @Configuration public class WebConfig extends MainConfig { @Bean @ConditionalOnMissingBean(RequestContextListener.class) public RequestContextListener requestContextListener() { return new RequestContextListener(); } @Bean @UIScope public VaadinSpringExampleUi exampleUi() { return new VaadinSpringExampleUi(helloService()); } } The final step is to create a dedicated WebApplicationInitializer. Spring Boot already offers a concrete implementation, we just need to reference our previous configuration classes as well as those provided by Spring Vaadin, namely VaadinAutoConfiguration and VaadinConfiguration. public class ApplicationInitializer extends SpringBootServletInitializer { @Override protected SpringApplicationBuilder configure(SpringApplicationBuilder application) { return application.showBanner(false) .sources(MainConfig.class) .sources(VaadinAutoConfiguration.class, VaadinConfiguration.class) .sources(WebConfig.class); } } At this point, we demonstrated a working Spring Vaadin sample application. Code for this article can be browsed and forked on Github.
March 10, 2014
by Nicolas Fränkel
· 13,553 Views
article thumbnail
WebLogic Classloader Analysis Tool
The WebLogic Server has a built-in webapp called Classloader Analysis Tool, and you may access it through http://localhost:7001/wls-cat You need to login with same user as you configured for the /console webapp. With the CAT, you may check what classes are loaded by your application in the server. This is extremely handy if your app is loading jar that's already loaded by the server. For example, if you include your own Apache commons-lang.jar in a webapp and deploy it, you will see that org.apache.commons.lang.time.DateUtils is not from your webapp! If you ever get an error saying DateUtils#addDay() doesn't exist or signature not match, then likely you are using different version than the one comes with WLS. In this case, you will need to add "WEB-INF/weblogic.xml" that change classloading behavior. Like this: true Another cool thing you can use this webapp to check is resources packaged inside any jars. For resource file, you must use # prefix to it. For example try look up #log4j.properties and you will see where it's loading from. You may read more about this tool and related material here:http://docs.oracle.com/cd/E24329_01/web.1211/e24368/classloading.htm
March 10, 2014
by Zemian Deng
· 30,249 Views · 2 Likes
article thumbnail
Exporting Spring Data JPA Repositories as REST Services using Spring Data REST
Spring Data modules provides various modules to work with various types of datasources like RDBMS, NOSQL stores etc in unified way. In my previous article SpringMVC4 + Spring Data JPA + SpringSecurity configuration using JavaConfig I have explained how to configure Spring Data JPA using JavaConfig. Now in this post let us see how we can use Spring Data JPA repositories and export JPA entities as REST endpoints using Spring Data REST. First let us configure spring-data-jpa and spring-data-rest-webmvc dependencies in our pom.xml. org.springframework.data spring-data-jpa 1.5.0.RELEASE org.springframework.data spring-data-rest-webmvc 2.0.0.RELEASE Make sure you have latest released versions configured correctly, otherwise you will encounter the following error: java.lang.ClassNotFoundException: org.springframework.data.mapping.SimplePropertyHandler Create JPA entities. @Entity @Table(name = "USERS") public class User implements Serializable { private static final long serialVersionUID = 1L; @Id @GeneratedValue(strategy = GenerationType.IDENTITY) @Column(name = "user_id") private Integer id; @Column(name = "username", nullable = false, unique = true, length = 50) private String userName; @Column(name = "password", nullable = false, length = 50) private String password; @Column(name = "firstname", nullable = false, length = 50) private String firstName; @Column(name = "lastname", length = 50) private String lastName; @Column(name = "email", nullable = false, unique = true, length = 50) private String email; @Temporal(TemporalType.DATE) private Date dob; private boolean enabled=true; @OneToMany(fetch=FetchType.EAGER, cascade=CascadeType.ALL) @JoinColumn(name="user_id") private Set roles = new HashSet<>(); @OneToMany(mappedBy = "user") private List contacts = new ArrayList<>(); //setters and getters } @Entity @Table(name = "ROLES") public class Role implements Serializable { private static final long serialVersionUID = 1L; @Id @GeneratedValue(strategy = GenerationType.IDENTITY) @Column(name = "role_id") private Integer id; @Column(name="role_name",nullable=false) private String roleName; //setters and getters } @Entity @Table(name = "CONTACTS") public class Contact implements Serializable { private static final long serialVersionUID = 1L; @Id @GeneratedValue(strategy = GenerationType.IDENTITY) @Column(name = "contact_id") private Integer id; @Column(name = "firstname", nullable = false, length = 50) private String firstName; @Column(name = "lastname", length = 50) private String lastName; @Column(name = "email", nullable = false, unique = true, length = 50) private String email; @Temporal(TemporalType.DATE) private Date dob; @ManyToOne @JoinColumn(name = "user_id") private User user; //setters and getters } Configure DispatcherServlet using AbstractAnnotationConfigDispatcherServletInitializer. Observe that we have added RepositoryRestMvcConfiguration.class to getServletConfigClasses() method. RepositoryRestMvcConfiguration is the one which does the heavy lifting of looking for Spring Data Repositories and exporting them as REST endpoints. package com.sivalabs.springdatarest.web.config; import javax.servlet.Filter; import org.springframework.data.rest.webmvc.config.RepositoryRestMvcConfiguration; import org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter; import org.springframework.web.servlet.support.AbstractAnnotationConfigDispatcherServletInitializer; import com.sivalabs.springdatarest.config.AppConfig; public class SpringWebAppInitializer extends AbstractAnnotationConfigDispatcherServletInitializer { @Override protected Class[] getRootConfigClasses() { return new Class[] { AppConfig.class}; } @Override protected Class[] getServletConfigClasses() { return new Class[] { WebMvcConfig.class, RepositoryRestMvcConfiguration.class }; } @Override protected String[] getServletMappings() { return new String[] { "/rest/*" }; } @Override protected Filter[] getServletFilters() { return new Filter[]{ new OpenEntityManagerInViewFilter() }; } } Create Spring Data JPA repositories for JPA entities. public interface UserRepository extends JpaRepository { } public interface RoleRepository extends JpaRepository { } public interface ContactRepository extends JpaRepository { } That's it. Spring Data REST will take care of rest of the things. You can use spring Rest Shell https://github.com/spring-projects/rest-shell or Chrome's Postman Addon to test the exported REST services. D:\rest-shell-1.2.1.RELEASE\bin>rest-shell http://localhost:8080:> Now we can change the baseUri using baseUri command as follows: http://localhost:8080:>baseUri http://localhost:8080/spring-data-rest-demo/rest/ http://localhost:8080/spring-data-rest-demo/rest/> http://localhost:8080/spring-data-rest-demo/rest/>list rel href ====================================================================================== users http://localhost:8080/spring-data-rest-demo/rest/users{?page,size,sort} roles http://localhost:8080/spring-data-rest-demo/rest/roles{?page,size,sort} contacts http://localhost:8080/spring-data-rest-demo/rest/contacts{?page,size,sort} Note: It seems there is an issue with rest-shell when the DispatcherServlet url mapped to "/" and issue list command it responds with "No resources found". http://localhost:8080/spring-data-rest-demo/rest/>get users/ { "_links": { "self": { "href": "http://localhost:8080/spring-data-rest-demo/rest/users/{?page,size,sort}", "templated": true }, "search": { "href": "http://localhost:8080/spring-data-rest-demo/rest/users/search" } }, "_embedded": { "users": [ { "userName": "admin", "password": "admin", "firstName": "Administrator", "lastName": null, "email": "[email protected]", "dob": null, "enabled": true, "_links": { "self": { "href": "http://localhost:8080/spring-data-rest-demo/rest/users/1" }, "roles": { "href": "http://localhost:8080/spring-data-rest-demo/rest/users/1/roles" }, "contacts": { "href": "http://localhost:8080/spring-data-rest-demo/rest/users/1/contacts" } } }, { "userName": "siva", "password": "siva", "firstName": "Siva", "lastName": null, "email": "[email protected]", "dob": null, "enabled": true, "_links": { "self": { "href": "http://localhost:8080/spring-data-rest-demo/rest/users/2" }, "roles": { "href": "http://localhost:8080/spring-data-rest-demo/rest/users/2/roles" }, "contacts": { "href": "http://localhost:8080/spring-data-rest-demo/rest/users/2/contacts" } } } ] }, "page": { "size": 20, "totalElements": 2, "totalPages": 1, "number": 0 } } You can find the source code at https://github.com/sivaprasadreddy/sivalabs-blog-samples-code/tree/master/spring-data-rest-demo For more Info on Spring Rest Shell: https://github.com/spring-projects/rest-shell
March 7, 2014
by Siva Prasad Reddy Katamreddy
· 30,040 Views
article thumbnail
The 5 Layers of PaaS
[Matt Butcher is a topic expert featured in the DZone 2014 Cloud Platform Research Report, which you can download for free.] Ask a cloud-savvy developer what PaaS is, and you will get an answer like this: A PaaS is a cloud service that lets developers deploy applications into the cloud without having to manage the underlying infrastructure layer. A year or two ago, PaaS systems were monolithic. A single vendor or solution, like Heroku, would provide one system that handled all aspects of PaaS. But things are changing. With a plethora of Open Source tools like Docker, Packer, Serf, CoreOS, Dokku, and Flynn, it is now possible to build your own PaaS. But what exactly makes up a PaaS? I will take a functional approach to defining PaaS by asking what are the things that a PaaS does? PaaS can be viewed as a workflow with several functional phases. Each phase accomplishes a specific goal in the process of moving an application onto a production platform. The phases are not necessarily serial steps. They may run in parallel, and not in the order listed below. The five functional phases of a PaaS are: Deployment Provisioning Lifecycle management Service management Reporting 1. Deployment The deployment phase is responsible for moving an application from its source (typically a developer's machine) to the PaaS. Some of the common ways of doing this include: Running a git remote on the PaaS and handling git push events from clients. (Heroku, OpenShift, Flynn and Dokku all use this method. Elastic Beanstalk uses a variation on this.) Sending the code as a bundle (often a gzipped tar). Cloud Foundry uses this method, as does Stackato. Compiling the code locally and copying the resulting executable to the PaaS. When a PaaS receives a deployment, it kicks off processes to move that app into a running state. The exact order of those processes varies, so I will keep them in the order in which they appeared above. 2. Provisioning In the provisioning phase, the PaaS sets up the infrastructure necessary for running the app. "Infrastructure" is a broad and sometimes nebulous term, but here are some common provisioning targets: Setting up containers and/or compute instances Configuring networking Installing or configuring operating system services (e.g. Apache) Installing or configuring libraries (e.g. Ruby Gems) Many PaaS systems spread provisioning responsibilities across multiple tools. One tool may create a compute instance, while another tool may install libraries. But all are sharing the same responsibility: create the environment in which the application will run. 3. Lifecycle Management Once the PaaS has a copy of the app as well as an environment capable of running the app, it needs to manage the execution of the app. This is lifecycle management. Common tasks of lifecycle management include: Starting the app Monitoring the app's running state Monitoring or reporting on the app's resource consumption Restarting an app upon failure Stopping or restarting the app on command Some minimal PaaS systems offer only basic lifecycle management (e.g. start and stop), while highly sophisticated ones may include autoscaling, auto-throttling, and hot (zero-downtime) deployments. 4. Service Management This phase is not one that all PaaS layers perform. In fact, I would go so far as to say that it is not a mandatory piece of PaaS, But it certainly is useful when present. All PaaS systems run applications (that is, after all, what they're for). But some go a step beyond and provide services that may be attached to an application. These services run outside of the application container or compute instance. Services might include: Databases Networked file systems Message queues Caches Aggregated logging "Old guard" systems (like Cloud Foundry) share a service (e.g. MySQL) across multiple applications. Some of the newer container-based approaches like CoreOS may supplant this model by making it simpler to run services in specially-designated containers. (Check out the Serf project for a similar approach.) Why don't all PaaS systems need this layer? One reason is that many cloud providers already have comparable services in the form of DBaaS, MQaaS, and so on. 5. Reporting and Monitoring This final phase is the most banal. Most of the application's lifecycle is not spent on deployment or provisioning or service management. It's spent running. During an applications life, there are many interesting things that can occur. There are lifecycle events that we'd like to know about, like restarts. There are environmental conditions of interest, like resource utilization and system performance. And, of course, there is application data that we would like to monitor, like log files and application metrics. Many, but by no means all, PaaS platforms provide at least some level of reporting. Here are some examples: Amazon Elastic Beanstalk integrates with AWS Cloud Watch, and also aggregates system log files per application. ActiveState Stackato provides a web console with copious logs, and can show real-time statistics about an application and its surrounding environment. Heroku can optionally send events to a Loggly backend (which is a service). Conclusion: PaaS and Mini-PaaS As we've seen, each functional phase of PaaS can be done to greater or lesser degrees of complexity. Old guard PaaS systems often come feature-packed. But with PaaS building blocks like Docker, Flynn, and CoreOS, building a special purpose tailored mini-PaaS is not out of the question. Just take a look at Deis and Dokku for solutions with varying degrees of complexity.
March 7, 2014
by Matt Butcher
· 21,800 Views · 5 Likes
article thumbnail
XML to Avro Conversion
We all know what XML is right? Just in case not, no problem here is what it is all about. 5 Now, what the computer really needs is the number five and some context around it. In XML you (human and computer) can see how it represents context to five. Now lets say instead you have a business XML document like FPML 32.00 150000 1.00 EUR 405000 2001-07-17Z NONE EUR 2.70 ISDA2002 ISDA2002Equity TODO GBEN Party A Party B That is a lot of extra unnecessary data points. Now lets look at this using Apache Avro. With Avro, the context and the values are separated. This means the schema/structure of what the information is does not get stored or streamed over and over and over and over (and over) again. The Avro schema is hashed. So the data structure only holds the value and the computer understands the fingerprint (the hash) of the schema and can retrieve the schema using the fingerprint. 0x d7a8fbb307d7809469ca9abcb0082e4f8d5651e46d3cdb762d02d0bf37c9e592 This type of implementation is pretty typical in the data space. When you do this you can reduce your data between 20%-80%. When I tell folks this they immediately ask, “why such a large gap of unknowns”. The answer is because not every XML is created the same. But that is the problem because you are duplicating the information the computer needs to understand the data. XML is nice for humans to read, sure … but that is not optimized for the computer. Here is a converter we are working on https://github.com/stealthly/xml-avro to help get folks off of XML and onto lower cost, open source systems. This allows you to keep parts of your systems (specifically the domain business code) using the XML and not having to be changed (risk mitigation) but store and stream the data with less overhead (optimize budget).
March 7, 2014
by Joe Stein
· 27,208 Views
article thumbnail
call_once for C#
One of the useful gems that made it into C++11 Standard Template Libraries (STD) is call_once, this nifty little method makes sure that specific code is called only once (duh) and it follows these 3 rules: Exactly one execution of exactly one of the functions (passed as f to the invocations in the group) is performed. It is undefined which function will be selected for execution. The selected function runs in the same thread as thecall_once invocation it was passed to. No invocation in the group returns before the abovementioned execution of the selected function is completed successfully, that is, doesn't exit via an exception. If the selected function exits via exception, it is propagated to the caller. Another function is then selected and executed. I needed something similar – I had a method that should only be called once (initialize) and I wanted to implement something similar to the call_once I’ve been using for my C++ development. My first object was to try and make it as preferment as possible and so I’ve looked for a solution that does not involve locks: public static class Call { public static void Once(OnceFlag flag, Action action) { if (flag.CheckIfNotCalledAndSet) { action.Invoke(); } } } since I was trying to mimic the C++ code I wrote two objects Call (above) and OnceFlag which has all of the magic inside using Interlocked: public class OnceFlag { private const int NotCalled = 0; private const int Called = 1; private int _state = NotCalled; internal bool CheckIfCalledAndSet { get { var prev = Interlocked.Exchange(ref _state, Called); return prev == NotCalled; } } internal void Reset() { Interlocked.Exchange(ref _state, NotCalled); } } I’m using Interlocked as a thread-safe way to check & set the value making sure that only once it would return true – try it: class Program { static OnceFlag _flag = new OnceFlag(); static void Main(string[] args) { var t1 = new Thread(() => DoOnce(1)); var t2 = new Thread(() => DoOnce(2)); var t3 = new Thread(() => DoOnce(3)); var t4 = new Thread(() => DoOnce(4)); t1.Start(); t2.Start(); t3.Start(); t4.Start(); t1.Join(); t2.Join(); t3.Join(); t4.Join(); } private static void DoOnce(int index) { Call.Once(_flag, () => Console.WriteLine("Callled (" + index + ")")); } } It’s very simple solution unfortunately not entirely correct – the method used will only be called once, but requirements 2 & 3 were not implemented. Luckily for me I didn’t need to make sure that exception enable another call to pass through nor did I need to block other calls until the first call finishes. But I wanted to try and write a proper implementation, unfortunately not as preferment due to the use of locks: public static void Once(OnceFlagSimple flag, Action action) { lock (flag) { if (flag.CheckIfNotCalled) { action.Invoke(); flag.Set(); } } } But it works, and since I’m already using lock I can split the check and Set methods and use a bool value inside the flag instead of Interlocked. All other threads are blocked due to lock until first finish running – check! In case of exception other method can execute the once block – check! If exited properly the block would only execute once – check! But not very good performance due to locking even after the first time run. I’m still looking for a better way to implement call_once – it’s a good exercise in threading and I might find a cool new ways to use the classes under Threading or Task namespaces. please let me know if you have a better implementation – that’s what the comments are for…
March 6, 2014
by Dror Helper
· 7,052 Views
article thumbnail
Convert CSV Data to Avro Data
In one of my previous posts I explained how we can convert json data to avro data and vice versa using avro tools command line option. Today I was trying to see what options we have for converting csv data to avro format, as of now we don't have any avro tool option to accomplish this . Now, we can either write our own java program (MapReduce program or a simple java program) or we can use various SerDe's available with Hive to do this quickly and without writing any code :) To convert csv data to Avro data using Hive we need to follow the steps below: Create a Hive table stored as textfile and specify your csv delimiter also. Load csv file to above table using "load data" command. Create another Hive table using AvroSerDe. Insert data from former table to new Avro Hive table using "insert overwrite" command. To demonstrate this I will use use the data below (student.csv): 0,38,91 0,65,28 0,78,16 1,34,96 1,78,14 1,11,43 Now execute below queries in Hive: --1. Create a Hive table stored as textfile USE test; CREATE TABLE csv_table ( student_id INT, subject_id INT, marks INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE; --2. Load csv_table with student.csv data LOAD DATA LOCAL INPATH "/path/to/student.csv" OVERWRITE INTO TABLE test.csv_table; --3. Create another Hive table using AvroSerDe CREATE TABLE avro_table ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' TBLPROPERTIES ( 'avro.schema.literal'='{ "namespace": "com.rishav.avro", "name": "student_marks", "type": "record", "fields": [ { "name":"student_id","type":"int"}, { "name":"subject_id","type":"int"}, { "name":"marks","type":"int"}] }'); --4. Load avro_table with data from csv_table INSERT OVERWRITE TABLE avro_table SELECT student_id, subject_id, marks FROM csv_table; Now you can get data in Avro format from Hive warehouse folder. To dump this file to local file system use below command: hadoop fs -cat /path/to/warehouse/test.db/avro_table/* > student.avro If you want to get json data from this avro file you can use avro tools command: java -jar avro-tools-1.7.5.jar tojson student.avro > student.json So we can easily convert csv to avro and csv to json also by just writing 4 HQLs.
March 5, 2014
by Rishav Rohit
· 39,718 Views · 1 Like
article thumbnail
How to Install R Packages with Ansible
Here is a short snippet of Ansible playbook that installs R and any required packages to any nodes of the cluster: - name: Making sure R is installed apt: pkg=r-base state=installed - name: adding a few R packages command: /usr/bin/Rscript --slave --no-save --no-restore-history -e "if (! ('{{item}' %in% installed.packages()[,'Package'])) install.packages(pkgs={{item}, repos=c('http://www.freestatistics.org/cran/'))" with_items: - rjson - rPython - plyr - psych - reshape2 You should replace the repos with one chosen from the list of Cran mirrors. Note that the command above installs each package only if it is not already present, but messes up the “changed” status of Ansible’s PLAY RECAP by incorrectly reporting a change per R package at every run. Find more big data technical posts on my blog.
March 5, 2014
by Svend Vanderveken
· 6,196 Views
article thumbnail
Lessons Learned: ActiveMQ, Apache Camel and Connection Pooling
Every once in a while, I run into an interesting problem related to connections and pooling with ActiveMQ, and today I’d like to discuss something that is not always very clear and could potentially cause you to drink heavily when using ActiveMQ and Camel JMS. Not to say that you won’t want to drink heavily when using ActiveMQ and Camel anyway… in celebration of how delightful integration and messaging become when using them of course. So first up. Connection pooling. Sure, you’ve always heard to pool your connections. What does that really mean, and why do you want to do it? Opening up a connection to an ActiveMQ broker is a relativley expensive operation when compared to other actions like creating a session or consumer. So when sending or receiving messages and generally interacting with the broker, you’d like to reuse existing connections if possible. What you don’t want to do is rely on a JMS library (like Spring JmsTemplate for example) that opens and closes connections for each send or receive of a message… unless you can pool/cache your connections. So if we can agree that pooling connections is a good idea, take a look at an example config: You may even want to use Apache Camel and its wonderful camel-jms component because doing otherwise would just be silly. So maybe you want to set up a JMS config similar to so: This config basically means for consumers, set up 15 concurrent consumers, use transactions (local), use PERSISTENT messages for producers, set a timeout for 10000 for request-reply etc, etc. Huge note: If you want a more thorough taste of the configs for the jms component, especially around caching consumers, transactions and more, please take a look at Torsten’s excellent blog on Camel JMS with transactions – lesson learned. Maybe you should also spend some time poking around his blog as he’s got lots of good Camel/ActiveMQ stuff too Awesome so far. We have a connection pool of 10 connections, we will expect 10 sessions per connection (for a total of 100 sessions if we needed that…), and 15 concurrent consumers. We should be able to deal with some serious load, right? Take a look at this route here. It’s simple enough, exposes the activemq component (which will use the jmsConfig from above, so 15 concurrent consumers) and just does some logging: from("activemq:test.queue") .routeId("test.queue.routeId") .to("log:org.apache.camel.blog?groupSize=100"); Try and run this. You will find your consumers blocked up right away and stack traces will show this beauty: "Camel (camel-1) thread #1 - JmsConsumer[test.queue]" daemon prio=5 tid=7f81eb4bc000 nid=0x10abbb000 in Object.wait() [10abba000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <7f40e9070> (a org.apache.commons.pool.impl.GenericKeyedObjectPool$Latch) at java.lang.Object.wait(Object.java:485) at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1151) - locked <7f40e9070> (a org.apache.commons.pool.impl.GenericKeyedObjectPool$Latch) at org.apache.activemq.pool.ConnectionPool.createSession(ConnectionPool.java:146) at org.apache.activemq.pool.PooledConnection.createSession(PooledConnection.java:173) at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196) .... How can that possibly be? We have connection pooling… we have sessions per connection set to 10 per connection, so how are we all blocked up on creating new sessions? The answer is you’re exhausting the number of sessions, as you can expect by the stack trace. But how? And how much do I need to drink to resolve this? Well hold on now. Grab a beer and hear me out. First understand this. ActiveMQ’s pooling implementation uses commons-pool and the maxActiveSessionsPerConnection attribute is actually mapped to the maxActive property of the underlying pool. From the docs this means: maxActive controls the maximum number of objects (per key) that can allocated by the pool (checked out to client threads, or idle in the pool) at one time. The key here is “key” (literally… the ‘per key’ clause of the documentation). So in the ActiveMQ implementation the key is an object that represents 1) whether the session mode is transacted and 2) what the acknowledgement mode is () as seen here. So in plain terms, you’ll end up with a “maxActive” sessions for each key that’s used on that connection.. so if you have clients that use transactions, no transactions, client-ack, auto-ack, transacted-session, dups-okay, etc you can start to see that you’d end up with “maxActive” sessions for each permutation. So if you have maxActiveSesssionsPerConnection set to 10, you could really end up with 10 x 2 x 4 == 80 sessions. This is something to tuck away in the back of your mind. The second key here is that when the camel-jms component sets up consumers, it ends up sharing a single connection among all the consumers specified by the concurrentConsumers session. This is an interesting point, because camel-jms uses the underlying Spring framework’s DefaultMessageListenerContainer and unfortunately this restriction comes from that library. So if you have 15 concurrent consumers, they will all share a single connection (even if pooling… it will grab one connection from the pool and hold it). So if you have 15 consumers that each share a connection, each share a transacted mode, each share an ack mode, then you end up trying to create 15 sessions for that one connection. And you end up with the above. So my rule of thumb for avoiding these scenarios: Understand exactly what each of your producers and consumers are doing, what their TX and ACK modes are Always tune the max sessions param when you NEED to (too many session threads? i dunno..) but always do concurrentConsumers+1 as the value AT LEAST If producers and consumers are producing/consuming the same destination SPLIT UP THE CONNECTION POOL: one pool for consumers, one pool for producers Dunno how valuable this info will be, but I wanted to jot it down for myself. If someone else finds it valuable, or has questions, let me know in the comments.
March 4, 2014
by Christian Posta
· 26,447 Views · 2 Likes
article thumbnail
When to Use MongoDB Rather than MySQL (or Other RDBMS): The Billing Example
NoSQL has been a hot topic a pretty long time (well, it's not only a buzz anymore). However, when should we really use it instead of an RDBMS?
March 3, 2014
by Moshe Kaplan
· 378,986 Views · 12 Likes
article thumbnail
Step-by-Step: Live Migrate Multiple (Clustered) VMs in One Line of PowerShell - Revisited
A while back, I wrote an article showing how to Live Migrate Your VMs in One Line of Powershell between non-clustered Windows Server 2012 Hyper-V hosts using Shared Nothing Live Migration. Since then, I’ve been asked a few times for how this type of parallel Live Migration would be performed for highly available virtual machines between Hyper-V hosts within a cluster. In this article, we’ll walk through the steps of doing exactly that … via Windows PowerShell on Windows Server 2012 or 2012 R2 or our FREE Hyper-V Server 2012 R2 bare-metal, enterprise-grade hypervisor in a clustered configuration. Wait! Do I need PowerShell to Live Migrate multiple VMs within a Cluster? Well, actually … No. You could certainly use the Failover Cluster Manager GUI tool to select multiple highly available virtual machines, right-click and select Move | Live Migration … Failover Cluster Manager – Performing Multi-VM Live Migration But, you may wish to script this process for other reasons … perhaps to efficiently drain all VM’s from a host as part of a maintenance script that will be performing other tasks. Can I use the same PowerShell cmdlets for Live Migrating within a Cluster? Well, actually … No again. When VMs are made highly available resources within a cluster, they’re managed as cluster group resources instead of being standalone VM resources. As a result, we have a different set of Cluster-aware PowerShell cmdlets that we use when managing these cluster groups. To perform a scripted multi-VM Live Migration, we’ll be leveraging three of these cmdlets: Get-ClusterNode, Get-ClusterGroup and Move-ClusterVirtualMachineRole Now, let’s see that one line of PowerShell! Before getting to the point of actually performing the multi-VM Live Migration in a single PowerShell command line, we first need to setup a few variables to handle the "what" and "where" of moving these VMs. First, let’s specify the name of the cluster with which we’ll be working. We’ll store it in a $clusterName variable. $clusterName = read-host -Prompt "Cluster name" Next, we’ll need to select the cluster node to which we’ll be Live Migrating the VMs. Lets use the Get-ClusterNode and Out-GridView cmdlets together to prompt for the cluster node and store the value in a $targetClusterNode variable. $targetClusterNode = Get-ClusterNode -Cluster $clusterName | Out-GridView -Title "Select Target Cluster Node" ` -OutputMode Single And then, we’ll need to create a list of all the VMs currently running in the cluster. We can use the Get-ClusterGroup cmdlet to retrieve this list. Below, we have an example where we are combining this cmdlet with a Where-Object cmdlet to return only the virtual machine cluster groups that are running on any node except the selected target cluster node. After all, it really doesn’t make any sense to Live Migrate a VM to the same node on which it’s currently running! $haVMs = Get-ClusterGroup -Cluster $clusterName | Where-Object {($_.GroupType -eq "VirtualMachine") ` -and ($_.OwnerNode -ne $targetClusterNode.Name)} We’ve stored the resulting list of VMs in a $haVMs variable. Ready to Live Migrate! OK … Now we have all of our variables defined for the cluster, the target cluster node and the list of VMs from which to choose. Here’s our single line of PowerShell to do the magic … $haVMs | Out-GridView -Title "Select VMs to Move" –PassThru | Move-ClusterVirtualMachineRole -MigrationType Live ` -Node $targetClusterNode.Name -Wait 0 Proceed with care: Keep in mind that your target cluster node will need to have sufficient available resources to run the VM's that you select for Live Migration. Of course, it's best to initially test tasks like this in your lab environment first. Here’s what is happening in this single PowerShell command line: We’re passing the list of VMs stored in the $haVMs variable to the Out-GridView cmdlet. Out-GridView prompts for which VMs to Live Migrate and then passes the selected VMs down the PowerShell object pipeline to the Move-ClusterVirtualMachineRole cmdlet. This cmdlet initiates the Live Migration for each selected VM, and because it’s using a –Wait 0 parameter, it initiates each Live Migration one-after-another without waiting for the prior task to finish. As a result, all of the selected VMs will Live Migrate in parallel, up to the maximum number of concurrent Live Migrations that you’ve configured on these cluster nodes. The VMs selected beyond this maximum will simply queue up and wait their turn. Unlike some competing hypervisors, Hyper-V doesn't impose an artificial hard-coded limit on how many VMs for you can Live Migrate concurrently. Instead, it's up to you to set the maximum to a sensible value based on your hardware and network capacity. Do you have your own PowerShell automation ideas for Hyper-V? Feel free to share your ideas in the Comments section below. See you in the Clouds! - Keith
March 3, 2014
by Keith Mayer
· 10,743 Views
article thumbnail
Java 8: Lambda Expressions vs Auto Closeable
If you used earlier versions of Neo4j via its Java API with Java 6 you probably have code similar to the following to ensure write operations happen within a transaction: public class StylesOfTx { public static void main( String[] args ) throws IOException { String path = "/tmp/tx-style-test"; FileUtils.deleteRecursively(new File(path)); GraphDatabaseService db = new GraphDatabaseFactory().newEmbeddedDatabase( path ); Transaction tx = db.beginTx(); try { db.createNode(); tx.success(); } finally { tx.close(); } } } In Neo4j 2.0 Transaction started extending AutoCloseable which meant that you could use ‘try with resources’ and the ‘close’ method would be automatically called when the block finished: public class StylesOfTx { public static void main( String[] args ) throws IOException { String path = "/tmp/tx-style-test"; FileUtils.deleteRecursively(new File(path)); GraphDatabaseService db = new GraphDatabaseFactory().newEmbeddedDatabase( path ); try ( Transaction tx = db.beginTx() ) { Node node = db.createNode(); tx.success(); } } } This works quite well although it’s still possible to have transactions hanging around in an application when people don’t use this syntax – the old style is still permissible. In Venkat Subramaniam’s Java 8 book he suggests an alternative approach where we use a lambda based approach: public class StylesOfTx { public static void main( String[] args ) throws IOException { String path = "/tmp/tx-style-test"; FileUtils.deleteRecursively(new File(path)); GraphDatabaseService db = new GraphDatabaseFactory().newEmbeddedDatabase( path ); Db.withinTransaction(db, neo4jDb -> { Node node = neo4jDb.createNode(); }); } static class Db { public static void withinTransaction(GraphDatabaseService db, Consumer fn) { try ( Transaction tx = db.beginTx() ) { fn.accept(db); tx.success(); } } } } The ‘withinTransaction’ function would actually go on GraphDatabaseService or similar rather than being on that Db class but it was easier to put it on there for this example. A disadvantage of this style is that you don’t have explicit control over the transaction for handling the failure case – it’s assumed that if ‘tx.success()’ isn’t called then the transaction failed and it’s rolled back. I’m not sure what % of use cases actually need such fine grained control though. Brian Hurt refers to this as the ‘hole in the middle pattern‘ and I imagine we’ll start seeing more code of this ilk once Java 8 is released and becomes more widely used.
March 3, 2014
by Mark Needham
· 8,360 Views
article thumbnail
Python CSV Files: Reading and Writing
Learn to parse CSV (Comma Separated Values) files with Python examples using the csv module's reader function and DictReader class.
March 3, 2014
by Mike Driscoll
· 375,754 Views · 6 Likes
article thumbnail
Generating a War File From a Plain IntelliJ Web Project
Sometimes you just want to create a quick web project in IntelliJ IDEA, and you would use their wizard and with web or Java EE module as starter project. But these projects will not have Ant nor Maven script generated for you automatically, and the IDEA Build would only compile your classes. So if you want an war file generated, try the following: 1) Menu: File > Project Structure > Artifacts 2) Click the green + icon and create a "Web Application: Archive", then OK 3) Menu: Build > Build Artifacts ... > Web: war By default it should generate it under your /out/artifacts/web_war.war Note that IntelliJ also allows you to setup "Web Application: Exploded" artifact, which great for development that run and deploy to an application server within your IDE.
March 3, 2014
by Zemian Deng
· 76,334 Views · 2 Likes
  • Previous
  • ...
  • 1508
  • 1509
  • 1510
  • 1511
  • 1512
  • 1513
  • 1514
  • 1515
  • 1516
  • 1517
  • ...
  • Next
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook
×