DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

The Latest Coding Topics

article thumbnail
From Vaadin to Docker - A Novice's Journey
I’m a huge Vaadin fan and I’ve created a Github workshop I can demo at conferences. A common issue with such kind of workshops is that attendees have to prepare their workstations in advance… and there’s always a significant part of them that comes with not everything ready. At this point, two options are available to the speaker: either wait for each of the attendee to finish the preparation – too bad for the people who took the time at home to do that, or start anyway – and lose the not-ready part. Given the current buzz around Docker, I thought that could be a very good way to make the workshop preparation quicker – only one step, and hasslefree – no problem regarding the quirks of your operation system. The required steps I ask the attendees are the following: Install Git Install Java, Maven and Tomcat Clone the git repo Build the project (to prepare the Maven repository) Deploy the built webapp Start Tomcat These should directly be automated into Docker. As I wasted much time getting this to work, here’s the tale of my journey in achieving this (be warned, it’s quite long). If you’ve got similar use-cases, I hope it will be useful in you getting things done faster. Starting with Docker The first step was to get to know the basics about Docker. Fortunately, I had the chance to attend a Docker workshop by David Gageot at Duchess Swiss. This included both Docker installation and basics of Dockerfile. I assume readers have likewise a basic understanding of Docker. For those who don’t, I guess browsing the Docker’s official documentation is a nice idea: Installation Dockerfile reference Building my first Dockerfile The Docker image can be built with the following command ran into the directory of the Dockerfile: $ docker build -t vaadinworkshop . The first issues one can encounter when playing with Docker the first time, is to get the following error message: Get http:///var/run/docker.sock/v1.14/containers/json: dial unix /var/run/docker.sock: no such file or directory The reason is because one didn’t export the required environment variables displayed by the boot2docker information message. If you lost the exact data, no worry, just use the shellinit boot2docker parameter: $ boot2docker shellinit Writing /Users/i303869/.docker/boot2docker-vm/ca.pem: Writing /Users/i303869/.docker/boot2docker-vm/cert.pem: Writing /Users/i303869/.docker/boot2docker-vm/key.pem: export DOCKER_HOST=tcp://192.168.59.103:2376 export DOCKER_CERT_PATH=/Users/i303869/.docker/boot2docker-vm Copy-paste the export lines above will solve the issue. These can also be set in one’s .bashrc script as it seems these values seldom change. Next in line is the following error: Get http://192.168.59.103:2376/v1.14/containers/json: malformed HTTP response "x15x03x01x00x02x02" This error message seems to be because of a mismatch between versions of the client and the server. It seems it is because of a bug on Mac OSX when upgrading. For a long term solution, reinstall Docker from scratch; for a quick fix, use the --tls flag with the docker command. As it is quite cumbersome to type it everything, one can alias it: $ alias docker="docker --tls" My last mistake when building the image comes from building the Dockerfile from a not empty directory. Docker sends every file it finds in the directory of the Dockerfile to the Docker container for build: $ docker --tls build -t vaadinworkshop . Sending build context to Docker daemon Too many kB Fix: do not try this at home and start from a directory container the Dockerfile only. Starting from scratch Dockerfiles describe images – images are built as a layered list of instructions. Docker images are designed around single inheritance: one image has to be set a single parent. An image requiring no parent starts from scratch, but Docker provides 4 base official distributions: busybox, debian, ubuntu and centos (operating systems are generally a good start). Whatever you want to achieve, it is necessary to choose the right parent. Given the requirements I set for myself (Java, Maven, Tomcat and Git), I tried to find the right starting image. Many Dockerfiles are already available online on the Docker hub. The browsing app is quite good, but to be really honest, the search can really be improved. My intention was to use the image that matched the most of my requirements, then fill the gap. I could find no image providing Git, but I thought the dgageot/maven Dockerfile would be a nice starting point. The problem is that the base image is a busybox and provides no installer out-of-the-box (apt-get, yum, whatever). For this reason, David uses a lot of curl to get Java 8 and Maven in his Dockerfiles. I foolishly thought I could use a different flavor of busybox that provides the opkg installer. After a while, I accumulated many problems, resolving one heading to another. In the end, I finally decided to use the OS I was most comfortable with and to install everything myself: FROM ubuntu:utopic Scripting Java installation Installing git, maven and tomcat packages is very straightforward (if you don’t forget to use the non-interactive options) with RUN and apt-get: RUN apt-get update && \ apt-get install -y --force-yes git maven tomcat8 Java doesn’t fall into this nice pattern, as Oracle wants you to accept the license. Nice people did however publish it to a third-party repo. Steps are the following: Add the needed package repository Configure the system to automatically accept the license Configure the system to add un-certified packages Update the list of repositories At last, install the package Also add a package for Java 8 system configuration. RUN echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu precise main" | tee -a /etc/apt/sources.list && \ echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections && \ apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886 RUN apt-get update && \ apt-get install -y --force-yes oracle-java8-installer oracle-java8-set-default Building the sources Getting the workshop’s sources and building them is quite straightforward with the following instructions: RUN git clone https://github.com/nfrankel/vaadin7-workshop.git WORKDIR /vaadin7-workshop RUN mvn package The drawback of this approach is that Maven will start from a fresh repository, and thus download the Internet the first time it is launched. At first, I wanted to mount a volume from the host to the container to share the ~/.m2/repository folder to avoid this, but I noticed this could only be done at runtime through the -v option as the VOLUME instruction cannot point to a host directory. Starting the image The simplest command to start the created Docker image is the following: $ docker run -p 8080:8080 Do not forget the port forwarding from the container to the host, 8080 for the standard HTTP port. Also, note that it’s not necessary to run the container as a daemon (with the -d option). The added value of that is that the standard output of the CMD (see below) will be redirected to the host. When running as a daemon and wanting to check the logs, one has to execute bash in the container, which requires a sequence of cumbersome manipulations. Configuring and launching Tomcat Tomcat can be launched when starting the container by just adding the following instruction to the Dockerfile: CMD ["catalina.sh", "run"] However, trying to start the container at this point will result in the following error: Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/common/classes], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/common], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/server/classes], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/server], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/shared/classes], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/shared], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.Catalina initDirs SEVERE: Cannot find specified temporary folder at /usr/share/tomcat8/temp Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.Catalina load WARNING: Unable to load server configuration from [/usr/share/tomcat8/conf/server.xml] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.Catalina initDirs SEVERE: Cannot find specified temporary folder at /usr/share/tomcat8/temp Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.Catalina load WARNING: Unable to load server configuration from [/usr/share/tomcat8/conf/server.xml] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.Catalina start SEVERE: Cannot start server. Server instance is not configured. I have no idea why, but it seems Tomcat 8 on Ubuntu is not configured in any meaningful way. Everything is available but we need some symbolic links here and there as well as creating the temp directory. This translates into the following instruction in the Dockerfile: RUN ln -s /var/lib/tomcat8/common $CATALINA_HOME/common && \ ln -s /var/lib/tomcat8/server $CATALINA_HOME/server && \ ln -s /var/lib/tomcat8/shared $CATALINA_HOME/shared && \ ln -s /etc/tomcat8 $CATALINA_HOME/conf && \ mkdir $CATALINA_HOME/temp The final trick is to connect the exploded webapp folder created by Maven to Tomcat’s webapps folder, which it looks for deployments: RUN mkdir $CATALINA_HOME/webapps && \ ln -s /vaadin7-workshop/target/workshop-7.2-1.0-SNAPSHOT/ $CATALINA_HOME/webapps/vaadinworkshop At this point, the Holy Grail is not far away, you just have to browse the URL… if only we knew what the IP was. Since running on Mac, there’s an additional VM beside the host and the container that’s involved. To get this IP, type: $ boot2docker ip The VM's Host only interface IP address is: 192.168.59.103 Now, browsing http://192.168.59.103:8080/vaadinworkshop/ will bring us to the familiar workshop screen: Developing from there Everything works fine but didn’t we just forget about one important thing, like how workshop attendees are supposed to work on the sources? Easy enough, just mount the volume when starting the container: docker run -v /Users//vaadin7-workshop:/vaadin7-workshop -p 8080:8080 vaadinworkshop Note that the host volume must be part of /Users and if on OSX, it must use boot2docker v. 1.3+. Unfortunately, it seems now is the showstopper, as mounting an empty directory from the host to the container will not make the container’s directory available from the host. On the contrary, it will empty the container’s directory given that the host’s directory doesn’t exist… It seems there’s an issue in Docker on Mac. The installation of JHipster runs into the same problem, and proposes to use the Samba Docker folder sharing project. I’m afraid I was too lazy to go further at this point. However, this taught me much about Docker, its usages and use-cases (as well as OSX integration limitations). For those who are interested, you’ll find below the Docker file. Happy Docker! FROM ubuntu:utopic MAINTAINER Nicolas Frankel # Config to get to install Java 8 w/o interaction RUN echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu precise main" | tee -a /etc/apt/sources.list && echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections && apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886 RUN apt-get update && apt-get install -y --force-yes git oracle-java8-installer oracle-java8-set-default maven tomcat8 RUN git clone https://github.com/nfrankel/vaadin7-workshop.git WORKDIR /vaadin7-workshop RUN git checkout v7.2-1 RUN mvn package ENV JAVA_HOME /usr/lib/jvm/java-8-oracle ENV CATALINA_HOME /usr/share/tomcat8 ENV PATH $PATH:$CATALINA_HOME/bin # Configure Tomcat 8 directories RUN ln -s /var/lib/tomcat8/common $CATALINA_HOME/common && ln -s /var/lib/tomcat8/server $CATALINA_HOME/server && ln -s /var/lib/tomcat8/shared $CATALINA_HOME/shared && ln -s /etc/tomcat8 $CATALINA_HOME/conf && mkdir $CATALINA_HOME/temp && mkdir $CATALINA_HOME/webapps && ln -s /vaadin7-workshop/target/workshop-7.2-1.0-SNAPSHOT/ $CATALINA_HOME/webapps/vaadinworkshop VOLUME ["/vaadin7-workshop"] CMD ["catalina.sh", "run"] # docker build -t vaadinworkshop . # docker run -v ~/vaadin7-workshop training/webapp -p 8080:8080 vaadinworkshop
November 25, 2014
by Nicolas Fränkel
· 13,084 Views
article thumbnail
Spring Integration Java DSL 1.0 GA Released
[This article was written by Artem Bilan.] Dear Spring community, As we promised in the Release Candidate blog post, we are pleased to announce that the Spring Integration Java DSL 1.0 GA is now available. As usual, use the Release Repository with Maven or Gradle, or download a distribution archive, to give it a spin. See the project home page for more information. First of all, we are glad to share with you that on Nov 12, 2014, DZone research recognized Spring Integration as the leader in the ESB / Integration framework space, leading with 42% marketshare, in a publication of their recent survey results. And the report is the most popular DZone Guide in November, with more than 12 000 downloads already! Don't miss it: very exciting. We hope the release of the Spring Integration Java DSL adds more excitement!. Many thanks to all contributors, including several who are new to the community. The release includes just a few bug fixes, since the release candidate, and a lot of JavaDocs! Not specifically related to the the release, I want to present here some resources on the matter. We are observing many valuable DSL questions on Stack Overflow. Josh Long's tech tip showing how we can use together Spring Boot, REST, Spring Integration 4.1 WebSocket support and Spring Integration Java DSL plus Java 8 features. The Jdbc Splitter implementation in the project tests. My gist to demonstrate how we can use Reactor Streams together with the Spring Integration Java DSL. Dave Syer has started to use Spring Integration Java DSL in the Spring Cloud Bus project. Don't miss the si4demo to see the evolution of Spring Integration including the Java DSL, as shown at the 2014 SpringOne/2GX Conference. (Video should be available soon). Especial thanks to Biju Kunjummen who has done some nice articles on DZone to introduce Spring Integration Java DSL: https://dzone.com/articles/spring-integration-java-dsl, https://dzone.com/articles/spring-integration-java-dsl-0. And of course, with the latest Spring XD, we can build Modules based on @Configuration including Spring Integration Java DSL IntegrationFlow definitions. Just after this announcement I'm going to publish a DSL Tutorial to explain concepts and features using the Java DSL version of the Cafe Demo sample as material. As always, we look forward to your comments and feedback (StackOverflow (spring-integration tag), Spring JIRA, GitHub) and we very much welcome contributions!
November 25, 2014
by Pieter Humphrey
· 5,156 Views
article thumbnail
How Does Elasticsearch Real-time Search?
Compared to other features, real-time search capability is undoubtedly one of the most important features in Elasticsearch. Today we’ll look closely how is provided real-time search by Elasticsearch. Real time First of all, if we need to explain the concept of real-time, in general, we can say that the delay between input and out time in the information is small at real-time systems. This means, data is taken without data accumulation, processed in real time. Today, the best solution Elasticsearch known for real-time search, when a record is added to it for storage makes it searchable in 1 second. How? As is known, the disks are able to create a risk of bottleneck for I/O operations at the data persistence step. Also some mechanisms used for prevent any loss of data increases cost of time. At this point Elasticsearch uses the file-system cache that sitting between itself and the disk for overcome the risk of bottleneck and ensure the a new document can be searched in real time. A new segment is written to the file-system cache first and only later it flushed to disk by Elasticsearch. This lightweight process of writing and opening a new segment is called a refresh in Elasticsearch. By default, all shards is refreshed automatically once every second. In this way, Elasticsearch support real-time search. Test time Above digression about the time of refresh of the shards you can bring to mind the following questions: What happens, when a new document is requested in less than 1 second time? Can be documents requested, without having to depend of the refresh period shards of managed by Elasticsearch? Short answers. Elasticsearch does not return the document. Yes. Now let’s get clarity on this issue is a simple example. hakdogan$ curl -XPUT localhost:9200/kodcucom/document/1 -d'{ > "title": "Document A" > }' We sent a document to Elasticsearch. The index name is kodcucom, type document, id value 1. The title field is only field in the document and the value of "Document A". Let’s take this document from Elasticsearch. hakdogan$ curl -XGET localhost:9200/kodcucom/document/1?pretty { "_index" : "kodcucom", "_type" : "document", "_id" : "1", "_version" : 1, "found" : true, "_source":{ "title": "Document A" } } As expected, the document was returned to us. Well, if we keep short the time between document recording and get request than default shard refresh time what will happen? Let’s see. hakdogan$ curl -XPUT localhost:9200/kodcucom/document/2 -d'{"title": "Document B"}'; curl -XGET localhost:9200/kodcucom/_search?pretty {"_index":"kodcucom","_type":"document","_id":"2","_version":1,"created":true}{ "took" : 38, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 1.0, "hits" : [ { "_index" : "kodcucom", "_type" : "document", "_id" : "1", "_score" : 1.0, "_source":{ "title": "Document A" } } ] } } As can be seen, only the previous document was returned to us by Elasticsearch when we do concurrently create and get request. Well, how can I get the document concurrently? Let’s see. hakdogan$ curl -XPUT localhost:9200/kodcucom/document/3 -d'{"title": "Document C"}'; curl -XGET localhost:9200/kodcucom/_refresh; curl -XGET localhost:9200/kodcucom/_search?pretty {"_index":"kodcucom","_type":"document","_id":"3","_version":1,"created":true}{"_shards":{"total":10,"successful":5,"failed":0}{ "took" : 3, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 3, "max_score" : 1.0, "hits" : [ { "_index" : "kodcucom", "_type" : "document", "_id" : "1", "_score" : 1.0, "_source":{ "title": "Document A" } }, { "_index" : "kodcucom", "_type" : "document", "_id" : "2", "_score" : 1.0, "_source":{"title": "Document B"} }, { "_index" : "kodcucom", "_type" : "document", "_id" : "3", "_score" : 1.0, "_source":{"title": "Document C"} } ] } } In this command, we perform to refresh operation on kodcucom index before the search request. In this way, the document was returned to us. Auto refresh time can be changed. By setting the index.refresh_interval parameter in the configuration file. Applies to all indices in the cluster. A per-index basis by updated index setting. In addition to these, you can turn off automatic refresh. An important point to keep in mind about the refresh time of the shards, the refresh operation is costly in terms of system resources. If you wished to make changes to the auto-refresh time, this situation should be taken into account. Extension of the automatic refresh time, enables faster indexing but new documents and changes made to the existing documents will not appear in searches during specified period of time.
November 25, 2014
by Hüseyin Akdoğan DZone Core CORE
· 17,596 Views
article thumbnail
Externalizing Session State for a Spring Boot Application Using Spring-Session
Spring-session is a very cool new project that aims to provide a simpler way of managing sessions in Java based web applications. One of the features that I explored with spring-session recently was the way it supports externalizing session state without needing to fiddle with the internals of specific web containers like Tomcat or Jetty. To test spring-session I have used a shopping cart type application(available here) which makes heavy use of session by keeping the items added to the cart as a session attribute, as can be seen from these screenshots: Consider first a scenario without Spring session. So this is how I have exposed my application: I am using nginx to load balance across two instances of this application. This set-up is very easy to run using Spring boot, I brought up two instances of the app up using two different server ports, this way: mvn spring-boot:run -Dserver.port=8080 mvn spring-boot:run -Dserver.port=8082 and this is my nginx.conf to load balance across these two instances: events { worker_connections 1024; } http { upstream sessionApp { server localhost:8080; server localhost:8082; } server { listen 80; location / { proxy_pass http://sessionApp; } } } I display the port number of the application in the footer just to show which instance is handling the request. If I were to do nothing to move the state of the session out the application then the behavior of the application would be erratic as the session established on one instance of the application would not be recognized by the other instance - specifically if Tomcat receives a session id it does not recognize then the behavior is to create a new session. Introducing Spring session into the application There are container specific ways to introduce a external session stores - One example is here, where Redis is configured as a store for Tomcat. PivotalGemfire provides a module to externalize Tomcat's session state. The advantage of using Spring-session is that there is no dependence on the container at all - maintaining session state becomes an application concern. The instructions on configuring an application to use Spring session is detailed very well at the Spring-session site, just to quickly summarize how I have configured my Spring Boot application, these are first the dependencies that I have pulled in: org.springframework.session spring-session 1.0.0.BUILD-SNAPSHOT org.springframework.session spring-session-data-redis 1.0.0.BUILD-SNAPSHOT org.springframework.data spring-data-redis 1.4.1.RELEASE redis.clients jedis 2.4.1 and my configuration to use Spring-session for session support, note the Spring Boot specific FilterRegistrationBean which is used to register the session repository filter: import org.springframework.boot.context.embedded.FilterRegistrationBean; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.core.annotation.Order; import org.springframework.data.redis.connection.jedis.JedisConnectionFactory; import org.springframework.session.data.redis.config.annotation.web.http.EnableRedisHttpSession; import org.springframework.session.web.http.SessionRepositoryFilter; import org.springframework.web.filter.DelegatingFilterProxy; import java.util.Arrays; @Configuration @EnableRedisHttpSession public class SessionRepositoryConfig { @Bean @Order(value = 0) public FilterRegistrationBean sessionRepositoryFilterRegistration(SessionRepositoryFilter springSessionRepositoryFilter) { FilterRegistrationBean filterRegistrationBean = new FilterRegistrationBean(); filterRegistrationBean.setFilter(new DelegatingFilterProxy(springSessionRepositoryFilter)); filterRegistrationBean.setUrlPatterns(Arrays.asList("/*")); return filterRegistrationBean; } @Bean public JedisConnectionFactory connectionFactory() { return new JedisConnectionFactory(); } } And that is it! magically now all session is handled by Spring-session, and neatly externalized to Redis. If I were to retry my previous configuration of using nginx to load balance two different Spring-Boot applications using the common Redis store, the application just works irrespective of the instance handling the request. I look forward to further enhancements to this excellent new project. The sample application which makes use of Spring-session is available here: https://github.com/bijukunjummen/shopping-cart-cf-app.git
November 24, 2014
by Biju Kunjummen
· 41,259 Views · 2 Likes
article thumbnail
Spring component scan for beans with no no-args constructor
Suppose you have the following spring set-up component scan enabled a class (MailServer) with some constructors (no no-args constructor) annotated with @Component (or @Service / @Named) declaration of above class in the spring-config (Even though you have component-scan enabled, suppose you need to configure the default mail server with a name "defaultMailServer" ) injecting the above "defaultMailServer" to a property of another class (i.e. MailSender) Spring Configuration MailServer.java @Component public class MailServer { private String host; private int port; private String protocol; public MailServer(final String host, final int port, final String protocol) { this.host = host; this.port = port; this.protocol = protocol; } //getters } MailSender.java @Service public class MailSenderImpl implements MailSender { @Autowired @Qualifier("defaultMailServer") private MailServer mailServer; @Override public void send() { // TODO } } Main.java public class Main { private static final String CONFIG_PATH = "classpath*:application-config.xml"; public static void main(final String[] args) { final ApplicationContext context = new ClassPathXmlApplicationContext(CONFIG_PATH); final MailSender mailSender = context.getBean(MailSenderImpl.class); mailSender.send(); } } Nothing looks wrong. But when you execute the program you will get the following exception. org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [me.fahimfarook.mail.MailServer]: No default constructor found; This error message is misleading because we are not accessing MailServer with default constructor anywhere in our code. We have injected "defaultMailServer" which is properly defined with constructor-args. However, if you add a no-args constructor to MailServer this exception will go away. That means spring context is trying to create a MailServer object using no-args constructor somewhere even though we have defined a MailServer with constructor-args. But we have defined a MailServer bean with constructor-args and accessing that bean only? The reason for that is - we have defined a bean explicitly in the spring config xml while we have enabled component scanning. Spring will create the "defaultMailServer" bean successfully since constructor-args have been defined in the spring config xml. However, since component scanning has been enabled, spring will try to automatically discover and register another MailServer bean. However, this will fail because spring can't create an object using the defined constructor as we have not auto-wired the constructor / parameters in the constructor. As the second step, spring will try to create a MailServer with no-args constructor. Therefore you need to have auto-wired the constructor which you defined in the spring-config file in order to get rid of this error. MailServer.java - Fixed @Component public class MailServer { private String host; private int port; private String protocol; @Autowired public MailServer(@Value("")final String host, @Value("#{new Integer(-1)}")final int port, @Value("")final String protocol) { this.host = host; this.port = port; this.protocol = protocol; } } Now that you have auto-wired the constructor which is defined with "defaultMailServer", spring can create this bean and it will not demand a default constructor. However, if spring had thrown BeanInstantiationException with a message like "Constructor for defaultMailServer could not be found" instead of "No default constructor found", the exact issue with the code could have been identified easily. In summary: If component scanning is enabled, spring will try to create a bean even though a bean of that class has already been defined in the spring config xml. However if the bean defined in the spring config file and the auto-discovered bean have the same name, spring will not to create a new bean while it does component scanning. If a bean does not have a no-args constructor, at-least one of the constructors must be auto-wired. If no constructor is auto-wired, spring will try to create an object using default no-args constructor. Original post here
November 23, 2014
by Fahim Farook
· 43,216 Views · 1 Like
article thumbnail
Spring - Accessing injected properties from constructor
In Spring, a bean is instantiated before its properties are injected. That is: Instantiate the bean first Inject the properties This is because spring uses setter methods of the instantiated bean in order to inject properties. (This is true if the property injection is declared in the spring configuration file, however if the bean properties are auto-wired - no setter methods are required. But still spring instantiates the bean first before auto-wiring properties.) Therefore you cannot access the properties from your constructor (unless properties are injected through constructor-args) as the properties are still holding null/ default values when accessed within the constructor. Consider the following example. Spring configuration file Engine.java package me.fahimfarook.sample.spring; import java.util.logging.Logger; public class Engine { private String make; private int cylinders; private static Logger LOGGER = Logger.getLogger(Engine.class.getName()); public void cleanup() { LOGGER.info("Engine::cleanup - cleaning-up engine."); } public String getMake() { return make; } public void setMake(final String make) { this.make = make; } public int getCylinders() { return cylinders; } public void setCylinders(final int cylinders) { this.cylinders = cylinders; } } Car.java package me.fahimfarook.sample.spring; public class Car { private String brand; private int year; private Engine engine; public Car() { this.engine.cleanup(); } public String getBrand() { return brand; } public void setBrand(final String brand) { this.brand = brand; } public int getYear() { return year; } public void setYear(final int year) { this.year = year; } public Engine getEngine() { return engine; } public void setEngine(final Engine engine) { this.engine = engine; } } As you can see I'm calling engine.start() from the constructor of the Car. Main.java package me.fahimfarook.sample.spring; import org.springframework.context.ApplicationContext; import org.springframework.context.support.ClassPathXmlApplicationContext; public class Main { private static final String CONFIG_PATH = "classpath*:configs/spring/applicaton-config.xml"; public static void main(String[] args) { final ApplicationContext context = new ClassPathXmlApplicationContext(CONFIG_PATH); final Car car = (Car)context.getBean("car"); } } When you execute this main method you will get a NullPointerException. Caused by: java.lang.NullPointerException at me.fahimfarook.sample.spring.Car.start(Car.java:18) at me.fahimfarook.sample.spring.Car.(Car.java:14) Even though we have configured spring to inject engine into car, by the time of instantiation (of car bean), spring has not injected the engine. We have two workarounds to solve this problem. 1. From a design perspective this NullPointerException indicates a code smell. That is the relationship between the car and engine should be "composition" rather than "aggregation". The engine must cease to exist when car is destroyed. Also, the engine cannot exist without a car. In order to enforce composition between car and engine we need the following 2 changes. Add a constructor to Car with an Engine parameter Remove the setEngine() mutator Here's the fixed code. Spring config - Fixed Note that engine is injected via a constructor-arg and is removed. Car.java - Fixed package me.fahimfarook.sample.spring; public class Car { private String brand; private int year; private Engine engine; public Car(final Engine engine) { this.engine = engine; this.engine.start(); } public String getBrand() { return brand; } public void setBrand(final String brand) { this.brand = brand; } public int getYear() { return year; } public void setYear(final int year) { this.year = year; } public Engine getEngine() { return engine; } // public void setEngine(final Engine engine) { // this.engine = engine; // } } Note that the setter of engine has been commented out and and Car is having a constructor with an Engine parameter. Now if you execute the main method you will not get the NullPointerException anymore as we have forced spring to inject an engine while construction of the car. 2. The other workaround would be to move the logic in the constructor to a separate method and annotate that method with @PostConstruct annotation. @PostConstruct is a standard Java annotation which is supported by spring. @PostConstruct indicates that the method annotated with @PostConstruct must be invoked once all its dependencies are injected only. However in our example you need to have enabled in order to support annotations. You can find more information about this annotation here.
November 23, 2014
by Fahim Farook
· 20,994 Views · 2 Likes
article thumbnail
Adding Gzip Compression in CXF APIs and Interceptors
Nowadays it has become mandatory to Gzipping the APIs response due to huge amount of data we are sending in response. It saves network bandwidth and delivery time and off course space over the internet. While using CXF; it provides an option to use the Gzip Compression in no of ways. Blueprint Annotation Blueprint: Annotation: First you need to register the GZIPOutInterceptor in out interceptors list. For that you need to hook into CXF initialization classes. public class InterceptorManager extends AbstractFeature { private static final Logger LOGGER = Logger.getLogger( "simcore" ); private static final Interceptor< Message > GZIP = new GZIPOutInterceptor(); //private static final Interceptor< Message > GZIP = new GZIPOutInterceptor(512); /* (non-Javadoc) * @see org.apache.cxf.feature.AbstractFeature#initializeProvider(org.apache.cxf.interceptor.InterceptorProvider, org.apache.cxf.Bus) */ @Override protected void initializeProvider( InterceptorProvider provider, Bus bus ) { /** * Adding Gzip interceptor to all outbound requests/responses */ LOGGER.debug( " ############## Adding Gzip as OUT Interceptor ##############" ); provider.getOutInterceptors().add( GZIP ); } } GZIPOutInterceptor comes with an option to set the Threshold value as no of Bytes. If response size will be below this threshold value then it will not be compressed. It is extremely useful when we will be sending empty lists and status messages/codes only. Because compressing those small responses will be overhead at server side. But there is another factor which is no of users requesting the response. So set this value by thinking over all the cases in mind. @GZIP Now we can use this annotation on any of our web-services controller to implement compression on all the APIs provided in that class. @WebService @Consumes ( { MediaType.TEXT_PLAIN, MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON } ) @Produces ( MediaType.APPLICATION_JSON ) @GZIP public interface WebServicesController { @GET @Path ( "/myGzipData" ) @Produces ( { MediaType.APPLICATION_JSON } ) Response getZipData( ); } Moreover we can set different parameters in Gzip annotation. @GZIP ( force = true, threshold = 512 )
November 22, 2014
by Shan Arshad
· 13,509 Views · 1 Like
article thumbnail
A XSS filter for Java EE web apps
Cross Site Scripting, or XSS, is a fairly common vector used to attack web sites. It involves user generated code being redisplayed by a website with all the privileges and security rights that a browser assigns to code originating from the current host. If the user code is something like , then you have a problem. OWASP is an organisation that provides guidance on web security, and they have a page that provides a suggested method for avoiding XSS in JavaEE web app. You can read this document at https://www.owasp.org/index.php/How_to_add_validation_logic_to_HttpServletRequest. The library being demonstrated here is based off the ideas presented in that article, but fleshed out to be more flexible and easy to deploy. We call this library the (unimaginatively named) Parameter Validation Filter, or PVF. PVF is implemented as a Servlet filter that intercepts requests to web pages, runs submitted parameters through a configurable sequence of validation rules, and either sanitises the parameters before they are sent through to the web application, or returns a HTTP error code if validation errors were detected. We have made the following assumptions when developing this library: Client side validation will prevent legitimate users from submitting invalid data. The PVF library should prevent further processing if invalid data is submitted in the majority of cases. Occasionally it might be appropriate to sanitise submitted data, but any sanitisation should be trivial (like the removal of whitespace). To make use of the PVF library, you’ll need to add it to your project. This artifact is currently in the Sonatype staging repo, so you'll need to add that repo to your Maven config. See http://stackoverflow.com/questions/13945757/how-do-you-import-a-maven-dependency-from-sonatype-org for details. com.matthewcasperson parameter_validation_filter LATEST The filter then needs to be added to the web.xml file with the following settings. You may want to configure the url-pattern to match the pages that you actually want to protect. ParameterValidationFilter com.matthewcasperson.validation.filter.ParameterValidationFilter configFile /WEB-INF/xml/pvf.xml ParameterValidationFilter *.jsp Finally you need to create a file called WEB-INF/xml/pvf.xml. This file defines the custom validation rules applied to the parameters being sent to your web applications. true com.matthewcasperson.validation.ruleimpl.TrimTextValidationRule com.matthewcasperson.validation.ruleimpl.FailIfNotCanonicalizedValidationRule com.matthewcasperson.validation.ruleimpl.FailIfContainsHTMLValidationRule .* .* false false The XML has been commented to make it easier to understand, but there are a few interesting elements: paramNamePatternString, which has been configured to enable the validation chain to match all parameters requestURIPatternString, which has been configured to enable the chain to match all URIs The three elements called validationRuleName, which reference the full class name of the validation rules that will be applied to each parameter passed into our web application Although this is a simple example, the three validation rules that have been implemented (TrimTextValidationRule, FailIfNotCanonicalizedValidationRule and FailIfContainsHTMLValidationRule) are quite effective at preventing a malicious user from submitting parameters that contain XSS code. The first rule, TrimTextValidationRule, simply strips away any whitespace on either side of the parameter. This uses the trim() function any developer should be familiar with. The second rule, FailIfNotCanonicalizedValidationRule, will prevent further processing if the supplied parameter has already been encoded. No legitimate user will have a need to supply text like %3Cscript%3EdoEvil()%3B%3C%2Fscript%3E, so any time encoded text is found we simply return with a HTTP 400 error code. This rule makes use of the ESAPI library supplied by OWASP. Like the second rule, the third rule will prevent further processing if the supplied parameter has any special HTML characters. If you would like your customers to be able to pass through characters like &, this rule is too broad. However, it is almost always valid to block special HTML characters. If you want to see how effective this simple validation chain is, check out the live demo at http://pvftest-matthewcasperson.rhcloud.com/. You may want to take a look at https://www.owasp.org/index.php/XSS_Filter_Evasion_Cheat_Sheet to find some XSS patterns that are often used to bypass XSS filters. Moving forward we will be looking to implement more targeted validation rules, especially those that can’t be easily implemented as regex matches (like making sure a date if after today, or that a number is between two values etc). If you have any suggestions, or find any bugs, feel free to fork the code from our GitHub repo . We do hope to get some public feedback in order to make this library as robust as it can be.
November 22, 2014
by Matthew Casperson
· 21,742 Views · 3 Likes
article thumbnail
Angular JS: Two Ways to Initialize an Angular App
This article represents code samples along with related concepts for two different ways in which an Angular app can be defined.
November 21, 2014
by Ajitesh Kumar
· 35,011 Views
article thumbnail
NewTypes Aren't As Cool As You Think
My last post talked about what’s wrong with type classes (in general, but also specifically in Haskell). This post generated some great feedback on Reddit, including some valid criticism that I didn’t explain why I hated on newtypes so much. I took some of that feedback and incorporated it into a revised version of the post, but I have even more to say about “newtypes," so I decided to write another blog post. What’s in a Newtype Newtypes are a feature of Haskell that let you define a new type in terms of an existing type. In the following example, I create a newtype for Email, which “holds” a String. newtype Email = Email String They are similar to type synonyms (type Email = String), except that type synonyms don’t create new types, they just allow you to refer to existing types by other names. Every newtype can be easily translated into a data declaration. In fact, only the keyword changes: data Email = Email String There’s a slight semantic difference between the two, but for purposes of this blog post, any criticism I have against newtypes apply equally to similar constructs modeled with type or data. The Promise of Newtypes Newtypes are used to provide and select between alternate implementations of type classes for some base types. I think that’s a hack (albeit a necessary one), but I’ve already talked about this so I won’t belabor it here. The other promise of newtypes is that we can use them to make our code more type safe. Instead of passing around String as an email, for example, we can create a super lightweight “wrapper” around String called Email, and make it an error to use a String wherever an Email is expected. This practice isn’t restricted to Haskell. Even in Java, it’s considered good coding practice to wrap primitives with classes whose names denote the meaning of the wrapper (Email, SSN, Address, etc.). There’s a part of this promise that’s certainly true. If I have to define a function accepting four parameters, and three of them are strings, but one of those strings denotes an email, then I have two choices: Model the email parameter with a String. In this case, I may accidentally use the email where I intended to use the other two string parameters, or I may use one of the other two string parameters where I intended to use the email. Considering just these choices, there are five ways my program may go wrong if I use the wrong name in the wrong position. Model the email parameter with a newtype. In this case, I cannot use the email where I intended to use the other two string parameters, because the compiler may stop me. Similarly, I cannot use the other two string parameters where I intended to use the email, for the same reason. Looking at just these choices, there are 0 ways my program may go wrong. Thus, newtypes, like all good FP practices, reduce the number of ways my program can go wrong. Unfortunately, in my opinion, they don’t go nearly far enough. False Security For most intent and purposes, newtypes are isomorphic to the single value they hold. In my preceding example, given a String, I can get an email (Email "foo"). Given an Email, I can also get a String, e.g. by pattern matching on the Email constructor. Stated differently, and also approximately because I’m ignoring bottom: the String and Email types are isomorphic; they contain the same inhabitants, for any useful definition of “same”. The only substantive difference between the preceding String and Email is the name of the data constructor (call Email an AbergrackleFoozyWatzit, and what has changed?). Hence, my previous criticism of newtypes as “programming by name”. By themselves, newtypes don’t really reduce the number of ways my program can go wrong. They just make it a bit harder to go wrong. But any newtype is isomorphic to the value it holds, and it’s trivial to convert between the two. In fact, if my code doesn’t need to convert between the two (either directly or indirectly), then it’s better off generic. That is, if I never need to convert an Email to a String, or a String to an Email, then I should really write the code generically to work with any value (even if that means making data structures or functions more polymorphic). Parametricity provides a massive reduction in the number of ways my program can go wrong. Newtypes, on the other hand, just make it a bit harder to go wrong, by adding one layer of indirection. In this example, as with many newtypes, I’ve created a bad isomorphism. The domain model of an email is not isomorphic to the data model of a string. But by using a newtype, I have implicitly declared that they are isomorphic. Calling a string an email may make me feel better, because of the different name, but fundamentally, with a newtype, it’s still a string, and I’m only ever one more step away from going wrong. In my experience, too many newtypes create an isomorphism between things that, properly modeled, are not isomorphic. Fortunately, there’s a well-worn workaround that lets us get more mileage out of newtypes. Smart Constructors If I define Email in a module, I can make its data constructor private, and export a helper function to construct an Email. Such helper functions are called smart constructors. They can be used to break the natural isomorphisms created by newtyping. An example is shown below: newtype Email = MkEmail String mkEmail :: String -> Maybe Email mkEmail s = ... In this example, I create a smart constructor which does not promise that it can turn every string into an email. It promises only that it might be able to turn a string into an email, by returning a Maybe Email. With the smart constructor approach, I’ve modeled the fact that while every email has a string representation, not every string has an email representation. Going back to my earlier example of passing same-typed parameters to a function, if I use a smart constructor, then while I can still use an email anywhere a string is expected (by converting), I can’t use a string anywhere an email is expected. (Well, ignoring the fromJust abomination!) Smart Constructors, Dumb Data Smart constructors take us one step closer toward modeling data in a type safe fashion. Unfortunately, I still don’t think it’s far enough. With smart constructors, our data model is fundamentally underconstrained, so we patch that up by restricting who can create the data. That’s putting a band-aid on the real problem. Why not just solve the root issue — viz., that our data model is underconstrained? Dumb Constructors, Smart Data The best solution to a great many newtype problems, I believe, is creating a data model where there is a true isomorphism between the entity modeled by our data and the values passed to the data constructor. That is, creating a data model such that there exists no regions in our data’s state space which correspond to invalid states. Email is a simple example, because there are well-defined models for what constitutes a data model, which can be translated into data declarations in straightforward, if tedious fashion. (To some extent, it’s a failure of most languages I know that such specifications cannot be easily translated into code without tedious boilerplate!) When our data declaration precisely fits our data model specification, there’s no need for smart constructors, and no need for newtypes. There’s far fewer ways that code can go wrong, and because our domain model is captured precisely by our data model, we can transform that data model in ways that make semantic sense (e.g. transforming just the name part of an email, since we’re now in the realm of structured data). Summary As I’ve explained in this blog post, I don’t really hate newtypes. I think they’re very useful, and I do use them, because they make it more difficult for my programs to go wrong. Ultimately, however, I think a lot of problems solved with newtypes (modeling coordinates, positions, emails, etc.) are better solved by more precise data modeling. That is, by making our programs stop lying about isomorphisms. Precision may be tedious due to limitations of the languages we work in, but honestly, what’s more tedious than debugging broken code?
November 20, 2014
by John De Goes
· 10,883 Views
article thumbnail
(C# code snippet) How to create USB web camera viewer and stream to remote locations
in this brief tutorial you will learn how to develop a camera viewer application in c# that allows you to display the image of your usb webcam and to stream the camera image to remote pcs and smartphones. instead of presenting a long article, i would rather show how to implement such application with a few lines of c# code by using the prewritten components of a c# camera library. prerequisites a visual c# wpf application created in visual studio the voipsdk.dll added to the references. (it can be found on the official website of this c# camera library .) a media player supporting rtsp streaming (e.g. vlc) installed on a remote pc first of all let’s build the gui. if you follow the content of the mainwindow.xaml file line-by-line, you will see how to create user all the necessary gui elements that allows the user to be able to connect to a usb camera and display its image, and to set the listen address (including 2 textboxes for the ip address and the port number) that makes rtsp streaming possible. (the following figure illustrates the gui that can be created by using this code snippet.) in the mainwindow.xaml.cs file you will see how to implement the camera viewer functionality and how to turn your application as a video server. to test your application run the program, click the connect button, then when the camera image is displayed, enter the ipv4 address of your pc as listening address, and specify ’554’ as a port number. thereafter open the vlc media player on an other pc or smartphone, and open the network media stream by entering the following network url: rtsp://192.168.115.1:554 (that is: rtsp://youripv4address/portnumber). the result can be seen below: i hope my code snippet was useful! happy programming! // mainwindow.xaml // mainwindow.xaml.cs using system; using system.collections.generic; using system.linq; using system.runtime.interopservices; using system.text; using system.threading.tasks; using system.windows; using system.windows.controls; using system.windows.data; using system.windows.documents; using system.windows.input; using system.windows.media; using system.windows.media.imaging; using system.windows.navigation; using system.windows.shapes; using ozeki.media.ipcamera; using ozeki.media.mediahandlers; using ozeki.media.mediahandlers.video; using ozeki.media.mjpegstreaming; using ozeki.media.video.controls; namespace basic_cameraviewer { /// /// interaction logic for mainwindow.xaml /// public partial class mainwindow : window { private videoviewerwpf _videoviewerwpf; private bitmapsourceprovider _provider; private iipcamera _ipcamera; private webcamera _webcamera; private mediaconnector _connector; private myserver _server; private ivideosender _videosender; public mainwindow() { initializecomponent(); _connector = new mediaconnector(); _provider = new bitmapsourceprovider(); _server = new myserver(); setvideoviewer(); } private void setvideoviewer() { _videoviewerwpf = new videoviewerwpf { horizontalalignment = horizontalalignment.stretch, verticalalignment = verticalalignment.stretch, background = brushes.black }; camerabox.children.add(_videoviewerwpf); _videoviewerwpf.setimageprovider(_provider); } #region usb camera connect/disconnect private void connectusbcamera_click(object sender, routedeventargs e) { _webcamera = webcamera.getdefaultdevice(); if (_webcamera == null) return; _connector.connect(_webcamera, _provider); _videosender = _webcamera; _webcamera.start(); _videoviewerwpf.start(); } private void disconnectusbcamera_click(object sender, routedeventargs e) { if (_webcamera == null) return; _videoviewerwpf.stop(); _webcamera.stop(); _webcamera.dispose(); _connector.disconnect(_webcamera, _provider); } #endregion private void guithread(action action) { dispatcher.begininvoke(action); } private void startserver_click(object sender, routedeventargs e) { var ipadress = ipaddresstext.text; var port = int.parse(porttext.text); _server.videosender = _videosender; _server.onclientcountchanged += server_onclientcountchanged; _server.start(); _server.setlistenaddress(ipadress, port); } void server_onclientcountchanged(object sender, eventargs e) { guithread(() => { connectedclientlist.items.clear(); foreach (var client in _server.connectedclients) connectedclientlist.items.add("end point: " + client.transportinfo.remoteendpoint); }); } private void stopserver_click(object sender, routedeventargs e) { _server.onclientcountchanged -= server_onclientcountchanged; _server.stop(); } } }
November 19, 2014
by Timothy Walker
· 27,692 Views
article thumbnail
How to Compress Responses in Java REST API with GZip and Jersey
There may be cases when your REST api provides responses that are very long, and we all know how important transfer speed and bandwidth still are on mobile devices/networks. I think this is the first performance optimization point one needs to address, when developing REST apis that support mobile apps. Guess what? Because responses are text, we can compress them. And with today’s power of smartphones and tablets uncompressing them on the client side should not be a big deal… So in this post I will present how you can SELECTIVELY compress your REST API responses, if you’ve built it in Java with Jersey, which is the JAX-RS Reference Implementation (and more)… 1. Jersey filters and interceptors Well, thanks to Jersey’s powerful Filters and Interceptors features, the implementation is fairly easy. Whereas filters are primarily intended to manipulate request and response parameters like HTTP headers, URIs and/or HTTP methods, interceptors are intended to manipulate entities, via manipulating entity input/output streams. You’ve seen the power of filters in my posts How to add CORS support on the server side in Java with Jersey, where I’ve shown how to CORS-enable a REST API and How to log in Spring with SLF4J and Logback, where I’ve shown how to log requests and responses from the REST API , but for compressing will be using a GZip WriterInterceptor. A writer interceptor is used for cases where entity is written to the “wire”, which on the server side as in this case, means when writing out a response entity. 1.1. GZip Writer Interceptor So let’s have a look at our GZip Writer Interceptor: package org.codingpedia.demo.rest.interceptors; import java.io.IOException; import java.io.OutputStream; import java.util.zip.GZIPOutputStream; import javax.ws.rs.WebApplicationException; import javax.ws.rs.core.MultivaluedMap; import javax.ws.rs.ext.WriterInterceptor; import javax.ws.rs.ext.WriterInterceptorContext; @Provider @Compress public class GZIPWriterInterceptor implements WriterInterceptor { @Override public void aroundWriteTo(WriterInterceptorContext context) throws IOException, WebApplicationException { MultivaluedMap headers = context.getHeaders(); headers.add("Content-Encoding", "gzip"); final OutputStream outputStream = context.getOutputStream(); context.setOutputStream(new GZIPOutputStream(outputStream)); context.proceed(); } } Note: it implements the WriterInterceptor, which is an interface for message body writer interceptors that wrap around calls to javax.ws.rs.ext.MessageBodyWriter.writeTo providers implementing WriterInterceptor contract must be either programmatically registered in a JAX-RS runtime or must be annotated with @Provider annotation to be automatically discovered by the JAX-RS runtime during a provider scanning phase. @Compress is the name binding annotation, which we will discuss more detailed in the coming paragraph “The interceptor gets a output stream from the WriterInterceptorContext and sets a new one which is a GZIP wrapper of the original output stream. After all interceptors are executed the output stream lastly set to the WriterInterceptorContext will be used for serialization of the entity. In the example above the entity bytes will be written to the GZIPOutputStream which will compress the stream data and write them to the original output stream. The original stream is always the stream which writes the data to the “wire”. When the interceptor is used on the server, the original output stream is the stream into which writes data to the underlying server container stream that sends the response to the client.” [2] “The overridden method aroundWriteTo() gets WriterInterceptorContext as a parameter. This context contains getters and setters for header parameters, request properties, entity, entity stream and other properties.” [2]; when you compress your response you should set the “Content-Encoding” header to “gzip” 1.2. Compress annotation Filters and interceptors can be name-bound. Name binding is a concept that allows to say to a JAX-RS runtime that a specific filter or interceptor will be executed only for a specific resource method. When a filter or an interceptor is limited only to a specific resource method we say that it is name-bound. Filters and interceptors that do not have such a limitation are called global. In our case we’ve built the @Compress annotation: package org.codingpedia.demo.rest.interceptors; import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy; import javax.ws.rs.NameBinding; //@Compress annotation is the name binding annotation @NameBinding @Retention(RetentionPolicy.RUNTIME) public @interface Compress {} and used it to mark methods on resources which should be gzipped (e.g. when GET-ing all the podcasts with the PodcastsResource): @Component @Path("/podcasts") public class PodcastsResource { @Autowired private PodcastService podcastService; ........................... /* * *********************************** READ *********************************** */ /** * Returns all resources (podcasts) from the database * * @return * @throws IOException * @throws JsonMappingException * @throws JsonGenerationException * @throws AppException */ @GET @Compress @Produces({ MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML }) public List getPodcasts( @QueryParam("orderByInsertionDate") String orderByInsertionDate, @QueryParam("numberDaysToLookBack") Integer numberDaysToLookBack) throws IOException, AppException { List podcasts = podcastService.getPodcasts( orderByInsertionDate, numberDaysToLookBack); return podcasts; } ........................... } 2. Testing 2.1. SOAPui Well, if you are testing with SOAPui, you can issue the following request against the PodcastsResource Request: GET http://localhost:8888/demo-rest-jersey-spring/podcasts/?orderByInsertionDate=DESC HTTP/1.1 Accept-Encoding: gzip,deflate Accept: application/json, application/xml Host: localhost:8888 Connection: Keep-Alive User-Agent: Apache-HttpClient/4.1.1 (java 1.5) Response: HTTP/1.1 200 OK Content-Type: application/json Content-Encoding: gzip Content-Length: 409 Server: Jetty(9.0.7.v20131107) [ { "id": 2, "title": "Quarks & Co - zum Mitnehmen", "linkOnPodcastpedia": "http://www.podcastpedia.org/quarks", "feed": "http://podcast.wdr.de/quarks.xml", "description": "Quarks & Co: Das Wissenschaftsmagazin", "insertionDate": "2014-10-29T10:46:13.00+0100" }, { "id": 1, "title": "- The Naked Scientists Podcast - Stripping Down Science", "linkOnPodcastpedia": "http://www.podcastpedia.org/podcasts/792/-The-Naked-Scientists-Podcast-Stripping-Down-Science", "feed": "feed_placeholder", "description": "The Naked Scientists flagship science show brings you a lighthearted look at the latest scientific breakthroughs, interviews with the world top scientists, answers to your science questions and science experiments to try at home.", "insertionDate": "2014-10-29T10:46:02.00+0100" } ] SOAPui recognizes the Content-Type: gzip header, we’ve added in the GZIPWriterInterceptor and automatically uncompresses the response and displays it readable to the human eye. Well, that’s it. You’ve learned how Jersey makes it straightforward to compress the REST api responses. Tip: If you want really learn how to design and implement REST API in Java read the following Tutorial – REST API design and implementation in Java with Jersey and Spring
November 18, 2014
by Adrian Matei
· 62,909 Views · 2 Likes
article thumbnail
Creating a Microsoft Login Button Using PHP
in this tutorial i will show you how to create a microsoft login button for your website using php. to start with, let’s answer the question: what is oauth? oauth is a protocol used to allow secure authorization to websites and applications to access user information on other websites. there are two versions of oauth, 1.0 and 2.0. in this post we will use oauth 2.0 to build a microsoft login system. what is microsoft log-in? microsoft log-in means asking user to grant access to his/her microsoft live information like email id, username etc. once your website has been granted access and has all these information about the user it can allow the users to access protected pages on your website. setting up directory and files before we get started you need to create a php file named redirect.php. place this file anywhere in your webspace. creating a microsoft app if your website is allowing login using microsoft then your website is considered as an microsoft app. so you have your website ready now its time to register you website as a microsoft app. follow this steps to create a microsoft app: visit microsoft apps page . now create a microsoft app select api settings and for redirect url pass url pointing to the redirect.php file. you can find the client id and client secret under app settings creating login with microsoft button when user clicks on login button you need to run this code to redirect user to microsoft live website so that user can grant permission to your app to access their information $client_id = ""; 2 $redirect_uri = ""; 3 $scopes = "wl.basic,wl.offline_access,wl.signin,wl.emails"; 4 5 header("location: " . "https://login.live.com/oauth20_authorize.srf?client_id=" . $client_id . "&scope=" . $scopes . "&response_type=token&redirect_uri=" . $redirect_uri); scopes represent the list of permissions for the app. you need to pass a comma separated list of permissions. list of all scopes . populate the $client_id and $redirect_uri variables. redirecting back to the app once user has given access to the app, microsoft will redirect user back to the redirect uri. now you need to retrieve an access token which acts like a permission to get user information. in the redirect.php file you can retrieve access token by running this code $client_id, "redirect_uri" => $redirect_uri, "client_secret" => $client_secret, "code" => $_get["code"], "grant_type" => "authorization_code"); foreach($fields as $key=>$value) { $fields_string .= $key."=".$value."&"; } rtrim($fields_string, "&"); $ch = curl_init(); curl_setopt($ch,curlopt_url, $url); curl_setopt($ch,curlopt_httpheader, array("content-type: application/x-www-form-urlencoded")); curl_setopt($ch,curlopt_post, count($fields)); curl_setopt($ch,curlopt_postfields, $fields_string); curl_setopt($ch,curlopt_returntransfer,1); $result = curl_exec($ch); $result = json_decode($result); curl_close($ch); //this is the refresh token used to access microsoft live rest apis $access_token = $result->access_token; $refresh_token = $result->refresh_token; } else { echo "an error occured"; } ?> populate variable $client_id , $client_secret and $redirect_uri . finally we got $access_token and $refresh_token . $access_token usually expires in 1 hour therefore $refresh_token is used to get a new access token after every 1 hour. if access token is expired then you are likely to get an error in http response content while making requests to rest apis. you can retrieve new access token using this function function new_access_token($refresh_token) { $url = "https://login.live.com/oauth20_token.srf"; $fields = array("client_id" => $client_id, "redirect_uri" => $redirect_uri, "client_secret" => $client_secret, "grant_type" => "refresh_token", "refresh_token" => $refresh_token); foreach($fields as $key=>$value) { $fields_string .= $key."=".$value."&"; } rtrim($fields_string, "&"); $ch = curl_init(); curl_setopt($ch,curlopt_url, $url); curl_setopt($ch,curlopt_httpheader, array("content-type: application/x-www-form-urlencoded")); curl_setopt($ch,curlopt_post, count($fields)); curl_setopt($ch,curlopt_postfields, $fields_string); curl_setopt($ch,curlopt_returntransfer,1); $result = curl_exec($ch); $result = json_decode($result); curl_close($ch); $access_token = $result->access_token; return $access_token; } making calls to rest api you can find list of all rest apis at microsoft rest api reference . all the requests to these apis must be made using the access token. to retrieve user profile information you need to make a get request of such kind 1 echo file_get_contents ( " https://apis.live.net/v5.0/me?access_token= " . $access_token ); integrating microsoft login in wordpress wordpress is made on php therefore all code will be same for authorizing user and getting profile information. to create a redirect url in wordpress use wordpress ajax api . final thoughts if you want to more than just login then increase the permissions in permission list and store the access token and refresh token in database for further use. make sure you update the access token when its refreshed. don’t share the client secret with anyone.
November 17, 2014
by Zachary Bedell
· 10,245 Views · 5 Likes
article thumbnail
Immutability Through Interfaces
It is often desirable to have immutable objects, objects that cannot be modified once constructed. Typically, an immutable object has fields that are declared as final and are set in the object constructor. There are getters for the fields, but no setters since the values cannot be saved. Once created, the object state doesn’t change and the objects can be shared across different threads since the state is fixed. There are plenty of caveats to that statement, for example if you have a list, while the list field reference may be final, the list itself may be able to change and have values added and removed which would spoil the immutability. Achieving this state can often be difficult because we rely on a number of tools and frameworks that may not support immutability such as any framework that builds an object by creating it and then setting values on it. However, one way around this would be to take a mutable object and make it immutable through interfaces. We do this by creating an interface that represents all the getters for an object, but none of the setters. Given a domain entity of a person with an id, first and last name fields we can define an immutable interface : public interface ImmutablePerson { public int getId(); public String getFirstName(); public String getLastName(); } We can then implement this interface in our person class : public class PersonEntity implements ImmutablePerson { private int id; private String firstName; private String lastName; public int getId() { return id; } public void setId(int id) { this.id = id; } public String getFirstName() { return firstName; } public void setFirstName(String firstName) { this.firstName = firstName; } } Our API deals with the PersonEntity class internally, but exposes the object as the ImmutablePerson only. public ImmutablePerson loadPerson(int id) { PersonEntity result = someLoadingMechanism.getPerson(id); result.setxxxxx(somevalue); //we can change it internally return result; //once it has been returned it is only accessed as an immutable person } To return a mutable instance for modification, you can return the PersonEntity. Alternatively, you can add aMutablePerson interface with just the setters and implement that. Unless you want to keep casting between the Immutable/Mutable types, the MutablePerson interface can extend the ImmutablePerson interface so the writable version is also readable. public interface MutablePerson extends ImmutablePerson { public void setId(int id) { public void setFirstName(String firstName); public void setLastName(String lastName); } Granted, someone can easily cast the object to a PersonEntity or MutablePerson and change the values, but then, you can also change static final variables with reflection if you wanted to. The ability to cast the object to make it writable only could be an internal-use only feature. This mechanism can be used to return immutable objects from persistence layers. There are some caveats with certain ORM tools since they use lazy loading mechanisms that would break the immutability. Underneath the interface it is still a mutable object. With some additional code, you could create immutable collections by wrapping the underlying collections in read only collections and returning them to the user. Some of the JVM benefits of immutability granted upon variables marked as final would not be given to this type of solution since the underlying variables are not final or immutable.
November 15, 2014
by Andy Gibson
· 14,045 Views · 1 Like
article thumbnail
Using Eclipse's Link Source Feature
NOTE: Apparently a bundle with a linked source will not be exported or built in an update site built. also Tycho will complain that it can't find linked sources, which severely limits the possibilities. A workaround is to export the bundles as plain old jars (this works fine for some reason), but the problem is far from ideal. See bug reports: https://bugs.eclipse.org/bugs/show_bug.cgi?id=457192 https://bugs.eclipse.org/bugs/show_bug.cgi?id=66177 Introduction I've been using Eclipse for more than ten years now, and so I like to think I know my way around it's offerings, but every now and then I get pleasantly surprised by discovering a feature – which usually had been there all along- but for which I finally have made the time to investigate. In this case, I am talking about the 'Link Source' feature in the Project Properties tab. Most experienced Eclipse users will at some point wander through the project properties, for instance when certain libraries are not found by the compiler, or when a plugin project starts to behave unexpectedly. The Project Properties tab comes into play when the Manifest.MF file no longer provides the answers for certain problems you face, and you need to delve deeper into the classpath and project settings. It also becomes topical when you need to make a custom project. At Project Chaupal we are currently maintaining and updating the code from Project JXTA. JXTA has been around for quite a bit in the open source community, and the development has had its ups and downs, so the code could do with a makeover here and there. I've been involved with keeping the code available in OSGI since 2006 or so – also with its ups and downs- and one of my ideals would be to automatically generate the OSGI bundles straight from the JXTA sources, without any handwork. The JXTA jar ships with a large number of third party libraries (e.g. Jetty and Log4j), some of which are available as OSGI bundles, so I don't want to include them in the JXTA OSGI bundle I make. A list of dependencies in the Manifest should be enough! Some of the third party libraries also aim to provide the same functionality (e.g. the database functionality provided by Derby and H2), so I would prefer to divide this over two bundles, and then just select the bundle that is needed. Ever since JXTA 2.6, the code has been mavenised, and so the code now conforms to the typical structure that Maven requires, with a specific location for JAVA code (src/main/java), resources (src/main/resources) and tests (src/main/test). I prefer to use Tycho for my OSGI bundles, so the regular Eclipse tooling is leading. As a result my goal is to: Create separate bundles for the core and the test source code Add the required resources, such as .properties files and the likes Split the core source code over different bundles, so that every bundle depends on one third party library at the most. It took me a day or two to get everything the way I wanted it, but in the end it was surprisingly easy, so I thought I'd share the experience.This tutorial assumes that you are well-versed in Eclipse and OSGI development. If not, a good tutorial on the subject can be found here. Preparation As was described earlier, the plan is to use a Maven project (available on GitHub) as a source for a number of OSGI bundles. For starters, we need to do the following; Prepare an Eclipse IDE with Egit and, optionally, with support for GitHub. As always, Lars Vogel's tutorial provides an excellent guide to achieve this. With the GitHub support you can actually search for the required repository, and clone it in your workspace within minutes. Add Maven Support for Eclipse. The tutorial can be found here. We now have an Eclipse IDE with one project loaded in the workspace, which conforms to the Maven structure. Now we can start to do the magic! Extracting Source Files in an OSGI Project First create a new plugin project, using the wizard (File → new → Plugin Project). Fill in the required details as requested (target platform→OSGI framework) and press 'finish'. We now have a standard textbook OSGI bundle project. For the sake of argument, let's call this bundle org.mybundle.host. Now we are going to add java source files and resources from the Maven project: open the project properties tab (right mouse click → properties) select the 'Java build path' option and choose the 'source' tab press 'Link Source' and browse to the 'main/java' subfolder of the Maven project. Close the project properties Include the source folder in the build.properties file and clean the workspace Update the Manifest to include the required dependencies, and export the packages as needed As you can see, the java files have been included in the bundle project, and will compile in a normal fashion. TIP: Currently the source file will by default have the same name as the folder. You can change this in the 'link source' wizard. For instance, you can delete the 'src' folder that is created by default, and replace it with the linked source if you want. This should only be considered if you are not going to make specific java files in the bundle. Next we are going to include the resources, such as .properties files that are included in the Maven project. As an exercise, we will exclude all html files that may be included. Open the project properties and follow the steps described previously, but now select the main/resources folder. Then press the 'next' tab, instead of closing You can now select which files to include or exclude in your bundle. Select the 'exclude' tab, and enter the following pattern: **/*.htm*. Close the project properties, update the build path and clean the project We have now included the desired resources, and in principle the bundle should now work as desired. With the include and exclude tabs, you can determine which files and folders you want to add to your bundle. The inclusion and exclusion patterns follow the conventions used by Apache ANT. TIP: You can check if the bundle has the correct source and resource files by opening a file explorer (in Windows) and browsing to the 'bin' folder of your bundle project. If you first refresh your bundle project (F5 in the Eclipse IDE), the correct class and resource files should be present there. NOTE: Although I would not recommend it, it is possible to link the sources of multiple non-OSGI code sources this way. Even though the folders need different names, they will be built as if they are one source folder. Now we create a second plugin project for the test files. We will call this bundle org.mybundle.test Open the project properties and follow the steps described previously, but now select the main/test folder. If required, you can exclude certain tests in the 'exclude' tab Close the project properties, update the build path and clean the project In the manifest editor, include the dependencies to org.mybundle.host, and for instance JUnit and JMock. When there are no more compiler errors, your two bundles should behave as regular OSGI bundles, with the only difference that the sources are extracted from the Maven project. TIP: It is also possible to make fragment bundles this way, and you can include library resources in your bundle (such as third party jars). This way you can restructure a non-OSGI project at will Using Variables When you store your projects in the cloud, such as on gitHub, you may often find multiple versions of your workspace scattered over different computers, and your repositories stored on different drives. This means that linking your sources with absolute paths, as we have done previously, is not a very versatile approach. Especially with linking this may become problematic, as the linked source project (e.g. the maven project) can be stored on a different location than the project that uses the source files. Luckily eclipse allows you to define variables in your project, which can help either to standardise the relative locations or, if this is not possible, to easily modify the links. In order to achive this, follow these steps: Select the project properties, and select the 'Java Build Path' option. Add or Edit a link source, and select the 'variables' button. Add one or more new variables by pressing the 'New' button and entering the locations. Then press 'OK' As an example, the following three variables point to two GitHub projects, one which holds the Maven project, while the bundle project is located in the same subfolder: - GITHUB_LOC: C:/Users/MyName/MyGithubLocation - MYPROJECT_LOC: ${GITHUB_LOC}/MyProject - MYSOURCEPROJECT_LOC: ${MYPROJECT_LOC}/MySource Now all you have to do is change the 'Linked folder location' to: MYSOURCEPROJECT_LOC/src/main/java in order to include the Maven project in your bundle.You can then also add a resource folder: MYSOURCEPROJECT_LOC/src/main/resources TIP: In the above example with a Maven project, the linked folder name will default to 'java' (and the resources to 'resources'). It is recommended to leave it that way, because you can then use the 'src' folder for bundle specific code that yuo may want to add, like a decalarative service. Also remember to update the build path to include the folders your project needs. Conclusion The 'link source' option provides a powerful way to make non-OSGI code accessible as OSGI bundles. The inclusion and exclusion patterns allow you to customise the bundles to your needs.
November 13, 2014
by Kees Pieters
· 16,380 Views · 1 Like
article thumbnail
Coldfusion Example: Using jQuery UI Accordion with a ColdFusion Query
A reader pinged me yesterday with a simple problem that I thought would be good to share on the blog. He had a query of events that he wanted to use with jQuery UI's Accordion control. The Accordion control simply takes content and splits into various "panes" with one visible at a time. For his data, he wanted to split his content into panes designated by a unique month and year. Here is a quick demo of that in action. I began by creating a query to store my data. I created a query with a date and title property and then random chose to add 0 to 3 "events" over the next twelve months. I specifically wanted to support 0 to ensure my demo handled noticing months without any data. 01. 04. 05.q = queryNew("date,title"); 06.for(i=1; i<12; i++) { 07. //for each month, we add 0-3 events (some months may not have data) 08. toAdd = randRange(0, 3); 09. 10. for(k=0; k To handle creating the accordion, I had to follow the rules jQuery UI set up for the control. Basically - wrap the entire set of data in a div, and separate each "pane" with an h3 and inner div. To handle this, I have to know when a new unique month/year "block" starts. I store this in a variable, lastDateStr, and just check it in every iteration over the query. I also need to ensure that on the last row I close the div. 01. 02. 03. 04. 05. 06. 07. 08. 09. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. #thisDateStr# 30. 31. 32. 33. 34. 35. 36. #title# 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. And the end result: So, not rocket science, but hopefully helpful to someone. Here is the entire template if you want to try it yourself. 01. 04. 05.q = queryNew("date,title"); 06.for(i=1; i<12; i++) { 07. //for each month, we add 0-3 events (some months may not have data) 08. toAdd = randRange(0, 3); 09. 10. for(k=0; k 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. #thisDateStr# 46. 47. 48. 49. 50. 51. 52. #title# 53. 54. 55. 56. 57. 58. 59. 60. 61. 62. 63.
November 13, 2014
by Raymond Camden
· 4,591 Views
article thumbnail
How to Deal with MySQL Deadlocks
Originally Written by Peiran Song A deadlock in MySQL happens when two or more transactions mutually hold and request for locks, creating a cycle of dependencies. In a transaction system, deadlocks are a fact of life and not completely avoidable. InnoDB automatically detects transaction deadlocks, rollbacks a transaction immediately and returns an error. It uses a metric to pick the easiest transaction to rollback. Though an occasional deadlock is not something to worry about, frequent occurrences call for attention. Before MySQL 5.6, only the latest deadlock can be reviewed using SHOW ENGINE INNODB STATUS command. But with Percona Toolkit’s pt-deadlock-logger you can have deadlock information retrieved from SHOW ENGINE INNODB STATUS at a given interval and saved to a file or table for late diagnosis. For more information on using pt-deadlock-logger, see this post. With MySQL 5.6, you can enable a new variable innodb_print_all_deadlocks to have all deadlocks in InnoDB recorded in mysqld error log. Before and above all diagnosis, it is always an important practice to have the applications catch deadlock error (MySQL error no. 1213) and handle it by retrying the transaction. How to diagnose a MySQL deadlock A MySQL deadlock could involve more than two transactions, but the LATEST DETECTED DEADLOCK section only shows the last two transactions. Also it only shows the last statement executed in the two transactions, and locks from the two transactions that created the cycle. What are missed are the earlier statements that might have really acquired the locks. I will show some tips on how to collect the missed statements. Let’s look at two examples to see what information is given. Example 1: 1 141013 6:06:22 2 *** (1) TRANSACTION: 3 TRANSACTION 876726B90, ACTIVE 7 sec setting auto-inc lock 4 mysql tables in use 1, locked 1 5 LOCK WAIT 9 lock struct(s), heap size 1248, 4 row lock(s), undo log entries 4 6 MySQL thread id 155118366, OS thread handle 0x7f59e638a700, query id 87987781416 localhost msandbox update 7 INSERT INTO t1 (col1, col2, col3, col4) values (10, 20, 30, 'hello') 8 *** (1) WAITING FOR THIS LOCK TO BE GRANTED: 9 TABLE LOCK table `mydb`.`t1` trx id 876726B90 lock mode AUTO-INC waiting 10 *** (2) TRANSACTION: 11 TRANSACTION 876725B2D, ACTIVE 9 sec inserting 12 mysql tables in use 1, locked 1 13 876 lock struct(s), heap size 80312, 1022 row lock(s), undo log entries 1002 14 MySQL thread id 155097580, OS thread handle 0x7f585be79700, query id 87987761732 localhost msandbox update 15 INSERT INTO t1 (col1, col2, col3, col4) values (7, 86, 62, "a lot of things"), (7, 76, 62, "many more") 16 *** (2) HOLDS THE LOCK(S): 17 TABLE LOCK table `mydb`.`t1` trx id 876725B2D lock mode AUTO-INC 18 *** (2) WAITING FOR THIS LOCK TO BE GRANTED: 19 RECORD LOCKS space id 44917 page no 529635 n bits 112 index `PRIMARY` of table `mydb`.`t2` trx id 876725B2D lock mode S locks rec but not gap waiting 20 *** WE ROLL BACK TRANSACTION (1) Line 1 gives the time when the deadlock happened. If your application code catches and logs deadlock errors,which it should, then you can match this timestamp with the timestamps of deadlock errors in application log. You would have the transaction that got rolled back. From there, retrieve all statements from that transaction. Line 3 & 11, take note of Transaction number and ACTIVE time. If you log SHOW ENGINE INNODB STATUS output periodically(which is a good practice), then you can search previous outputs with Transaction number to hopefully see more statements from the same transaction. The ACTIVE sec gives a hint on whether the transaction is a single statement or multi-statement one. Line 4 & 12, the tables in use and locked are only with respect to the current statement. So having 1 table in use does not necessarily mean that the transaction involves 1 table only. Line 5 & 13, this is worth of attention as it tells how many changes the transaction had made, which is the “undo log entries” and how many row locks it held which is “row lock(s)”. These info hints the complexity of the transaction. Line 6 & 14, take note of thread id, connecting host and connecting user. If you use different MySQL users for different application functions which is another good practice, then you can tell which application area the transaction comes from based on the connecting host and user. Line 9, for the first transaction, it only shows the lock it was waiting for, in this case the AUTO-INC lock on table t1. Other possible values are S for shared lock and X for exclusive with or without gap locks. Line 16 & 17, for the second transaction, it shows the lock(s) it held, in this case the AUTO-INC lock which was what TRANSACTION (1) was waiting for. Line 18 & 19 shows which lock TRANSACTION (2) was waiting for. In this case, it was a shared not gap record lock on another table’s primary key. There are only a few sources for a shared record lock in InnoDB: 1) use of SELECT … LOCK IN SHARE MODE 2) on foreign key referenced record(s) 3) with INSERT INTO… SELECT, shared locks on source table The current statement of trx(2) is a simple insert to table t1, so 1 and 3 are eliminated. By checking SHOW CREATE TABLE t1, you could confirm that the S lock was due to a foreign key constraint to the parent table t2. Example 2: With MySQL community version, each record lock has the record content printed: 1 2014-10-11 10:41:12 7f6f912d7700 2 *** (1) TRANSACTION: 3 TRANSACTION 2164000, ACTIVE 27 sec starting index read 4 mysql tables in use 1, locked 1 5 LOCK WAIT 3 lock struct(s), heap size 360, 2 row lock(s), undo log entries 1 6 MySQL thread id 9, OS thread handle 0x7f6f91296700, query id 87 localhost ro ot updating 7 update t1 set name = 'b' where id = 3 8 *** (1) WAITING FOR THIS LOCK TO BE GRANTED: 9 RECORD LOCKS space id 1704 page no 3 n bits 72 index `PRIMARY` of table `tes t`.`t1` trx id 2164000 lock_mode X locks rec but not gap waiting 10 Record lock, heap no 4 PHYSICAL RECORD: n_fields 5; compact format; info bit s 0 11 0: len 4; hex 80000003; asc ;; 12 1: len 6; hex 000000210521; asc ! !;; 13 2: len 7; hex 180000122117cb; asc ! ;; 14 3: len 4; hex 80000008; asc ;; 15 4: len 1; hex 63; asc c;; 16 17 *** (2) TRANSACTION: 18 TRANSACTION 2164001, ACTIVE 18 sec starting index read 19 mysql tables in use 1, locked 1 20 3 lock struct(s), heap size 360, 2 row lock(s), undo log entries 1 21 MySQL thread id 10, OS thread handle 0x7f6f912d7700, query id 88 localhost r oot updating 22 update t1 set name = 'c' where id = 2 23 *** (2) HOLDS THE LOCK(S): 24 RECORD LOCKS space id 1704 page no 3 n bits 72 index `PRIMARY` of table `tes t`.`t1` trx id 2164001 lock_mode X locks rec but not gap 25 Record lock, heap no 4 PHYSICAL RECORD: n_fields 5; compact format; info bit s 0 26 0: len 4; hex 80000003; asc ;; 27 1: len 6; hex 000000210521; asc ! !;; 28 2: len 7; hex 180000122117cb; asc ! ;; 29 3: len 4; hex 80000008; asc ;; 30 4: len 1; hex 63; asc c;; 31 32 *** (2) WAITING FOR THIS LOCK TO BE GRANTED: 33 RECORD LOCKS space id 1704 page no 3 n bits 72 index `PRIMARY` of table `tes t`.`t1` trx id 2164001 lock_mode X locks rec but not gap waiting 34 Record lock, heap no 3 PHYSICAL RECORD: n_fields 5; compact format; info bit s 0 35 0: len 4; hex 80000002; asc ;; 36 1: len 6; hex 000000210520; asc ! ;; 37 2: len 7; hex 17000001c510f5; asc ;; 38 3: len 4; hex 80000009; asc ;; 39 4: len 1; hex 62; asc b;; Line 9 & 10: The ‘space id’ is tablespace id, ‘page no’ gives which page the record lock is on inside the tablespace. The ‘n bits’ is not the page offset, instead the number of bits in the lock bitmap. The page offset is the ‘heap no’ on line 10, Line 11~15: It shows the record data in hex numbers. Field 0 is the cluster index(primary key). Ignore the highest bit, the value is 3. Field 1 is the transaction id of the transaction which last modified this record, decimal value is 2164001 which is TRANSACTION (2). Field 2 is the rollback pointer. Starting from field 3 is the rest of the row data. Field 3 is integer column, value 8. Field 4 is string column with character ‘c’. By reading the data, we know exactly which row is locked and what is the current value. What else can we learn from analysis? Since most MySQL deadlocks happen between two transactions, we could start the analysis based on that assumption. In Example 1, trx (2) was waiting on a shared lock, so trx (1) either held a shared or exclusive lock on that primary key record of table t2. Let’s say col2 is the foreign key column, by checking the current statement of trx(1), we know it did not require the same record lock, so it must be some previous statement in trx(1) that required S or X lock(s) on t2’s PK record(s). Trx (1) only made 4 row changes in 7 seconds. Then you learned a few characteristics of trx(1): it does a lot of processing but a few changes; changes involve table t1 and t2, a single record insertion to t2. These information combined with other data could help developers to locate the transaction. Where else can we find previous statements of the transactions? Besides application log and previous SHOW ENGINE INNODB STATUS output, you may also leverage binlog, slow log and/or general query log. With binlog, if binlog_format=statement, each binlog event would have the thread_id. Only committed transactions are logged into binlog, so we could only look for Trx(2) in binlog. In the case of Example 1, we know when the deadlock happened, and we know Trx(2) started 9 seconds ago. We can run mysqlbinlog on the right binlog file and look for statements with thread_id = 155097580. It is always good to then cross refer the statements with the application code to confirm. $ mysqlbinlog -vvv --start-datetime=“2014-10-13 6:06:12” --stop-datatime=“2014-10-13 6:06:22” mysql-bin.000010 > binlog_1013_0606.out With Percona Server 5.5 and above, you can set log_slow_verbosity to include InnoDB transaction id in slow log. Then if you have long_query_time = 0, you would be able to catch all statements including those rolled back into slow log file. With general query log, the thread id is included and could be used to look for related statements. How to avoid a MySQL deadlock There are things we could do to eliminate a deadlock after we understand it. – Make changes to the application. In some cases, you could greatly reduce the frequency of deadlocks by splitting a long transaction into smaller ones, so locks are released sooner. In other cases, the deadlock rises because two transactions touch the same sets of data, either in one or more tables, with different orders. Then change them to access data in the same order, in another word, serialize the access. That way you would have lock wait instead of deadlock when the transactions happen concurrently. – Make changes to the table schema, such as removing foreign key constraint to detach two tables, or adding indexes to minimize the rows scanned and locked. – In case of gap locking, you may change transaction isolation level to read committed for the session or transaction to avoid it. But then the binlog format for the session or transaction would have to be ROW or MIXED.
November 12, 2014
by Peter Zaitsev
· 31,662 Views
article thumbnail
Ext JS Grid Grouping Tutorial
this tutorial shows you how to create a grouped grid in ext js. ext js grid grouping is a common scenario in business applications where users need to analyze tabular data grouped by different attributes. the ext js application that you will create in this tutorial will render an ext js grid containing fictional data describing model cars. this article is part of a series of ext js tutorials that i have posted on jorgeramon.me. creating an ext js project from scratch we will get started by creating the directories for our sample ext js application. let’s go ahead and create the directories as depicted in the screenshot below. this is a typical ext js project structure with an app directory under which we’ve placed the model, store and view directories. we will not create a controller folder because for this simple example we will use the ext.application instance that we will create as the only controller in the app. next, we will create the file that hosts our ext js application. let’s name it grid-grouping.html and place it in the root directory of the project. here’s the code that goes in the file: remember to make sure that the css and script references to the ext js library in the head section of the html document are pointing to the correct directories in your development workstation. the references above work for my development environment. in the head section we also have an entry for the app.js file, which we need to add to the root directory of the project, as shown below. let’s open app.js and type the following ext js application definition: ext.application({ name: 'app', autocreateviewport: true, launch: function () { } }); this is a super simple ext js application. we are using the autocreateviewport config to instruct the app to load and instantiate the viewport, which we will create in a few minutes, before firing the launch function. we will leave the launch function empty because in this example we do not need to execute any logic within the function. the ext js model to represent the model cars data that we will ultimately render in the grid, we will use the modelcar ext js model class. this class goes in the modelcar.js file that we will add to the model directory of the project. this is how we are going to define the modelcar model: ext.define('app.model.modelcar', { extend: 'ext.data.model', fields: [ { name: 'id', type: 'int' }, { name: 'category', type: 'string' }, { name: 'name', type: 'string' }, { name: 'vendor', type: 'string' } ] }); back in the app.js file we need to add a models config to the ext js application just so the app knows that it has a dependency on the modelcar module and it needs to load it as part of the app. ext.application({ name: 'app', models: ['modelcar'], autocreateviewport: true, launch: function () { } }); creating a store to feed the ext js grid now we are going to focus on the ext js store that will feed the grid. let’s add a modelcars.js file to the store directory of the project. we will type the following definition in the file: ext.define('app.store.modelcars', { extend: 'ext.data.store', model: 'app.model.modelcar', groupfield: 'category', groupdir: 'asc', sorters: ['name'], data: [ { 'id': 1, 'category': 'trucks and buses', 'name': '1940 ford pickup truck', 'vendor': 'motor city art classics' }, { 'id': 2, 'category': 'trucks and buses', 'name': '1957 chevy pickup', 'vendor': 'gearbox collectibles' }, { 'id': 3, 'category': 'classic cars', 'name': '1972 alfa romeo gta', 'vendor': 'motor city art classics' }, { 'id': 4, 'category': 'motorcycles', 'name': '2003 harley-davidson eagle drag bike', 'vendor': 'studio m art models' }, { 'id': 5, 'category': 'motorcycles', 'name': '1996 moto guzzi 1100i', 'vendor': 'motor city art classics' }, { 'id': 6, 'category': 'classic cars', 'name': '1952 alpine renault 1300', 'vendor': 'studio m art models' }, { 'id': 7, 'category': 'classic cars', 'name': '1993 mazda rx-7', 'vendor': 'motor city art classics' }, { 'id': 9, 'category': 'classic cars', 'name': '1965 aston martin db5', 'vendor': 'motor city art classics' }, { 'id': 10, 'category': 'classic cars', 'name': '1998 chrysler plymouth prowler', 'vendor': 'unimax art galleries' }, { 'id': 11, 'category': 'trucks and buses', 'name': '1926 ford fire engine', 'vendor': 'studio m art models' }, { 'id': 12, 'category': 'trucks and buses', 'name': '1962 volkswagen microbus', 'vendor': 'unimax art galleries' }, { 'id': 13, 'category': 'trucks and buses', 'name': '1980’s gm manhattan express', 'vendor': 'motor city art classics' }, { 'id': 13, 'category': 'motorcycles', 'name': '1997 bmw f650 st', 'vendor': 'gearbox collectibles' }, { 'id': 13, 'category': 'motorcycles', 'name': '1974 ducati 350 mk3 desmo', 'vendor': 'motor city art classics' }, { 'id': 13, 'category': 'motorcycles', 'name': '2002 yamaha yzr m1', 'vendor': 'motor city art classics' } ] }) the ext js model config of the store is set to the modelcar model that we created a couple of minutes ago. no surprises there. a couple of things that i don’t want you to miss in the store’s definition are the groupfield and groupdir configs, which tell the store which field and in which direction to group by: groupfield: 'category', groupdir: 'asc', also note how we use the sorters config to sort the grouped records by the name field. finally, as the data config indicates, we are using a hard-coded array as the data for the store. we are using hard-coded data in order to keep the example short. let’s make the application aware of the modelcars store by adding the stores config in the app.js file: ext.application({ name: 'app', models: ['modelcar'], stores: ['modelcars'], autocreateviewport: true, launch: function () { } }); creating the ext js viewport the app’s viewport is where we will define the ext js grouped grid. let’s create the viewport.js file in the project’s view directory: here’s the code that we will add to the viewport’s file: ext.define('app.view.viewport', { extend: 'ext.container.viewport', requires: ['ext.grid.panel'], style: 'padding:25px', layout: 'vbox', items: [ { xtype: 'gridpanel', width: 650, height: 475, title: 'ext js grid grouping', store: 'modelcars', columns: [ { text: 'id', hidden: true, dataindex: 'id' }, { text: 'name', sortable: true, dataindex: 'name', flex: 3 }, { text: 'vendor', sortable: true, dataindex: 'vendor', flex:2 }, { text: 'category', sortable: true, dataindex: 'category', flex: 2 } ], features: [{ ftype: 'grouping', // you can customize the group's header. groupheadertpl: '{name} ({children.length})', enablenogroups:true }] } ] }); let’s review this viewport definition in detail. first, we are using the viewport’s requires config to tell the ext js loader that it needs to load the ext.grid.panel class as part of the application. then, we define our grid instance within the items config of the viewport. we are telling the grid to use the modelcars ext js store as its data source by setting the store config to ‘modelcars’. this works because ext js components pass the value of the store config to the ext.data.storemanager class, which can look up or create a store, depending on whether the store config’s value is a store id, a store’s instance or a store’s configuration. we are using the grid’s columns config to define four columns: id, name, vendor and category. setting the hidden config of the id column to true will make the column invisible to the user. the sortable config of the visible columns is true to allow users to sort by those columns. setting up ext js grid grouping the features config allows us to specify one or more features to be added to the grid. as explained in the ext js docs, an ext js feature is a class that provides hooks that you can use to inject functionality into the grid at different points during the grid’s creation cycle. this is a clever approach to maintaining the grid’s core as lightweight as possible, while allowing for advanced functionality to be “plugged in” as needed. currently, extjs provides the following grid features: grouping, through the ext.grid.feature.grouping class. grouping summary, though the ext.grid.feature.groupingsummary class. row body, though the ext.grid.feature.rowbody class. summary, through the ext.grid.feature.summary class. in our app we are only using the grouping feature. we set the ftype property to ‘grouping’ to signal the grid that we will use the grouping feature. the groupheadertpl config allows us to enter a template for the group headers of the grid: groupheadertpl: '{name} ({children.length})' this template will render the name of the group along with the number of records in the group, producing a result similar to this picture: setting the enablenogroups config to true will let the user turn off grouping in the grid. note that if we wanted to specify multiple features, we would need to add them to the features config in array notation. one last step before we are ready to test the app. let’s return to the app.js file and the add views config, where we will place a reference to the viewport class that we just created: ext.application({ name: 'app', models: ['modelcar'], stores: ['modelcars'], views: ['viewport'], autocreateviewport: true, launch: function () { } }); now we can test the app on a browser. upon application load, the grid’s records are grouped by the category field, and sorted by the name field: summary and next steps in this tutorial we reviewed the steps needed to create a grouped grid in ext js, and we learned that this process involves attaching a “grouping” feature to an ext js grid that is connected to a grouped ext js store. as next steps, i would suggest that you explore how to use the “grouping summary” and “summary” features of the ext js grid, which are commonly used together with the “grouping” feature. i will cover these features in future tutorials. don’t forget to sign up for my mailing list so you can be among the first to know when i publish the next update. download the source code download the ext js grouped grid example here: ext js grid grouping tutorial on github
November 11, 2014
by Jorge Ramon
· 17,663 Views
article thumbnail
Plotting Data Online via Plotly and Python
I don’t do a lot of plotting in my job, but I recently heard about a website called Plotly that provides a plotting service for anyone’s data. They even have a plotly package for Python (among others)! So in this article we will be learning how to plot with their package. Let’s have some fun making graphs! Getting Started You will need the plotly package to follow along with this article. You can use pip to get the package and install it: pip install plotly Now that you have it installed, you’ll need to go to the Plotly website and create a free account. Once that’s done, you will get an API key. To make things super simple, you can use your username and API key to create a credentials file. Here’s how to do that: import plotly.tools as tls tls.set_credentials_file( username="your_username", api_key="your_api_key") # to get your credentials credentials = tls.get_credentials_file() If you don’t want to save your credentials, then you can also sign in to their service by doing the following: import plotly.plotly as py py.sign_in('your_username','your_api_key') For the purposes of this article, I’m assuming you have created the credentials file. I found that makes interacting with their service a bit easier to use. Creating a Graph Plotly seems to default to a Scatter Plot, so we’ll start with that. I decided to grab some data from a census website. You can download any US state’s population data, along with other pieces of data. In this case, I downloaded a CSV file that contained the population of each county in the state of Iowa. Let’s take a look: import csv import plotly.plotly as py #---------------------------------------------------------------------- def plot_counties(csv_path): """ http://census.ire.org/data/bulkdata.html """ counties = {} county = [] pop = [] counter = 0 with open(csv_path) as csv_handler: reader = csv.reader(csv_handler) for row in reader: if counter == 0: counter += 1 continue county.append(row[8]) pop.append(row[9]) trace = dict(x=county, y=pop) data = [trace] py.plot(data, filename='ia_county_populations') if __name__ == '__main__': csv_path = 'ia_county_pop.csv' plot_counties(csv_path) If you run this code, you should see a graph that looks like this: <br> You can also view the graph here. Anyway, as you can see in the code above, all I did was read the CSV file and extract out the county name and the population. Then I put that data into two different Python lists. Finally I created a dictionary of those lists and then wrapped that dictionary in a list. So you end up with a list that contains a dictionary that contains two lists! To make the Scatter Plot, I passed the data to plotly’s plot method. Converting to a Bar Chart Now let’s see if we can change the ScatterPlot to a Bar Chart. First off, we’ll play around with the plot data. The following was done via the Python interpreter: >>> scatter = py.get_figure('driscollis', '0') >>> print scatter.to_string() Figure( data=Data([ Scatter( x=[u'Adair County', u'Adams County', u'Allamakee County', u'..', ], y=[u'7682', u'4029', u'14330', u'12887', u'6119', u'26076', '..' ] ) ]) ) This shows how we can grab the figure using the username and the plot’s unique number. Then we printed out the data structure. You will note that it doesn’t print out the entire data structure. Now let’s do the actual conversion to a Bar Chart: from plotly.graph_objs import Data, Figure, Layout scatter_data = scatter.get_data() trace_bar = Bar(scatter_data[0]) data = Data([trace_bar]) layout = Layout(title="IA County Populations") fig = Figure(data=data, layout=layout) py.plot(fig, filename='bar_ia_county_pop') This will create a bar chart at the following URL: https://plot.ly/~driscollis/1. Here’s the image of the graph: <br> This code is slightly different than the code we used originally. In this case, we explicitly created a Bar object and passed it the scatter plot’s data. Then we put that data into a Data object. Next we created a Layout object and gave our chart a title. Then we created a Figure object using the data and layout objects. Finally we plotted the bar chart. Saving the Graph to Disk Plotly also allows you to save your graph to your hard drive. You can save it in the following formats: png, svg, jpeg, and pdf. Assuming you still have the Figure object from the previous example handy, you can do the following: py.image.save_as(fig, filename='graph.png') If you want to save using one of the other formats, then just use that format’s extension in the filename. Wrapping Up At this point you should be able to use the plotly package pretty well. There are many other graph types available, so be sure to read Plotly’s documentation thoroughly. They also support streaming graphs. As I understand it, Plotly allows you to create 10 graphs for free. After that you would either have to delete some of your graphs or pay a monthly fee. Additional Reading Plotly Python documentation Plotly User Guide
November 10, 2014
by Mike Driscoll
· 10,936 Views
article thumbnail
Missing Stack Traces for Repeated Exceptions
A long while ago a optimisation was added to the JVM so that if the same exception is thrown again and again and again a single instance of the Exception is created without the stack trace filled in in order to increase performance. This is an excellent idea unless you are trying to diagnose a problem and you have missed the original error. If you forgot about this optimisation you send the afternoon looking at the following log output and weeping slightly. (In my defence I have a little one in the house hence the fuzzy brain and lack of blogging action this year) java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException ... java.lang.Exception: Uncaught exceptions during test at oracle.jdevstudio-testware-tests.level0.testADFcMATS(/.../work/mw9111/jdeveloper/jdev/extensions/oracle.jdevstudio-testware-tests/abbot/common-adf/CreateNewCustomAppAndProjectWithName.xml:25) at oracle.jdevstudio-testware-tests.level0.testADFcMATS(/.../work/mw9111/jdeveloper/jdev/extensions/oracle.jdevstudio-testware-tests/abbot/level0/testADFcMATS.xml:94) at oracle.abbot.JDevScriptFixture.runTest(JDevScriptFixture.java:555) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at junit.textui.TestRunner.doRun(TestRunner.java:116) at junit.textui.TestRunner.doRun(TestRunner.java:109) at oracle.abbot.AbbotRunner.run(AbbotRunner.java:614) at oracle.abbot.AbbotAddin$IdeAbbotRunner.run(AbbotAddin.java:634) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ArrayIndexOutOfBoundsException [Crickets] It turns out that you can turn off this optimisation with a simple flag: java -XX:-OmitStackTraceInFastThrow .... In my particular case the actual exception causing this trouble was a NPE from the GlyphView code in JDK8. (One that is being caused by a glitch in hotspot it seems) But that in turn was causing the AIOOBE in some logging code clouding the issue even more. This in particular is a good flag to add by default when running your automated tests, particularly in combination with the stack trace length override I have talked about before.
November 9, 2014
by Gerard Davison
· 11,631 Views · 1 Like
  • Previous
  • ...
  • 805
  • 806
  • 807
  • 808
  • 809
  • 810
  • 811
  • 812
  • 813
  • 814
  • ...
  • Next
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook
×