This example explains to display HTML in Android TextView. Many times while you design an application, you may encounter a place where you will like to use HTML content in your screen. This may be to display a static “eula” or “help” content. In android there is a lovely class android.text.HTML that processes HTML strings into displayable styled text. Currently android doesn’t support all HTML tags. Android API documentation does not stipulate what HTML tags are supported. I have looked into the android Source code and from a quick look at the source code, here’s what seems to be supported as of now. http://grepcode.com/file/repository.grepcode.com/java/ext/com.google.android/android/2.2_r1.1/android/text/Html.java , , , , , , , , , , , , , , , , , , , , From HTML method returns displayable styled text from the provided HTML string. As per );"="" android’s official Documentations any tags in the HTML will display as a generic replacement image which your program can then go through and replace with real images. Html.formHtml method takes an Html.TagHandler and an Html.ImageGetter as arguments as well as the text to parse. We can parse null as for the Html.TagHandler but you’d need to implement your own Html.ImageGetter as there isn’t a default implementation. The Html.ImageGetterneeds to run synchronously and if you’re downloading images from the web you’ll probably want to do that asynchronously. But in my example I am using the images from resources to make my ImageGetter implementation simpler. package com.javatechig.example.ui; import android.os.Bundle; import android.app.Activity; import android.graphics.drawable.Drawable; import android.text.Html; import android.view.Menu; import android.widget.TextView; /* * @author: nilanchala * http://javatechig.com/ */ public class MainActivity extends Activity { private final String htmlText = " Heading TextThis tutorial " + "explains how to display " + "HTML text in android text view. " + "" + " Example from " + "Javatechig.com"; @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_main); TextView htmlTextView = (TextView)findViewById(R.id.html_text); htmlTextView.setText(Html.fromHtml(htmlText, new ImageGetter(), null)); } @Override public boolean onCreateOptionsMenu(Menu menu) { // Inflate the menu; this adds items to the action bar if it is present. getMenuInflater().inflate(R.menu.main, menu); return true; } private class ImageGetter implements Html.ImageGetter { public Drawable getDrawable(String source) { int id; if (source.equals("hughjackman.jpg")) { id = R.drawable.hughjackman; } else { return null; } Drawable d = getResources().getDrawable(id); d.setBounds(0,0,d.getIntrinsicWidth(),d.getIntrinsicHeight()); return d; } }; }

I sometimes help out users in Quartz Scheduler forums. Once in a while some one would ask how can he/she setup the Quartz inside a web application. This is actualy a fairly simple thing to do. The library already comes with a ServletContextListener that you can use to start a Scheduler. I will show you a simple webapp example here. First create a Maven pom.xml file. 4.0.0 quartz-web-demo quartz-web-demo war 1.0-SANPSHOT org.quartz-scheduler quartz 2.2.0 Then you need to create a src/main/webapp/META-INF/web.xml file. quartz:config-file quartz.properties quartz:shutdown-on-unload true quartz:wait-on-shutdown true quartz:start-on-load true org.quartz.ee.servlet.QuartzInitializerListener And lastly, you need a src/main/resources/quartz.properties config file for Scheduler. # Main Quartz configuration org.quartz.scheduler.skipUpdateCheck = true org.quartz.scheduler.instanceName = MyQuartzScheduler org.quartz.scheduler.jobFactory.class = org.quartz.simpl.SimpleJobFactory org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool org.quartz.threadPool.threadCount = 5 You may configure many other things with Quartz, but above should get you started as in In-Memory scheduler. Now you should able to compile and run it. bash> mvn compile bash> mvn org.apache.tomcat.maven:tomcat7-maven-plugin:2.1:run -Dmaven.tomcat.port=8081 How to configure logging for Quartz Scheduler Another frequently asked question is how do they setup logging and see the DEBUG level messages. The Quartz Schedulers uses SLF4J, so you have many loggers options to choose. I will show you how to setup Log4j for example below. First, add this to your pom.xml org.slf4j slf4j-log4j12 1.7.5 Then add src/main/resources/log4j.properties file to show messages onto STDOUT. log4j.rootLogger=INFO, stdout log4j.logger.org.quartz=DEBUG log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%5p [%t] (%F:%L) - %m%n Restart your web application on command line, and now you should see all the DEBUG level logging messages coming from Quartz library. With everything running, your next question might be asking how do you access the scheduler from your web application? Well, when the scheduler is created by the servlet context listener, it is stored inside the web app’s ServletContext space with org.quartz.impl.StdSchedulerFactory.KEY key. So you may retrieve it and use it in your own Servlet like this: public class YourServlet extends HttpServlet { public init(ServletConfig cfg) { String key = "org.quartz.impl.StdSchedulerFactory.KEY"; ServletContext servletContext = cfg.getServletContext(); StdSchedulerFactory factory = (StdSchedulerFactory) servletContext.getAttribute(key); Scheduler quartzScheduler = factory.getScheduler("MyQuartzScheduler"); // TODO use quartzScheduler here. } } Now you are on your way to build your next scheduling application! Have fun!

Our problem began when we tried to read a certain cell that contained the value 929 as a numeric field and store it into an integer.

In a complex system such as a software development team, it's easy for fear to arise.

inner interface is also called nested interface, which means declare an interface inside of another interface. for example, the entry interface is declared in the map interface. public interface map { interface entry{ int getkey(); } void clear(); } why use inner interface? there are several compelling reasons for using inner interface: it is a way of logically grouping interfaces that are only used in one place. it increases encapsulation. nested interfaces can lead to more readable and maintainable code. one example of inner interface used in java standard library is java.util.map and java.util.map.entry. here java.util.map is used also as a namespace. entry does not belong to the global scope, which means there are many other entities that are entries and are not necessary map’s entries. this indicates that entry represents entries related to the map. how inner interface works? to figure out how inner interface works, we can compare it with nested classes. nested classes can be considered as a regular method declared in outer class. since a method can be declared as static or non-static, similarly nested classes can be static and non-static. static class is like a static method, it can only access outer class members through objects. non-static class can access any member of the outer class. because an interface can not be instantiated, the inner interface only makes sense if it is static. therefore, by default inter interface is static, no matter you manually add static or not. a simple example of inner interface? map.java public interface map { interface entry{ int getkey(); } void clear(); } mapimpl.java public class mapimpl implements map { class implentry implements map.entry{ public int getkey() { return 0; } } @override public void clear() { //clear } }

Apache Wicket is known for its capability of transparently handling the state of web applications on server side, and can be easily adopted to create RESTful services.

in this article i’ll introduce the basic coding that require to create jquery datatable using json passed by simple servlet. datatable is very powerful jquery based grid with advance features which can be build in short span of time with customize features. installation 1. download latest jquery datatable download 2. above download will provide two jquery plugin jquery.js and querytables.js 3. default stylesheet which shipped with latest datatable download package note: you can download full source code from github link creating the datatable we can write below code to create the basic datatable with data feedsummary.jsp ========================== $(document).ready will ready to execute the javascript and var otable = $(‘#tableid’).datatable says that write datatable on tableid place. datatables will adding sorting, filtering, paging and information to your table by default, providing the end user of your web-site with the ability to control the display of the table and find the information that they want from it as quickly as possible. the pointer tableid and column name will be defined in table tag as below feedsummary.jsp ===================== first namelast nameaddress 1address 2 above datatable code invoke feedservlet which will return json string as defined below feedservlet.java =============== protected void dopost(httpservletrequest request, httpservletresponse response) throws servletexception, ioexception { printwriter out = response.getwriter(); string json = "{ \"demo\":[[\"first name\",\"last name\","+ +\"address1\",\"address2\"],[\"first name\",\"last name\",\"address1\",\"address2\"]]}"; out.println(json); } now either we can use servlet annotation or web.xml as below to register above feedservlet web.xml ========= feedservlet feedservlet feedservlet feedservlet /feedservlet running incorporate the above point and deploy with server to view the result as follows: http://localhost:8080/exampledatatablejson/feedsummary.jsp jquery datatable image conclusion you can download full source code from github link and most welcome to fork or update the same. references: http://datatables.net/examples/

I came across this interesting problem today and want to capture it. Suppose this simple Camel route, consuming from ActiveMQ and logging the message: After deploying this to ServiceMix (JBoss Fuse 6.0 in my test) it works just fine. The problem occurs when stopping this Camel route (via an osgi:stop of the corresponding bundle). The connection into the ActiveMQ broker does not get closed! Drilling into the JMX view of the broker the connection is still registered in JMX. Even worse each restart and stop of the Camel route leaks another connection. I did a bit of root cause analysis and found: When the Camel route is stopped it calls into SingleConnectionFactory.destroy(). This cleans up the internally held ActiveMQConnection. At this stage the connection is properly removed from the broker's JMX view, which you will probably only notice when debugging through the code. However the Spring JMS listener used by the Camel JMS consumer is still running and it detects that the connection is down and tries to transparently reconnect. This calls into SingleConnectionFactory.createConnection again (full stack trace below [1]). The SingleConnectionFactory does happily reopen a new connection into the broker which then remains registered in JMX and open although the route gets stopped. So how do you resolve this? Instead of using Springs SingleConnectionFactory I recommend to use ActiveMQs PooledConnectionFactory instead. So the above Camel route configuration becomes When using the ActiveMQs PooledConnectionFactory instead, things behave pretty much the same with one subtle but important difference. Similar to above, stopping the Camel bundle calls into PooledConnectionFactory.stop(). This internally closes all ActiveMQConnections (only one in this example case but potentially more) which also unregisters the connection from the brokers JMX view. Now, Springs JMS listener used by the Camel JMS consumer is still running and detects the connection closure and tries to transparently reconnect. This calls into PooledConnectionFactory.createConnection(). This implementation however contains the following check: if (stopped.get()) { LOG.debug("PooledConnectionFactory is stopped, skip create new connection."); return null; } AtomicBoolean stopped will be set to true and no new connection is established! Springs SingleConnectionFactory does not have this logic. It happily reopens a new connection after it got destroyed. Please note the properties init-method="start" destroy-method="stop" on the PooledConnectionFactory bean definition are important as otherwise you may also leak connections when shutting down your bundles. [1] Daemon Thread [Camel (camelContext) thread #21 - JmsConsumer[EwdTest1]] (Suspended (breakpoint at line 280 in org.springframework.jms.connection.SingleConnectionFactory)) owns: java.lang.Object (id=9254) owns: java.lang.Object (id=9255) owns: java.lang.Object (id=9286) org.springframework.jms.connection.SingleConnectionFactory.initConnection() line: 280 org.springframework.jms.connection.SingleConnectionFactory.createConnection() line: 225 org.apache.camel.component.jms.DefaultJmsMessageListenerContainer(org.springframework.jms.support.JmsAccessor).createConnection() line: 184 org.apache.camel.component.jms.DefaultJmsMessageListenerContainer(org.springframework.jms.listener.AbstractJmsListeningContainer).createSharedConnection() line: 404 org.apache.camel.component.jms.DefaultJmsMessageListenerContainer(org.springframework.jms.listener.AbstractJmsListeningContainer).refreshSharedConnection() line: 389 org.apache.camel.component.jms.DefaultJmsMessageListenerContainer(org.springframework.jms.listener.DefaultMessageListenerContainer).refreshConnectionUntilSuccessful() line: 869 org.apache.camel.component.jms.DefaultJmsMessageListenerContainer(org.springframework.jms.listener.DefaultMessageListenerContainer).recoverAfterListenerSetupFailure() line: 851 org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run() line: 982 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) line: 895 java.util.concurrent.ThreadPoolExecutor$Worker.run() line: 918 java.lang.Thread.run() line: 680

This post was originally authored by Rob Winch from SpringSource. This is my last post in a two part series on Spring Security 3.2.0.RC1. My previous post discussed Spring Security's CSRF protection. In this post we will discuss how to use Spring Security to add various response headers to help secure your application. SECURITY HEADERS Many of the new Spring Security features in 3.2.0.RC1 are implemented by adding headers to the response. The foundation for these features came from hard work from Marten Deinum. If the name sounds familiar, it may because one of his 10K+ posts on the Spring Forums has helped you out. If you are using XML configuration, you can add all of the default headers using Spring Security's element with no child elements to add all the default headers to the response: ... If you are using Spring Security's Java configuration, all of the default security headers are added by default. They can be disabled using the Java configuration below: @EnableWebSecurity @Configuration public class WebSecurityConfig extends WebSecurityConfigurerAdapter { @Override protected void configure(HttpSecurity http) throws Exception { http .headers().disable() ...; } } The remainder of this post will discuss each of the default headers in more detail: Cache Control Content Type Options HTTP Strict Transport Security X-Frame-Options X-XSS-PROTECTION Cache Control In the past Spring Security required you to provide your own cache control for your web application. This seemed reasonable at the time, but browser caches have evolved to include caches for secure connections as well. This means that a user may view an authenticated page, log out, and then a malicious user can use the browser history to view the cached page. To help mitigate this Spring Security has added cache control support which will insert the following headers into you response. Cache-Control: no-cache, no-store, max-age=0, must-revalidate Pragma: no-cache Simply adding the element with no child elements will automatically add Cache Control and quite a few other protections. However, if you only want cache control, you can enable this feature using Spring Security's XML namespace with the element. ... Similarly, you can enable only cache control within Java Configuration with the following: @EnableWebSecurity @Configuration public class WebSecurityConfig extends WebSecurityConfigurerAdapter { @Override protected void configure(HttpSecurity http) throws Exception { http .headers() .cacheControl() .and() ...; } } If you actually want to cache specific responses, your application can selectively invokeHttpServletResponse.setHeader(String,String) to override the header set by Spring Security. This is useful to ensure things like CSS, JavaScript, and images are properly cached. When using Spring Web MVC, this is typically done within your configuration. For example, the following configuration will ensure that the cache headers are set for all of your resources: @EnableWebMvc public class WebMvcConfiguration extends WebMvcConfigurerAdapter { @Override public void addResourceHandlers(ResourceHandlerRegistry registry) { registry .addResourceHandler("/resources/**") .addResourceLocations("/resources/") .setCachePeriod(31556926); } // ... } Content Type Options Uploading Files There are many additional things one should do (i.e. only display the document in a distinct domain, ensure Content-Type header is set, sanitize the document, etc) when allowing content to be uploaded. However, these measures are out of the scope of what Spring Security provides. It is also important to point out when disabling content sniffing, you must specify the content type in order for things to work properly. Historically browsers, including Internet Explorer, would try to guess the content type of a request using content sniffing. This allowed browsers to improve the user experience by guessing the content type on resources that had not specified the content type. For example, if a browser encountered a JavaScript file that did not have the content type specified, it would be able to guess the content type and then execute it. The problem with content sniffing is that this allowed malicious users to use polyglots (i.e. a file that is valid as multiple content types) to execute XSS attacks. For example, some sites may allow users to submit a valid postscript document to a website and view it. A malicious user might create a postscript document that is also a valid JavaScript file and execute a XSS attack with it. Content sniffing can be disabled by adding the following header to our response: X-Content-Type-Options: nosniff Just as with the cache control element, the nosniff directive is added by default when using the element with no child elements. However, if you want more control over which headers are added you can use the element as shown below: ... The X-Content-Type-Options header is added by default with Spring Security Java configuration. If you want more control over the headers, you can explicitly specify the content type options with the following: @EnableWebSecurity @Configuration public class WebSecurityConfig extends WebSecurityConfigurerAdapter { @Override protected void configure(HttpSecurity http) throws Exception { http .headers() .contentTypeOptions() .and() ...; } } HTTP Strict Transport Security (HSTS) When you type in your bank's website, do you enter mybank.example.com or do you enter https://mybank.example.com? If you omit the https protocol, you are potentially vulnerable toMan in the Middle attacks. Even if the website performs a redirect to https://mybank.example.com a malicious user could intercept the initial HTTP request and manipulate the response (i.e. redirect to https://mibank.example.com and steal their credentials). Many users omit the https protocol and this is why HTTP Strict Transport Security (HSTS)was created. Once mybank.example.com is added as a HSTS host, a browser can know ahead of time that any request to mybank.example.com should be interpreted as https://mybank.example.com. This greatly reduces the possibility of a Man in the Middle attack occurring. HSTS Notes In accordance with RFC6797, the HSTS header is only injected into HTTPS responses. In order for the browser to acknowledge the header, the browser must first trust the CA that signed the SSL certificate used to make the connection (not just the SSL certificate). One way for a site to be marked as a HSTS host is to have the host preloaded into the browser. Another is to add the "Strict-Transport-Security" header to the response. For example the following would instruct the browser to treat the domain as an HSTS host for a year (there are approximately 31536000 seconds in a year): Strict-Transport-Security: max-age=31536000 ; includeSubDomains The optional includeSubDomains directive instructs Spring Security that subdomains (i.e. secure.mybank.example.com) should also be treated as an HSTS domain. As with the other headers, Spring Security adds the previous header to the response when the element is specified with no child elements. It is also automatically added when you are using Java Configuration. You can also only use HSTS headers with the element as shown below: ... Similarly, you can enable only HSTS headers with Java Configuration: @EnableWebSecurity @Configuration public class WebSecurityConfig extends WebSecurityConfigurerAdapter { @Override protected void configure(HttpSecurity http) throws Exception { http .headers() .hsts() .and() ...; } } X-Frame-Options Content Security Policy Another modern approach to dealing with clickjacking is using a Content Security Policy. Spring Security does not provide support for this as the specification is not released and it is quite a bit more complicated. To stay up to date with this issue and to see how you can implement it with Spring Security refer to SEC-2117 Allowing your website to be added to a frame can be a security issue. For example, using clever CSS styling users could be tricked into clicking on something that they were not intending (video demo). For example, a user that is logged into their bank might click a button that grants access to other users. This sort of attack is known asClickjacking. There are a number ways to mitigate clickjacking attacks. For example, to protect legacy browsers from clickjacking attacks you can use frame breaking code. While not perfect, the frame breaking code is the best you can do for the legacy browsers. A more modern approach to address clickjacking is to use X-Frame-Options header: X-Frame-Options: DENY The X-Frame-Options response header instructs the browser to prevent any site with this header in the response from being rendered within a frame. As with the other response headers, this is automatically included when the element is specified with no child elements. You can also explicitly specify the element to control which headers are added to the response. ... Similarly, you can enable only frame options within Java Configuration with the following: @EnableWebSecurity @Configuration public class WebSecurityConfig extends WebSecurityConfigurerAdapter { @Override protected void configure(HttpSecurity http) throws Exception { http .headers() .frameOptions() .and() ...; } } X-XSS-Protection Some browsers have built in support for filtering out reflected XSS attacks. This is by no means full proof, but does assist in XSS protection. The filtering is typically enabled by default, so adding the header typically just ensures it is enabled and instructs the browser what to do when a XSS attack is detected. For example, the filter might try to change the content in the least invasive way to still render everything. At times, this type of replacement can become a XSS vulnerability in itself. Instead, it is best to block the content rather than attempt to fix it. To do this we can add the following header: X-XSS-Protection: 1; mode=block This header is included by default when the element is specified with no child elements. We can explicitly state it using the element as shown below: ... Similarly, you can enable only xss protection within Java Configuration with the following: @EnableWebSecurity @Configuration public class WebSecurityConfig extends WebSecurityConfigurerAdapter { @Override protected void configure(HttpSecurity http) throws Exception { http .headers() .xssProtection() .and() ...; } } FEEDBACK PLEASE If you encounter a bug, have an idea for improvement, etc please do not hesitate to bring it up! We want to hear your thoughts so we can ensure we get it right before the code is generally available. Trying out new features early is a good and simple way to give back to the community. This also ensures that the features you want are present and working as you think they should. Please log any issues or feature requests to the Spring Security JIRA. After logging a JIRA, we encourage (but do not require) you to submit your changes in a pull request. You can read more about how to do this in the Contributor Guidelines If you have questions on how to do something, please use the Spring Security forums orStack Overflow with the tag spring-security (I will be monitoring them closely). If you have specific comments questions about this blog, feel free to leave a comment. Using the appropriate tools will help make it easier for everyone. CONCLUSION You should have a good understanding of the new features present in Spring Security 3.2.RC1.

TL;DR: This blog post features a small demo project on github: neo4j-uuid and explains how to automatically assign UUIDs to nodes and relationships in Neo4j. A very brief introduction into Neo4j 1.9′s KernelExtensionFactory is included as well. A Little Rant on Neo4j Node/Relationship IDs In a lot of use cases there is demand for storing a reference to a Neo4j node or relationship in a third party system. The first naive idea probably is to use the internal node/relationship id that Neo4j provides. Do not do that! Ever! You ask why? Well, Neo4j’s id is basically a offset in one of the store files Neo4j uses (with some math involved). Assume you delete couple of nodes. This produces holes in the store files that Neo4j might reclaim when creating new nodes later on. And since the id is a file offset there is a chance that the new node will have exactly the same id like the previously deleted node. If you don’t synchronously update all node id references stored elsewhere, you’re in trouble. If neo4j would be completely redeveloped from scratch the getId() method would not be part of the public API. As long as you use node ids only inside a request of an application for example, there’s nothing wrong. To repeat myself: Never ever store a node id in a third party system. I have officially warned you. UUIDs Enough of ranting, let’s see what we can do to safely store node references in an external system. Basically we need an identifier that has no semantics in contrast to the node id. A common approach to this is using Universally Unique Identifiers (UUID). Java JDK offers a UUID implementation, so we could potentially use UUID.randomUUID(). Unfortunately random UUIDs are slow to generate. A preferred approach is to use the machine’s MAC and a timestamp as base for the UUID – this should provide enough uniqueness. There a nice library out there at http://wiki.fasterxml.com/JugHome providing exactly what we need. Automatic UUID Assignments For convenience it would be great if all fresh created nodes and relationships get automatically assigned a uuid property without doing this explicitly. Fortunately Neo4j supports TransactionEventHandlers, a callback interface pluging into transaction handling. A TransactionEventHandler has a chance to modify or veto any transaction. It’s a sharp tool which can have significant negative performance impact if used the wrong way. I’ve implemented a UUIDTransactionEventHandler that performs the following tasks: Populate a UUID property for each new node or relationship Reject a transaction if a manual modification of a UUID is attempted; either assignment or removal public class UUIDTransactionEventHandler implements TransactionEventHandler { public static final String UUID_PROPERTY_NAME = "uuid"; private final TimeBasedGenerator uuidGenerator = Generators.timeBasedGenerator(); @Override public Object beforeCommit(TransactionData data) throws Exception { checkForUuidChanges(data.removedNodeProperties(), "remove"); checkForUuidChanges(data.assignedNodeProperties(), "assign"); checkForUuidChanges(data.removedRelationshipProperties(), "remove"); checkForUuidChanges(data.assignedRelationshipProperties(), "assign"); populateUuidsFor(data.createdNodes()); populateUuidsFor(data.createdRelationships()); return null; } @Override public void afterCommit(TransactionData data, java.lang.Object state) { } @Override public void afterRollback(TransactionData data, java.lang.Object state) { } /** * @param propertyContainers set UUID property for a iterable on nodes or relationships */ private void populateUuidsFor(Iterable propertyContainers) { for (PropertyContainer propertyContainer : propertyContainers) { if (!propertyContainer.hasProperty(UUID_PROPERTY_NAME)) { final UUID uuid = uuidGenerator.generate(); final StringBuilder sb = new StringBuilder(); sb.append(Long.toHexString(uuid.getMostSignificantBits())).append(Long.toHexString(uuid.getLeastSignificantBits())); propertyContainer.setProperty(UUID_PROPERTY_NAME, sb.toString()); } } } private void checkForUuidChanges(Iterable> changeList, String action) { for (PropertyEntry removedProperty : changeList) { if (removedProperty.key().equals(UUID_PROPERTY_NAME)) { throw new IllegalStateException("you are not allowed to " + action + " " + UUID_PROPERTY_NAME + " properties"); } } } } Setting up Using KernelExtensionFactory There are two remaining tasks for full automation of UUID assignments: We need to setup autoindexing for uuid properties to have a convenient way to look up nodes or relationships by UUID We need to register UUIDTransactionEventHandler with the graph database Since version 1.9 Neo4j has the notion of KernelExtensionFactory. Using KernelExtensionFactory you can supply a class that receives lifecycle callbacks when e.g. Neo4j is started or stopped. This is the right place for configuring autoindexing and setting up the TransactionEventHandler. Since JVM’s ServiceLoader is used KernelExtenstionFactories need to be registered in a file META-INF/services/org.neo4j.kernel.extension.KernelExtensionFactory by listing all implementations you want to use: org.neo4j.extension.uuid.UUIDKernelExtensionFactory KernelExtensionFactories can declare dependencies, therefore declare a inner interface (“Dependencies” in code) below that just has getters. Using proxies Neo4j will implement this class and supply you with the required dependencies. The dependencies are match on requested type, see Neo4j’s source code what classes are supported for being dependencies. KernelExtensionFactories must implement a newKernelExtension method that is supposed to return a instance of LifeCycle. For our UUID project we return a instance of UUIDLifeCycle: package org.neo4j.extension.uuid; import org.neo4j.graphdb.GraphDatabaseService; import org.neo4j.graphdb.PropertyContainer; import org.neo4j.graphdb.event.TransactionEventHandler; import org.neo4j.graphdb.factory.GraphDatabaseSettings; import org.neo4j.graphdb.index.AutoIndexer; import org.neo4j.graphdb.index.IndexManager; import org.neo4j.kernel.configuration.Config; import org.neo4j.kernel.lifecycle.LifecycleAdapter; import java.util.Map; /** * handle the setup of auto indexing for UUIDs and registers a {@link UUIDTransactionEventHandler} */ class UUIDLifeCycle extends LifecycleAdapter { private TransactionEventHandler transactionEventHandler; private GraphDatabaseService graphDatabaseService; private IndexManager indexManager; private Config config; UUIDLifeCycle(GraphDatabaseService graphDatabaseService, Config config) { this.graphDatabaseService = graphDatabaseService; this.indexManager = graphDatabaseService.index(); this.config = config; } /** * since {@link org.neo4j.kernel.NodeAutoIndexerImpl#start()} is called *after* {@link org.neo4j.extension.uuid.UUIDLifeCycle#start()} it would apply config settings for auto indexing. To prevent this we change config here. * @throws Throwable */ @Override public void init() throws Throwable { Map params = config.getParams(); params.put(GraphDatabaseSettings.node_auto_indexing.name(), "true"); params.put(GraphDatabaseSettings.relationship_auto_indexing.name(), "true"); config.applyChanges(params); } @Override public void start() throws Throwable { startUUIDIndexing(indexManager.getNodeAutoIndexer()); startUUIDIndexing(indexManager.getRelationshipAutoIndexer()); transactionEventHandler = new UUIDTransactionEventHandler(); graphDatabaseService.registerTransactionEventHandler(transactionEventHandler); } @Override public void stop() throws Throwable { stopUUIDIndexing(indexManager.getNodeAutoIndexer()); stopUUIDIndexing(indexManager.getRelationshipAutoIndexer()); graphDatabaseService.unregisterTransactionEventHandler(transactionEventHandler); } void startUUIDIndexing(AutoIndexer autoIndexer) { autoIndexer.startAutoIndexingProperty(UUIDTransactionEventHandler.UUID_PROPERTY_NAME); } void stopUUIDIndexing(AutoIndexer autoIndexer) { autoIndexer.stopAutoIndexingProperty(UUIDTransactionEventHandler.UUID_PROPERTY_NAME); } } Most of the code is pretty much straight forward, l.44/45 set up autoindexing for uuid property. l48 registers the UUIDTransactionEventHandler with the graph database. Not that obvious is the code in the init() method. Neo4j’s NodeAutoIndexerImpl configures autoindexing itself and switches it on or off depending on the respective config option. However we want to have autoindexing always switched on. Unfortunately NodeAutoIndexerImpl is run after our code and overrides our settings. That’s we l.37-40 tweaks the config settings to force nice behaviour of NodeAutoIndexerImpl. Looking up Nodes or Relationships for UUID For completeness the project also contains a trivial unmanaged extension for looking up nodes and relationships using the REST interface, see UUIDRestInterface. By sending a HTTP GET to http://localhost:7474/db/data/node/ the node’s internal id returned. Build System and Testing For building the project, Gradle is used; build.gradle is trivial. Of course couple of tests are included. As a long standing addict I’ve obviously used Spock for testing. See the test code here. Final Words A downside of this implementation is that each and every node and relationships gets indexed. Indexing always trades write performance for read performance. Keep that in mind. It might make sense to get rid of unconditional auto indexing and put some domain knowledge into the TransactionEventHandler to assign only those nodes uuids and index them that are really used for storing in an external system.

In one of our weekly book clubs, we worked on an Open/Closed Principle kata. What's the OCP? The Open/Closed Principle, as proposed by Bertrand Meyer, says that systems should be open for extension, but closed for the modification of existing components. This notion commonly maps to the addition of new code to implement a new requirement, tied with the minimal modifications of existing code. For example, adding a new class without touching existing source file apart from some configuration. The related Kata turns the OCP dial up to eleven, and requires you to implement a series of requirements in a strictly OCP way; it's not that code cannot be modified, but refactoring steps where it is extended must be separated from extension steps where new classes are added to the codebase. Here are some links giving some examples of the process: http://www.slideshare.net/giordano/xpug-coding-dojo-katayahtzee-in-ocp-way http://matteo.vaccari.name/blog/archives/293 The problem Given the unfamiliarity of the some of the team with the principle, we picked up a very simple problem: FizzBuzz*. However, it is not trivial to develop it by following OCP by the book*link repository; by the end of the 4 Pomodoros we had completed the main text and added a new requirement (42 is the Answer, not BuzzBang). One thing that saves much time in this kind of exercises is to start with predefined scenarios (1st requirement: number itself; 2nd requirement: 3 is Buzz; and so on). Only show the current requirement, bring it to completion, and go to the next; unless of course you're exercising in splitting requirements... Method The process must be followed, even blindly, to get a nice workout from this exercise; this is the point of a kata, and you should be a Nazi in avoiding violating the constraints that make the exercise difficult (for the duration of this exercise alone). Tell them their code will be thrown away at the end of the session, not put into production, and that exercising at the maximum viable effort is just for their benefit. The protocol to follow is: read the new requirement and understand it. Add a new test for it, so that you now have N test and only 1 of them is failing. Refactor at will, but at any time only N - 1 tests must be green, and the new one must still be failing ("refactoring" move). Add new code and/or modify Factory classes that setup the system as a graph of objects or closures (as your only "new code" move). Show a green bar. If you can't complete the "new code" move, come back to the refactoring step and repeat it differently. The constraints are: In the "refactoring" move there should still be N-1 tests passing and the new one failing (the form changes but not its function, to put it into Alexander's terms.) In the "new code" move nothing outside of Factories can be modified, while of course new classes can be added at will. Effect The first benefit of exercising on single principles and patterns is that you are then able to talk about OCP during daily work, within your team. Naming things is one of the first steps in mastering them. You can say to a colleague, ""You should first refactor the X package before adding functionality B, since it is a variation over functionality A that it was developed for. But first make it extensible, remember OCP?" A side effect is that you can target solutions that follow this method and stimulate them during pair programming. Both participants need to be aware of the roads that can be followed to get at a solution together. Finally, OCP-like solutions where new requirements are added only by adding code (and without adding duplication) are certainly in my list of code perfumes* (to be cherished, as the opposite of smells to be removed). Conclusions Working strictly and iteratively in the two OCP phases for a couple of hours doesn't mean this is always the right way to face a programming problem. However, when you're training your quadriceps, you don't suddenly go off cycling and come back every few minutes (at least the brain does not have muscles that you rotate in a single Pomodoro.) So no excuses for going off track: you can train other muscles and movements, as important as these, but at another time. The OCP contraints are also a nice alternative for the usual kata on prime numbers or bowling, or to execute katas without a goal. One possible and frequent output of katas is that you end up with very dense 3 lines of code, that will not accept a new requirements even if menaced with a shotgun. The purpose of those exercises is to show incremental development at work in microslices, but the OCP one forces the development of some architectural skill instead of working on a single, monolithic class.

A couple of weeks ago I was trying to write a test around some OAuth code that we have on an internal application and I was using Jersey Client to send the various requests. I initially started with the following code: Client = Client.create(); ClientResponse response = client.resource( "http://localhost:59680" ).get( ClientResponse.class ); But when I ran the test I was getting the following exception: com.sun.jersey.api.client.ClientHandlerException: java.net.ProtocolException: Server redirected too many times (20) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:151) at com.sun.jersey.api.client.Client.handle(Client.java:648) at com.sun.jersey.api.client.WebResource.handle(WebResource.java:680) at com.sun.jersey.api.client.WebResource.get(WebResource.java:191) at com.neotechnology.testlab.manager.webapp.AuthenticationIntegrationTest.shouldRedirectToGitHubForAuthentication(AuthenticationIntegrationTest.java:81) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at com.neotechnology.kirkaldy.testing.Resources$1.evaluate(Resources.java:84) at com.neotechnology.kirkaldy.testing.FailureOutput$2.evaluate(FailureOutput.java:37) at org.junit.rules.RunRules.evaluate(RunRules.java:18) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.junit.runner.JUnitCore.run(JUnitCore.java:157) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63) Caused by: java.net.ProtocolException: Server redirected too many times (20) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1446) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:249) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149) ... 28 more If we check the traffic going across port 59680 we can see what’s going wrong: $ sudo ngrep -d lo0 port 59680 interface: lo0 (127.0.0.0/255.0.0.0) filter: (ip) and ( port 59680 ) ##### T 127.0.0.1:59704 -> 127.0.0.1:59680 [AP] GET / HTTP/1.1..User-Agent: Java/1.6.0_45..Host: localhost:59680..Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2..Connection: keep-alive.... ## T 127.0.0.1:59680 -> 127.0.0.1:59704 [AP] HTTP/1.1 302 Found..Set-Cookie: JSESSIONID=mdyw3a4fmqc1b6p53birm4dd;Path=/..Expires: Thu, 01 Jan 1970 00:00:00 GMT..Location: http://localhost:59679/authorize?client_id=basic-client&state=the-state&scope=user%2Crepo..Content-Length : 0..Server: Jetty(8.1.8.v20121106).... ########### T 127.0.0.1:59707 -> 127.0.0.1:59680 [AP] GET /auth/callback?code=timey-wimey&state=the-state HTTP/1.1..User-Agent: Java/1.6.0_45..Host: localhost:59680..Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2..Connection: keep-alive.... ## T 127.0.0.1:59680 -> 127.0.0.1:59707 [AP] HTTP/1.1 302 Found..Cache-Control: no-cache..Set-Cookie: JSESSIONID=8gggez0ns9ftiex4314mbgz9;Path=/..Expires: Thu, 01 Jan 1970 00:00:00 GMT..Location: http://localhost:59680/..Content-Length: 0..Server: Jetty(8.1.8.v20121106).... ########### T 127.0.0.1:59713 -> 127.0.0.1:59680 [AP] GET / HTTP/1.1..User-Agent: Java/1.6.0_45..Host: localhost:59680..Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2..Connection: keep-alive.... ## The response we receive includes a direction to the client to store a cookie but we can see on the next request that the cookie hasn’t been included. I came across this post, which had a few suggestions on how to get around the problem, but the only approach that worked for me was to use jersey-apache-client for which I added the following dependency: com.sun.jersey.contribs jersey-apache-client 1.13 jar I then change my client code to read like this: ApacheHttpClientConfig config = new DefaultApacheHttpClientConfig(); config.getProperties().put(ApacheHttpClientConfig.PROPERTY_HANDLE_COOKIES, true); ApacheHttpClient client = ApacheHttpClient.create( config ); client.setFollowRedirects(true); client.getClientHandler().getHttpClient().getParams().setBooleanParameter( HttpClientParams.ALLOW_CIRCULAR_REDIRECTS, true ); ClientResponse response = client.resource( "http://localhost:59680" ).get( ClientResponse.class ); If we run that and watch the output using ngrep we can see that it now handles cookies correctly: $ sudo ngrep -d lo0 port 59680 Password: interface: lo0 (127.0.0.0/255.0.0.0) filter: (ip) and ( port 59680 ) ##### T 127.0.0.1:60372 -> 127.0.0.1:59680 [AP] GET / HTTP/1.1..User-Agent: Jakarta Commons-HttpClient/3.1..Host: localhost:59680.... ## T 127.0.0.1:59680 -> 127.0.0.1:60372 [AP] HTTP/1.1 302 Found..Set-Cookie: JSESSIONID=vn8zzf9ep3x4mtw66ydm0n6a;Path=/..Expires: Thu, 01 Jan 1970 00:00:00 GMT..Location: http://localhost:60322/authorize?client_id=basic-client&state=the-state&scope=user%2Crepo..Content-Length : 0..Server: Jetty(8.1.8.v20121106).... ## T 127.0.0.1:60372 -> 127.0.0.1:59680 [AP] GET /auth/callback?code=timey-wimey&state=the-state HTTP/1.1..User-Agent: Jakarta Commons-HttpClient/3.1..Host: localhost:59680..Cookie: $Version=0; JSESSIONID=vn8zzf9ep3x4mtw66ydm0n6a; $Path=/.... ## T 127.0.0.1:59680 -> 127.0.0.1:60372 [AP] HTTP/1.1 302 Found..Cache-Control: no-cache..Location: http://localhost:59680/..Content-Length: 0..Server: Jetty(8.1.8.v20121106).... ## T 127.0.0.1:60372 -> 127.0.0.1:59680 [AP] GET / HTTP/1.1..User-Agent: Jakarta Commons-HttpClient/3.1..Host: localhost:59680..Cookie: $Version=0; JSESSIONID=vn8zzf9ep3x4mtw66ydm0n6a; $Path=/.... ## T 127.0.0.1:59680 -> 127.0.0.1:60372 [AP] HTTP/1.1 200 OK..Vary: Accept-Encoding..Accept-Ranges: bytes..Content-Type: text/html..Content-Length: 2439..Last-Modified: Tue, 23 Jul 2013 10:48:15 GMT..Server: Jetty(8.1.8.v20121106)....... . . . . . . . . . . . ....

Doing code reviews is a great way to discover things that people might struggle to comprehend. While proof-reading OpenStack patches recently, I spotted that people were not correctly using the various decorators Python provides for methods. So, here's my attempt at providing me with a link to send in my next code reviews. :-) How Methods Work in Python A method is a function that is stored as a class attribute. You can declare and access such a function this way: >>> class Pizza(object): ... def __init__(self, size): ... self.size = size ... def get_size(self): ... return self.size ... >>> Pizza.get_size What Python tells you here is that the attribute get_size of the class Pizza is a method that is unbound. What does this mean? We'll know as soon as we try to call it: >>> Pizza.get_size() Traceback (most recent call last): File "", line 1, in TypeError: unbound method get_size() must be called with Pizza instance as first argument (got nothing instead) We can't call it because it's not bound to any instance of Pizza. A method wants an instance as its first argument (in Python 2 it must be an instance of that class and in Python 3 it could be anything). Let's try to do that, then: >>> Pizza.get_size(Pizza(42)) 42 It worked! We called the method with an instance as its first argument, so everything's fine. But you will agree with me if I say this is not a very handy way to call methods -- we have to refer to the class each time we want to call a method. And if we don't know what class our object is, this is not going to work for very long. So what Python does for us is that it binds all the methods from the class Pizza to any instance of this class. This means that the attribute get_size of an instance of Pizza is a bound method: a method for which the first argument will be the instance itself. >>> Pizza(42).get_size > >>> Pizza(42).get_size() 42 As expected, we don't have to provide any argument to get_size, since it's bound, its self argument is automatically set to our Pizza instance. Here's an even better proof of that: >>> m = Pizza(42).get_size >>> m() 42 Indeed, you don't even have to keep a reference to your Pizza object. Its method is bound to the object, so the method is sufficient to itself. But what if you wanted to know which object this bound method is bound to? Here's a little trick: >>> m = Pizza(42).get_size >>> m.__self__ <__main__.Pizza object at 0x7f3138827910> >>> # You could guess, look at this: ... >>> m == m.__self__.get_size True Obviously, we still have a reference to our object, and we can find it back if we want. In Python 3, the functions attached to a class are not considered an unbound method anymore, but as simple functions that are bound to an object if required. So the principle stays the same, the model is just simplified. >>> class Pizza(object): ... def __init__(self, size): ... self.size = size ... def get_size(self): ... return self.size ... >>> Pizza.get_size Static Methods Static methods are a special case of methods. Sometimes, you'll write code that belongs to a class, but that doesn't use the object itself at all. For example: class Pizza(object): @staticmethod def mix_ingredients(x, y): return x + y def cook(self): return self.mix_ingredients(self.cheese, self.vegetables) In such a case, writing mix_ingredients as a non-static method would work too, but it would provide a self argument that would not be used. Here, the decorator @staticmethod buys us several things: Python doesn't have to instantiate a bound method for each Pizza object we instiantiate. Bound methods are objects too, and creating them has a cost. Having a static method avoids that: >>> Pizza().cook is Pizza().cook False >>> Pizza().mix_ingredients is Pizza.mix_ingredients True >>> Pizza().mix_ingredients is Pizza().mix_ingredients True It eases the readability of the code: seeing @staticmethod, we know that the method does not depend on the state of object itself. It allows us to override the mix_ingredients method in a subclass. If we used a function mix_ingredients defined at the top-level of our module, a class inheriting from Pizza wouln't be able to change the way we mix ingredients for our pizza without overriding cook itself. Class Methods Having said that, what are class methods? Class methods are methods that are not bound to an object, but to … a class! >>> class Pizza(object): ... radius = 42 ... @classmethod ... def get_radius(cls): ... return cls.radius ... >>> >>> Pizza.get_radius > >>> Pizza().get_radius > >>> Pizza.get_radius is Pizza().get_radius True >>> Pizza.get_radius() 42 No matter which way you use to access this method, it will always be bound to the class it is attached too, and its first argument will be the class itself (remember that classes are objects, too). When to use this kind of method? Well, class methods are mostly useful for two types of methods: Factory methods that are used to create an instance for a class using, for example, some sort of pre-processing. If we use a @staticmethod instead, we would have to hard-code the Pizza class name in our function, making any class inheriting from Pizza unable to use our factory for its own use. class Pizza(object): def __init__(self, ingredients): self.ingredients = ingredients @classmethod def from_fridge(cls, fridge): return cls(fridge.get_cheese() + fridge.get_vegetables()) Static methods calling static methods: if you split a static method in to several static methods, you shouldn't hard-code the class name but you should use class methods. Using this way to declare your method, the Pizza name is never directly referenced, and inheritance and method overriding will work flawlessly class Pizza(object): def __init__(self, radius, height): self.radius = radius self.height = height @staticmethod def compute_circumference(radius): return math.pi * (radius ** 2) @classmethod def compute_volume(cls, height, radius): return height * cls.compute_circumference(radius) def get_volume(self): return self.compute_volume(self.height, self.radius) Abstract Methods An abstract method is a method defined in a base class, but that may not provide any implementation. In Java, it would describe the methods of an interface. So, the simplest way to write an abstract method in Python is: class Pizza(object): def get_radius(self): raise NotImplementedError Any class inheriting from Pizza should implement and override the get_radius method, otherwise an exception would be raised. This particular way of implementing an abstract method has a drawback. If you write a class that inherits from Pizza and forget to implement get_radius, the error will only be raised when you try to use that method. >>> Pizza() <__main__.Pizza object at 0x7fb747353d90> >>> Pizza().get_radius() Traceback (most recent call last): File "", line 1, in File "", line 3, in get_radius NotImplementedError There's a way to trigger this way earlier, when the object is being instantiated, using the ABC module that's provided with Python. import abc class BasePizza(object): __metaclass__ = abc.ABCMeta @abc.abstractmethod def get_radius(self): """Method that should do something.""" Using abc and its special class, as soon as you try to instantiate BasePizza or any class inheriting from it, you'll get a TypeError. >>> BasePizza() Traceback (most recent call last): File "", line 1, in TypeError: Can't instantiate abstract class BasePizza with abstract methods get_radius Mixing Static, Class and Abstract Methods When building classes and inheritances, the time will come when you will have to mix all of these method decorators. So, here's some tips about such a situation. Keep in mind that declaring a class as abstract doesn't freeze the prototype of that method. That means that it must be implemented, but it can be implemented with any argument list. import abc class BasePizza(object): __metaclass__ = abc.ABCMeta @abc.abstractmethod def get_ingredients(self): """Returns the ingredient list.""" class Calzone(BasePizza): def get_ingredients(self, with_egg=False): egg = Egg() if with_egg else None return self.ingredients + egg This is valid, since Calzone fulfills the interface requirement we defined for BasePizza objects. That means that we could also implement it as being a class or a static method, for example: import abc class BasePizza(object): __metaclass__ = abc.ABCMeta @abc.abstractmethod def get_ingredients(self): """Returns the ingredient list.""" class DietPizza(BasePizza): @staticmethod def get_ingredients(): return None This is also correct and fulfills the contract we have with our abstract BasePizza class. The fact that the get_ingredients method doesn't need to know about the object to return a result is an implementation detail, not a criteria to have our contract fulfilled. Therefore, you can't force an implementation of your abstract method to be a regular, class or static method, and arguably you shouldn't. Starting with Python 3 (this won't work as you would expect in Python 2, see issue 5867), it's now possible to use the @staticmethod and @classmethod decorators on top of @abstractmethod: import abc class BasePizza(object): __metaclass__ = abc.ABCMeta ingredient = ['cheese'] @classmethod @abc.abstractmethod def get_ingredients(cls): """Returns the ingredient list.""" return cls.ingredients Don't misread this: If you think this is going to force your subclasses to implement get_ingredients as a class method, you are wrong. This simply implies that your implementation of get_ingredients in the BasePizza class is a class method. An implementation in an abstract method? Yes! In Python, contrary to methods in Java interfaces, you can have code in your abstract methods and call it via super(): import abc class BasePizza(object): __metaclass__ = abc.ABCMeta default_ingredients = ['cheese'] @classmethod @abc.abstractmethod def get_ingredients(cls): """Returns the ingredient list.""" return cls.default_ingredients class DietPizza(BasePizza): def get_ingredients(self): return ['egg'] + super(DietPizza, self).get_ingredients() In such a case, every pizza you will build by inheriting from BasePizza will have to override the get_ingredients method, but will be able to use the default mechanism to get the ingredient list by using super().

There are many good benefits in using slf4j library as your Java application logging API layer. Here I will show few examples on how to use and configure it.

introduction openstack is one of the most popular open source cloud computing projects to provide infrastructure as a service solution. its key components are compute (nova), networking (neutron, formerly known as quantum), storage (object and block storage, swift and cinder, respectively), openstack dashboard (horizon), identity service (keystone) and image service (glance). there are other official incubated projects like metering (celiometer) and orchestration and service definition (heat). savanna is a hadoop as a service for openstack introduced by mirantis . it is still in an early phase (version .02 was released in summer 2013) and according to its roadmap version 1.0 is targeted for official openstack incubation. in principle, heat also could be used for hadoop cluster provisioning but savanna is especially tuned for providing hadoop-specific api functionality while heat is meant to be used for generic purposes. savanna architecture savanna is integrated with the core openstack components such as keystone, nova, glance, swift and horizon. it has a rest api that supports the hadoop cluster provisioning steps. savanna api is implemented as a wsgi server that, by default, listens to port 8386. in addition, savanna can also be integrated with horizon, the openstack dashboard to create a hadoop cluster from the management console. savanna also comes with a vanilla plugin that deploys a hadoop cluster image. the standard out-of-the-box vanilla plugin supports hadoop 1.1.2 version. installing savanna the simplest option to try out savanna is to use devstack in a virtual machine. i was using an ubuntu 12.04 virtual instance in my tests. in that environment we need to execute the following commands to install devstack and savanna api: $ sudo apt-get install git-core $ git clone https://github.com/openstack-dev/devstack.git $ vi localrc # edit localrc admin_password=nova mysql_password=nova rabbit_password=nova service_password=$admin_password service_token=nova # enable swift enabled_services+=,swift swift_hash=66a3d6b56c1f479c8b4e70ab5c2000f5 swift_replicas=1 swift_data_dir=$dest/data # force checkout prerequsites # force_prereq=1 # keystone is now configured by default to use pki as the token format which produces huge tokens. # set uuid as keystone token format which is much shorter and easier to work with. keystone_token_format=uuid # change the floating_range to whatever ips vm is working in. # in nat mode it is subnet vmware fusion provides, in bridged mode it is your local network. floating_range=192.168.55.224/27 # enable auto assignment of floating ips. by default savanna expects this setting to be enabled extra_opts=(auto_assign_floating_ip=true) # enable logging screen_logdir=$dest/logs/screen $ ./stack.sh # this will take a while to execute $ sudo apt-get install python-setuptools python-virtualenv python-dev $ virtualenv savanna-venv $ savanna-venv/bin/pip install savanna $ mkdir savanna-venv/etc $ cp savanna-venv/share/savanna/savanna.conf.sample savanna-venv/etc/savanna.conf # to start savanna api: $ savanna-venv/bin/python savanna-venv/bin/savanna-api --config-file savanna-venv/etc/savanna.conf to install savanna ui integrated with horizon, we need to run the following commands: $ sudo pip install savanna-dashboard $ cd /opt/stack/horizon/openstack-dashboard $ vi settings.py horizon_config = { 'dashboards': ('nova', 'syspanel', 'settings', 'savanna'), installed_apps = ( 'savannadashboard', .... $ cd /opt/stack/horizon/openstack-dashboard/local $ vi local_settings.py savanna_url = 'http://localhost:8386/v1.0' $ sudo service apache2 restart provisioning a hadoop cluster as a first step, we need to configure keystone-related environment variables to get the authentication token: ubuntu@ip-10-59-33-68:~$ vi .bashrc $ export os_auth_url=http://127.0.0.1:5000/v2.0/ $ export os_tenant_name=admin $ export os_username=admin $ export os_password=nova ubuntu@ip-10-59-33-68:~$ source .bashrc ubuntu@ip-10-59-33-68:~$ ubuntu@ip-10-59-33-68:~$ env | grep os os_password=nova os_auth_url=http://127.0.0.1:5000/v2.0/ os_username=admin os_tenant_name=admin ubuntu@ip-10-59-33-68:~$ keystone token-get +-----------+----------------------------------+ | property | value | +-----------+----------------------------------+ | expires | 2013-08-09t20:31:12z | | id | bdb582c836e3474f979c5aa8f844c000 | | tenant_id | 2f46e214984f4990b9c39d9c6222f572 | | user_id | 077311b0a8304c8e86dc0dc168a67091 | +-----------+----------------------------------+ $ export auth_token="bdb582c836e3474f979c5aa8f844c000" $ export tenant_id="2f46e214984f4990b9c39d9c6222f572" then we need to create the glance image that we want to use for our hadoop cluster. in our example we have used mirantis's vanilla image but we can also build our own image: $ wget http://savanna-files.mirantis.com/savanna-0.2-vanilla-1.1.2-ubuntu-12.10.qcow2 $ glance image-create --name=savanna-0.2-vanilla-hadoop-ubuntu.qcow2 --disk-format=qcow2 --container-format=bare < ./savanna-0.2-vanilla-1.1.2-ubuntu-12.10.qcow2 ubuntu@ip-10-59-33-68:~/devstack$ glance image-list +--------------------------------------+-----------------------------------------+-------------+------------------+-----------+--------+ | id | name | disk format | container format | size | status | +--------------------------------------+-----------------------------------------+-------------+------------------+-----------+--------+ | d0d64f5c-9c15-4e7b-ad4c-13859eafa7b8 | cirros-0.3.1-x86_64-uec | ami | ami | 25165824 | active | | fee679ee-e0c0-447e-8ebd-028050b54af9 | cirros-0.3.1-x86_64-uec-kernel | aki | aki | 4955792 | active | | 1e52089b-930a-4dfc-b707-89b568d92e7e | cirros-0.3.1-x86_64-uec-ramdisk | ari | ari | 3714968 | active | | d28051e2-9ddd-45f0-9edc-8923db46fdf9 | savanna-0.2-vanilla-hadoop-ubuntu.qcow2 | qcow2 | bare | 551699456 | active | +--------------------------------------+-----------------------------------------+-------------+------------------+-----------+--------+ $ export image_id=d28051e2-9ddd-45f0-9edc-8923db46fdf9 then we have installed httpie , an open source http client that can be used to send rest requests to savanna api: $ sudo pip install httpie from now on we will use httpie to send savanna commands. we need to register the image with savanna: $ export savanna_url="http://localhost:8386/v1.0/$tenant_id" $ http post $savanna_url/images/$image_id x-auth-token:$auth_token username=ubuntu http/1.1 202 accepted content-length: 411 content-type: application/json date: thu, 08 aug 2013 21:28:07 gmt { "image": { "os-ext-img-size:size": 551699456, "created": "2013-08-08t21:05:55z", "description": "none", "id": "d28051e2-9ddd-45f0-9edc-8923db46fdf9", "metadata": { "_savanna_description": "none", "_savanna_username": "ubuntu" }, "mindisk": 0, "minram": 0, "name": "savanna-0.2-vanilla-hadoop-ubuntu.qcow2", "progress": 100, "status": "active", "tags": [], "updated": "2013-08-08t21:28:07z", "username": "ubuntu" } } $ http $savanna_url/images/$image_id/tag x-auth-token:$auth_token tags:='["vanilla", "1.1.2", "ubuntu"]' http/1.1 202 accepted content-length: 532 content-type: application/json date: thu, 08 aug 2013 21:29:25 gmt { "image": { "os-ext-img-size:size": 551699456, "created": "2013-08-08t21:05:55z", "description": "none", "id": "d28051e2-9ddd-45f0-9edc-8923db46fdf9", "metadata": { "_savanna_description": "none", "_savanna_tag_1.1.2": "true", "_savanna_tag_ubuntu": "true", "_savanna_tag_vanilla": "true", "_savanna_username": "ubuntu" }, "mindisk": 0, "minram": 0, "name": "savanna-0.2-vanilla-hadoop-ubuntu.qcow2", "progress": 100, "status": "active", "tags": [ "vanilla", "ubuntu", "1.1.2" ], "updated": "2013-08-08t21:29:25z", "username": "ubuntu" } } then we need to create a nodegroup templates (json files) that will be sent to savanna. there is one template for the master nodes ( namenode , jobtracker ) and another template for the worker nodes such as datanode and tasktracker . the hadoop version is 1.1.2. $ vi ng_master_template_create.json { "name": "test-master-tmpl", "flavor_id": "2", "plugin_name": "vanilla", "hadoop_version": "1.1.2", "node_processes": ["jobtracker", "namenode"] } $ vi ng_worker_template_create.json { "name": "test-worker-tmpl", "flavor_id": "2", "plugin_name": "vanilla", "hadoop_version": "1.1.2", "node_processes": ["tasktracker", "datanode"] } $ http $savanna_url/node-group-templates x-auth-token:$auth_token < ng_master_template_create.json http/1.1 202 accepted content-length: 387 content-type: application/json date: thu, 08 aug 2013 21:58:00 gmt { "node_group_template": { "created": "2013-08-08t21:58:00", "flavor_id": "2", "hadoop_version": "1.1.2", "id": "b3a79c88-b6fb-43d2-9a56-310218c66f7c", "name": "test-master-tmpl", "node_configs": {}, "node_processes": [ "jobtracker", "namenode" ], "plugin_name": "vanilla", "updated": "2013-08-08t21:58:00", "volume_mount_prefix": "/volumes/disk", "volumes_per_node": 0, "volumes_size": 10 } } $ http $savanna_url/node-group-templates x-auth-token:$auth_token < ng_worker_template_create.json http/1.1 202 accepted content-length: 388 content-type: application/json date: thu, 08 aug 2013 21:59:41 gmt { "node_group_template": { "created": "2013-08-08t21:59:41", "flavor_id": "2", "hadoop_version": "1.1.2", "id": "773b2cfb-1e05-46f4-923f-13edc7d6aac6", "name": "test-worker-tmpl", "node_configs": {}, "node_processes": [ "tasktracker", "datanode" ], "plugin_name": "vanilla", "updated": "2013-08-08t21:59:41", "volume_mount_prefix": "/volumes/disk", "volumes_per_node": 0, "volumes_size": 10 } } the next step is to define the cluster template: $ vi cluster_template_create.json { "name": "demo-cluster-template", "plugin_name": "vanilla", "hadoop_version": "1.1.2", "node_groups": [ { "name": "master", "node_group_template_id": "b3a79c88-b6fb-43d2-9a56-310218c66f7c", "count": 1 }, { "name": "workers", "node_group_template_id": "773b2cfb-1e05-46f4-923f-13edc7d6aac6", "count": 2 } ] } $ http $savanna_url/cluster-templates x-auth-token:$auth_token < cluster_template_create.json http/1.1 202 accepted content-length: 815 content-type: application/json date: fri, 09 aug 2013 07:04:24 gmt { "cluster_template": { "anti_affinity": [], "cluster_configs": {}, "created": "2013-08-09t07:04:24", "hadoop_version": "1.1.2", "id": "{ "name": "cluster-1", "plugin_name": "vanilla", "hadoop_version": "1.1.2", "cluster_template_id" : "64c4117b-acee-4da7-937b-cb964f0471a9", "user_keypair_id": "stack", "default_image_id": "3f9fc974-b484-4756-82a4-bff9e116919b" }", "name": "demo-cluster-template", "node_groups": [ { "count": 1, "flavor_id": "2", "name": "master", "node_configs": {}, "node_group_template_id": "b3a79c88-b6fb-43d2-9a56-310218c66f7c", "node_processes": [ "jobtracker", "namenode" ], "volume_mount_prefix": "/volumes/disk", "volumes_per_node": 0, "volumes_size": 10 }, { "count": 2, "flavor_id": "2", "name": "workers", "node_configs": {}, "node_group_template_id": "773b2cfb-1e05-46f4-923f-13edc7d6aac6", "node_processes": [ "tasktracker", "datanode" ], "volume_mount_prefix": "/volumes/disk", "volumes_per_node": 0, "volumes_size": 10 } ], "plugin_name": "vanilla", "updated": "2013-08-09t07:04:24" } } now we are ready to create the hadoop cluster: $ vi cluster_create.json { "name": "cluster-1", "plugin_name": "vanilla", "hadoop_version": "1.1.2", "cluster_template_id" : "64c4117b-acee-4da7-937b-cb964f0471a9", "user_keypair_id": "savanna", "default_image_id": "d28051e2-9ddd-45f0-9edc-8923db46fdf9" } $ http $savanna_url/clusters x-auth-token:$auth_token < cluster_create.json http/1.1 202 accepted content-length: 1153 content-type: application/json date: fri, 09 aug 2013 07:28:14 gmt { "cluster": { "anti_affinity": [], "cluster_configs": {}, "cluster_template_id": "64c4117b-acee-4da7-937b-cb964f0471a9", "created": "2013-08-09t07:28:14", "default_image_id": "d28051e2-9ddd-45f0-9edc-8923db46fdf9", "hadoop_version": "1.1.2", "id": "d919f1db-522f-45ab-aadd-c078ba3bb4e3", "info": {}, "name": "cluster-1", "node_groups": [ { "count": 1, "created": "2013-08-09t07:28:14", "flavor_id": "2", "instances": [], "name": "master", "node_configs": {}, "node_group_template_id": "b3a79c88-b6fb-43d2-9a56-310218c66f7c", "node_processes": [ "jobtracker", "namenode" ], "updated": "2013-08-09t07:28:14", "volume_mount_prefix": "/volumes/disk", "volumes_per_node": 0, "volumes_size": 10 }, { "count": 2, "created": "2013-08-09t07:28:14", "flavor_id": "2", "instances": [], "name": "workers", "node_configs": {}, "node_group_template_id": "773b2cfb-1e05-46f4-923f-13edc7d6aac6", "node_processes": [ "tasktracker", "datanode" ], "updated": "2013-08-09t07:28:14", "volume_mount_prefix": "/volumes/disk", "volumes_per_node": 0, "volumes_size": 10 } ], "plugin_name": "vanilla", "status": "validating", "updated": "2013-08-09t07:28:14", "user_keypair_id": "savanna" } } after a while we can run the nova command to check if the instances are created and running: $ nova list +--------------------------------------+-----------------------+--------+------------+-------------+----------------------------------+ | id | name | status | task state | power state | networks | +--------------------------------------+-----------------------+--------+------------+-------------+----------------------------------+ | 1a9f43bf-cddb-4556-877b-cc993730da88 | cluster-1-master-001 | active | none | running | private=10.0.0.2, 192.168.55.227 | | bb55f881-1f96-4669-a94a-58cbf4d88f39 | cluster-1-workers-001 | active | none | running | private=10.0.0.3, 192.168.55.226 | | 012a24e2-fa33-49f3-b051-9ee2690864df | cluster-1-workers-002 | active | none | running | private=10.0.0.4, 192.168.55.225 | +--------------------------------------+-----------------------+--------+------------+-------------+----------------------------------+ now we can log in to the hadoop master instance and run the required hadoop commands: $ ssh -i savanna.pem [email protected] $ sudo chmod 777 /usr/share/hadoop $ sudo su hadoop $ cd /usr/share/hadoop $ hadoop jar hadoop-example-1.1.2.jar pi 10 100 savanna ui via horizon in order to create nodegroup templates, cluster templates and the cluster itself we used a command line tool -- httpie -- to send rest api calls. the same functionality is also available via horizon, the standard openstack dashboard. first we need to register the image with savanna: then we need to create the nodegroup templates: after that we have to create the cluster template: and finally we have to create the cluster:

In a previous article about how you can remove whitesapce from a string, I spoke about using the functions ltrim() and rtrim(). These work by passing in a string to remove whitespace. Using the ltrim() function will remove the whitespace from the start of the string, using the rtrim() function will remove the whitespace from the end of the string. But you can also use these functions to remove characters from a string. These functions take a second parameter that allows you to specify what characters to remove. // This will search for the word start at the beginning of the string and remove it ltrim($string, 'start'); // This will search for the word end at the end of the string and remove it rtrim($string, 'end'); Remove Trailing Slashes From a String A common use for this functionality is to remove the trailing slash from a URL. Below is a code snippet that allows you to easily do this using the rtrim() function. function remove_trailing_slashes( $url ) { return rtrim($url, '/'); } A common use for the ltrim() function is to remove the "http://" from a URL. Use the function below to remove both "http" and "https" from a URL: function remove_http( $url ) { $url = ltrim($url, 'http://'); $url = ltrim($url, 'https://'); return $url; }

Here is how you can convert asciidoc text using a Groovy script: // filename: RunAsciidoc.groovy @Grab('org.asciidoctor:asciidoctor-java-integration:0.1.3') import org.asciidoctor.* def asciidoctor = Asciidoctor.Factory.create() def output = asciidoctor.renderFile(new File(args[0]), [:]) println(output); Now you may run it groovy RunAsciidoc.groovy myarticle.txt Many thanks to the asciidoctor.org project!

You might be asking yourself, 'why do I need a MongoDB-like experience for DynamoDB when there are already full-MongoDB cloud services like MMS, MongoLab, MongoHQ and MongoDirector? One developer believes there is a need and has set up an experimental project called DyngoDB. It provides the MongoDB-style interface in front of Amazon's DynamoDB and their CloudSearch service. Apparently, in the developer's case, he only wants the MongoDB interface but prefers the DynamoDB storage engine. We'll have to see if other developers also have this specific set of preferences.

java.util.Optional in Java 8 is a poor cousin of scala.Option[T] and Data.Maybe in Haskell. But this doesn’t mean it’s not useful.

In the upcoming release of Apache Camel 2.12 we have introduced an SPI that allows users to plugin different schedulers for schedule-based consumers. The motivation for this feature came from the fact that some Camel components have scheduled consumers. Usually file and FTP consumers. By default, they use the scheduler from the JVM that can schedule based on a fixed period. Now, with the SPI, we allow a different scheduler to be used instead. We have two cron-based schedulers ready out-of-box in the camel-quartz2 and camel-spring components. So pick your favorite, or dive in and build your own scheduler. CRON expression If you want to pickup files during working hours (polling every 10th second) on weekdays, you can easily do this now (notice we use + as a space separator). Example with Spring: ... Example with Quartz: ... To implement a similar solution in older releases of Camel, you would need to use sort in order to use a route policy. There is a cron-based route policy that can be used to setup cron expressions when a route should be started and when it should be stopped. With this new functionality in Camel 2.12, it's even easier to just define the cron expression in the endpoint URI directly. There are more details in the Camel docs: - Polling Consumer - Quarz2 Component Camel-quartz2 is also a new component in the upcoming Apache Camel 2.12 release. In fact, we already have 14 new components. You can take a peek at the work-in-progress release notes to see what is coming down the road.