DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

The Latest Testing, Deployment, and Maintenance Topics

article thumbnail
JBoss 5 to 7 in 11 steps
Introduction Some time ago we decided to upgrade our application from JBoss 5 to 7 (technically 7.2). In this article I going to describe several things which we found problematic. At the end I also provided a short list of benefits we gained in retrospect. First some general information about our application. It was built using EJB 3.0 technology. We have 2 interfaces for communicating with other components – JMS and JAX-WS. We use JBoss AS 5 as our messaging broker which is started as a separate JVM process. This part of the system we were not allowed to change. Finally – we use JPA to store processing results to Oracle DB. Step #1 – Convince your Product Owner Although our application was rather small and built on JEE5 standard it took us 4 weeks to migrate it to JEE6 and JBoss 7. So you can't do it as a maintenance ticket – it's simply too big. There is always problem with providing Business Value of such migration for Product Owners as well as for key Stakeholders. There are several aspects which might help you convincing them. One of the biggest benefits is processing time. JBoss 7 is simply faster and has better caching (Infinispan over Ehcache). Another one is startup time (our server is ready to go in 5-6 seconds opposed to 1 minute in JBoss 5). Finally – development is much faster (EJB 3.1 is much better then 3.0). The last one might be translated to “time to market”. Having above arguments I'm pretty sure you'll convince them. Step #2 – Do some reading Here is a list on interesting links which are worth reading before the migration: JBoss 5 -> 7 migration guide: https://docs.jboss.org/author/display/AS7/How+do+I+migrate+my+application+from+AS5+or+AS6+to+AS7 JBoss 7 vs EAP libraries: https://access.redhat.com/site/articles/112673 JBoss EAP Faq: http://www.jboss.org/jbossas/faq Cache implementation benchmarks: http://sourceforge.net/p/nitrocache/blog/2012/05/performance-benchmark-nitrocache--ehcache--infinispan--jcs--cach4j/ JBoss 7 performence tuning: http://www.mastertheboss.com/jboss-performance/jboss-as-7-performance-tuning JBoss caching: http://www.mastertheboss.com/hibernate-howto/using-hibernate-second-level-cache-with-jboss-as-5-6-7 Step #3 – Off you go – change Maven dependencies JBoss 5 isn't packaged very well, so I suppose you many dependencies included in your classpath (either directly or by transitive dependencies). This is the first big change in JBoss 7. Now I strongly advice you to use this artifact in your dependency management section: org.jboss.as jboss-as-parent 7.2.0.Final pom import We also decided to stick only to JEE6 spec and configure all additional JBoss 7 options with proper XML files. If it sounds good for your project too, just add this dependency and you're done with this step: org.jboss.spec jboss-javaee-6.0 1.0.0.Final pom provided After cleaning up dependencies your code probably won't compile for a couple of days or even weeks. It takes time to clean this up. Step #4 – EJB 3.0 to 3.1 migration Dependency Injection is a heart of the application, so it is worth to start with it. Almost all of your code should work, but you'll have some problems with beans annotated with @Service (these are singletons with JBoss 5 EJB Extended API). You just need to replace them with @Singleton annotations and put @PostConstruct annotation on your init method. One last thing – remember to use proper concurrency strategy. We decided to use @ConcurrencyManagement(BEAN) and leave the implementation as is. Step #5 – Upgrade to JPA 2.0 If you used JPA 1.0 with Hibernate, I'm pretty sure you have a lot of non standard annotations defining caching or cascading. All of them might be successfully replaced with JPA 2.0 annotations and finally you might get rid of Hibernate from compile classpath and depend only on JPA 2.0. Here are several standard things to do: Get rid of Hibernate's Session.evict and switch to EntityManager.detach Get rid of Hibernate's @Cache annotation and replace it with @Cachable Fix Cascades (now delete orphan is a part of @XXXToYYY annotations) Remove Hibernate dependency and stick with JEE6 spec Step #6 – Fix Hibernate's sequencer Migrating Hibernate 3 to 4 is a bit tricky because of the way it uses sequences (fields annotated with @Id). Hibernate by default uses a pool of ids instead of incrementing sequence. An example will be more descriptive: Some_DB_Sequence.nextval -> 1 Hibernate 3: 1*50 = 50; IDs to be used = 50, 51, 52.…, 99 Some_DB_Sequence.nextval -> 2 Hibernate 3: 2*50 = 100; IDs to be used = 100, 101, 102.…, 149 In Hibernate 4.x there is a new sequence generator that uses new IDs that are 1:1 related to DB sequence. Typically it's disabled by default... but not in JBoss 7.1. So after migration, Hibernate tries to insert entities using IDs read from sequence (using new sequence generator) that were already used which causes constraint violation. The fastest solution is to switch Hibernate to the old method of sequence generation (described in example above), that requires following change in persistence.xml: Step #7 – Caching Infinispan is shipped with JBoss 7 and does not require much configuration. There is only one setting in persistence.xml which needs to be set and the others might be removed: Infinispan itself might require some extra configuration – just use standalone-full-ha.xml as guide. Step #8 – RMI with JBoss 5 If you're using a lot of RMI communicating with other JBoss 5 servers – I have bad information for you – JBoss 5 and 7 are totally different and this kind of comminication will not work. I strongly recommend to switch to some other technology like JAX-WS. In the retrospect we are very glad we decided to do it. Step #9 – JMS migration We thought it would be really hard to connect with JMS server based on JBoss 5. It turned out that you have 2 options and both work fine: Start HornetQ server on your own instance and create a bridge to JBoss 5 instance Use Generic JMS adapter: https://github.com/jms-ra/generic-jms-ra Step #10 – Fix EAR layout In JBoss 5 it does not matter where all jars are being placed. All EJBs are being started. It does not work with JBoss 7 anymore. All EJB which should start must be added as modules. Step #11 – JMX console Bad information – it's not present in JBoss 7. We liked it very much, but we had to switch to jvisualvm to invoke our JMX operations. There is a ticket in WildFly Jira opened for that: https://issues.jboss.org/browse/WFLY-1197. Unfortunately at moment of writing this article it is not resolved. Some thoughts in retrospect It is really time consuming task to migrate from JBoss 5 to 7. Although in my opinion it is worth it. Now we have better caching for cluster solutions (Infinispan), better DI (EJB 3.1) and better Web Services (CXF instead of JBoss WS). Processing time decreased by 25% without any code change. Development speed increased in my opinion (it is really hard to measure it) by 50% and we are much more productive (faster server restarts). Memory footprint lowered from 1GB to 512MB. Finally automatic application redeployment finally works! However there is always a price to pay – the migration took us 4 weeks (2 sprints). We didn't write any code for our business in that period. So make sure you prepare well for such migration and my last advice – invest some time to write good automatic functional tests (we use Arquillian for that). Once they're green again – you're almost crossing finishing line.
January 9, 2014
by Sebastian Laskawiec
· 47,067 Views
article thumbnail
Hunting for an SWT Test Framework? Say Hello to Red Deer
This is the first in a series of posts on the new “Red Deer” (https://github.com/jboss-reddeer/reddeer) open source testing framework for Eclipse. In this post, we’ll introduce Red Deer, and take a look at the some of the advantages that it offers by building a sample test program from scratch. Some of the features that Red Deer automated offers are: An easy to use, high-level API for testing standard Eclipse components Support for creating custom extensions for your own applications A requirements validation mechanism to assist you in configuring complex tests Eclipse Tooling to Assist in Creating new Projects A record and playback tool to enable you to quickly create automated tests An integration with Selenium for testing web based applications Support for running tests in a Jenkins CI environment Note that as of this writing, Red Deer is in an incubation stage. The current release is at level 0.5. The target date for the 1.0 release of Red Deer is late 2014. But, as a community-based, open source project, now is a great time to try Red Deer and make suggestions or even contribute code! A Look at Red Deer’s Architecture The Red Deer project itself is comprised of utilities and the API that supports the development and execution of automated tests. The API (the parts of the above diagram that are enclosed in dashed line boxes) can be thought of as having three layers: The top layer consists of extensions to Red Deer’s abstract classes or implementations for Eclipse components such as Views, Editors, Wizards, or Shells. For example, if you are writing tests for a feature that uses a custom Eclipse View, you can extend Red Deer’s View class by adding support for the specific functions of the feature. The advantage that this API layer gives you is that your test programs do not have to focus on manipulating the individual UI elements directly to perform operations. Your programs can instead instantiate an instance of an Eclipse component such as a View, and then use that instance’s methods to perform operations on the View. This layer of abstraction makes your test programs easier to write, understand, and maintain. The middle layer consists of the Red Deer implementations for SWT UI elements such as: Button, Combo, Label, Menu, Shell, TabItem, Table, ToolBar, Tree. This API layer supports the API’s higher level by providing the building blocks for the API’s Views, Editors, Shells, and WIzards. This middle layer of the API also provides Red Deer packages that enable your tests to enforce requirements, so that necessary setup tasks are performed before a test is run. The bottom layer consists of Red Deer packages that support the execution of tests such as: Conditions, Matchers, Widgets, Workbench, and Red Deer extensions to JUnit. What Makes Red Deer different from other Tools? A Layer of Abstraction The top-most layer of the API enables you to instantiate Eclipse UI elements as objects, and then manipulate them through their methods. The resulting code is easier to read and maintain, instead of being brittle and subject to failures when the UI changes. For example, for a test that has to open a view and press a button, without Red Deer, the test would have to navigate the top level menu, find the view menu, then the view type in that menu, then find the view open dialog, then locate the “OK” button, etc. Your test would have to spend a lot of time navigating through the UI elements before it could even begin to perform the test’s steps. With Red Deer, the code to open a view (in this case, the servers view) is simply: ServersView view = new ServersView(); view.open(); Furthermore, within that ServersView, your test program can perform operations on the View through methods which are defined in the view (and are incidentally also well debugged by the Red Deer team), instead of having to explicitly locate and manipulate the UI elements directly. For example, to obtain a list of all the servers, instead of locating the UI tree that contains the server list, and extracting that list of servers into an array, your Red Deer program can simply call the “getServers()” method. Likewise, the code to open a PackageExplorer, and then select a project within that PackageExplorer is as follows: PackageExplorer packageExplorer = new PackageExplorer(); packageExplorer.open(); packageExplorer.getProject("myTestProject").select(); And, the code to retrieve all the projects within that PackageExplorer is simply: packageExplorer.getProjects(); The result are that your tests are easier to write and maintain and you can focus on testing your application’s logic instead of writing brittle code to navigate through the application. Installing Red Deer The only prerequisites to using Red Deer are Eclipse and Java. In this post, we’ll use Eclipse Kepler and OpenJDK 1.7, running on Red Hat Enterprise Linux (RHEL) 6. To install Red Deer 0.4 (this is the latest stable milestone version as of this writing) follow these steps: Open up Eclipse Navigate to: Help->Install New Software Define a new download site using the Red Deer update site URL: http://download.jboss.org/jbosstools/updates/stable/kepler/core/reddeer/0.4.0/ Select Red Deer, click on the Finish button and Red Deer will install Now that you have Red Deer installed, let’s move onto building a new Red Deer test. Building your First Red Deer Test To create a new Red Deer test project, you make use of the Red Deer UI tooling and select New->Project->Other->Red Deer Test: Before we move on, let’s take a look at the WEB-INF/MANIFEST.MF file that is created in the project: Manifest-Version: 1.0 Bundle-ManifestVersion: 2 Bundle-Name: com.example.reddeer.sample Bundle-SymbolicName: com.example.reddeer.sample;singleton:=true Bundle-Version: 1.0.0.qualifier Bundle-ActivationPolicy: lazy Bundle-Vendor: Sample Co Bundle-RequiredExecutionEnvironment: JavaSE-1.6 Require-Bundle: org.junit, org.jboss.reddeer.junit, org.jboss.reddeer.swt, org.jboss.reddeer.eclipse The line we’re interested in is the final line in the file. These are the bundles that are required by Red Deer. After the empty project is created by the wizard, you can define a package and create a test class. Here's the code for a minimal functional test. The test will verify that the eclipse configuration is not empty. package com.example.reddeer.sample; import static org.junit.Assert.assertFalse; import java.util.List; import org.jboss.reddeer.swt.api.TreeItem; import org.jboss.reddeer.swt.impl.button.PushButton; import org.jboss.reddeer.swt.impl.menu.ShellMenu; import org.jboss.reddeer.swt.impl.tree.DefaultTree; import org.junit.Test; import org.junit.runner.RunWith; import org.jboss.reddeer.junit.runner.RedDeerSuite; @RunWith(RedDeerSuite.class) public class SimpleTest { @Test public void TestIt() { new ShellMenu("Help", "About Eclipse Platform").select(); new PushButton("Installation Details").click(); DefaultTree ConfigTree = new DefaultTree(); List ConfigItems = ConfigTree.getAllItems(); assertFalse ("The list is empty!", ConfigItems.isEmpty()); for (TreeItem item : ConfigItems) { System.out.println ("Found: " + item.getText()); } } } After you save the test's source file, you can run the test. To run the test, select the Run As->Red Deer Test option: And - there's the green bar! Simplifying Tests with Requirements Red Deer requirements enable you to define actions that you want happen before a test is executed. The advantage to using requirements is that you define the actions with annotations instead of using a @BeforeClass method. The result is that your test code is easier to read and maintain. The biggest difference between a Red Deer requirement and the the @BeforeClass annotation from the JUnit framework is that if a requirement cannot be fulfilled the test is not executed. Like everything else in Red Deer, you can make use of predefined requirements, or you can extend the feature by adding your own custom requirements. These custom requirements can be made complex and for convenience can be stored in external properties files. (We’ll take a look at defining custom requirements in a later post in this series when we examine how to create and contribute extensions to Red Deer.) The current milestone release of Red Deer provides predefined requirements that enable you to clean out your current workspace and open a perspective. Let’s add these to our example. To do this, we need to add these import statements: import org.jboss.reddeer.eclipse.ui.perspectives.JavaBrowsingPerspective; import org.jboss.reddeer.requirements.cleanworkspace.CleanWorkspaceRequirement.CleanWorkspace; import org.jboss.reddeer.requirements.openperspective.OpenPerspectiveRequirement.OpenPerspective; And these annotations: @CleanWorkspace @OpenPerspective(JavaBrowsingPerspective.class) And, we also have to a reference to org.jboss.reddeer.requirements to the required bundle list in our example’s MANIFEST.MF file: Require-Bundle: org.junit, org.jboss.reddeer.junit, org.jboss.reddeer.swt, org.jboss.reddeer.eclipse, org.jboss.reddeer.requirements When we’re done, our example looks like this: package com.example.reddeer.sample; import static org.junit.Assert.assertFalse; import java.util.List; import org.jboss.reddeer.swt.api.TreeItem; import org.jboss.reddeer.swt.impl.button.PushButton; import org.jboss.reddeer.swt.impl.menu.ShellMenu; import org.jboss.reddeer.swt.impl.tree.DefaultTree; import org.junit.Test; import org.junit.runner.RunWith; import org.jboss.reddeer.junit.runner.RedDeerSuite; import org.jboss.reddeer.eclipse.ui.perspectives.JavaBrowsingPerspective; import org.jboss.reddeer.requirements.cleanworkspace.CleanWorkspaceRequirement.CleanWorkspace; import org.jboss.reddeer.requirements.openperspective.OpenPerspectiveRequirement.OpenPerspective; @RunWith(RedDeerSuite.class) @CleanWorkspace @OpenPerspective(JavaBrowsingPerspective.class) public class SimpleTest { @Test public void TestIt() { new ShellMenu("Help", "About Eclipse Platform").select(); new PushButton("Installation Details").click(); DefaultTree ConfigTree = new DefaultTree(); List ConfigItems = ConfigTree.getAllItems(); assertFalse ("The list is empty!", ConfigItems.isEmpty()); for (TreeItem item : ConfigItems) { System.out.println ("Found: " + item.getText()); } } } Notice how we were able to add those functions to the test code, while only adding a very small amount of actual new code? Yes, it can pay to be a lazy programmer. ;-) What’s Next? What’s next for Red Deer is its continued development as it progresses through its incubation stage until its 1.0 release. What’s next for this series of posts will be discussions about: The Red Deer Recorder - To enable you to capture manual actions and convert them into test programs How you can Extend Red Deer - To provide test coverage for your plugins’ specific functions. And How you can Contribute these extensions to the Red Deer project. How you can Define Complex Requirements - To enable you to perform setup tasks for your tests. Red Deer’s Integration with Selenium - To enable you to test web interfaces provided by your plugins. Running Red Deer tests with Jenkins - To enable you to take advantage of Jenkins’ Continuous Integration (CI) test framework. Author’s Acknowledgements I’d like to thank all the contributors to Red Deer for their vision and contributions. It’s a new project, but it is growing fast! The contributors (in alphabetic order) are: Stefan Bunciak, Radim Hopp, Jaroslav Jankovic, Lucia Jelinkova, Marian Labuda, Martin Malina, Jan Niederman, Vlado Pakan, Jiri Peterka, Andrej Podhradsky, Milos Prchlik, Radoslav Rabara, Petr Suchy, and Rastislav Wagner.
January 7, 2014
by Len DiMaggio
· 7,826 Views
article thumbnail
Introduction to Codenvy
what is codenvy exactly? well, their website states: codenvy is a cloud environment for coding, building, and debugging apps. basically, it’s an ide in the cloud (“ide as a service?”) accessible by all the major browsers . it started out as an additional feature to the exo platform in early 2009 and gained a lot of traction after the first paas (openshift) and git integration was added mid-2011. codenvy targets me as a (java) software developer to run and debug applications in their hosted cloud ide, while being able to share and collaborate during development and finally publish to a repository – e.g. git – or a number of deployment platforms – e.g. amazon, openshift or google app engine. i first encountered their booth at javaone last september, but they couldn’t demo their product right there on the spot over the wifi, because their on-line demo workspace never finished loading well i got the t-shirt instead then, but now’s the time to see what codenvy has in store as a cloud ide. signing up signing up took 3 seconds. all you have to do is go to codenvy.com , use the “sign up” button, choose an email address and a name for your workspace , confirm the email they’ll send you and you’re done. the “workspace” holds all your projects and is part of the url codenvy will create for you, like “ https://codenvy.com/ide/ . although not very clear during the registration process – which of course nowadays is usually minimalistic as can be – it seems that i’ve signed up for codenvy’s free community plan , which gives me an unlimited number of public projects. you can even start coding without registration. after confirming the registration mail, i’m in. finally i’ll end up in the browser where your (empty) workspace has been opened. empty workspace a few options a possible for here on, as seen in the figure above: create a new project from scratch – generate an empty project from predefined project types import from github – import projects from your github account clone a git repository – create a new project from any public git reposiroty browse documentation invite people – get team members on board support – questions, feedback and troubleshooting let’s… create a new project from scratch this option allows you to name the new project – e.g. “myproject”, choose a technology and a paas . the technology is a defined set of languages of frameworks to develop with. available technologies at the moment the technologies are: java jar java war java spring javascript ruby on rails python php node.js android maven multi-module at the time of writing java 1.6 is supported. available paas at the moment the available platforms are: amazon webservices (aws) elastic beanstalk savvis cloud appfrog cloudbees google app engine (gae) heroku manymo android emulator red hat’s openshift none depending on the choice of technology, or or more paas options become available. a single jar can not be deployed onto any of the platforms, leaving only the option “none” available. a java web application (war) can be deployed onto any number of platforms, except heroku and manymo. node.js can only be deployed to openshift. creating a simple jar project after having selected a jar (and no platform) one can select a project template . e.g. if webapplication (war) would have been selected, codenvy would present project templates, such as google app engine java project illustrating simple examples that use the search api , java web project with datasource usage or a demonstration of accessing amazon s3 buckets using the java sdk . the jar technology has only one project: simple jar project . after having finished the wizard, our jar project has been created in our workspace. we’ll see two views of our project: a project explorer and a package explorer. project- and package explorer what we can see is that our jar project has been given a maven pom.xml with the following content: view source print ? 01. < project xmlns = " http://maven.apache.org/pom/4.0.0 " xmlns:xsi = " http://www.w3.org/2001/xmlschema-instance " 02. xsi:schemalocation = " http://maven.apache.org/pom/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd " > 03. < modelversion >4.0.0 04. < groupid >com.codenvy.workspaceyug8g52wjwb5im13 05. < artifactid >testjarproject 06. < version >1.0-snapshot 07. < packaging >jar 08. 09. < name >sample-lib 10. 11. < properties > 12. < project.build.sourceencoding >utf-8 13. 14. 15. < dependencies > 16. < dependency > 17. < groupid >junit 18. < artifactid >junit 19. < version >3.8.1 20. < scope >test 21. 22. 23. we have a generated group id com.codenvy.workspaceyug8g52wjwb5im13 , our own artifact id and the junit dependency, which is a decent choice for many java developers use it as a testing framework. the source encoding has already been set to utf-8, which is also a sensible choice. as a convenience we’ve also been given a hello.sayhello class, so we know we’re actually in a java project say hello file & project management so what about the browser-based editor we’re working in? on top we’re seeing a few menu’s, like file, project, edit, view, run, git, paas, window, share and help . i’ll be highlighting a few. file- and project menu the file menu allows to creating folders , packages and various kind of filetypes , such as text, xml (1.0 at time of writing) , html (4.1) , css (2.0), java classes and jsp’s (2.1). although i’m in a jar project, i am still also able to create here e.g. ruby, php or python files. a very convenient feature is to upload existing files to the workspace, either separately or in zip archives. i’ve tried dropping a file onto the package explorer from the file system, but the browser (in this case, chrome) tries to open it instead the project menu allows to create new projects, either by launching the create project wizard again, but also allows for importing from github . in order to clone a repository, you’ll have to authorize codenvy to access github.com to be able to import a project. after having authorized github, codeenvy presents me with a list of projects to choose from. after having imported all necessary stuff, it somehow needs to know what kind of project i’m importing. selecting a file type after importing a project from github the project i imported didn’t give codenvy any clues as to what kind of project it is (which is right since i only had a readme.md in it), so it lists a few options to choose from. i chose the maven multi-module type after which the output window shows: [email protected]:tvinke/examples.git was successfully cloned. [info] project type updated. if you’d have a pom.xml in the root of your project, it would immediately recognize it a s a maven project. apart from going through the project > import from github option, you can also go directly to the git menu, and choose clone repository . this allows you to manually enter the remote repository uri, wanted project name and the remote name (e.g. “origin”). cloning a repository one you have pulled in a git project, the git menu allows all kinds of common operations, such as adding and removing files, committing, pushing, pulling and much more. git menu the ssh keys can be found under menu window > preferences where you can view the github.com entry, where one can view the details or delete it. also a new key can be either generated or uploaded here. sharing the project one of the unique selling points of codenvy are their collaboration possibilities which come along with any project. you can: invite other developers with read-only rights or full read-write rights to your workspace and every project in it.when you’re pair-programming like this, or co-editing a file with a colleague, you can also send each other code pointers – small shortcuts to code lines. use factories to create temporary workspaces , through cloning, off one source project (“factory”) and represent the cloning mechanism as a url which can be given to other developers. a use case might be to get a colleague quickly started on a project by providing a fully working development environment.there’s a lot more about creating factories in the docs (such as through rest), but the nice thing is that once you have a factory url, you can embed it as a button, send it through email of publish it somewhere for others! a factory url to load up e.g. their twitter bootstrap sample – as they use on their website themselves – looks like: https://codenvy.com/factory?v=1.0&pname=sample-twitterbootstrap&wname=codenvy-factories&vcs=git&vcsurl=http%3a%2f%2fcodenvy.com%2fgit%2f04%2f0f%2f7f%2fworkspacegcpv6cdxy1q34n1i%2fsample-twitterbootstrap&idcommit=c1443ecea63471f5797f172c081cd802bac6e6b0&action=openproject&ptype=javascript conclusion applications are run in the cloud nowadays, so why not create them there too? codenvy brings some interesting features, such as being able to instantly provision workspaces (through factory urls) and share projects in real-time. it supports common operations with projects, files and version control. with a slew of languages and platforms and as an ide being always accessible through the internet, it could lower the barrier to actually code anytime and anywhere. in a future post i will try and see whether or not it can actually replace my conventional desktop ide for java development.
January 4, 2014
by Ted Vinke
· 8,012 Views
article thumbnail
Unit Testing Asynchronous Web API Action Methods Using MS Test
Since Entity Framework now has a very nice support of performing all its actions asynchronously, the methods in the repositories in our projects will turn into asynchronous methods soon and so will be the code depending on it. Tom Fitzmacken did a nice job by putting together a tutorial on unit testing Web API 2 Controllers on official ASP.NET site. The tutorial discusses on testing synchronous action methods. The same techniques can be applied to test asynchronous action actions as well. In this post, we will see how easy it is to test asynchronous Web API action methods using MS Test. I created a simple repository interface with just one method in it. The implementation class uses Entity Framework to get a list of contacts from the database. public interface IRepository { Task> GetAllContactsAsync(); } public class Repository : IRepository { ContactsContext context = new ContactsContext(); public async Task> GetAllContactsAsync() { return await context.Contacts.ToArrayAsync(); } } Following is the ASP.NET Web API controller that uses the above repository: public class ContactsController : ApiController { IRepository repository; public ContactsController() : this(new Repository()) { } public ContactsController(IRepository _repository) { repository = _repository; } [Route("api/contacts/plain")] public async Task> GetContactsListAsync() { IEnumerable contacts; try { contacts = await repository.GetAllContactsAsync(); } catch (Exception) { throw; } return contacts; } [Route("api/contacts/httpresult")] public async Task GetContactsHttpActionResultAsync() { IEnumerable contacts; try { contacts = await repository.GetAllContactsAsync(); } catch (Exception ex) { return InternalServerError(ex); } return Ok(contacts); } } As we see, the controller has two action methods performing the same task, but the way they return the results is different. Since both of the action methods respond to HTTP GET method, I used attribute routing to distinguish them. I used poor man’s dependency injection to instantiate the repository; it can be easily replaced using an IoC container. Before writing unit tests for the above action methods, we need to create a mock repository. public class MockRepository:IRepository { List contacts; public bool FailGet { get; set; } public MockRepository() { contacts = new List() { new Contact(){Id=1, Title="Title1", PhoneNumber="1992637281", CustomerId=1}, new Contact(){Id=2, Title="Title2", PhoneNumber="9172735171", SupplierId=2}, new Contact(){Id=3, Title="Title3", PhoneNumber="8361910353", CustomerId=2}, new Contact(){Id=4, Title="Title4", PhoneNumber="7801274518", SupplierId=3} }; } public async Task> GetAllContactsAsync() { if (FailGet) { throw new InvalidOperationException(); } await Task.Delay(1000); return contacts; } } The property FailGet in the above class is used to force the mock to throw an exception. This is done just to cover more test cases. In the test class, we need a TestInitialize method to arrange the objects needed for unit testing. [TestClass] public class ContactsControllerTests { MockRepository repository; ContactsController contactsApi; [TestInitialize] public void InitializeForTests() { repository = new MockRepository(); contactsApi = new ContactsController(repository); } } Let us test the GetContactsListAsync method first. Testing this method seems to be straight forward, as it returns either a plain generic list or throws an exception. But the test method can’t just return void like other tests, as the method is asynchronous. To test an asynchronous method, the test method should also be made asynchronous and return a Task. Following test checks if the controller action returns a collection of length 4: [TestMethod] public async Task GetContacts_Should_Return_List_Of_Contacts() { var contacts = await contactsApi.GetContactsListAsync(); Assert.AreEqual(contacts.Count(), 4); } If the repository encounters an exception, the exception is re-thrown from the GetContactsListAsync method as well. This case can be checked using the ExpectedException attribute. [TestMethod] [ExpectedException(typeof(InvalidOperationException))] public async Task GetContacts_Should_Throw_Exception() { repository.FailGet = true; var contacts = await contactsApi.GetContactsListAsync(); } Now let’s test the GetContactsHttpActionResultAsync method. Though this method does the same thing as the previous method, it doesn’t return the plain .NET objects. To test this method, we need to extract the result from the IHttpActionResult object obtained from the action method. Following test checks if the action result contains a collection when the repository is able to fetch results. Return type of Ok() method used above is OkNegotiatedContentResult. IHttpActionresult has to be converted to this type to check for the result obtained: [TestMethod] public async Task GetContactsHttpActionResult_Should_Return_HttpResult_With_Contacts() { var contactsResult = await contactsApi.GetContactsHttpActionResultAsync() as OkNegotiatedContentResult>; Assert.AreEqual(contactsResult.Content.Count(), 4); } Similarly, in case of error, we are calling InternalServerError() method to return the exception for us. We need to convert the result to ExceptionResult type to be able to check the type of exception thrown. It is shown below: [TestMethod] public async Task GetContactsHttpActionResult_Should_Return_HttpResult_With_Exception() { repository.FailGet = true; var contactsResult = await contactsApi.GetContactsHttpActionResultAsync() as ExceptionResult; Assert.IsInstanceOfType(contactsResult.Exception,typeof(InvalidOperationException)); } Happy coding!
December 24, 2013
by Rabi Kiran Srirangam
· 32,892 Views
article thumbnail
Storing Objects in Android
One alternative to using SQLite on Android is to store Java objects in SharedPreferences.
December 19, 2013
by Tony Siciliani
· 47,791 Views · 1 Like
article thumbnail
Make Your Progress Bar Smoother in Android
Want to smooth out that progress bar in Android? Here's how to get that done.
December 10, 2013
by Antoine Merle
· 31,321 Views
article thumbnail
The GO Product Roadmap – a New Agile Product Management Tool
A product roadmap is a high-level, strategic plan, which provides a longer-term outlook on the product. This creates a continuity of purpose, and it helps product managers and owners acquire funding for their product; it sets expectations, aligns stakeholders, and facilitates prioritization; it makes it easier to coordinate the development and launch of different products, and it provides reassurance to the customers (if the product roadmap is made public). Unfortunately, I find that many product managers and product owners struggle with their roadmaps, as they are dominated by features: There are too many features, and the features are often too detailed. This turns a roadmap into a tactical planning tool that competes with the Product Canvas or product backlog. What’s more, the features are sometimes regarded as a commitment by senior management than part of a high-level plan that is likely to change. The GO Product Roadmap Explained Faced with this situation, I have developed a new goal-oriented agile roadmap — the GO product roadmap, or “GO” for short. GO is based on my experience of teaching and coaching product managers and product owners, as well as using product roadmaps in my own business. The following pictures shows what the GO product roadmap looks like. You can download a PDF and Excel template by simply clicking on the picture. The first row of the GO roadmap depicted above contains the date or timeframe for the upcoming releases. You can work with a specific date such as 1st of March, or a period such as the first or second quarter. The second row states the name or version of the releases, for instance, iOS 7 or Windows 8.1. The third row provides the goal of each release, the reason why it is worthwhile to develop and launch it. Sample goals are to acquire or to activate users, to retain users by enhancing the user experience, or to accelerate development by removing technical debt. Working with goals shifts the conversation from debating individual features to agreeing on desired benefits making strategic product decisions. The development team, the stakeholders, and the management sponsor should all buy into the goals. The fourth row provides the features necessary to reach the goal. The features are means to an end, but not an end in themselves: They serve to create value and to reach the goal. Try to limit the number of features for each release to three, but do not state more than five. Refrain from detailing the features, and focus on the product capabilities that are necessary to meet the goal. Your product roadmap should be a high-level plan. The details should be covered in the Product Canvas or product backlog, and commitments should be limited to individual sprints. The last row states the metrics, the measurements or key performance indicators (KPIs) that help determine if the goal has been met, and if the release was successful. Make sure that the metrics you select allow you to measure if and to which extent you have met the goal. A Sample GO Product Roadmap To illustrate how the GO template can be applied, imagine we are about to develop a new dance game for girls aged eight to 12 years. The app should be fun and educational allowing the players to modify the characters, change the music, dance with remote players, and choreograph new dances. Here is what the corresponding GO roadmap could look like: While the roadmap above will have to be updated and refined at a later stage (particularly the metrics), I find it good enough to show how the product may evolve and make an investment decision. When creating your GO roadmap make sure you determine the goal of each release before you identify the features. This ensures that the features do serve the goal. Filling in the roadmap template from top to bottom and from left to right works well for me. Wrap-up The GO product roadmap provides a new, powerful way to do product roadmapping. Rather than focussing on features, GO emphasizes the importance of shared goals. This makes it easier to communicate the roadmap, create alignment, and use it as a strategic planning tool that provides an umbrella for the Product Canvas and the product backlog. The metrics provided by the tool ensure that the goals are measurable rather than lofty and fuzzy ideas. Download the template now, and try it out! You can learn more about creating effective product roadmap and working with the GO product roadmap by attending my Agile Product Planning training course. I would love to hear your questions about the roadmap and your experiences of creating product roadmaps. Please leave a comment below, or contact me.
December 3, 2013
by Roman Pichler
· 15,341 Views
article thumbnail
Disable Tests for Mule Studio Maven Projects
One of the most welcoming features of the new Mule Studio 3.4 is the Maven support. I was very keen to try out this new feature. I grabbed one of the projects I was working on, and imported it into Mule Studio through File -> Import -> Existing Maven Projects. Everything is as good as it gets. However I had one issue. Every time I wanted to run my flows as Mule Application, Mule Studio was doing a whole build of my project, including running the tests. Since this was a large project, I wanted to avoid running all the tests every time I needed to start the application. So I started by adding -DskipTests=true as a VM argument in the run configuration, but this did not work. My second attempt was to add the MAVEN_OPTS environmental variable and set it to -DskipTests=true, so back to the Mule Studio run configuration, clicked the Environment Variables table, set it there. Again, unfortunately this did not work. Worry not, there is a way. The third and final attempt was to check if the Mule Studio Maven support provides its own configuration, and luckily it does. So to fix it, Window -> Preferences (or MuleStudio -> Preference if you are using a MAC), navigate to Mule Studio on the left hand panel, expand that, and choose “Maven Settings”. In the configuration panel on the right hand side, you can type -DskipTests=true in the text box labelled as “MAVEN_OPTS environment variable”. Running my flow now does not run the tests. One small tip, -DskipTests=true and -Dmaven.test.skip=true are slightly different. If you go for the second option, Maven won’t even build your test classes, hence if you try to run any JUnit test from Mule Studio after a build, it will fail with ClassNotFoundException. Therefore I recommend the first option.
December 2, 2013
by Alan Cassar
· 14,801 Views
article thumbnail
Deconstructing the Azure Point-to-Site VPN for Command Line usage
when configuring an azure virtual network one of the most common things you'll want to do is setup a point-to-site vpn so that you can actually get to your servers to manage and maintain them. azure point-to-site vpns use client certificates to secure connections which can be quite complicated to configure so microsoft has gone the extra mile to make it easy for you to configure and get setup – sadly at the cost of losing the ability to connect through the command line or through powershell – let's change that. current state of play == no command line vpn connections normally when you want to launch a vpn from the cli or powershell in windows you can simply use the following command: rasdial "my home vpn" the azure pre-packaged vpn doesn't allow this because it's really just not a normal vpn. it's something else , something mysterious - not a normal native windows vpn connection. when you run the azure vpn through the command line you get this (you'll see a hint as to why i'd be using azure point-to-site in this screenshot): azure vpns don't appear to support this. if you want to keep your servers behind a private network in azure and use continuous deployment to get your code into production this makes it hard to deploy without a human being around. not really the best case scenario – especially when you remind yourself that automated builds aim to do away with human error altogether. what the azure point-to-site looks like out of the box when you first go to setup a point-to-site vpn into your azure virtual network microsoft points you at a page that walks you through creating a client certificate on your local machine to use as authentication. they then get you to download a package for setting up the azure vpn ras dialler on your local machine. this is accessed from within the azure "networks" page for your virtual network. you install this package and then whenever connecting you're greeted with a connection screen that you might of seen in a previous life. and by seen i don't mean that windows azure virtual networks have been around for ages. but more that the login screen may look familiar. this is because this login screen is a microsoft " connection manager " login screen and has been around for a while. example from technet (note extremely dated bitmap awesomeness): connection manager is used to pre-package vpn and dial up connections for easy-install distribution in a large organisation. this also means we can reconstruct the underlying vpn connection and use it as a normal vpn – claiming back our cli super powers. digging through the details so what we really want to know is: what is this mystical vpn technology the people at microsoft have bestowed upon us? here's how i started getting more information about the implementation: connecting once successfully then disconnect. open it up again to connect and click on properties then clicking on view log you'll then be greeted by something that looks like this: ****************************************************************** operating system : windows nt 6.2 dialler version : 7.2.9200.16384 connection name : my azure virtual network all users/single user : single user start date/time : 24/11/2013, 7:50:31 ****************************************************************** module name, time, log id, log item name, other info for connection type, 0=dial-up, 1=vpn, 2=vpn over dial-up ****************************************************************** [cmdial32] 7:50:31 03 pre-init event callingprocess = c:\windows\system32\cmmon32.exe [cmdial32] 7:50:39 04 pre-connect event connectiontype = 1 [cmdial32] 7:50:39 06 pre-tunnel event username = myclientsslcertificate domain = dunsetting = [obfuscated azure gateway id] tunnel devicename = tunneladdress = [obfuscated azure gateway id].cloudapp.net [cmdial32] 7:50:44 07 connect event [cmdial32] 7:50:44 08 custom action dll actiontype = connect actions description = to update your routing table actionpath = c:\users\doug\appdata\roaming\microsoft\network\connections\cm\[obfuscated azure gateway id]\cmroute.dll returnvalue = 0x0 [cmmon32] 7:56:21 23 external disconnect [cmdial32] 7:56:21 13 disconnect event callingprocess = c:\windows\explorer.exe more importantly you'll see this path included in the connection: within this folder is all the magic connection manager odds and ends. apologies for the [obfuscated], simply the path contains information to my azure endpoint. within this folder you'll see a bunch of files: most importantly there is a pbk file – a personal phonebook. this is what stores the connect settings for the vpn as is a commonly distributed way of sending out connection settings in the enterprise. if you run this on its own you'll actually be able to connect to the vpn directly (without your network routes being updated). this phonebook is where we can steal our settings from to recreate a command line driven connection. setting it up open up the properties of your azure point-to-site vpn phonebook above, and copy the connection address. it will look like this: azuregateway-[guid].cloudapp.net open network sharing centre , and create a new connection. then select connect to a workplace . select that you'll "use my internet connection". then enter your azure point-to-site vpn address and then give your new connection a name. remember this name for later then click create to save your vpn. now open the connection properties for your newly created vpn. this is where we'll use the settings in your azure diallers config to setup your connection. i'll save you the hassle of showing you me copying the settings from one connection to another and instead i'll just focus on what you need to set them to. flick over to the options tab and then click ppp settings . click the 2 missing options enable software compression and negotiate multi-link for single-link connections . set the type of vpn to secure socket tunnelling protocol (sstp), turn on eap and select microsoft: smart card of other certificate as the authentication type. then click on properties . select "use a certificate on this computer", un-tick "connect to these servers", and then select the certificate that uses your azure endpoint uri as its certificate name and then save out. then flick over to the network tab. open tcp/ipv4 then advanced then untick use default gateway on remote network . this setting stops internet traffic going over the vpn while you're connected so you can still surf reddit while managing your azure environment. close the vpn configuration panel. you now have a working vpn connection to azure. when you connect using windows you'll be asked to select the name of the client certificate you'll be authenticating with. you select the certificate you created and uploaded into azure before you setup your connection. when you connect using the command line you don't need to specify your certificate: rasdial "azure vpn" but there's one catch: your local machine's route table doesn't know when to send any traffic to your azure virtual network. the network link is there, but windows doesn't know what to send over your internet link and what to send over the vpn link. you see microsoft did a few things when they packaged your connection manager, and one of these things was to also copy a file called "cmroute.dll" and call this after connection to route your traffic onto your virtual network. this file altered your routing table to route traffic to your virtual network subnets through the vpn connection . we can do the same thing – so lets go about it. what's this about routing... rooting (for the english speakers in the room) my azure virtual network consists of the following network range: 10.0.0.0/8 i also have the following subnets for different machines groups. 10.0.1.0/24 (web servers) 10.0.2.0/24 (application servers) 10.0.3.0/24 (management services) my pptp connections, or point-to-site connections sit on the range: 172.16.0/24 this means that when i connect to the azure vpn i will get an ip address in this range. example: 172.16.0.17 when this happens we need to tell windows to route all traffic going to my 10.0.x.x range ip addresses through the ip address that has been given to us by azure's vpn rras service. you can see your current routing table by entering route print into a command prompt or powershell console. automating the routing additions luckily the windows task scheduler supports event listeners that allow us to watch for vpn connections and run commands off the back of them. take the below powershell script below and save it for arguments sake in c:\scripts\updateroutetableforazurevpn.ps1 ############################################################# # adds ip routes to azure vpn through the point-to-site vpn ############################################################# # define your azure subnets $ips = @("10.0.1.0", "10.0.2.0","10.0.3.0") # point-to-site ip address range # should be the first 4 octets of the ip address '172.16.0.14' == '172.16.0. $azurepptprange = "172.16.0." # find the current new dhcp assigned ip address from azure $azureipaddress = ipconfig | findstr $azurepptprange # if azure hasn't given us one yet, exit and let u know if (!$azureipaddress){ "you do not currently have an ip address in your azure subnet." exit 1 } $azureipaddress = $azureipaddress.split(": ") $azureipaddress = $azureipaddress[$azureipaddress.length-1] $azureipaddress = $azureipaddress.trim() # delete any previous configured routes for these ip ranges foreach($ip in $ips) { $routeexists = route print | findstr $ip if($routeexists) { "deleting route to azure: " + $ip route delete $ip } } # add our new routes to azure virtual network foreach($subnet in $ips) { "adding route to azure: " + $subnet echo "route add $ip mask 255.255.255.0 $azureipaddress" route add $subnet mask 255.255.255.0 $azureipaddress } now execute the following from an elevated command prompt window. this tells windows to add an event listener based task that looks for events to our "azure vpn" connection and if it sees them, it runs our powershell script. schtasks /create /f /tn "vpn connection update" /tr "powershell.exe -noninteractive -command c:\scripts\updateroutetableforazurevpn.ps1" /sc onevent /ec application /mo "*[system[(level=4 or level=0) and (eventid=20225)]] and *[eventdata[data='azure vpn']] " if i then connect to my vpn the above script should execute. after connecting if i check my routing table by entering route print into a console application we have our routes to azure added correctly. we're done! with that we're now able to fully use an azure point-to-site vpn simply from the command line. this means we can use it as part of a build server deployment, or if you're working on it all the time you can simply set it up to connect every time you login to windows . command line usage rasdial "[connection name]" rasdial "[connection name]" /disconnect for my connection named "azure vpn" this command line usage becomes: rasdial "azure vpn" rasdial "azure vpn" /disconnect
November 29, 2013
by Douglas Rathbone
· 10,661 Views
article thumbnail
Make Jenkins Windows Service use your Preferred JRE
recently i was working on installing and configuring a new instance of jenkins . for some reason, which is out of this post’s context, i wanted to make jenkins run with a specific version of the java environment. fortunately it was something really easy. this post is mainly a reminder to me, next time i’d like to do the same jenkins by default uses the jre which located under the jre sub-directory of your jenkins installation home ( %jenkins_home ). to change this find the file named jenkins.xml in which is located in your %jenkins_home directory. edit it and look for the following section %base%\jre\bin\java now change the content of the executable property to point to your favorite jre. you can describe it as an absolute or relative path or you can even use, environment variables. save the file and restart jenkins. that’s it! enjoy!
November 26, 2013
by Patroklos Papapetrou
· 16,806 Views · 2 Likes
article thumbnail
Integration vs. Orchestration
Applications are at the center of the IT universe. As IT shifts its primary goal from connectivity to experience, it will require tighter collaboration between the various infrastructure elements that support application workloads. There are two philosophical approaches to how this orchestration might take place: through a tightly-integrated system, or through a more loose coupling of heterogeneous components. But how should architects make the choice between these approaches? The principles of architecture tend to be most vehemently argued by the vendors competing to sell the underlying solutions. IT vendors generally (and networking in particular) tend to turn these principle discussions into tit-for-tat FUD wars, arguing in absolution that one approach or another is the right way to go. But the ones who put their careers on the line when they select an architectural approach should understand more fully what drives specific architectural selections. The difference between tightly-integrated systems and more loosely federated components is really performance. Whenever two components come together, that boundary is defined by some interface. If you need to extract performance out of the coupled system, you have to make changes on one or both sides of said interface. As a vendor, if you can twiddle the bits on only one side, you can improve the overall system performance up to but not beyond whatever the other side can do. So when performance is the primary objective, you will tend to see solutions where both sides of that interface are owned (or at least controlled) by the same party. The ability to make changes on both sides of the interface is the only way to maximize performance. When the primary objective is not performance, you will see a generalized interface that sits between a decoupled pairing of solution components. Enter SDN. Or network virtualization. Or NFV. Or DevOps. When we talk about performance as an industry, we usually mean capacity and speed. But performance is more than bandwidth and latency. The whole reason any of the SDN technologies is emerging is to satisfy operational issues. Getting applications provisioned, monitored, troubleshot, billed, upgraded, and so on has taken over the top spot on the pain list for many companies. The question we ought to be asking is what are the operational performance requirements. The answer isn't black or white. What does performance even mean in an operational setting? It seems at least plausible that operational performance translates to things like the rate of change (think provisioning changes per second or call setup and teardown rates, for example) or the rate of polling (queries per second, as with monitoring or billing). For some environments, it might be that the scale of configuration management or data querying is quite high. Any company that is doing fine-grained monitoring or rapid state-based network changes, for example, might have very high operational performance requirements. Meanwhile, most normal networks will likely have a much lower performance bar. For the former, the objective has to be to eke out every bit of operational performance from the system. This will demand a more tightly-integrated solution. Both sides of the resource boundary (network and storage, as an example) might need to be within the same system, and the interface between them should appropriately be very specific to the implementation. For the latter, a more generalized interface between infrastructure elements should be more than sufficient. The primary goal is not to maximize performance but rather enable collaboration between components. In these architectures, the generalized interface is the most important thing as it will optimize choice and flexibility between the individual system elements. Both are absolutely valid use cases; there is no judgment in which is the more noble cause. But architects ought to be clear about what it is they are optimizing for. Selecting a generalized interface merely because it is open could be disastrous if it turns out that the performance requirements exceed what that interface provides. Conversely, selecting a tightly-integrated system might be more costly or limiting than is necessary if the real problem is orchestration rather than performance. So where do architects start? Everything starts with requirements. Is the objective to achieve a specific rate of change? Or is the objective merely to make tasks like provisioning and troubleshooting more coordinated across infrastructure silos? Are you planning to do anything exotic in terms of polling data on the system elements? Or are you expecting data to be accessed at a more casual rate? The real point here is that architects should start to express their orchestration requirements in terms of both capability and performance. We do this instinctively when we think about how we move bits back and forth, or how we access storage, or how we allot cycles on a server. But when it comes to management, because our collective capabilities have been so lacking, we have ignored performance. As SDN and other technologies continue to advance, operational performance will take on a more important role. And without knowing what the requirements are, designers will really be flying blind, making tradeoffs that might not even be necessary. [Today's fun fact: In ancient Rome, it was considered a sign of leadership to be born with a crooked nose. If Mike Tyson were born earlier, we'd call him Emperor.]
November 20, 2013
by Mike Bushong
· 8,980 Views · 1 Like
article thumbnail
Python: Making scikit-learn and Pandas Play Nice
In the last post I wrote about Nathan and my attempts at the Kaggle Titanic Problem, I mentioned that our next step was to try out scikit-learn, so I thought I should summarize where we’ve got up to. We needed to write a classification algorithm to work out whether a person onboard the Titanic survived, and luckily, scikit-learn has extensive documentation on each of the algorithms. Unfortunately almost all those examples use numpy data structures and we’d loaded the data using pandas and didn’t particularly want to switch back! Luckily it was really easy to get the data into numpy format by calling ‘values’ on the pandas data structure, something we learnt from a reply on Stack Overflow. For example if we were to wire up an ExtraTreesClassifier which worked out survival rate based on the ‘Fare’ and ‘Pclass’ attributes we could write the following code: import pandas as pd from sklearn.ensemble import ExtraTreesClassifier from sklearn.cross_validation import cross_val_score train_df = pd.read_csv("train.csv") et = ExtraTreesClassifier(n_estimators=100, max_depth=None, min_samples_split=1, random_state=0) columns = ["Fare", "Pclass"] labels = train_df["Survived"].values features = train_df[list(columns)].values et_score = cross_val_score(et, features, labels, n_jobs=-1).mean() print("{0} -> ET: {1})".format(columns, et_score)) To start with with read in the CSV file which looks like this: $ head -n5 train.csv PassengerId,Survived,Pclass,Name,Sex,Age,SibSp,Parch,Ticket,Fare,Cabin,Embarked 1,0,3,"Braund, Mr. Owen Harris",male,22,1,0,A/5 21171,7.25,,S 2,1,1,"Cumings, Mrs. John Bradley (Florence Briggs Thayer)",female,38,1,0,PC 17599,71.2833,C85,C 3,1,3,"Heikkinen, Miss. Laina",female,26,0,0,STON/O2. 3101282,7.925,,S 4,1,1,"Futrelle, Mrs. Jacques Heath (Lily May Peel)",female,35,1,0,113803,53.1,C123,S Next we create our classifier which “fits a number of randomized decision trees (a.k.a. extra-trees) on various sub-samples of the dataset and use averaging to improve the predictive accuracy and control over-fitting.“ i.e. a better version of a random forest. On the next line we describe the features we want the classifier to use, then we convert the labels and features into numpy format so we can pass the to the classifier. Finally we call the cross_val_score function which splits our training data set into training and test components and trains the classifier against the former and checks its accuracy using the latter. If we run this code we’ll get roughly the following output: $ python et.py ['Fare', 'Pclass'] -> ET: 0.687991021324) This is actually a worse accuracy than we’d get by saying that females survived and males didn’t. We can introduce ‘Sex’ into the classifier by adding it to the list of columns: columns = ["Fare", "Pclass", "Sex"] If we re-run the code we’ll get the following error: $ python et.py An unexpected error occurred while tokenizing input The following traceback may be corrupted or invalid The error message is: ('EOF in multi-line statement', (514, 0)) ... Traceback (most recent call last): File "et.py", line 14, in et_score = cross_val_score(et, features, labels, n_jobs=-1).mean() File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/cross_validation.py", line 1152, in cross_val_score for train, test in cv) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/externals/joblib/parallel.py", line 519, in __call__ self.retrieve() File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/externals/joblib/parallel.py", line 450, in retrieve raise exception_type(report) sklearn.externals.joblib.my_exceptions.JoblibValueError/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/externals/joblib/my_exceptions.py:26: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message, : JoblibValueError ___________________________________________________________________________ Multiprocessing exception: ... ValueError: could not convert string to float: male ___________________________________________________________________________ This is a slightly verbose way of telling us that we can’t pass non numeric features to the classifier – in this case ‘Sex’ has the values ‘female’ and ‘male’. We’ll need to write a function to replace those values with numeric equivalents. train_df["Sex"] = train_df["Sex"].apply(lambda sex: 0 if sex == "male" else 1) Now if we re-run the classifier we’ll get a slightly more accurate prediction: $ python et.py ['Fare', 'Pclass', 'Sex'] -> ET: 0.813692480359) The next step is to use the classifier against the test data set, so let’s load the data and run the prediction: test_df = pd.read_csv("test.csv") et.fit(features, labels) et.predict(test_df[columns].values) Now if we run that: $ python et.py ['Fare', 'Pclass', 'Sex'] -> ET: 0.813692480359) Traceback (most recent call last): File "et.py", line 22, in et.predict(test_df[columns].values) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/ensemble/forest.py", line 444, in predict proba = self.predict_proba(X) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/ensemble/forest.py", line 479, in predict_proba X = array2d(X, dtype=DTYPE) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/utils/validation.py", line 91, in array2d X_2d = np.asarray(np.atleast_2d(X), dtype=dtype, order=order) File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/core/numeric.py", line 235, in asarray return array(a, dtype, copy=False, order=order) ValueError: could not convert string to float: male which is the same problem we had earlier! We need to replace the ‘male’ and ‘female’ values in the test set too so we’ll pull out a function to do that now. def replace_non_numeric(df): df["Sex"] = df["Sex"].apply(lambda sex: 0 if sex == "male" else 1) return df Now we’ll call that function with our training and test data frames: train_df = replace_non_numeric(pd.read_csv("train.csv")) ... test_df = replace_non_numeric(pd.read_csv("test.csv")) If we run the program again: $ python et.py ['Fare', 'Pclass', 'Sex'] -> ET: 0.813692480359) Traceback (most recent call last): File "et.py", line 26, in et.predict(test_df[columns].values) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/ensemble/forest.py", line 444, in predict proba = self.predict_proba(X) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/ensemble/forest.py", line 479, in predict_proba X = array2d(X, dtype=DTYPE) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/utils/validation.py", line 93, in array2d _assert_all_finite(X_2d) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/utils/validation.py", line 27, in _assert_all_finite raise ValueError("Array contains NaN or infinity.") ValueError: Array contains NaN or infinity. There are missing values in the test set so we’ll replace those with average values from our training set using an Imputer: from sklearn.preprocessing import Imputer imp = Imputer(missing_values='NaN', strategy='mean', axis=0) imp.fit(features) test_df = replace_non_numeric(pd.read_csv("test.csv")) et.fit(features, labels) print et.predict(imp.transform(test_df[columns].values)) If we run that it completes successfully: $ python et.py ['Fare', 'Pclass', 'Sex'] -> ET: 0.813692480359) [0 1 0 0 1 0 0 1 1 0 0 0 1 0 1 1 0 0 1 1 0 0 1 0 1 0 1 0 1 0 0 0 1 0 1 0 0 0 0 1 0 0 0 1 1 0 0 0 1 1 0 0 1 1 0 0 0 0 0 1 0 0 0 1 0 1 1 0 0 1 1 0 1 0 1 0 0 1 0 1 1 0 0 0 0 0 1 0 1 0 1 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 1 1 1 1 0 1 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 0 0 0 0 1 1 1 1 1 0 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 1 0 0 1 0 1 0 0 0 0 1 0 0 1 0 1 0 1 0 1 0 1 0 0 1 0 0 0 1 0 0 0 0 1 0 1 1 1 1 1 0 0 0 1 0 1 0 1 0 0 0 0 0 0 0 1 0 0 0 1 1 0 0 0 0 0 0 1 0 1 1 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 1 1 1 0 1 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 1 0 1 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 1 0 1 0 1 0 1 1 0 0 0 1 0 1 0 0 1 0 1 1 0 1 0 0 0 1 1 0 1 0 0 1 1 0 0 0 0 0 0 0 1 0 1 0 0 0 0 1 1 0 0 0 1 0 1 0 0 1 0 1 0 0 1 0 0 1 1 1 1 0 0 1 0 0 0] The final step is to add these values to our test data frame and then write that to a file so we can submit it to Kaggle. The type of those values is ‘numpy.ndarray’ which we can convert to a pandas Series quite easily: predictions = et.predict(imp.transform(test_df[columns].values)) test_df["Survived"] = pd.Series(predictions) We can then write the ‘PassengerId’ and ‘Survived’ columns to a file: test_df.to_csv("foo.csv", cols=['PassengerId', 'Survived'], index=False) Then output file looks like this: $ head -n5 foo.csv PassengerId,Survived 892,0 893,1 894,0 The code we’ve written is on github in case it’s useful to anyone.
November 14, 2013
by Mark Needham
· 32,253 Views · 1 Like
article thumbnail
Spring Static Application Context
Introduction I had an interesting conversation the other day about custom domain-specific languages and we happened to talk about a feature of Spring that I’ve used before but doesn’t seem to be widely known: the static application context. This post illustrates a basic example I wrote that introduces the static application context and shows how it might be useful. It’s also an interesting topic as it shows some of the well-architected internals of the Spring framework. Most uses of Spring start with XML or annotations and wind up with an application context instance. Behind the scenes, Spring has been working hard to instantiate objects, inject properties, invoke context aware listeners, and so forth. There are a set of classes internal to Spring to help this process along, as Spring needs to hold all of the configuration data about beans before any beans are instantiated. (This is because the beans may be defined in any order, and Spring doesn’t have the exhaustive set of dependencies until all beans are defined.) Spring Static Application Context Spring offers a class called StaticApplicationContext that gives programmatic access from Java to this whole configuration and registration process. This means we can define an entire application context from pure Java code, without using XML or Java annotations or any other tricks. The Javadoc for StaticApplicationContext is here, but an example is coming. Why might we use this? As the Javadoc says, it’s mainly useful for testing. Spring uses it for its own testing, but I’ve found it useful for testing applications that use Spring or other dependency management frameworks. Often, for unit testing, we want to inject different objects into a class from those used in production (e.g. mock objects, or objects that simulate remote invocation, database, or messaging). Of course, we can just keep a separate Spring XML configuration file for testing, but it’s very nice to have our whole configuration right there in the Java unit test class as it makes it easier to maintain. Example I’ve added an example to my intro-to-java repository on GitHub. I created aStaticContext class that provides a very basic Java domain-specific language (DSL) for Spring beans. This is just to make it easier to use from the unit test. The DSL only includes the most basic Spring capabilities: register a bean, set properties, and wire dependencies. package org.anvard.introtojava.spring; import org.springframework.beans.MutablePropertyValues; import org.springframework.beans.factory.config.ConstructorArgumentValues; import org.springframework.beans.factory.config.RuntimeBeanReference; import org.springframework.beans.factory.support.RootBeanDefinition; import org.springframework.context.ApplicationContext; import org.springframework.context.support.StaticApplicationContext; public class StaticContext { public class BeanContext { private String name; private Class beanClass; private ConstructorArgumentValues args; private MutablePropertyValues props; private BeanContext(String name, Class beanClass) { this.name = name; this.beanClass = beanClass; this.args = new ConstructorArgumentValues(); this.props = new MutablePropertyValues(); } public BeanContext arg(Object arg) { args.addGenericArgumentValue(arg); return this; } public BeanContext arg(int index, Object arg) { args.addIndexedArgumentValue(index, arg); return this; } public BeanContext prop(String name, Object value) { props.add(name, value); return this; } public BeanContext ref(String name, String beanRef) { props.add(name, new RuntimeBeanReference(beanRef)); return this; } public void build() { RootBeanDefinition def = new RootBeanDefinition(beanClass, args, props); ctx.registerBeanDefinition(name, def); } } private StaticApplicationContext ctx; private StaticContext() { this.ctx = new StaticApplicationContext(); } public static StaticContext create() { return new StaticContext(); } public ApplicationContext build() { ctx.refresh(); return ctx; } public BeanContext bean(String name, Class beanClass) { return new BeanContext(name, beanClass); } } This class uses several classes that are normally internal to Spring: StaticApplicationContext: Holds bean definitions and provides regular Java methods for registering beans. ConstructorArgumentValues: A smart list for a bean’s constructor arguments. Can hold both wire-by-type and indexed constructor arguments. MutablePropertyValues: A smart list for a bean’s properties. Can hold regular objects and references to other Spring beans. RuntimeBeanReference: A reference by name to a bean in the context. Used for wiring beans together because it allows Spring to delay resolution of a dependency until it’s been instantiated. The StaticContext class uses the builder pattern and provides for method chaining. This makes for cleaner use from our unit test code. Here’s the simplest example: @Test public void basicBean() { StaticContext sc = create(); sc.bean("basic", InnerBean.class).prop("prop1", "abc"). prop("prop2", "def").build(); ApplicationContext ctx = sc.build(); assertNotNull(ctx); InnerBean bean = (InnerBean) ctx.getBean("basic"); assertNotNull(bean); assertEquals("abc", bean.getProp1()); assertEquals("def", bean.getProp2()); } A slightly more realistic example that includes wiring beans together is not much more complicated: @Test public void innerBean() { StaticContext sc = create(); sc.bean("outer", OuterBean.class).prop("prop1", "xyz"). ref("inner", "inner").build(); sc.bean("inner", InnerBean.class).prop("prop1", "ghi"). prop("prop2", "jkl").build(); ApplicationContext ctx = sc.build(); assertNotNull(ctx); InnerBean inner = (InnerBean) ctx.getBean("inner"); assertNotNull(inner); assertEquals("ghi", inner.getProp1()); assertEquals("jkl", inner.getProp2()); OuterBean outer = (OuterBean) ctx.getBean(OuterBean.class); assertNotNull(outer); assertEquals("xyz", outer.getProp1()); assertEquals(inner, outer.getInner()); } Note that once we build the context, we can use it like any other Spring application context, including fetching beans by name or type. Also note that the two contexts we created here are completely separate, which is important for unit testing. Conclusion Much like my post on custom Spring XML, the static application context is a specialty feature that isn’t intended for everyday users of Spring. But I’ve found it convenient when unit testing and it provides an interesting peek into how Spring works.
November 13, 2013
by Alan Hohn
· 34,728 Views · 6 Likes
article thumbnail
Alternative to JUnit Parameterized Classes: junit-dataprovider
we all know junit test-classes can be parameterized , which means that for a given set of test-elements, the test class is instantiated a few times, but using constructors for that isn’t always what you want. i’ve taken the stringsorttest from this blog as an example. @runwith(parameterized.class) public class stringsorttest { @parameters public static collection data() { return arrays.aslist(new object[][] { { "abc", "abc" }, { "cba", "abc" }, }); } private final string input; private final string expected; public stringsorttest(final string input, final string expected) { this.input = input; this.expected = expected; } @test public void testsort() { assertequals(expected, mysortmethod(input)); } } this is pretty darn obnoxious sometimes if you have multiple sets of data for various tests, which all go through the constructor, which would force you to write multiple test classes. testng solves this better by allowing you to provide separate data sets to individual test methods using the @dataprovider annotation . but don’t worry, now you can achieve the same with the junit-dataprovider , available on github. pull in the dependency with e.g. maven. com.tngtech.java junit-dataprovider 1.5.0 test the above example now could be rewritten as: @runwith(dataproviderrunner.class) public class stringsorttest { @dataprovider public static object[][] data() { return new object[][] { { "abc", "abc" }, { "cba", "abc" }, }; } @test @usedataprovider("data") public void testsort(final string input, final string expected) { assertequals(expected, mysortmethod(input)); } } you’ll see: no constructor. in this example it doesn’t have many benefits, except maybe for less boiler-plate, but now you can create as many @dataprovider-annotated methods which are fed directly to your @usedataprovider-annotated testmethod(s). sidenote for eclipse: if you’re using that ide, the junit plugin is unable to map the names of the passed/failed testmethods to the ones in the testclass correctly. if a method fails, you’ll have to find it back manually in the testclass (or vote for the patch andreas smidt created to get this fixed in eclipse). in the mean time, if you’re stuck with junit and you’d love to use this feature you’re so accustomed to using with testng, go ahead and try the junit-dataprovider now.
November 13, 2013
by Ted Vinke
· 39,528 Views · 2 Likes
article thumbnail
Data Access Module using Groovy with Spock testing
This blog is more of a tutorial where we describe the development of a simple data access module, more for fun and learning than anything else. All code can be found here for those who don’t want to type along: https://github.com/ricston-git/tododb As a heads-up, we will be covering the following: Using Groovy in a Maven project within Eclipse Using Groovy to interact with our database Testing our code using the Spock framework We include Spring in our tests with ContextConfiguration A good place to start is to write a pom file as shown here. The only dependencies we want packaged with this artifact are groovy-all and commons-lang. The others are either going to be provided by Tomcat or are only used during testing (hence the scope tags in the pom). For example, we would put the jar with PostgreSQL driver in Tomcat’s lib, and tomcat-jdbc and tomcat-dbcp are already there. (Note: regarding the postgre jar, we would also have to do some minor configuration in Tomcat to define a DataSource which we can get in our app through JNDI – but that’s beyond the scope of this blog. See here for more info). Testing-wise, I’m depending on spring-test, spock-core, and spock-spring (the latter is to get spock to work with spring-test). Another significant addition in the pom is the maven-compiler-plugin. I have tried to get gmaven to work with Groovy in Eclipse, but I have found the maven-compiler-plugin to be a lot easier to work with. With your pom in an empty directory, go ahead and mkdir -p src/main/groovy src/main/java src/test/groovy src/test/java src/main/resources src/test/resources. This gives us a directory structure according to the Maven convention. Now you can go ahead and import the project as a Maven project in Eclipse (install the m2e plugin if you don’t already have it). It is important that you do not mvn eclipse:eclipse in your project. The .classpath it generates will conflict with your m2e plugin and (at least in my case), when you update your pom.xml the plugin will not update your dependencies inside Eclipse. So just import as a maven project once you have your pom.xml and directory structure set up. Okay, so our tests are going to be integration tests, actually using a PostgreSQL database. Since that’s the case, lets set up our database with some data. First go ahead and create a tododbtest database which will only be used for testing purposes. Next, put the following files in your src/test/resources: Note, fill in your username/password: DROP TABLE IF EXISTS todouser CASCADE; CREATE TABLE todouser ( id SERIAL, email varchar(80) UNIQUE NOT NULL, password varchar(80), registered boolean DEFAULT FALSE, confirmationCode varchar(280), CONSTRAINT todouser_pkey PRIMARY KEY (id) ); insert into todouser (email, password, registered, confirmationCode) values ('[email protected]', 'abc123', FALSE, 'abcdefg') insert into todouser (email, password, registered, confirmationCode) values ('[email protected]', 'pass1516', FALSE, '123456') insert into todouser (email, password, registered, confirmationCode) values ('[email protected]', 'anon', FALSE, 'codeA') insert into todouser (email, password, registered, confirmationCode) values ('[email protected]', 'anon2', FALSE, 'codeB') Basically, testContext.xml is what we’ll be configuring our test’s context with. The sub-division into datasource.xml and initdb.xml may be a little too much for this example… but changes are usually easier that way. The gist is that we configure our data source in datasource.xml (this is what we will be injecting in our tests), and the initdb.xml will run the schema.sql and test-data.sql to create our table and populate it with data. So lets create our test, or should I say, our specification. Spock is specification framework that allows us to write more descriptive tests. In general, it makes our tests easier to read and understand, and since we’ll be using Groovy, we might as well make use of the extra readability Spock gives us. package com.ricston.blog.sample.model.spec; import javax.sql.DataSource import org.springframework.beans.factory.annotation.Autowired import org.springframework.test.annotation.DirtiesContext import org.springframework.test.annotation.DirtiesContext.ClassMode import org.springframework.test.context.ContextConfiguration import spock.lang.Specification import com.ricston.blog.sample.model.data.TodoUser import com.ricston.blog.sample.model.dao.postgre.PostgreTodoUserDAO // because it supplies a new application context after each test, the initialize-database in initdb.xml is // executed for each test/specification @DirtiesContext(classMode=ClassMode.AFTER_EACH_TEST_METHOD) @ContextConfiguration('classpath:testContext.xml') class PostgreTodoUserDAOSpec extends Specification { @Autowired DataSource dataSource PostgreTodoUserDAO postgreTodoUserDAO def setup() { postgreTodoUserDAO = new PostgreTodoUserDAO(dataSource) } def "findTodoUserByEmail when user exists in db"() { given: "a db populated with a TodoUser with email [email protected] and the password given below" String email = '[email protected]' String password = 'anon' when: "searching for a TodoUser with that email" TodoUser user = postgreTodoUserDAO.findTodoUserByEmail email then: "the row is found such that the user returned by findTodoUserByEmail has the correct password" user.password == password } } One specification is enough for now, just to make sure that all the moving parts are working nicely together. The specification itself is easy enough to understand. We’re just exercising the findTodoUserByEmail method of PostgreTodoUserDAO – which we will be writing soon. Using the ContextConfiguration from Spring Test we are able to inject beans defined in our context (the dataSource in our case) through the use of annotations. This keeps our tests short and makes them easier to modify later on. Additionally, note the use of DirtiesContext. Basically, after each specification is executed, we cannot rely on the state of the database remaining intact. I am using DirtiesContext to get a new Spring context for each specification run. That way, the table creation and test data insertions happen all over again for each specification we run. Before we can run our specification, we need to create at least the following two classes used in the spec: TodoUser and PostgreTodoUserDAO package com.sample.data import org.apache.commons.lang.builder.ToStringBuilder class TodoUser { long id; String email; String password; String confirmationCode; boolean registered; @Override public String toString() { ToStringBuilder.reflectionToString(this); } } package com.ricston.blog.sample.model.dao.postgre import groovy.sql.Sql import javax.sql.DataSource import com.ricston.blog.sample.model.dao.TodoUserDAO import com.ricston.blog.sample.model.data.TodoUser class PostgreTodoUserDAO implements TodoUserDAO { private Sql sql public PostgreTodoUserDAO(DataSource dataSource) { sql = new Sql(dataSource) } /** * * @param email * @return the TodoUser with the given email */ public TodoUser findTodoUserByEmail(String email) { sql.firstRow """SELECT * FROM todouser WHERE email = $email""" } } package com.ricston.blog.sample.model.dao; import com.ricston.blog.sample.model.data.TodoUser; public interface TodoUserDAO { /** * * @param email * @return the TodoUser with the given email */ public TodoUser findTodoUserByEmail(String email); } We’re just creating a POGO in TodoUser, implementing its toString using common’s ToStringBuilder. In PostgreTodoUserDAO we’re using Groovy’s SQL to access the database, for now, only implementing the findTodoUserByEmail method. PostgreTodoUserDAO implements TodoUserDAO, an interface which specifies the required methods a TodoUserDAO must have. Okay, so now we have all we need to run our specification. Go ahead and run it as a JUnit test from Eclipse. You should get back the following error message: org.codehaus.groovy.runtime.typehandling.GroovyCastException: Cannot cast object '{id=3, [email protected], password=anon, registered=false, confirmationcode=codeA}' with class 'groovy.sql.GroovyRowResult' to class 'com.ricston.blog.sample.model.data.TodoUser' due to: org.codehaus.groovy.runtime.metaclass.MissingPropertyExceptionNoStack: No such property: confirmationcode for class: com.ricston.blog.sample.model.data.TodoUser Possible solutions: confirmationCode at com.ricston.blog.sample.model.dao.postgre.PostgreTodoUserDAO.findTodoUserByEmail(PostgreTodoUserDAO.groovy:23) at com.ricston.blog.sample.model.spec.PostgreTodoUserDAOSpec.findTodoUserByEmail when user exists in db(PostgreTodoUserDAOSpec.groovy:37) Go ahead and connect to your tododbtest database and select * from todouser; As you can see, our confirmationCode varchar(280), ended up as the column confirmationcode with a lower case ‘c’. In PostgreTodoUserDAO’s findTodoUserByEmail, we are getting back GroovyRowResult from our firstRow invocation. GroovyRowResult implements Map and Groovy is able to create a POGO (in our case TodoUser) from a Map. However, in order for Groovy to be able to automatically coerce the GroovyRowResult into a TodoUser, the keys in the Map (or GroovyRowResult) must match the property names in our POGO. We are using confirmationCode in our TodoUser, and we would like to stick to the camel case convention. What can we do to get around this? Well, first of all, lets change our schema to use confirmation_code. That’s a little more readable. Of course, we still have the same problem as before since confirmation_code will not map to confirmationCode by itself. (Note: remember to change the insert statements in test-data.sql too). One way to get around this is to use Groovy’s propertyMissing methods as show below: def propertyMissing(String name, value) { if(isConfirmationCode(name)) { this.confirmationCode = value } else { unknownProperty(name) } } def propertyMissing(String name) { if(isConfirmationCode(name)) { return confirmationCode } else { unknownProperty(name) } } private boolean isConfirmationCode(String name) { 'confirmation_code'.equals(name) } def unknownProperty(String name) { throw new MissingPropertyException(name, this.class) } By adding this to our TodoUser.groovy we are effectively tapping in on how Groovy resolves property access. When we do something like user.confirmationCode, Groovy automatically calls getConfirmationCode(), a method which we got for free when declared the property confirmationCode in our TodoUser. Now, when user.confirmation_code is invoked, Groovy doesn’t find any getters to invoke since we never declared the property confirmation_code, however, since we have now implemented the propertyMissing methods, before throwing any exceptions it will use those methods as a last resort when resolving properties. In our case we are effectively checking whether a get or set on confirmation_code is being made and mapping the respective operations to our confirmationCode property. It’s as simple as that. Now we can keep the auto coercion in our data access object and the property name we choose to have in our TodoUser. Assuming you’ve made the changes to the schema and test-data.sql to use confirmation_code, go ahead and run the spec file and this time it should pass. That’s it for this tutorial. In conclusion, I would like to discuss some finer points which someone who’s never used Groovy’s SQL before might not know. As you can see in PostgreTodoUserDAO.groovy, our database interaction is pretty much a one-liner. What about resource handling (e.g. properly closing the connection when we’re done), error logging, and prepared statements? Resource handling and error logging are done automatically, you just have to worry about writing your SQL. When you do write your SQL, try to stick to using triple quotes as used in the PostgreTodoUserDAO.groovy example. This produces prepared statements, therefore protecting against SQL injection and avoids us having to put ‘?’ all over the place and properly lining up the arguments to pass in to the SQL statement. Note that transaction management is something which the code using our artifact will have to take care of. Finally, note that a bunch of other operations (apart from findTodoUserByEmail) are implemented in the project on GitHub: https://github.com/ricston-git/tododb. Additionally, there is also a specification test for TodoUser, making sure that the property mapping works correctly. Also, in the pom.xml, there is some maven-surefire-plugin configuration in order to get the surefire-plugin to pick up our Spock specifications as well as any JUnit tests which we might have in our project. This allows us to run our specifications when we, for example, mvn clean package. After implementing all the operations you require in PostgreTodoUserDAO.groovy, you can go ahead and compile the jar or include in a Maven multi-module project to get a data access module you can use in other applications.
November 6, 2013
by Justin Calleja
· 21,252 Views
article thumbnail
Android 4.4 KitKat, the Browser and the Chrome WebView
Android 4.4 has made a big change in the OS’ internals for HTML5 development: it has replaced its original WebKit-based WebView with modern Chromium. The new Android Browser is also powered by Chromium, but it’s not clear yet its future. Besides the good news, not everything looks exciting in these changes: let’s see why. Every web developer that has played with native webapps, PhoneGap and the Android’s WebView knows how terrible it was in terms of performance and HTML5 compatibility. These are the same problems that most web developers suffer right now with the Android Browser, which is reported to be 32% of the mobile web browsing market share, compared with just 5% of the modern Chrome for Android according to Akamai. I’ve been talking about this problem in a recent post this year: Android Browser: the eternal mobile browser. Therefore I’m the first one to celebrate the beginning of the end for this dying web platform and the Chrome team now in charge of Android’s web runtimes. Chroming Android From Android 4.4, Chromium 30 is the web engine for the WebView native widget, including the V8 JavaScript engine. Let’s start with good news: Support for remote debugging Support for new HTML5 features Better performance Now why we should take this change with moderated excitement: We will still deal with the old WebView for a couple of years. It won’t be upgraded without an OS upgrade There might be some compatibility issues Where is My Browser? Everybody at Android and Chrome team is talking about the new WebView but nobody is even mentioning what will happen to the browser. We all want Chrome as the default browser, but it seems it’s not there yet (licenses issues, I guess). I’ve even seen a couple of members of the Chrome team saying that the stock Android Browser didn’t exist in the latest previous versions, which is not true. From Google’s perspective, Android Browser sounds much like IE6 and nobody wants to talk about it. They give us the idea that Chrome has been powering web browsing in Android for a while, but that is only true for some particular Android devices - Nexuses and devices from top manufactures. However, as I’ve mentioned before, the relationship between users browsing with Android Browser and Chrome is still 7 to 1. Besides what some people believe, the previous version of Android, 4.3, included minor upgrades to the Browser, so it is there for sure. The question is: what will happen on 4.4 with the stock browser? We know that the Nexus 5 has Google Chrome by default; the question here is what will happen with other devices having in mind that average users don’t download browsers from the store and use what the devices offers for browsing. Based on the emulator, the Android Browser is still there on the emulator and it’s using the classic browser UI with the Chromium 30 engine (it can coexist with Chrome but they will be radically different) Unfortunately, there is no mention of this on docs and blogs on Android 4.4. I hope we can get a real answer from the Android team soon about the future of the browser itself. The Good News Remote debugging Finally we have the ability to debug remotely Android native webviews, including PhoneGap apps, and the Android Browser works smoothly both from real devices and from the emulator. When we have an Android app opened with a web view or the Android Browser, the Chrome remote debugger tools will recognize it as a “Chrome 30” session and we have the full package of excellent tools for debug, profile and test our webapps. HTML5 new features Compared with the classic web view and the Android Browser until 4.3, we now have support for: Server Sent events Web Sockets Web Workers Advanced form input selectors, such as date and time FileSystem API IndexedDB MediaCapture Stream ??? test Animation Timing API Page Visibility API Canvas Blend modes CSS3 Flexbox (latest version) CSS3 Filters Even matching Chrome 30 for Android, the Web View (and potentially the Android Browser) will not have support (no reasons given) for: WebGL WebRTC WebAudio FullScreen Form validation Compared with the classic Web View, the new one doesn’t have Network Information API Performance difference Having V8 as the JavaScript engine for the new web view, the JavaScript performance if much better, besides general performance on CSS thanks to hardware acceleration. The Not so Good News The Classic Web View is still alive Don’t get so excited. We will deal with the old Web View (known as “classic”) for a couple of years. In fact, some devices such as Galaxy Nexus that are today on 4.3 will not get the update. And remember that still today 30% of Android users are on 2.x after 2 years of being replaced by 4.0, so it’s fair to guess that at the beginning of 2016 we will still have around a third of the users on the “classic” WebView that we hate today. The migration on the market will be slow based on Android’s fragmentation. WebView upgrade The KitKat WebView is based on Chromium 30 and it won’t be updated. That means you are stuck with it unless to get an upgrade in the future of the whole OS to next version. Even Google has announced OS delta updates without vendors’ intervention, but it seems the WebView will not get that deal yet. Therefore and based on Chrome's release cycle, in one year we will have Chrome 40 and the WebView will still be in 30. In a couple of years we might be complaining about an “old and outdated” webview again Compatibility issues Because there are changes between the old WebKit-based rendering engine and the modern Chromium engine, you should test your native webapp on KitKat to make sure it’s still working great. To reduce problems, if our app was packaged before KitKat the WebView will enter a “quirks mode” (any similarity with IE6 is pure coincidence) that will reduce the risk of incompatibilities while still getting the new APIs. In fact, this compatibility mode will get in action if the configuration file of your app has a target SDK lower than 19 (the API number for KitKat). To get more detailed information on migration and compatibility issues you can try the new Guides at Android and Chrome websites: http://developer.android.com/guide/webapps/migrating.html http://developers.google.com/chrome/mobile/docs/webview Looking Forward I’m really looking forward to remove the old WebKit and Android Browser from the market. The Chrome team is doing a great job empowering the mobile web (just remember homescreen webapps from Chrome 31), but sometimes the Android ecosystem is slowing down HTML5 penetration and helping promoting companies to avoid using web technologies. I hope this is the beginning of a change.
November 4, 2013
by Maximiliano Firtman
· 35,049 Views
article thumbnail
How to Integrate Apache Shiro into a Web Application
Apache Shiro can be used in a wide range of applications as part of the Java Security Framework.
November 4, 2013
by Hüseyin Akdoğan DZone Core CORE
· 39,431 Views · 2 Likes
article thumbnail
EasyNetQ: Publisher Confirms
Publisher confirms are a RabbitMQ addition to AMQP to guarantee message delivery. You can read all about them here and here. In short they provide a asynchronous confirmation that a publish has successfully reached all the queues that it was routed to. To turn on publisher confirms with EasyNetQ set the publisherConfirms connection string parameter like this: var bus = RabbitHutch.CreateBus("host=localhost;publisherConfirms=true"); When you set this flag, EasyNetQ will wait for the confirmation, or a timeout, before returning from the Publish method: bus.Publish(new MyMessage { Text = "Hello World!" }); // here the publish has been confirmed. Nice and easy. There’s a problem though. If I run the above code in a while loop without publisher confirms, I can publish around 4000 messages per second, but with publisher confirms switched on that drops to around 140 per second. Not so good. With EasyNetQ 0.15 we introduced a new PublishAsync method that returns a Task. The Task completes when the publish is confirmed: bus.PublishAsync(message).ContinueWith(task => { if (task.IsCompleted) { Console.WriteLine("Publish completed fine."); } if (task.IsFaulted) { Console.WriteLine(task.Exception); } }); Using this code in a while loop gets us back to 4000 messages per second with publisher confirms on. Happy confirms!
November 1, 2013
by Mike Hadlow
· 8,953 Views
article thumbnail
Securing Docker’s Remote API
One piece to Docker that is interesting AMAZING is the Remote API that can be used to programatically interact with docker. I recently had a situation where I wanted to run many containers on a host with a single container managing the other containers through the API. But the problem I soon discovered is that at the moment when you turn networking on it is an all or nothing type of thing… you can’t turn networking off selectively on a container by container basis. You can disable IPv4 forwarding, but you can still reach the docker remote API on the machine if you can guess the IP address of it. One solution I came up with for this is to use nginx to expose the unix socket for docker over HTTPS and utilize client-side ssl certificates to only allow trusted containers to have access. I liked this setup a lot so I thought I would share how it’s done. Disclaimer: assumes some knowledge of docker! Generate The SSL Certificates We’ll use openssl to generate and self-sign the certs. Since this is for an internal service we’ll just sign it ourselves. We also remove the password from the keys so that we aren’t prompted for it each time we start nginx. # Create the CA Key and Certificate for signing Client Certs openssl genrsa -des3 -out ca.key 4096 openssl rsa -in ca.key -out ca.key # remove password! openssl req -new -x509 -days 365 -key ca.key -out ca.crt # Create the Server Key, CSR, and Certificate openssl genrsa -des3 -out server.key 1024 openssl rsa -in server.key -out server.key # remove password! openssl req -new -key server.key -out server.csr # We're self signing our own server cert here. This is a no-no in production. openssl x509 -req -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt # Create the Client Key and CSR openssl genrsa -des3 -out client.key 1024 openssl rsa -in client.key -out client.key # no password! openssl req -new -key client.key -out client.csr # Sign the client certificate with our CA cert. Unlike signing our own server cert, this is what we want to do. openssl x509 -req -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out client.crt Another option may be to leave the passphrase in and provide it as an environment variable when running a docker container or through some other means as an extra layer of security. We’ll move ca.crt, server.key and server.crt to /etc/nginx/certs. Setup Nginx The nginx setup for this is pretty straightforward. We just listen for traffic on localhost on port 4242. We require client-side ssl certificate validation and reference the certificates we generated in the previous step. And most important of all, set up an upstream proxy to the docker unix socket. I simply overwrote what was already in /etc/nginx/sites-enabled/default. upstream docker { server unix:/var/run/docker.sock fail_timeout=0; } server { listen 4242; server localhost; ssl on; ssl_certificate /etc/nginx/certs/server.crt; ssl_certificate_key /etc/nginx/certs/server.key; ssl_client_certificate /etc/nginx/certs/ca.crt; ssl_verify_client on; access_log on; error_log /dev/null; location / { proxy_pass http://docker; proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; client_max_body_size 10m; client_body_buffer_size 128k; proxy_connect_timeout 90; proxy_send_timeout 120; proxy_read_timeout 120; proxy_buffer_size 4k; proxy_buffers 4 32k; proxy_busy_buffers_size 64k; proxy_temp_file_write_size 64k; } } One important piece to make this work is you should add the user nginx runs as to the docker group so that it can read from the socket. This could be www-data, nginx, or something else! Hack It Up! With this setup and nginx restarted, let’s first run a curl command to make sure that this setup correctly. First we’ll make a call without the client cert to double check that we get denied access then a proper one. # Is normal http traffic denied? curl -v http://localhost:4242/info # How about https, sans client cert and key? curl -v -s -k https://localhost:4242/info # And the final good request! curl -v -s -k --key client.key --cert client.crt https://localhost:4242/info For the first two we should get some run of the mill 400 http response codes before we get a proper JSON response from the final command! Woot! But wait there’s more… let’s build a container that can call the service to launch other containers! For this example we’ll simply build two containers: one that has the client certificate and key and one that doesn’t. The code for these examples are pretty straightforward and to save space I’ll leave the untrusted container out. You can view the untrusted container on github (although it is nothing exciting). First, the node.js application that will connect and display information: https = require 'https' fs = require 'fs' options = host: 172.42.1.62 port: 4242 method: 'GET' path: '/containers/json' key: fs.readFileSync('ssl/client.key') cert: fs.readFileSync('ssl/client.crt') headers: { 'Accept': 'application/json'} # not required, but being semantic here! req = https.request options, (res) -> console.log res req.end() And the Dockerfile used to build the container. Notice we add the client.crt and client.key as part of building it! FROM shykes/nodejs MAINTAINER James R. Carr ADD ssl/client* /srv/app/ssl ADD package.json /srv/app/package.json ADD app.coffee /srv/app/app.coffee RUN cd /srv/app && npm install . CMD cd /srv/app && npm start That’s about it. Run docker build . and docker run -n >IMAGE ID< and we should see a json dump to the console of the actively running containers. Doing the same in the untrusted directory should present us with some 400 error about not providing a client ssl certificate. I’ve shared a project with all this code plus a vagrant file on github for your own prusual. Enjoy!
October 31, 2013
by James Carr
· 14,313 Views
article thumbnail
Writing Git Hooks Using Python
Since git hooks can be any executable script with an appropriate #! line, Python is more than suitable for writing your git hooks. Simply stated, git hooks are scripts which are called at different points of time in the life cycle of working with your git repository. Let’s start by creating a new git repository: ~/work> git init git-hooks-exp Initialized empty Git repository in /home/gene/work/git-hooks-exp/.git/ ~/work> cd git-hooks-exp/ ~/work/git-hooks-exp (master)> tree -al .git/ .git/ ├── branches ├── config ├── description ├── HEAD ├── hooks │ ├── applypatch-msg.sample │ ├── commit-msg.sample │ ├── post-update.sample │ ├── pre-applypatch.sample │ ├── pre-commit.sample │ ├── prepare-commit-msg.sample │ ├── pre-rebase.sample │ └── update.sample ├── info │ └── exclude ├── objects │ ├── info │ └── pack └── refs ├── heads └── tags 9 directories, 12 files Inside the .git are a number of directories and files, one of them being hooks/ which is where the hooks live. By default, you will have a number of hooks with the file names ending in .sample. They may be useful as starting points for your own scripts. However, since they all have an extension .sample, none of the hooks are actually activated. For a hook to be activated, it must have the right file name and it should be executable. Let’s see how we can write a hook using Python. We will write a post-commit hook. This hook is called immediately after you have made a commit. We are going to do something fairly useless, but quite interesting in this hook. We will take the commit SHA1 of this commit, and print how it may look like in a more human form. I do the latter using the humanhash module. You will need to have it installed. Here is how the hook looks like: #!/usr/bin/python import subprocess import humanhash # get the last commit SHA and print it after humanizing it # https://github.com/zacharyvoase/humanhash print humanhash.humanize( subprocess.check_output( ['git','rev-parse','HEAD'])) I use the subprocess.check_output() function to execute the command git rev-parse HEAD so that I can get the commit SHA1 and then call the humanhash.humanize() function with it. Save the hook as a file, post-commit in your hooks/ directory and make it executable using chmod +x .git/hooks/post-commit. Let’s see the hook in action: ~/work/git-hooks-exp (master)> touch file ~/work/git-hooks-exp (master)> git add file ~/work/git-hooks-exp (master)> git commit -m "Added a file" carbon-network-connecticut-equal [master (root-commit) 2d7880b] Added a file 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 file The commit SHA1 for the commit turned out to be 2d7880be746a1c1e75844fc1aa161e2b8d955427. Let’s check it with the humanize function and check if we get the same message as above: >>> humanhash.humanize('2d7880be746a1c1e75844fc1aa161e2b8d955427') 'carbon-network-connecticut-equal' And you can see the same message above as well. For some of the hooks, you will see that they are called with some parameters. In Python you can access them using the sys.argv attribute from the sys module, with the first member being the name of the hook of course and the others will be the parameters that the hook is called with.
October 31, 2013
by Amit Saha
· 13,641 Views
  • Previous
  • ...
  • 576
  • 577
  • 578
  • 579
  • 580
  • 581
  • 582
  • 583
  • 584
  • 585
  • ...
  • Next
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook
×