DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Newsletter
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

The Latest Testing, Tools, and Frameworks Topics

article thumbnail
XP Values: Courage
In a complex system such as a software development team, it's easy for fear to arise.
August 28, 2013
by Giorgio Sironi
· 6,909 Views
article thumbnail
java.net.ProtocolException: Server Redirected Too Many Times
A couple of weeks ago I was trying to write a test around some OAuth code that we have on an internal application and I was using Jersey Client to send the various requests. I initially started with the following code: Client = Client.create(); ClientResponse response = client.resource( "http://localhost:59680" ).get( ClientResponse.class ); But when I ran the test I was getting the following exception: com.sun.jersey.api.client.ClientHandlerException: java.net.ProtocolException: Server redirected too many times (20) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:151) at com.sun.jersey.api.client.Client.handle(Client.java:648) at com.sun.jersey.api.client.WebResource.handle(WebResource.java:680) at com.sun.jersey.api.client.WebResource.get(WebResource.java:191) at com.neotechnology.testlab.manager.webapp.AuthenticationIntegrationTest.shouldRedirectToGitHubForAuthentication(AuthenticationIntegrationTest.java:81) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at com.neotechnology.kirkaldy.testing.Resources$1.evaluate(Resources.java:84) at com.neotechnology.kirkaldy.testing.FailureOutput$2.evaluate(FailureOutput.java:37) at org.junit.rules.RunRules.evaluate(RunRules.java:18) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.junit.runner.JUnitCore.run(JUnitCore.java:157) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63) Caused by: java.net.ProtocolException: Server redirected too many times (20) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1446) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:249) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149) ... 28 more If we check the traffic going across port 59680 we can see what’s going wrong: $ sudo ngrep -d lo0 port 59680 interface: lo0 (127.0.0.0/255.0.0.0) filter: (ip) and ( port 59680 ) ##### T 127.0.0.1:59704 -> 127.0.0.1:59680 [AP] GET / HTTP/1.1..User-Agent: Java/1.6.0_45..Host: localhost:59680..Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2..Connection: keep-alive.... ## T 127.0.0.1:59680 -> 127.0.0.1:59704 [AP] HTTP/1.1 302 Found..Set-Cookie: JSESSIONID=mdyw3a4fmqc1b6p53birm4dd;Path=/..Expires: Thu, 01 Jan 1970 00:00:00 GMT..Location: http://localhost:59679/authorize?client_id=basic-client&state=the-state&scope=user%2Crepo..Content-Length : 0..Server: Jetty(8.1.8.v20121106).... ########### T 127.0.0.1:59707 -> 127.0.0.1:59680 [AP] GET /auth/callback?code=timey-wimey&state=the-state HTTP/1.1..User-Agent: Java/1.6.0_45..Host: localhost:59680..Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2..Connection: keep-alive.... ## T 127.0.0.1:59680 -> 127.0.0.1:59707 [AP] HTTP/1.1 302 Found..Cache-Control: no-cache..Set-Cookie: JSESSIONID=8gggez0ns9ftiex4314mbgz9;Path=/..Expires: Thu, 01 Jan 1970 00:00:00 GMT..Location: http://localhost:59680/..Content-Length: 0..Server: Jetty(8.1.8.v20121106).... ########### T 127.0.0.1:59713 -> 127.0.0.1:59680 [AP] GET / HTTP/1.1..User-Agent: Java/1.6.0_45..Host: localhost:59680..Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2..Connection: keep-alive.... ## The response we receive includes a direction to the client to store a cookie but we can see on the next request that the cookie hasn’t been included. I came across this post, which had a few suggestions on how to get around the problem, but the only approach that worked for me was to use jersey-apache-client for which I added the following dependency: com.sun.jersey.contribs jersey-apache-client 1.13 jar I then change my client code to read like this: ApacheHttpClientConfig config = new DefaultApacheHttpClientConfig(); config.getProperties().put(ApacheHttpClientConfig.PROPERTY_HANDLE_COOKIES, true); ApacheHttpClient client = ApacheHttpClient.create( config ); client.setFollowRedirects(true); client.getClientHandler().getHttpClient().getParams().setBooleanParameter( HttpClientParams.ALLOW_CIRCULAR_REDIRECTS, true ); ClientResponse response = client.resource( "http://localhost:59680" ).get( ClientResponse.class ); If we run that and watch the output using ngrep we can see that it now handles cookies correctly: $ sudo ngrep -d lo0 port 59680 Password: interface: lo0 (127.0.0.0/255.0.0.0) filter: (ip) and ( port 59680 ) ##### T 127.0.0.1:60372 -> 127.0.0.1:59680 [AP] GET / HTTP/1.1..User-Agent: Jakarta Commons-HttpClient/3.1..Host: localhost:59680.... ## T 127.0.0.1:59680 -> 127.0.0.1:60372 [AP] HTTP/1.1 302 Found..Set-Cookie: JSESSIONID=vn8zzf9ep3x4mtw66ydm0n6a;Path=/..Expires: Thu, 01 Jan 1970 00:00:00 GMT..Location: http://localhost:60322/authorize?client_id=basic-client&state=the-state&scope=user%2Crepo..Content-Length : 0..Server: Jetty(8.1.8.v20121106).... ## T 127.0.0.1:60372 -> 127.0.0.1:59680 [AP] GET /auth/callback?code=timey-wimey&state=the-state HTTP/1.1..User-Agent: Jakarta Commons-HttpClient/3.1..Host: localhost:59680..Cookie: $Version=0; JSESSIONID=vn8zzf9ep3x4mtw66ydm0n6a; $Path=/.... ## T 127.0.0.1:59680 -> 127.0.0.1:60372 [AP] HTTP/1.1 302 Found..Cache-Control: no-cache..Location: http://localhost:59680/..Content-Length: 0..Server: Jetty(8.1.8.v20121106).... ## T 127.0.0.1:60372 -> 127.0.0.1:59680 [AP] GET / HTTP/1.1..User-Agent: Jakarta Commons-HttpClient/3.1..Host: localhost:59680..Cookie: $Version=0; JSESSIONID=vn8zzf9ep3x4mtw66ydm0n6a; $Path=/.... ## T 127.0.0.1:59680 -> 127.0.0.1:60372 [AP] HTTP/1.1 200 OK..Vary: Accept-Encoding..Accept-Ranges: bytes..Content-Type: text/html..Content-Length: 2439..Last-Modified: Tue, 23 Jul 2013 10:48:15 GMT..Server: Jetty(8.1.8.v20121106)....... . . . . . . . . . . . ....
August 21, 2013
by Mark Needham
· 33,719 Views
article thumbnail
Spock - Return Nested Spies / Mocks
Hi! Some time ago I have written an article about Mockito and using RETURNS_DEEP_STUBS when working with JAXB. Quite recently we have faced a similliar issue with deeply nesetd JAXB and the awesome testing framework written in Groovy called Spock. Natively Spock does not support creating deep stubs or spies so we needed to create a workaround for it and this article will show you how to do it. Project structure We will be working on the same data structure as in the RETURNS_DEEP_STUBS when working with JAXB article so the project structure will be quite simillar: As you can see the main difference is such that the tests are present in the /test/groovy/ folder instead of /test/java/ folder. Extended Spock Specification In order to use Spock as a testing framework you have to create Groovy test scripts that extend the Spock Specification class. The details of how to use Spock are available here. In this project I have created an abstract class that extends Specification and adds two additional methods for creating nested Test Doubles (I don't remember if I haven't seen a prototype of this approach somewhere on the internet). ExtendedSpockSpecification.groovy package com.blogspot.toomuchcoding.spock; import spock.lang.Specification /** * Created with IntelliJ IDEA. * User: MGrzejszczak * Date: 14.06.13 * Time: 15:26 */ abstract class ExtendedSpockSpecification extends Specification { /** * The method creates nested structure of spies for all the elements present in the property parameter. Those spies are set on the input object. * * @param object - object on which you want to create nested spies * @param property - field accessors delimited by a dot - JavaBean convention * @return Spy of the last object from the property path */ protected def createNestedSpies(object, String property) { def lastObject = object property.tokenize('.').inject object, { obj, prop -> if (obj[prop] == null) { def foundProp = obj.metaClass.properties.find { it.name == prop } obj[prop] = Spy(foundProp.type) } lastObject = obj[prop] } lastObject } /** * The method creates nested structure of mocks for all the elements present in the property parameter. Those mocks are set on the input object. * * @param object - object on which you want to create nested mocks * @param property - field accessors delimited by a dot - JavaBean convention * @return Mock of the last object from the property path */ protected def createNestedMocks(object, String property) { def lastObject = object property.tokenize('.').inject object, { obj, prop -> def foundProp = obj.metaClass.properties.find { it.name == prop } def mockedProp = Mock(foundProp.type) lastObject."${prop}" >> mockedProp lastObject = mockedProp } lastObject } } These two methods work in a very simillar manner. Assuming that the method's argument property looks as follows: "a.b.c.d" then the methods tokenize the string by "." and iterate over the array -["a","b","c","d"]. We then iterate over the properties of the Meta Class to find the one whose name is equal to prop (for example "a"). If that is the case we then use Spock's Mock/Spy method to create a Test Double of a given class (type). Finally we have to bind the mocked nested element to its parent. For the Spy it's easy since we set the value on the parent (lastObject = obj[prop]). For the mocks however we need to use the overloaded >> operator to record the behavior for our mock - that's why dynamically call the property that is present in the prop variable (lastObject."${prop}" >> mockedProp). Then we return from the closure the mocked/spied instance and we repeat the process for it Class to be tested Let's take a look at the class to be tested: PlayerServiceImpl.java package com.blogspot.toomuchcoding.service; import com.blogspot.toomuchcoding.model.PlayerDetails; /** * User: mgrzejszczak * Date: 08.06.13 * Time: 19:02 */ public class PlayerServiceImpl implements PlayerService { @Override public boolean isPlayerOfGivenCountry(PlayerDetails playerDetails, String country) { String countryValue = playerDetails.getClubDetails().getCountry().getCountryCode().getCountryCode().value(); return countryValue.equalsIgnoreCase(country); } } The test class And now the test class: PlayerServiceImplWrittenUsingSpockTest.groovy package com.blogspot.toomuchcoding.service import com.blogspot.toomuchcoding.model.* import com.blogspot.toomuchcoding.spock.ExtendedSpockSpecification /** * User: mgrzejszczak * Date: 14.06.13 * Time: 16:06 */ class PlayerServiceImplWrittenUsingSpockTest extends ExtendedSpockSpecification { public static final String COUNTRY_CODE_ENG = "ENG"; PlayerServiceImpl objectUnderTest def setup(){ objectUnderTest = new PlayerServiceImpl() } def "should return true if country code is the same when creating nested structures using groovy"() { given: PlayerDetails playerDetails = new PlayerDetails( clubDetails: new ClubDetails( country: new CountryDetails( countryCode: new CountryCodeDetails( countryCode: CountryCodeType.ENG ) ) ) ) when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } def "should return true if country code is the same when creating nested structures using spock mocks - requires CGLIB for non interface types"() { given: PlayerDetails playerDetails = Mock() ClubDetails clubDetails = Mock() CountryDetails countryDetails = Mock() CountryCodeDetails countryCodeDetails = Mock() countryCodeDetails.countryCode >> CountryCodeType.ENG countryDetails.countryCode >> countryCodeDetails clubDetails.country >> countryDetails playerDetails.clubDetails >> clubDetails when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } def "should return true if country code is the same using ExtendedSpockSpecification's createNestedMocks"() { given: PlayerDetails playerDetails = Mock() CountryCodeDetails countryCodeDetails = createNestedMocks(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode >> CountryCodeType.ENG when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } def "should return false if country code is not the same using ExtendedSpockSpecification createNestedMocks"() { given: PlayerDetails playerDetails = Mock() CountryCodeDetails countryCodeDetails = createNestedMocks(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode >> CountryCodeType.PL when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: !playerIsOfGivenCountry } def "should return true if country code is the same using ExtendedSpockSpecification's createNestedSpies"() { given: PlayerDetails playerDetails = Spy() CountryCodeDetails countryCodeDetails = createNestedSpies(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode = CountryCodeType.ENG when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } def "should return false if country code is not the same using ExtendedSpockSpecification's createNestedSpies"() { given: PlayerDetails playerDetails = Spy() CountryCodeDetails countryCodeDetails = createNestedSpies(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode = CountryCodeType.PL when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: !playerIsOfGivenCountry } } Let's move through the test methods one by one. First I present the code and then have a quick description of the presented snippet. def "should return true if country code is the same when creating nested structures using groovy"() { given: PlayerDetails playerDetails = new PlayerDetails( clubDetails: new ClubDetails( country: new CountryDetails( countryCode: new CountryCodeDetails( countryCode: CountryCodeType.ENG ) ) ) ) when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } Here you could find the approach of creating nested structures by using the Groovy feature of passing properties to be set in the constructor. def "should return true if country code is the same when creating nested structures using spock mocks - requires CGLIB for non interface types"() { given: PlayerDetails playerDetails = Mock() ClubDetails clubDetails = Mock() CountryDetails countryDetails = Mock() CountryCodeDetails countryCodeDetails = Mock() countryCodeDetails.countryCode >> CountryCodeType.ENG countryDetails.countryCode >> countryCodeDetails clubDetails.country >> countryDetails playerDetails.clubDetails >> clubDetails when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } Here you can find a test that creates mocks using Spock - mind you that you need CGLIB as a dependency when you are mocking non interface types. def "should return true if country code is the same using ExtendedSpockSpecification's createNestedMocks"() { given: PlayerDetails playerDetails = Mock() CountryCodeDetails countryCodeDetails = createNestedMocks(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode >> CountryCodeType.ENG when: booleanplayerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } Here you have an example of creating nested mocks using the createNestedMocks method. def "should return false if country code is not the same using ExtendedSpockSpecification createNestedMocks"() { given: PlayerDetails playerDetails = Mock() CountryCodeDetails countryCodeDetails = createNestedMocks(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode >> CountryCodeType.PL when: booleanplayerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: !playerIsOfGivenCountry } An example showing that creating nested mocks using the createNestedMocks method really works - should return false for improper country code. def "should return true if country code is the same using ExtendedSpockSpecification's createNestedSpies"() { given: PlayerDetails playerDetails = Spy() CountryCodeDetails countryCodeDetails = createNestedSpies(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode = CountryCodeType.ENG when: booleanplayerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } Here you have an example of creating nested spies using the createNestedSpies method. def "should return false if country code is not the same using ExtendedSpockSpecification's createNestedSpies"() { given: PlayerDetails playerDetails = Spy() CountryCodeDetails countryCodeDetails = createNestedSpies(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode = CountryCodeType.PL when: booleanplayerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: !playerIsOfGivenCountry } An example showing that creating nested spies using the createNestedSpies method really works - should return false for improper country code. Summary In this post I have shown you how you can create nested mocks and spies using Spock. It can be useful especially when you are working with nested structures such as JAXB. Still you have to bear in mind that those structures to some extend violate the Law of Demeter. If you check my previous article about Mockito you would see that: We are getting the nested elements from the JAXB generated classes. Although it violates the Law of Demeter it is quite common to call methods of structures because JAXB generated classes are in fact structures so in fact I fully agree with Martin Fowler that it should be called the Suggestion of Demeter. And in case of this example the idea is the same - we are talking about structures so we don't violate the Law of Demeter. Advantages With a single method you can mock/spy nested elements Code cleaner than creating tons of objects that you then have to manually set Disadvantages Your IDE won't help you with providing the property names since the properties are presented as Strings You have to set Test Doubles only in the Specification context (and sometimes you want to go outside this scope) Sources As usual the sources are available at BitBucket and GitHub.
August 8, 2013
by Marcin Grzejszczak
· 16,451 Views · 1 Like
article thumbnail
Jersey Client: Testing External Calls
Jim and I have been doing a bit of work over the last week which involved calling neo4j’s HA status URI to check whether or not an instance was a master/slave and we’ve been using jersey-client. The code looked roughly like this: class Neo4jInstance { private Client httpClient; private URI hostname; public Neo4jInstance(Client httpClient, URI hostname) { this.httpClient = httpClient; this.hostname = hostname; } public Boolean isSlave() { String slaveURI = hostname.toString() + ":7474/db/manage/server/ha/slave"; ClientResponse response = httpClient.resource(slaveURI).accept(TEXT_PLAIN).get(ClientResponse.class); return Boolean.parseBoolean(response.getEntity(String.class)); } } While writing some tests against this code we wanted to stub out the actual calls to the HA slave URI so we could simulate both conditions and a brief search suggested that mockito was the way to go. We ended up with a test that looked like this: @Test public void shouldIndicateInstanceIsSlave() { Client client = mock( Client.class ); WebResource webResource = mock( WebResource.class ); WebResource.Builder builder = mock( WebResource.Builder.class ); ClientResponse clientResponse = mock( ClientResponse.class ); when( builder.get( ClientResponse.class ) ).thenReturn( clientResponse ); when( clientResponse.getEntity( String.class ) ).thenReturn( "true" ); when( webResource.accept( anyString() ) ).thenReturn( builder ); when( client.resource( anyString() ) ).thenReturn( webResource ); Boolean isSlave = new Neo4jInstance(client, URI.create("http://localhost")).isSlave(); assertTrue(isSlave); } which is pretty gnarly but does the job. I thought there must be a better way so I continued searching and eventually came across this post on the mailing list which suggested creating a custom ClientHandler and stubbing out requests/responses there. I had a go at doing that and wrapped it with a little DSL that only covers our very specific use case: private static ClientBuilder client() { return new ClientBuilder(); } static class ClientBuilder { private String uri; private int statusCode; private String content; public ClientBuilder requestFor(String uri) { this.uri = uri; return this; } public ClientBuilder returns(int statusCode) { this.statusCode = statusCode; return this; } public Client create() { return new Client() { public ClientResponse handle(ClientRequest request) throws ClientHandlerException { if (request.getURI().toString().equals(uri)) { InBoundHeaders headers = new InBoundHeaders(); headers.put("Content-Type", asList("text/plain")); return createDummyResponse(headers); } throw new RuntimeException("No stub defined for " + request.getURI()); } }; } private ClientResponse createDummyResponse(InBoundHeaders headers) { return new ClientResponse(statusCode, headers, new ByteArrayInputStream(content.getBytes()), messageBodyWorkers()); } private MessageBodyWorkers messageBodyWorkers() { return new MessageBodyWorkers() { public Map> getReaders(MediaType mediaType) { return null; } public Map> getWriters(MediaType mediaType) { return null; } public String readersToString(Map> mediaTypeListMap) { return null; } public String writersToString(Map> mediaTypeListMap) { return null; } public MessageBodyReader getMessageBodyReader(Class tClass, Type type, Annotation[] annotations, MediaType mediaType) { return (MessageBodyReader) new StringProvider(); } public MessageBodyWriter getMessageBodyWriter(Class tClass, Type type, Annotation[] annotations, MediaType mediaType) { return null; } public List getMessageBodyWriterMediaTypes(Class tClass, Type type, Annotation[] annotations) { return null; } public MediaType getMessageBodyWriterMediaType(Class tClass, Type type, Annotation[] annotations, List mediaTypes) { return null; } }; } public ClientBuilder content(String content) { this.content = content; return this; } } If we change our test to use this code it now looks like this: @Test public void shouldIndicateInstanceIsSlave() { Client client = client().requestFor("http://localhost:7474/db/manage/server/ha/slave"). returns(200). content("true"). create(); Boolean isSlave = new Neo4jInstance(client, URI.create("http://localhost")).isSlave(); assertTrue(isSlave); } Is there a better way? In Ruby I’ve used WebMock to achieve this and Ashok pointed me towards WebStub which looks nice except I’d need to pass in the hostname + port rather than constructing that in the code.
August 1, 2013
by Mark Needham
· 10,924 Views
article thumbnail
AWS: Attaching an EBS volume on an EC2 instance and making it available for use
I recently wanted to attach an EBS volume to an existing EC2 instance that I had running and since it was for a one off tasks (famous last words) I decided to configure it manually. I created the EBS volume through the AWS console and one thing that initially caught me out is that the EC2 instance and EBS volume need to be in the same region and zone. Therefore if I create my EC2 instance in ‘eu-west-1b’ then I need to create my EBS volume in ‘eu-west-1b’ as well otherwise I won’t be able to attach it to that instance. I attached the device as /dev/sdf although the UI gives the following warning: Linux Devices: /dev/sdf through /dev/sdp Note: Newer linux kernels may rename your devices to /dev/xvdf through /dev/xvdp internally, even when the device name entered here (and shown in the details) is /dev/sdf through /dev/sdp. After attaching the EBS volume to the EC2 instance my next step was to SSH onto my EC2 instance and make the EBS volume available. The first step is to create a file system on the volume: $ sudo mkfs -t ext3 /dev/sdf mke2fs 1.42 (29-Nov-2011) Could not stat /dev/sdf --- No such file or directory The device apparently does not exist; did you specify it correctly? It turns out that warning was handy and the device has in fact been renamed. We can confirm this by callingfdisk: $ sudo fdisk -l Disk /dev/xvda1: 8589 MB, 8589934592 bytes 255 heads, 63 sectors/track, 1044 cylinders, total 16777216 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/xvda1 doesn't contain a valid partition table Disk /dev/xvdf: 53.7 GB, 53687091200 bytes 255 heads, 63 sectors/track, 6527 cylinders, total 104857600 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/xvdf doesn't contain a valid partition table /dev/xvdf is the one we’re interested in so I re-ran the previous command: $ sudo mkfs -t ext3 /dev/xvdf mke2fs 1.42 (29-Nov-2011) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 3276800 inodes, 13107200 blocks 655360 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=4294967296 400 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424 Allocating group tables: done Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done Once I’d done that I needed to create a mount point for the volume and I thought the best place was probably a directory under /mnt: $ sudo mkdir /mnt/ebs The final step is to mount the volume: $ sudo mount /dev/xvdf /mnt/ebs And if we run df we can see that it’s ready to go: $ df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda1 7.9G 883M 6.7G 12% / udev 288M 8.0K 288M 1% /dev tmpfs 119M 164K 118M 1% /run none 5.0M 0 5.0M 0% /run/lock none 296M 0 296M 0% /run/shm /dev/xvdf 50G 180M 47G 1% /mnt/ebs
July 31, 2013
by Mark Needham
· 12,097 Views
article thumbnail
Fake System Clock Pattern in Scala with Implicit Parameters
Fake system clock is a design pattern addressing testability issues of programs heavily relying on system time. If business logic flow depends on current system time, testing various flows becomes cumbersome or even impossible. Examples of such problematic scenarios include: certain business flow runs only (or is ignored) during weekends some logic is triggered only after an hour since some other event when two events occur at the exact same time (typically 1 ms precision), something should happen … Each scenario above poses unique set of challenges. Taken literally our unit tests would have to run only on specific day (1) or sleep for an hour to observe some behaviour. Scenario (3) might even be impossible to test under some circumstances since system clock can tick 1 millisecond at any time, thus making test unreliable. Fake system clock addresses these issues by abstracting system time over simple interface. Essentially you never call new Date(), new GregorianCalendar() or System.currentTimeMillis() but always rely on this: import org.joda.time.{DateTime, Instant} trait Clock { def now(): Instant def dateNow(): DateTime } As you can see I am depending on Joda Time library. Since we are already in the Scala land, one might consider scala-timeor nscala-time wrappers. Moreover the abstract name Clock is not a coincidence. It’s short and descriptive, but more importantly it mimics java.time.Clock class from Java 8 - that happens to address the same problem discussed here at the JDK level! But since Java 8 is still not here, let’s stay with our sweet and small abstraction. The standard implementation that you would normally use simply delegates to system time: import org.joda.time.{Instant, DateTime} object SystemClock extends Clock { def now() = Instant.now() def dateNow() = DateTime.now() } For the purposes of unit testing we will develop other implementations, but first let’s focus on usage scenarios. In a typical Spring/JavaEE applications fake system clock can be turned into a dependency that the container can easily inject. This makes dependence on system time explicit and manageable, especially in tests: @Controller class FooController @Autowired() (fooService: FooService, clock: Clock) { def postFoo(name: String) = fooService store new Foo(name, clock) } Here I am using Spring constructor injection asking the container to provide some Clock implementation. Of course in this case SystemClock is marked as @Service. In unit tests I can pass fake implementation and in integration tests I can place another, @Primary bean in the context, shadowing the SystemClock. This works great, but becomes painful for certain types of objects, namely entity/DTO beans and utility (static) classes. These are typically not managed by Spring so it can’t inject Clock bean to them. This forces us to pass Clock manually from the last “managed” layer: class Foo(fooName: String, clock: Clock) { val name = fooName val time = clock.dateNow() } similarly: object TimeUtil { def firstFridayOfNextMonth(clock: Clock) = //... } It’s not bad from design perspective. Both Foo constructor and firstFridayOfNextMonth() method do rely on system time so let’s make it explicit. On the other hand Clock dependency must be dragged, sometimes through many layers, just so that it can be used in one single method somewhere. Again, this is not bad per se. If your high level method has Clockparameter you know from the beginning that it relies on current time. But still is seems like a lot of boilerplate and overhead for little gain. Luckily Scala can help us here with: implicit parameters Let us refactor our solution a little bit so that Clock is an implicit parameter: @Controller class FooController(fooService: FooService) { def postFoo(name: String)(implicit clock: Clock) = fooService store new Foo(name) } @Service class FooService(fooRepository: FooRepository) { def store(foo: Foo)(implicit clock: Clock) = fooRepository storeInFuture foo } @Repository class FooRepository { def storeInFuture(foo: Foo)(implicit clock: Clock) = { val friday = TimeUtil.firstFridayOfNextMonth() //... } } object TimeUtil { def firstFridayOfNextMonth()(implicit clock: Clock) = //... } Notice how we call fooRepository storeInFuture foo ignoring second clock parameter. However this alone is not enough. We still have to provide some Clock instance as second parameter, otherwise compilation error strikes: could not find implicit value for parameter clock: com.blogspot.nurkiewicz.foo.Clock controller.postFoo("Abc") ^ not enough arguments for method postFoo: (implicit clock: com.blogspot.nurkiewicz.foo.Clock)Unit. Unspecified value parameter clock. controller.postFoo("Abc") ^ The compiler tried to find implicit value for Clock parameter but failed. However we are really close, the simplest solution is to use package object: package com.blogspot.nurkiewicz.foo package object foo { implicit val clock = SystemClock } Where SystemClock was defined earlier. Here is what happens: every time I call a function with implicit clock: Clock parameter inside com.blogspot.nurkiewicz.foo package, the compiler will discover foo.clock implicit variable and pass it transparently. In other words the following code snippets are equivalent but the second one provides explicit Clock, thus ignoring implicits: TimeUtil.firstFridayOfNextMonth() TimeUtil.firstFridayOfNextMonth()(SystemClock) also equivalent (first form is turned into the second by the compiler): fooService.store(foo) fooService.store(foo)(SystemClock) Interestingly in the bytecode level, implicit parameters aren’t any different from normal parameters so if you want to call such method from Java, passing Clock instance is mandatory and explicit. implicit clock parameter seems to work quite well. It hides ubiquitous dependency while still giving possibility to override it. For example in: fooService.store(foo) fooService.store(foo)(SystemClock) Tests The whole point of abstracting system time was to enable unit testing by gaining full control over time flow. Let us begin with a simple fake system clock implementation that always returns the same, specified time: class FakeClock(fixed: DateTime) extends Clock { def now() = fixed.toInstant def dateNow() = fixed } Of course you are free to put any logic here: advancing time by arbitrary value, speeding it up, etc. You get the idea. Now remember, the reason for implicit parameter was to hide Clock from normal production code while still being able to supply alternative implementation. There are two approaches: either pass FakeClock explicitly in tests: val fakeClock = new FakeClock( new DateTime(2013, 7, 15, 0, 0, DateTimeZone.UTC)) controller.postFoo("Abc")(fakeClock) or make it implicit but more specific to the compiler resolution mechanism: implicit val fakeClock = new FakeClock( new DateTime(2013, 7, 15, 0, 0, DateTimeZone.UTC)) controller.postFoo("Abc") The latter approach is easier to maintain as you don’t have to remember about passing fakeClock to method under test all the time. Of course fakeClock can be defined more globally as a field or even inside test package object. No matter which technique of providing fakeClock we choose, it will be used throughout all calls to service, repository and utilities. The moment we given explicit value to this parameter, implicit parameter resolution is ignored. Problems and summary Solution above to testing systems heavily dependant on time is not free from issues on its own. First of all the implicitClock parameter must be propagated throughout all the layers up to the client code. Notice that Clock is only needed in repository/utility layer while we had to drag it up to the controller layer. It’s not a big deal since the compiler will fill it in for us, but sooner or later most of our methods will include this extra parameter. Also Java and frameworks working on top of our code are not aware of Scala implicit resolution happening at compile time. Therefore e.g. our Spring MVC controller will not work as Spring is not aware of SystemClock implicit variable. It can be worked around though with WebArgumentResolver. Fake system clock pattern in general works only when used consistently. If you have even one place when real time is used directly as opposed to Clock abstraction, good luck in finding test failure reason. This applies equally to libraries and SQL queries. Thus if you are designing a library relying on current time, consider providing pluggable Clock abstraction so that client code can supply custom implementation like FakeClock. In SQL, on the other hand, do not rely on functions likeNOW() but always explicitly provide dates from your code (and thus from custom Clock).
July 18, 2013
by Tomasz Nurkiewicz
· 7,616 Views
article thumbnail
Spock Passes the Next Test - Painless Stubbing
In the last post I talked about our need for some improved testing tools, our choice of Spock as something to spike, and how mocking looks in Spock. As that blog got rather long, I saved the next installment for a separate post. Today I want to look at stubbing. Stubbing Mocking is great for checking outputs - in the example in the last post, we're checking that the process of encoding an array calls the right things on the way out, if you like - that the right stuff gets poked onto the bsonWriter. Stubbing is great for faking your inputs (I don't know why this difference never occurred to me before, but Colin's talk at Devoxx UK made this really clear to me). One of the things we need to do in the compatibility layer of the new driver is to wrap all the new style Exceptions that can be thrown by the new architecture layer and turn them into old-style Exceptions, for backwards compatibility purposes. Sometimes testing the exceptional cases is... challenging. So I opted to do this with Spock. class DBCollectionSpecification extends Specification { private final Mongo mongo = Mock() private final ServerSelectingSession session = Mock() private final DB database = new DB(mongo, 'myDatabase', new DocumentCodec()) @Subject private final DBCollection collection = new DBCollection('collectionName', database, new DocumentCodec()) def setup() { mongo.getSession() >> { session } } def 'should throw com.mongodb.MongoException if rename fails'() { setup: session.execute(_) >> { throw new org.mongodb.MongoException('The error from the new Java layer') } when: collection.rename('newCollectionName'); then: thrown(com.mongodb.MongoException) } } So here we can use a real DB class, but with a mock Mongo that will return us a "mock" Session. It's not actually a mock though, it's more of a stub because we want to tell it how to behave when it's called - in this test, we want to force it to throw an org.mongodb.MongoException whenever execute is called. It doesn't matter to us what get passed in to the execute method (that's what the underscore means on line 16), what matters is that when it gets called it throws the correct type of Exception. Like before, the when: section shows the bit we're actually trying to test. In this case, we want to callrename. Then finally the then: section asserts that we received the correct sort of Exception. It's not enormously clear, although I've kept the full namespace in to try and clarify, but the aim is that anyorg.mongodb.MongoException that gets thrown by the new architecture gets turned into the appropriate com.mongodb.MongoException. We're sort of "lucky" because the old code is in the wrong package structure, and in the new architecture we've got a chance to fix this and put stuff into the right place. Once I'd tracked down all the places Exceptions can escape and started writing these sorts of tests to exercise those code paths, not only did I feel more secure that we wouldn't break backwards compatibility by leaking the wrong Exceptions, but we also found our test coverage went up - and more importantly, in the unhappy paths, which are often harder to test. I mentioned in the last post that we already did some simple stubbing to help us test the data driver. Why not just keep using that approach? Well, these stubs end up looking like this: private static class TestAsyncConnectionFactory implements AsyncConnectionFactory { @Override public AsyncConnection create(final ServerAddress serverAddress) { return new AsyncConnection() { @Override public void sendMessage(final List byteBuffers, final SingleResultCallback callback) { throw new UnsupportedOperationException(); } @Override public void receiveMessage(final ResponseSettings responseSettings, final SingleResultCallback callback) { throw new UnsupportedOperationException(); } @Override public void close() { } @Override public boolean isClosed() { throw new UnsupportedOperationException(); } @Override public ServerAddress getServerAddress() { throw new UnsupportedOperationException(); } }; } } Ick. And you end up extending them so you can just override the method you're interested in (particularly in the case of forcing a method to throw an exception). Most irritatingly to me, these stubs live away from the actual tests, so you can't easily see what the expected behaviour is. In the Spock test, the expected stubbed behaviour is defined on line 16, the call that will provoke it is on line 19 and the code that checks the expectation is on line 22. It's all within even the smallest monitor's window. So stubbing in Spock is painless. Next!
July 11, 2013
by Trisha Gee
· 4,935 Views
article thumbnail
The Legacy Code Retreat
Last Saturday, I was a coach at the Legacy Code Retreat Milan 2013 organized by @gabrielelana, along with @filippo and @andreafrancia. Here's a recap of the goals of the event and of the experience. Why? A Code Retreat is a day dedicated to deliberate practice, where programmers reunite and work in pair programming on a single problems for each iteration. Each iteration is composed of a coding (or designing) section of 45' and of a retrospective held by the whole group. The focus of Code Retreats is to improve a particular skill in a setting with no pressure and deadlines; all code is deleted at the end of each iteration to pass the message that it is not the end result of running that is important (getting to the same place where you started from) but the path taken and the training that results. A Legacy Code Retreat is a particular kind of Code Retreat where legacy code is provided in the form of a source code repository containing a version of the Trivial Pursuit game (available for multiple programming languages). In our implementation the goal was different for each iteration: 1st iteration: produce a massive end-to-end automated test for the code, called Golden Master. This is strongly suggested by the existing material on the Legacy Code Retreat: you can only change code that is covered by automated tests. 2nd iteration: make it easy to add a new category of questions. 3rd iteration: add unit tests for the roll() function checking its output and final state. 4th iteration: find all the code smells, and fix 3 of them. 5th iteration: remove all duplication. 6th iteration: make the introduction of different penalty rules a one-line change (an Open/Closed Principle kata). Here is the presentation by Gabriele containing the introduction to the day and the goals. The golden master The first iteration is special, and serve as a setup for the rest of the day; the result of this iteration is not (at least conceptually) thrown away but instead kept and run as an end2end test to find out any regression. The chicken-and-egg problem of legacy code is always the same: it's difficult to test for unclear dependencies and global state, but you can't change it to simplify the job until you have some tests that ensure you're not breaking anything. Both testing and refactoring require each other. In fact, the goal of the first iteration is to produce such an end2end test without modifying the code in the repository: you could say "just add new source files" or "only use safe refactorings" for the languages that allow them. The code itself has no particular global state (such as database tables), unless for the fact that it calls random() functions to generate the result of dice throwing. This means executing the code multiple times always gives different results: "Chet is the current player","They have rolled a 6","Chet's new location is 6",... "Chet is the current player","They have rolled a 4","Chet's new location is 4",... The challenge of producing the golden master is to provide an initialization to the global state of the random number generator so that executions of the game can be repeated. Once you have 100 iterations with different initializations, you can store them and repeat the same process to find out any different in the output of the game. Any difference would signal that refactoring the legacy code has broken it and `git reset` or your favorite undo must be used. As to keep iterations always independent, at the end of the first one we provided a branch golden-master on the repository to all pairs. In this way even who didn't reach the goal (which is normal given the short time frame) or have a rough solution could use a tuned test case for the rest of the day; the presence of more golden masters allows also pairs to change at each iteration, and the components to switch programming languages. How it went? The event was sold out and the 30 people showed up (fortunately an even number so as to make pair programming easier). As coaches, we moved between pairs trying not to interrupt them if they were focused but offering help to the stuck pairs and to the ones wandering off the objective. The final retrospective brought out several goods: good format: each iteration is almost independent. Clearly defined goals. Variety of languages and people. Location and food (Talent Garden in Milan and breakfast offered by XPeppers). And several bads too, to resolve for the next editions: no theoretical introduction on how to work with the legacy code. Difficulties in using Extract Class, with respect to Extract Method and Extract Field which are local changes. Difficulties in introducing unit-level tests.
July 10, 2013
by Giorgio Sironi
· 4,607 Views
article thumbnail
Why Static is Bad and How to Avoid It
Everybody who worked with a project which included a StringUtil(s) class with only static methods, raise her hand! Thought so. Are those methods bad? Probably not so much, although I had a word to say about the name, after all if a class is not a utility it isn’t useful (by the definition of Wiktionary) and we hopefully haven’t much of that kind in our projects. But static methods turn bad, when they become more complex than the typical content of a StringUtil class. The problem is your code becomes hard wired to that static method. There is no easy way to replace the reference to the static method with something else, and if you are testing your code using automated tests, this is exactly what you want to do. If you don’t test your code using automated tests, do something about it NOW! Converting a static method to something easily mocked is straight forward once you’ve done it once or twice. Lets start with an example: public class Utility{ public static int doSomething(){ //… } } public class Client{ public void foo(){ //… Utility.doSomething(); //… } } The Client uses a static method in Utility and we want to get rid of that. The first step is to make the doSomethingmethod non-static. It is really as easy as removing the static modifier. Of course now the Client needs and instance ofUtility, so we just create one for now: public class Utility{ public int doSomething(){ //… } } public class Client{ public void foo(){ //… new Utility().doSomething(); //… } } Of course this doesn’t improve the situation much. We still have a static reference to the Utility class, since the constructor is just another static method. But now we can simply inject the dependency from the outside: public class Utility{ public int doSomething(){ //… } } public class Client{ private final Utility utility; public Client(Utility aUtility){ utility = aUtility; } public void foo(){ //… utility.doSomething(); //… } } Now you can replace Utility by a mocked instance for tests, you can use a wrapped instance for logging or make it implement an interface and so one. Basically you are back in OO world. Of course you can use your favorite DI-Framework to inject the dependency (just make sure you do it properly), or if you don’t mind the compile time dependency you can create an alternative constructor in the Client which uses the default implementation.
July 8, 2013
by Jens Schauder
· 168,792 Views · 7 Likes
article thumbnail
Writing Clean Predicates with Java 8
in-line predicates can create a maintenance nightmare. writing in-line lambda expressions and using the stream interfaces to perform common operations on collections can be awesome. assume the following example: list getadultmales (list persons) { return persons.stream().filter(p -> p.getage() > adult && p.getsex() == sexenum.male ).collect(collectors.tolist()); } that’s fun! but things like this also lead to software that is costly to maintain. at least in an enterprise application, where most of your code handles business logic, your development team will grow the tenancy to write the same similar set of predicate rules again and again. that is not what you want on your project. it breaks three important principles for growing maintainable and stable enterprise applications: dry (don’t repeat yourself): writing code more than once is not a good fit for a lazy developer it also makes your software more difficult to maintain because it becomes harder to make your business logic consistent readability : following clean-code best practices, 80% of writing code is reading the code that already exists. having complicated lambda expressions is still a bit hard to read compared to a simple one-line statement. testability : your business logic needs to be well-tested. it is adviced to unit-test your complex predicates. and that is just much easier to do when you separate your business predicate from your operational code. and from a personal point of view… that method still contains too much boilerplate code… imports to the rescue! fortunately, we have a very good suggestion in the world of unit testing on how we could improve on this. imagine the following example: import static somepackage.personpredicate; ... list getadultmales (list persons) { return persons.stream().filter( isadultmale() ).collect(collectors.tolist()); } what we did here was: create a personpredicate class define a “factory” method that creates the lambda predicate for us statically import the factory method into our old class this is how such a predicate class could look like, located next to your person domain entity: public personpredicate { public static predicate isadultmale() { return p -> p.getage() > adult && p.getsex() == sexenum.male; } } wait… why don’t we just create a “ismaleadult” boolean function on the person class itself like we would do in domain driven development? i agreed, that is also an option… but as time goes on and your software project becomes bigger and loaded with functionality and data… you will again break your clean code principles: the class becomes bloated with all kind of function and conditions your class and tests become huge, more difficult to handle and change (*) (*) and yes… even if you do your best to separate your concerns and use composition patterns adding some defaults… working with domain objects, we can imagine that some operations (such as filter) are often executed on domain entities. taking that into account, it would make sense to let our entities implement some interface that offers us some default methods. for example: public interface domainoperations { default list filter(predicate predicate) { return persons.stream().filter( predicate ) .collect(collectors.tolist()); } } when our person entity implements this interface, we can clean-up our code even more: list getadultmales (list persons) { return persons.filter( isadultmale() ); } and there we go… conclusion moving your predicates to a predicate helper class offers some good advantages in the long run: predicate classes are easy to test and change your domain objects remain clean and focussed on representing your domain, not your business logic you optimize the re-usability of your code and, in the end, reduce your maintenance you seperate your business from operational concerns references clean code: a handbook of agile software craftsmanship [robert c. martin] practical unit testing with junit and mockito [tomek kaczanowski] state of the collections [http://cr.openjdk.java.net/~briangoetz/lambda/collections-overview.html] notes the code above is served as an example to illustrate the principles i wanted to discuss. however, i did not proof-run this code yet (it’s still on my todo list). some modifications may be needed for your project.
July 2, 2013
by Kevin Chabot
· 156,216 Views · 8 Likes
article thumbnail
Integration of Amazon Redshift Data Warehouse with Talend Data Integration
In this blog post, I will show you how to "ETL" all kinds of data to Amazon’s cloud data warehouse Redshift wit Talend’s big data components. Let’s begin with a short introduction to Amazon Redshift (copied from website): "Amazon Redshift is [part of Amazon Web Services (AWS) and] a fast and powerful, fully managed, petabyte-scale data warehouse service in the cloud. With a few clicks in the AWS Management Console, customers can launch a Redshift cluster, starting with a few hundred gigabytes and scaling to a petabyte or more, for under $1,000 per terabyte per year. Traditional data warehouses require significant time and resource to administer, especially for large datasets. In addition, the financial cost associated with building, maintaining, and growing self-managed, on-premise data warehouses is very high. Amazon Redshift not only significantly lowers the cost of a data warehouse, but also makes it easy to analyze large amounts of data very quickly.“ Sounds interesting! And indeed, we already see companies using Talend’s Redshift connectors. From Talend perspective it is not much more than just another database. If you have ever used a Talend connector, you can integrate to Redshift within some minutes. In the next sections, I will describe all necessary steps and give some hints regarding configuration issues and performance improvements. Be aware: You need Talend Open Studio for Data Integration (Apache License, open source) or any Talend Enterprise Edition / Platform which contains the Cloud components to see and use Amazon Redshift connectors. The open source edition offers all connectors and functionality to integrate with Amazon Redshift. However, Enterprise versions offer some more features (e.g. versioning), comfort (e.g. wizards) and commercial support. Setup Amazon Redshift Setup of Amazon Redshift is very easy. Just follow Amazon‘s getting started guide: http://docs.aws.amazon.com/redshift/latest/gsg/welcome.html. Like every other AWS guide, it is very easy to understand and use. Be aware, that you just have to do step 1, 2 and 3 of the getting started guide for using it with Talend. Some hints: - Step 1 („before you begin“): Just sign up. Client tools and drivers are not necessary because they are already installed within Talend Studio. - Step 2 („launch a cluster“): Yes, please start your cluster! - Step 3(„authorize access“): If you are not sure what to do here, select Connection Type = CIDR/IP. Find out your IP address (http://whatismyipaddress.com) and enter it with „/32“ at the end. Example: „192.168.1.1/32“ Now you can connect to Amazon Redshift from your Talend Studio on your local computer. Step 4 (connect) and step 5 (create table, data, queries) are not necessary, this will be done from Talend Studio. Of course, you should not forget to delete your cluster (step 7) when you are done. Otherwise, you will pay for every hour, even if you do not access your DWH. Connect to Amazon Redshift from Talend Studio Create a new connection to Amazon Redshift database as you do with every other relational database. The easiest way is to use „DB Connection Wizard“ in metadata. Just enter your connection information and check if it works. You get all information about configuration from Amazon Web Console. The connection string looks something like this: „jdbc:paraccel://talend-demo-cluster.cp8t6c5.eu-west-1.redshift.amazonaws.com:5439/dev“ Next, right click on the created connection and select „retrieve schema“. „public“ is the default schema which you (have to) use. Now, you are ready to use this connection within Talend Jobs to write to Amazon Redshift and read from it. Create Talend Jobs (Write, Read, Delete) Amazon Redshift components work like any other Talend (relational) database components. Look at www.help.talend.com for more information if you have not used them before (or just try them out, they are very self-explanatory). You just have to drag&drop your connection from metadata . Afterwards, you can easily write data (tRedShiftOutput), read data (tRedshiftInput), or do any other queries such as delete or copy (tRedShiftRow). In the following job, I start with deleting all content in the Amazon Redshift table. Then, I read data from a MySQL table and insert it into an Amazon Redshift table. The table is created automatically (as I have configured it this way). After this subjob is finished, I read the data again, and store it to a CSV file (which is also created automatically). Of course, this is no business use case, but it shows how to use different Amazon Redshift components. Query Data from Amazon Redshift You can connect to Amazon Redshift directly from Talend Studio to explore and query data of the DWH. Thus, no other database tool is required. Just right click on your Amazon Redshift connection in metadata and select „edit queries“. Here you can define, execute and save SQL queries. Improve Performance Write performance of Amazon Redshift is relatively low compared to „classical“ relational databases (in your data center) as you have to upload all data into the cloud. Different alternatives exist to improve performance: - Bulk inserts: „Extended insert“ (in advanced settings) improves performance a lot, but still not to hyperspeed… Also, as it is bulk, you can just do inserts! It is not compatible to „rejects“ or „updates“ - AWS S3 and COPY command: S3 is Amazon’s „simple storage service“, a key-value store – also called NoSQL today – for storing very large objects. You can use Amazon Redshift’s COPY command (http://docs.aws.amazon.com/redshift/latest/dg/r_COPY.html) to transfer data from S3 to Amazon Redshift with good performance. Though, you still have to copy data to S3 before, same „cloud problem“ here. The COPY command can be used with tRedshiftRow, so no problem at all from Talend perspective. To transfer data to S3, you can either use the Talend S3 components from Talendforge, Talend’s open source community (http://www.talendforge.org/exchange), or use camel-s3, an Apache Camel component which is included in Talend ESB. The latter is an option, if you use Talend Data Services which combines Talend DI and Talend ESB in its unified platform. Summary You need not be a cloud or DWH expert, or an expert developer to integrate with Amazon’s cloud data warehouse Redshift. It is very easy with Talend’s integration solutions. Just drag&drop, configure, do some graphical mappings / transformations (if necessary), that’s it. Code is generated. Job runs. You can integrate Amazon Redshift almost as simple as any other relational database. Just be aware of some cloud specific security and performance issues. With Talend, you can easily „ETL“ all data from different sources to Redshift and store it there for under $1,000 per terabyte per year – even with the open source version! Best regards, Kai Wähner (Contact and feedback via @KaiWaehner, www.kai-waehner.de, LinkedIn / Xing) This is content from my blog: http://www.kai-waehner.de/blog/2013/06/26/integration-of-amazon-redshift-cloud-data-warehouse-aws-saas-dwh-with-talend-data-integration-di-big-data-bd-enterprise-service-bus-esb/
June 27, 2013
by Kai Wähner DZone Core CORE
· 20,740 Views · 1 Like
article thumbnail
Android Geofencing with Google Maps
a geofence is a virtual perimeter of interest that can be set up to fire notifications when it is entered or exited, or both. for example, a geofencing app can alert us that our kid has left a previously specified area, or send us a coupon (e.g. the "present this sms an get 20% off" offer type) when we happen to walk or drive in the proximity of a movie theater. now, with the new location apis , google's location algorithm has been rewritten to be more accurate and use significantly less battery life. there is just enough documentation plus sample code and a downloadable sample app ( geofencedetection ) to help us get started creating geofencing apps. prerequisites are: download google play services (via android's sdk manager) and set it up as a library get a google maps v2 api key, and maybe run the sample app. this short quick start guide might help download (again via sdk manager) the support library to cater to older android versions. for practical purposes, let's just start where the sample geofencing app ( geofencedetection ) stops, and introduce a few enhancements to make the app semi-decent and show a sample of possibilities with the new location api. 1. zoom and camera position first, a little taste of google maps api v2. let's choose zoom level and camera angle: import com.google.android.gms.maps.googlemap; import com.google.android.gms.maps.cameraupdatefactory; import com.google.android.gms.maps.model.cameraposition; import com.google.android.gms.maps.model.latlng; //... // inside class, for a given lat/lon cameraposition init = new cameraposition.builder() .target(new latlng(lat, lon)) .zoom( 17.5f ) .bearing( 300f) // orientation .tilt( 50f) // viewing angle .build(); // use googglemap mmap to move camera into position mmap.animatecamera( cameraupdatefactory.newcameraposition(init) ); the code above has a zoom level allowing the viewing of buildings in 3d. google maps v2 uses opengl for embedded systems ( opengl es v2) to render 2d and 3d computer graphics. 2. options menu even if we are not big fans of an options menu, it might be adequate in this case, since we would not want to clutter the map with too much "touch" functionality (we will have plenty of that shortly). we can toggle between "normal" and satellite view: /** * toggle view satellite-normal */ public static void toggleview(){ mmap.setmaptype( mmap.getmaptype() == googlemap.map_type_normal ? googlemap.map_type_satellite : googlemap.map_type_normal); } we can also provide a "flight mode", where we let the camera scroll away. not tremendously useful, but kind of cool nonetheless: import com.google.android.gms.maps.googlemap.cancelablecallback; //... private static cancelablecallback callback = new cancelablecallback() { @override public void onfinish() { scroll(); } @override public void oncancel() {} }; public static void scroll() { // we don't want to scroll too fast since // loading new areas in map takes time mmap.animatecamera( cameraupdatefactory.scrollby(10, -10), callback ); // 10 pix } 3. geocoding/reverse geocoding the sample is here to demonstrate features and makes heavy use of latitude/longitude coordinates. but we need to provide a more user-friendly way to interface with locations on the map, like a street address. we can use geocoding/reverse geocoding to transform a street address to coordinates and vice-versa using android's geocoder . 4. adding geofences ok, now on to geofences. thanks to geocoding, we can request an actual physical address from the user instead of coordinates. we will just change that address to a latitude/longitude pair internally to process user input. notice how we use transparent uis as much as possible to enhance what some might call the user experience. notice also that we provide a spinner so that the user can choose between predefined values. that saves the user some typing and it saves us from validating coordinates values each time. still, if we want to be even more user-friendly, we can give our users the possibility to pre-fill the address field by long-pressing a point on the map. we will then use reverse geocoding to translate the coordinates to a physical address for display (screen below on the right): processing long-presses is pretty straightforward: import com.google.android.gms.maps.googlemap.onmaplongclicklistener; //... public class mainactivity extends fragmentactivity implements onmaplongclicklistener { //... mmap.setonmaplongclicklistener(this); //... @override public void onmaplongclick(latlng point) { // reverse geocode point } } adding /removing geofences is pretty much covered in the sample app (by the geofencerequester and geofenceremover classes). the thing to remember is that the process of adding/removing fences is as follows : a connection to google's location services is requested by our app. once/if the connection is available, the request to add/remove a fence is done using a pendingintent . if a similar request made by our app is still underway, the operation fails. although the method we call (e.g. addgeofences() ) returns at once, we won't know if the request was successful until location services calls back into our app (e.g. onaddgeofencesresultlistener 's onaddgeofencesresult() ) with a success status code. finally, the preceding method will use a broadcast intent to notify other components of our app of success/failure. needless to say, we need to code defensively at almost every step of the way. now, once a geofence is added, we can add a marker (the default or a customized one) and choose between different shapes (circle, polygon, etc.) to delimit the geofence. for instance we can write this code to add the default marker and circle the fence within a specified radius: import com.google.android.gms.maps.model.circle; import com.google.android.gms.maps.model.circleoptions; import com.google.android.gms.maps.model.markeroptions; //... public static void addmarkerforfence(simplegeofence fence){ if(fence == null){ // display en error message and return return; } mmap.addmarker( new markeroptions() .position( new latlng(fence.getlatitude(), fence.getlongitude()) ) .title("fence " + fence.getid()) .snippet("radius: " + fence.getradius()) ).showinfowindow(); //instantiates a new circleoptions object + center/radius circleoptions circleoptions = new circleoptions() .center( new latlng(fence.getlatitude(), fence.getlongitude()) ) .radius( fence.getradius() ) .fillcolor(0x40ff0000) .strokecolor(color.transparent) .strokewidth(2); // get back the mutable circle circle circle = mmap.addcircle(circleoptions); // more operations on the circle... } here are the resulting screens, including the one we get once we "touch to edit" the info window: the sample app has all we need to fire notifications once the circled area above is entered or exited. notice how we set up the marker's info window to allow editing the geofence radius, or removing the geofence altogether. to implement a clickable custom info window, all we need is to create our own infowindowadapter and oninfowindowclicklistener . as for the notifications themselves, this is how they look like in the sample app: we can of course change a notification's appearance and functionality, and... that would be the subject of another article. hopefully, this one gave a glimpse of what is possible with the new location api. have fun with android geofences.
June 26, 2013
by Tony Siciliani
· 78,495 Views
article thumbnail
CDI | @Default and @Inject Annotations
cdi (context and dependency injection) is a complete and lightweight injection technology designed for java ee environment. special container objects (ejb,entitymanager), primitive data type elements and java class/objects written by you can be easily managed and injected as well through cdi. every defined java class in each application that configured in cdi standard is a candidate to become an injectable cdi object. this default behavior is provided by @default annotation that was installed per each java class secretly. there is an car class which has a vehicle implementation in the above uml diagram. a random int value is produced in sayvelocity() method and there is an output to the console such as “the car is running at the speed of x” in work() method, where x is represents a number produced randomly. @default // optional public class car implements vehicle { public string work() { return "car is working in "+ sayvelocity()+" kmh."; } public int sayvelocity(){ return threadlocalrandom.current().nextint(20, 240) ; } } if the above car class is in an application activated in cdi environment, it becomes a candidate to be an object managed by cdi. so, what is implied by the activation of cdi? first of all, necessary dependencies need to be included in the classpath for the activation of cdi environment. if you are using an application server like glassfish, cdi can be used without any extra definition because of the existence of cdi libraries on the application server. but if your application is a java se application or is running in lightweight containers such as tomcat, jetty; a cdi library must be added to the project. the reference library of cdi technology is the jboss weld archetype. for this reason, if jboss weld dependencies are added to the project such as the following, first phase of the cdi activation is realized. org.jboss.weld.se weld-se 1.1.10.final to activate the cdi environment, a blank cdi configuration file named beans.xml must be in the application as a requirement of the standard. this file must be located on the /meta-inf/beans.xml path for java se applications and /web-inf/beans.xml path for java ee web applications. the existence necessity of this file may sound silly initially, but the existence of beans.xml in the directories specified above can be considered as a permission given to the container in order to activate cdi environment. activation of cdi environment is automatically started in java ee web applications when beans.xml file is encountered in the /web-inf directory. but it is not the same for a java se application, starting a cdi container is the developer’s job. this case can be seen clearly if you pay attention to the following gallery class: public class gallery { @inject // injection point private vehicle vehicle; public static void main(string[] args) { weld weld = new weld(); weldcontainer weldcontainer = weld.initialize(); gallery gallery = weldcontainer.instance().select(gallery.class).get(); string message= gallery.vehicle.work(); system.out.println("> "+message); } } weldcontainer type object reference in gallery class access to a cdi object which represents the initiated container environment and after this point, cdi objects are accessible and can be made injection procedures through weldcontainer object. after this point, it is used by adopting a cdi object that is a gallery class type through the container instance. in here, a programmatic access to the gallery object is provided. programmatic access is required for the startup of java se applications (as in spring), but you can easily access to all cdi objects in the environment also by annotation-based injection method through the obtained gallery cdi object, e.g. [ private @inject vehicle vehicle; ] an output as the following occurs when the gallery class is being run; the real content and sample code can be accessed in http://en.kodcu.com/2013/06/cdi-default-and-inject-annotations/ hope to see you again..
June 25, 2013
by Altuğ Altıntaş
· 33,959 Views
article thumbnail
Mockito - Extra Interfaces with Annotations and Static Methods
In the code I have quite recently came across a really bad piece of code that based on class casting in terms of performing some actions on objects. Of course the code needed to be refactored but sometimes you can't do it / or don't want to do it (and it should be understandable) if first you don't have unit tests of that functionality. In the following post I will show how to test such code, how to refactor it and in fact what I think about such code ;) Let's take a look at the project structure: As presented in the post regarding Mocktio RETURNS_DEEP_STUBS Answer for JAXB yet again we have the JAXB generated classes by the JAXB compiler in thecom.blogspot.toomuchcoding.model package. Let's ommit the discussion over the pom.xml file since it's exactly the same as in the previous post. In the com.blogspot.toomuchcoding.adapter package we have adapters over the JAXB PlayerDetails class that provides access to the Player interface. There is the CommonPlayerAdapter.java package com.blogspot.toomuchcoding.adapter; import com.blogspot.toomuchcoding.model.Player; import com.blogspot.toomuchcoding.model.PlayerDetails; /** * User: mgrzejszczak * Date: 09.06.13 * Time: 15:42 */ public class CommonPlayerAdapter implements Player { private final PlayerDetails playerDetails; public CommonPlayerAdapter(PlayerDetails playerDetails){ this.playerDetails = playerDetails; } @Override public void run() { System.out.printf("Run %s. Run!%n", playerDetails.getName()); } public PlayerDetails getPlayerDetails() { return playerDetails; } } DefencePlayerAdapter.java package com.blogspot.toomuchcoding.adapter; import com.blogspot.toomuchcoding.model.DJ; import com.blogspot.toomuchcoding.model.DefensivePlayer; import com.blogspot.toomuchcoding.model.JavaDeveloper; import com.blogspot.toomuchcoding.model.PlayerDetails; /** * User: mgrzejszczak * Date: 09.06.13 * Time: 15:42 */ public class DefencePlayerAdapter extends CommonPlayerAdapter implements DefensivePlayer, DJ, JavaDeveloper { public DefencePlayerAdapter(PlayerDetails playerDetails){ super(playerDetails); } @Override public void defend(){ System.out.printf("Defence! %s. Defence!%n", getPlayerDetails().getName()); } @Override public void playSomeMusic() { System.out.println("Oops I did it again...!"); } @Override public void doSomeSeriousCoding() { System.out.println("System.out.println(\"Hello world\");"); } } OffensivePlayerAdapter.java package com.blogspot.toomuchcoding.adapter; import com.blogspot.toomuchcoding.model.OffensivePlayer; import com.blogspot.toomuchcoding.model.PlayerDetails; /** * User: mgrzejszczak * Date: 09.06.13 * Time: 15:42 */ public class OffensivePlayerAdapter extends CommonPlayerAdapter implements OffensivePlayer { public OffensivePlayerAdapter(PlayerDetails playerDetails){ super(playerDetails); } @Override public void shoot(){ System.out.printf("%s Shooooot!.%n", getPlayerDetails().getName()); } } Ok, now let's go to the more interesting part. Let us assume that we have a very simple factory of players: PlayerFactoryImpl.java package com.blogspot.toomuchcoding.factory; import com.blogspot.toomuchcoding.adapter.CommonPlayerAdapter; import com.blogspot.toomuchcoding.adapter.DefencePlayerAdapter; import com.blogspot.toomuchcoding.adapter.OffensivePlayerAdapter; import com.blogspot.toomuchcoding.model.Player; import com.blogspot.toomuchcoding.model.PlayerDetails; import com.blogspot.toomuchcoding.model.PositionType; /** * User: mgrzejszczak * Date: 09.06.13 * Time: 15:53 */ public class PlayerFactoryImpl implements PlayerFactory { @Override public Player createPlayer(PositionType positionType) { PlayerDetails player = createCommonPlayer(positionType); switch (positionType){ case ATT: return new OffensivePlayerAdapter(player); case MID: return new OffensivePlayerAdapter(player); case DEF: return new DefencePlayerAdapter(player); case GK: return new DefencePlayerAdapter(player); default: return new CommonPlayerAdapter(player); } } private PlayerDetails createCommonPlayer(PositionType positionType){ PlayerDetails playerDetails = new PlayerDetails(); playerDetails.setPosition(positionType); return playerDetails; } } Ok so we have the factory that builds Players. Let's take a look at the Service that uses the factory: PlayerServiceImpl.java package com.blogspot.toomuchcoding.service; import com.blogspot.toomuchcoding.factory.PlayerFactory; import com.blogspot.toomuchcoding.model.*; /** * User: mgrzejszczak * Date: 08.06.13 * Time: 19:02 */ public class PlayerServiceImpl implements PlayerService { private PlayerFactory playerFactory; @Override public Player playAGameWithAPlayerOfPosition(PositionType positionType) { Player player = playerFactory.createPlayer(positionType); player.run(); performAdditionalActions(player); return player; } private void performAdditionalActions(Player player) { if(player instanceof OffensivePlayer){ OffensivePlayer offensivePlayer = (OffensivePlayer) player; performAdditionalActionsForTheOffensivePlayer(offensivePlayer); }else if(player instanceof DefensivePlayer){ DefensivePlayer defensivePlayer = (DefensivePlayer) player; performAdditionalActionsForTheDefensivePlayer(defensivePlayer); } } private void performAdditionalActionsForTheOffensivePlayer(OffensivePlayer offensivePlayer){ offensivePlayer.shoot(); } private void performAdditionalActionsForTheDefensivePlayer(DefensivePlayer defensivePlayer){ defensivePlayer.defend(); try{ DJ dj = (DJ)defensivePlayer; dj.playSomeMusic(); JavaDeveloper javaDeveloper = (JavaDeveloper)defensivePlayer; javaDeveloper.doSomeSeriousCoding(); }catch(ClassCastException exception){ System.err.println("Sorry, I can't do more than just play football..."); } } public PlayerFactory getPlayerFactory() { return playerFactory; } public void setPlayerFactory(PlayerFactory playerFactory) { this.playerFactory = playerFactory; } } Let's admit it... this code is bad. Internally when you look at it (regardless of the fact whether it used instance of operator or not) you feel that it is evil :) As you can see in the code we have some class casts going on... How on earth can we test it? In the majority of testing frameworks you can't do such class casts on mocks since they are built with the CGLIB library and there can be some ClassCastExceptions thrown. You could still not return mocks and real implementations (assuming that those will not perform any ugly stuff in the construction process) and it could actually work but still - this is bad code :P Mockito comes to the rescue (although you shouldn't overuse this feature - in fact if you need to use it please consider refactoring it) with its extraInterfaces feature: extraInterfaces MockSettings extraInterfaces(java.lang.Class... interfaces) Specifies extra interfaces the mock should implement. Might be useful for legacy code or some corner cases. For background, see issue 51 hereThis mysterious feature should be used very occasionally. The object under test should know exactly its collaborators & dependencies. If you happen to use it often than please make sure you are really producing simple, clean & readable code. Examples: Foo foo = mock(Foo.class, withSettings().extraInterfaces(Bar.class, Baz.class)); //now, the mock implements extra interfaces, so following casting is possible: Bar bar = (Bar) foo; Baz baz = (Baz) foo; Parameters:interfaces - extra interfaces the should implement. Returns:settings instance so that you can fluently specify other settings Now let's take a look at the test: PlayerServiceImplTest.java package com.blogspot.toomuchcoding.service; import com.blogspot.toomuchcoding.factory.PlayerFactory; import com.blogspot.toomuchcoding.model.*; import org.junit.Test; import org.junit.runner.RunWith; import org.mockito.InjectMocks; import org.mockito.Mock; import org.mockito.invocation.InvocationOnMock; import org.mockito.runners.MockitoJUnitRunner; import org.mockito.stubbing.Answer; import static org.hamcrest.CoreMatchers.is; import static org.junit.Assert.assertThat; import static org.mockito.BDDMockito.*; /** * User: mgrzejszczak * Date: 08.06.13 * Time: 19:26 */ @RunWith(MockitoJUnitRunner.class) public class PlayerServiceImplTest { @Mock PlayerFactory playerFactory; @InjectMocks PlayerServiceImpl objectUnderTest; @Mock(extraInterfaces = {DJ.class, JavaDeveloper.class}) DefensivePlayer defensivePlayerWithDjAndJavaDevSkills; @Mock DefensivePlayer defensivePlayer; @Mock OffensivePlayer offensivePlayer; @Mock Player commonPlayer; @Test public void shouldReturnOffensivePlayerThatRan() throws Exception { //given given(playerFactory.createPlayer(PositionType.ATT)).willReturn(offensivePlayer); //when Player createdPlayer = objectUnderTest.playAGameWithAPlayerOfPosition(PositionType.ATT); //then assertThat(createdPlayer == offensivePlayer, is(true)); verify(offensivePlayer).run(); } @Test public void shouldReturnDefensivePlayerButHeWontBeADjNorAJavaDev() throws Exception { //given given(playerFactory.createPlayer(PositionType.GK)).willReturn(defensivePlayer); //when Player createdPlayer = objectUnderTest.playAGameWithAPlayerOfPosition(PositionType.GK); //then assertThat(createdPlayer == defensivePlayer, is(true)); verify(defensivePlayer).run(); verify(defensivePlayer).defend(); verifyNoMoreInteractions(defensivePlayer); } @Test public void shouldReturnDefensivePlayerBeingADjAndAJavaDev() throws Exception { //given given(playerFactory.createPlayer(PositionType.GK)).willReturn(defensivePlayerWithDjAndJavaDevSkills); doAnswer(new Answer
June 12, 2013
by Marcin Grzejszczak
· 21,931 Views · 2 Likes
article thumbnail
Mockito - RETURNS_DEEP_STUBS for JAXB
Sorry for not having written for some time but I was busy with writing the JBoss Drools Refcard for DZone and I am in the middle of writing a book about Mockito so I don't have too much time left for blogging... Anyway quite recently on my current project I had an interesting situation regarding unit testing with Mockito and JAXB structures. We have very deeply nested JAXB structures generated from schemas that are provided for us which means that we can't change it in anyway. Let's take a look at the project structure: The project structure is pretty simple - there is a Player.xsd schema file that thanks to using the jaxb2-maven-plugin produces the generated JAXB Java classes corresponding to the schema in the target/jaxb/ folder in the appropriate package that is defined in the pom.xml. Speaking of which let's take a look at the pom.xml file. The pom.xml : 4.0.0 com.blogspot.toomuchcoding mockito-deep_stubs 0.0.1-SNAPSHOT UTF-8 1.6 1.6 spring-release http://maven.springframework.org/release maven-us-nuxeo https://maven-us.nuxeo.org/nexus/content/groups/public junit junit 4.10 org.mockito mockito-all 1.9.5 test org.apache.maven.plugins maven-compiler-plugin 2.5.1 org.codehaus.mojo jaxb2-maven-plugin 1.5 xjc xjc com.blogspot.toomuchcoding.model ${project.basedir}/src/main/resources/xsd Apart from the previously defined project dependencies, as mentioned previously in the jaxb2-maven-plugin in the configuration node you can define the packageName value that defines to which package should the JAXB classes be generated basing on the schemaDirectory value where the plugin can find the proper schema files. Speaking of which let's check the Player.xsd schema file (simillar to the one that was present in the Spring JMS automatic message conversion article of mine): As you can see I'm defining some complex types that even though might have no business sense but you can find such examples in the real life :) Let's find out how the method that we would like to test looks like. Here we have the PlayerServiceImpl that implements the PlayerService interface: package com.blogspot.toomuchcoding.service; import com.blogspot.toomuchcoding.model.PlayerDetails; /** * User: mgrzejszczak * Date: 08.06.13 * Time: 19:02 */ public class PlayerServiceImpl implements PlayerService { @Override public boolean isPlayerOfGivenCountry(PlayerDetails playerDetails, String country) { String countryValue = playerDetails.getClubDetails().getCountry().getCountryCode().getCountryCode().value(); return countryValue.equalsIgnoreCase(country); } } We are getting the nested elements from the JAXB generated classes. Although it violates the Law of Demeter it is quite common to call methods of structures because JAXB generated classes are in fact structures so in fact I fully agree with Martin Fowler that it should be called the Suggestion of Demeter. Anyway let's see how you could test the method: @Test public void shouldReturnTrueIfCountryCodeIsTheSame() throws Exception { //given PlayerDetails playerDetails = new PlayerDetails(); ClubDetails clubDetails = new ClubDetails(); CountryDetails countryDetails = new CountryDetails(); CountryCodeDetails countryCodeDetails = new CountryCodeDetails(); playerDetails.setClubDetails(clubDetails); clubDetails.setCountry(countryDetails); countryDetails.setCountryCode(countryCodeDetails); countryCodeDetails.setCountryCode(CountryCodeType.ENG); //when boolean playerOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); //then assertThat(playerOfGivenCountry, is(true)); } The function checks if, once you have the same Country Code, you get a true boolean from the method. The only problem is the amount of sets and instantiations that take place when you want to create the input message. In our projects we have twice as many nested elements so you can only imagine the number of code that we would have to produce to create the input object... So what can be done to improve this code? Mockito comes to the rescue to together with the RETURN_DEEP_STUBS default answer to the Mockito.mock(...) method: @Test public void shouldReturnTrueIfCountryCodeIsTheSameUsingMockitoReturnDeepStubs() throws Exception { //given PlayerDetails playerDetailsMock = mock(PlayerDetails.class, RETURNS_DEEP_STUBS); CountryCodeType countryCodeType = CountryCodeType.ENG; when(playerDetailsMock.getClubDetails().getCountry().getCountryCode().getCountryCode()).thenReturn(countryCodeType); //when boolean playerOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetailsMock, COUNTRY_CODE_ENG); //then assertThat(playerOfGivenCountry, is(true)); } So what happened here is that you use the Mockito.mock(...) method and provide the RETURNS_DEEP_STUBS answer that will create mocks automatically for you. Mind you that Enums can't be mocked that's why you can't write in the Mockito.when(...) functionplayerDetailsMock.getClubDetails().getCountry().getCountryCode().getCountryCode().getValue(). Summing it up you can compare the readability of both tests and see how clearer it is to work with JAXB structures by using Mockito RETURNS_DEEP_STUBS default answer. Naturally sources for this example are available at BitBucket and GitHub.
June 11, 2013
by Marcin Grzejszczak
· 10,140 Views · 1 Like
article thumbnail
Serialization and injection
Serialization is a form of persistence: serialized data survives the process and the RAM where it was created and can be reconstituted inside different processes and machines that live in a different time or place. Sometimes serialization is a poor form of persistence in fact, one that confuses the boundary between the different schemas the data can fit in. However, what I found useful in the last years of development is to institute a strict separation: serialize Value Objects, Entities, and everything that represents the state of the application. Meanwhile, use Dependency Injection over services that are part of a larger object graph and never serialize this second kind of objects. In the discussion that follows, I make the assumption that serialization and deserialization occur on the same machine (e.g. like for web-oriented sessions.) The problem with serialization, which work transparently most of the time, is the need to serialize service objects instead of limiting the procedure to data structures. How can you store such objects? Not options Some options to solve this problems are really not options. Serialization by itself will fail because of the staleness of the references contained in these objects. For example, in PHP trying to serialize a database connections composed by a Repository or DAO object will rightly fail with an exception. Whenever an object represents a resource of the current machine, it cannot usually be serialized except in the case when the only resource involved is RAM. If the resource is disk space or other running processes such as a database daemon, the reconstitution of the object in another place and time will fail and it's best to just stop the developer immediately during storage. Quasi-options Some solutions to the problem try to avoid the staleness problem by serializing objects without their resources, and make them regrab a new version of them on deserialization. In PHP for example, this can be done with the __sleep() and __wakeup() magic methods, called automatically during serialization and deserializaton respectively. This deserialization mechanism introduces a dependency from the serialized Entity to external services: such a dependency is already in place when building the object the first time (passing the XService in the constructor) but it is aggravated when deserializing (depending on a XServiceFactory instead of just an XService). An improvement, from the dependencies point of view, is to reattach collaborators to deserialized objects like you would for other persistence-related tasks. For example, EntityRepository can inject the missing pieces of Entity every time its find() method is called. However, there is still another option, which is the most resilient from the modelling point of view and not only that of dependency management: injecting non-serializable collaborators through the stack. Objects can collaborate even without keeping field references to each other, and injecting dependencies as parameters move the dependency starting point from the server to the client object (which may or may not be desirable). What is most important is that Entities are relieved of having to manage external references in any context, not only that of persistence and in particular serialization. The metaphor for the 3rd option Misko Hevery likes to say: have you ever seen a credit card able to charge itself? If a CreditCard is an Entity in your domain, it would be very strange to keeping a wire attached to your wallet wherever you go. With the first option, you have the card spring a wire when it is taken out of the wallet, like in horror movies. This intelligent cable tries as its best to attach to the nearest Point of Sale (a bad case of bluetooth I think). With Repositories in mind, you're not dealing with automated wires anymore, but you're still attaching cables between cards and fixed devices. In reality, cards collaborate with the PoS in a fast process that does not last more than a few seconds. Actually, sometimes they don't touch it at all, as in all Internet-based purchases. Keeping services around to deal with external dependencies does not mean the API of your Domain Model has to be biased towards service objects: pos.charge(creditCard); // can equivalently be: creditCard.chargeOn(pos); This is a form of Double Dispatch since there are two objects collaborating and you can dispatch (send messages) to both, being polimorphic by substituting both objects. The sequence of calls is: client -> creditCard -> pos The client object still looks at CreditCard as a behaviorally complete object, but it is clear which dependency is necessary to run each use case (CreditCard method). You can persist a CreditCard easily and send it over the wire to caches or databases. When it comes the time to charge, it is the client that has to bring forward a service able to connect to a bank.
June 5, 2013
by Giorgio Sironi
· 7,397 Views
article thumbnail
Amazon S3 Parallel MultiPart File Upload
In this blog post, I will present a simple tutorial on uploading a large file to Amazon S3 as fast as the network supports. Amazon S3 is clustered storage service of Amazon. It is designed to make web-scale computing easier. Amazon S3 provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, secure, fast, inexpensive infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers. For using Amazon services, you'll need your AWS access key identifiers, which AWS assigned you when you created your AWS account. The following are the AWS access key identifiers: Access Key ID (a 20-character, alphanumeric sequence) For example: 022QF06E7MXBSH9DHM02 Secret Access Key (a 40-character sequence) For example: kWcrlUX5JEDGM/LtmEENI/aVmYvHNif5zB+d9+ct Caution Your Secret Access Key is a secret, which only you and AWS should know. It is important to keep it confidential to protect your account. Store it securely in a safe place. Never include it in your requests to AWS, and never e-mail it to anyone. Do not share it outside your organization, even if an inquiry appears to come from AWS or Amazon.com. No one who legitimately represents Amazon will ever ask you for your Secret Access Key. The Access Key ID is associated with your AWS account. You include it in AWS service requests to identify yourself as the sender of the request. The Access Key ID is not a secret, and anyone could use your Access Key ID in requests to AWS. To provide proof that you truly are the sender of the request, you also include a digital signature calculated using your Secret Access Key. The sample code handles this for you. Your Access Key ID and Secret Access Key are displayed to you when you create your AWS account. They are not e-mailed to you. If you need to see them again, you can view them at any time from your AWS account. To get your AWS access key identifiers Go to the Amazon Web Services web site at http://aws.amazon.com. Point to Your Account and click Security Credentials. Log in to your AWS account. The Security Credentials page is displayed. Your Access Key ID is displayed in the Access Identifiers section of the page. To display your Secret Access Key, click Show in the Secret Access Key column. You can use your Amazon keys from a properties file in your application. Here is a sample for properties file containing Amazon keys: # Fill in your AWS Access Key ID and Secret Access Key # http://aws.amazon.com/security-credentials accessKey = secretKey = Here is sample AmazonUtil class for getting AWS Credentials from properties file. public class AmazonUtil { private static final Logger logger = LogUtil.getLogger(); private static final String AWS_CREDENTIALS_CONFIG_FILE_PATH = ConfigUtil.CONFIG_DIRECTORY_PATH + File.separator + "aws-credentials.properties"; private static AWSCredentials awsCredentials; static { init(); } private AmazonUtil() { } private static void init() { try { awsCredentials = new PropertiesCredentials(IOUtil.getResourceAsStream(AWS_CREDENTIALS_CONFIG_FILE_PATH)); } catch (IOException e) { logger.error("Unable to initialize AWS Credentials from " + AWS_CREDENTIALS_CONFIG_FILE_PATH); } } public static AWSCredentials getAwsCredentials() { return awsCredentials; } } Amazon S3 has Multipart Upload service which allows faster, more flexible uploads into Amazon S3. Multipart Upload allows you to upload a single object as a set of parts. After all parts of your object are uploaded, Amazon S3 then presents the data as a single object. With this feature you can create parallel uploads, pause and resume an object upload, and begin uploads before you know the total object size. For more information on Multipart Upload, review the Amazon S3 Developer Guide In this tutorial, my sample application uploads each file parts to Amazon S3 with different threads for using network throughput as possible as much. Each file part is associated with a thread and each thread uploads its associated part with Amazon S3 API. Figure 1. Amazon S3 Parallel Multi-Part File Upload Mechanism Amazon S3 API suppots MultiPart File Upload in this way: 1. Send a MultipartUploadRequest to Amazon. 2. Get a response containing a unique id for this upload operation. 3. For i in ${partCount} 3.1. Calculate size and offset of split-i in whole file. 3.2. Build a UploadPartRequest with file offset, size of current split and unique upload id. 3.3. Give this request to a thread and starts upload by running thread. 3.3.1. Send associated UploadPartRequest to Amazon. 3.3.2. Get response after successful upload and save ETag property of response. 4. Wait all threads to terminate 5. Get ETags (ETag is an identifier for successfully completed uploads) of all terminated threads. 6. Send a CompleteMultipartUploadRequest to Amazon with unique upload id and all ETags. So Amazon joins all file parts as target objects. Here is implementation: public class AmazonS3Util { private static final Logger logger = LogUtil.getLogger(); public static final long DEFAULT_FILE_PART_SIZE = 5 * 1024 * 1024; // 5MB public static long FILE_PART_SIZE = DEFAULT_FILE_PART_SIZE; private static AmazonS3 s3Client; private static TransferManager transferManager; static { init(); } private AmazonS3Util() { } private static void init() { // ... s3Client = new AmazonS3Client(AmazonUtil.getAwsCredentials()); transferManager = new TransferManager(AmazonUtil.getAwsCredentials()); } // ... public static void putObjectAsMultiPart(String bucketName, File file) { putObjectAsMultiPart(bucketName, file, FILE_PART_SIZE); } public static void putObjectAsMultiPart(String bucketName, File file, long partSize) { List partETags = new ArrayList(); List uploaders = new ArrayList(); // Step 1: Initialize. InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(bucketName, file.getName()); InitiateMultipartUploadResult initResponse = s3Client.initiateMultipartUpload(initRequest); long contentLength = file.length(); try { // Step 2: Upload parts. long filePosition = 0; for (int i = 1; filePosition < contentLength; i++) { // Last part can be less than part size. Adjust part size. partSize = Math.min(partSize, (contentLength - filePosition)); // Create request to upload a part. UploadPartRequest uploadRequest = new UploadPartRequest(). withBucketName(bucketName).withKey(file.getName()). withUploadId(initResponse.getUploadId()).withPartNumber(i). withFileOffset(filePosition). withFile(file). withPartSize(partSize); uploadRequest.setProgressListener(new UploadProgressListener(file, i, partSize)); // Upload part and add response to our list. MultiPartFileUploader uploader = new MultiPartFileUploader(uploadRequest); uploaders.add(uploader); uploader.upload(); filePosition += partSize; } for (MultiPartFileUploader uploader : uploaders) { uploader.join(); partETags.add(uploader.getPartETag()); } // Step 3: complete. CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest(bucketName, file.getName(), initResponse.getUploadId(), partETags); s3Client.completeMultipartUpload(compRequest); } catch (Throwable t) { logger.error("Unable to put object as multipart to Amazon S3 for file " + file.getName(), t); s3Client.abortMultipartUpload( new AbortMultipartUploadRequest( bucketName, file.getName(), initResponse.getUploadId())); } } // ... private static class UploadProgressListener implements ProgressListener { File file; int partNo; long partLength; UploadProgressListener(File file) { this.file = file; } @SuppressWarnings("unused") UploadProgressListener(File file, int partNo) { this(file, partNo, 0); } UploadProgressListener(File file, int partNo, long partLength) { this.file = file; this.partNo = partNo; this.partLength = partLength; } @Override public void progressChanged(ProgressEvent progressEvent) { switch (progressEvent.getEventCode()) { case ProgressEvent.STARTED_EVENT_CODE: logger.info("Upload started for file " + "\"" + file.getName() + "\""); break; case ProgressEvent.COMPLETED_EVENT_CODE: logger.info("Upload completed for file " + "\"" + file.getName() + "\"" + ", " + file.length() + " bytes data has been transferred"); break; case ProgressEvent.FAILED_EVENT_CODE: logger.info("Upload failed for file " + "\"" + file.getName() + "\"" + ", " + progressEvent.getBytesTransfered() + " bytes data has been transferred"); break; case ProgressEvent.CANCELED_EVENT_CODE: logger.info("Upload cancelled for file " + "\"" + file.getName() + "\"" + ", " + progressEvent.getBytesTransfered() + " bytes data has been transferred"); break; case ProgressEvent.PART_STARTED_EVENT_CODE: logger.info("Upload started at " + partNo + ". part for file " + "\"" + file.getName() + "\""); break; case ProgressEvent.PART_COMPLETED_EVENT_CODE: logger.info("Upload completed at " + partNo + ". part for file " + "\"" + file.getName() + "\"" + ", " + (partLength > 0 ? partLength : progressEvent.getBytesTransfered()) + " bytes data has been transferred"); break; case ProgressEvent.PART_FAILED_EVENT_CODE: logger.info("Upload failed at " + partNo + ". part for file " + "\"" + file.getName() + "\"" + ", " + progressEvent.getBytesTransfered() + " bytes data has been transferred"); break; } } } private static class MultiPartFileUploader extends Thread { private UploadPartRequest uploadRequest; private PartETag partETag; MultiPartFileUploader(UploadPartRequest uploadRequest) { this.s3Client = s3Client; this.uploadRequest = uploadRequest; } @Override public void run() { partETag = s3Client.uploadPart(uploadRequest).getPartETag(); } private PartETag getPartETag() { return partETag; } private void upload() { start(); } } }
May 28, 2013
by Serkan Özal
· 57,530 Views · 3 Likes
article thumbnail
7 Agile Best Practices that You Don’t Need to Follow
There are many good ideas and practices in Agile development, ideas and practices that definitely work: breaking projects into Small Releases to manage risk and accelerate feedback; time-boxing to limit WIP and keep everyone focused; relying only on working software as the measure of progress; simple estimating and using velocity to forecast team performance; working closely and constantly with the customer; and Continuous Integration – and Continuous Delivery – to ensure that code is always working and stable. But there are other commonly accepted ideas and best practices that aren’t important: if you don’t follow them, nothing bad will happen to you and your project will still succeed. And there are a couple that you are better off not following at all. Test-Driven Development Teams that need to move quickly need to depend on a fast, efficient testing safety net. With Test First Development or Test-Driven Development (TDD), there’s no excuse for not writing tests – after all, you have to write a failing test before you write the code. So you end up with a good set of working automated tests that ensure a high level of coverage and regression protection. TDD is not only a way of ensuring that developers test their code. It is also advocated as a design technique that leads to better quality code and a simpler, cleaner design. A study of teams at Microsoft and IBM (Realizing Quality Improvement through Test Driven Development, Microsoft Research, 2008) found that while TDD increased upfront development costs between 15-35% (TDD demands developers change the way that they think and work, which slows developers down, at least at first), it reduced defect density by 40% (IBM) or as much as 60-90% (Microsoft) over teams that did not follow disciplined unit testing. But in Making Software Chapter 12 “How Effective is Test-Driven Development” researchers led by Burak Turhan found that while TDD improves external quality (measured by one or more of test cases passed, number of defects, defect density, defects per test, effort required to fix defects, change density, % of preventative changes) and can improve the quality of the tests (fewer mistakes in the tests, tests that are easier to maintain), TDD does not consistently improve the quality of the design. TDD seems to reduce code complexity and improve reuse, however it also negatively impacts coupling and cohesion. And while method and class-level complexity is better in code developed using TDD, project/package level complexity is worse. People who like TDD like it a lot, so if you like it, do it. And even if you are not TDD-infected, there are times when working test first is natural – when you have to solve a specific problem in a specific way, or if you’re fixing a bug where the failing test case is already written up for you. But the important thing is that you write a good set of tests and keep them up to date and run them frequently – it doesn't matter if you write them before, or after, you write the code. Pair Programming According to the VersionOne State of Agile Development Survey 2012, almost 1/3 of teams follow pair programming – a surprisingly high number, given how disciplined pair programming is, and how few teams follow XP (2%) or Scrum/XP Hybrid (11%) methods where pair programming would be prescribed. There are good reasons for pairing: information sharing and improving code quality through continuous, informal code reviews as developers work together. And there are natural times to pair developers, or sometimes developers and testers, together: when you’re working through a hard design problem; or on code that you’ve never seen before and somebody who has worked on it is available to help; or when you’re over your head in troubleshooting a high-pressure problem; or testing a difficult part of the system; or when a new person joins the team and needs to learn about the code and coding practices. Some (extroverted) people enjoy pairing up, the energy it creates and the opportunities it provides to get to know others on the team. But forcing people who prefer working on their own or who don’t like each other to work closely together is definitely not a good idea. There are real social costs in pairing: you have to be careful to pair people up by skill, experience, style, personality type and work ethic. And sustained pair programming can be exhausting, especially over the long term – one study (Vanhanen and Lassenius 2007) found that people only pair between 1.5 and 4 hours a day on average, because it’s too intense to do all day long. In Pair Programming Considered Harmful? Jon Evans says that pairing can have also negative effects on creativity: Research strongly suggests that people are more creative when they enjoy privacy and freedom from interruption … What distinguished programmers at the top-performing companies wasn’t greater experience or better pay. It was how much privacy, personal workspace and freedom from interruption they enjoyed,” says a New York Times article castigating “the new groupthink”. And in “Still Questioning Extreme Programming” Pete McBreen points out some other disadvantages and weaknesses of pair programming: Exploration of ideas is not encouraged, pairing makes a developer focus on writing the code, so unless there is time in the day for solo exploration the team gets a very superficial level of understanding of the code. Developers can come to rely too much on the unit tests, assuming that if the tests pass then the code is OK. (This follows on from the lack of exploration.) Corner cases and edge cases are not investigated in detail, especially if they are hard to write tests for. Code that requires detail thinking about the design is hard to do when pairing unless one partner completely dominates the session. With the usual tradeoff between partners, it is hard to build technically complex designs unless they have been already been worked out in a solo session. Personal styles matter when pairing, and not all pairings are as productive as others. Pairs with different typing skills and proficiencies often result in the better typist doing all of the coding with the other partner being purely passive. And of course pairing in distributed teams doesn't work well if at all (depending on distance, differences in time zones, culture, working styles, language), although some people still try. While pairing does improve code quality over solo programming, you can get the same improvements in code quality, and at least some of the information sharing advantages, through code reviews, at less cost. Code reviews – especially lightweight, offline reviews – are easier to schedule, less expensive and less intrusive than pairing. And as Jason Cohen points out even if developers are pair programming, you may still need to do code reviews, because pair programming is really about joint problem solving, and doesn’t cover all of the issues that a code review would. Back to Jon Evans for the final word on pair programming: The true answer is that there is no one answer; that what works best is a dynamic combination of solitary, pair, and group work, depending on the context, using your best judgement. Paired programming definitely has its place. (Betteridge’s Law strikes again!) In some cases that place may even be “much of most days.” But insisting on 100 percent pairing is mindless dogma, and like all mindless dogma, ultimately counterproductive. Emergent Design and Metaphor Incremental development works, and trying to keep design simple makes good sense, but attempting to define an architecture on the fly is foolish and impractical. There’s a reason that almost nobody actually follows Emergent Design: it doesn't work. Relying on a high-level metaphor (the system is an "assembly line" or a "bill of materials" or a "hive of bees") shared by the team as some kind of substitute for architecture is even more ridiculous. Research from Carnegie Mellon University found that … natural language metaphors are relatively useless for either fostering communication among technical and non-technical project members or in developing architecture. Almost no one understands what a system metaphor is any ways, or how it is to be used, or how to choose a meaningful metaphor or how to change it if you got it wrong (and how you would know if you got it wrong), including one of the people who helped come up with the idea: Okay I might as well say it publicly - I still haven't got the hang of this metaphor thing. I saw it work, and work well on the C3 project, but it doesn't mean I have any idea how to do it, let alone how to explain how to do it. Martin Fowler, Is Design Dead? Agile development methods have improved development success and shown better ways to approach many different software development problems – but not architecture and design. Daily Standups When you have a new team and everyone needs to get to know each other and more time to understand what the project is about; or when the team is working under emergency conditions trying to fix something or finish something under extreme pressure, then getting everyone together in regular meetings, maybe even more than once a day, is necessary and valuable. But whether everyone stands up or sits down and what they end up talking about in a meeting should be up to you. If your team has been working well together for a while and everyone knows each other and knows what they are working on, and if developers update cards on a task board or a Kanban board or the status in an electronic system as they get things done, and if they are grown up enough to ask for help when they need it, then you don’t need to make them all stand up in a room every morning. Collective Code Ownership Letting everyone work on all of the code isn't always practical (because not everyone on the team has the requisite knowledge or experience to work on every problem) and collective code ownership can have negative effects on code quality. Share code where it makes sense to do so, but realize that not everybody can – or should – work on every part of the system. Writing All Requirements as Stories The idea that every requirement specification can be written as User Stories in 1 or 2 lines on cards, that requirements should be too short on purpose (so that the developer has to talk to someone to explain what’s really needed) and insisting that they should all be in the same template form “As a type of user I want some goal so that some reason…” is silly and unnecessary. This is the same kind of simple minded orthodoxy that led everyone to try to capture all requirements in UML Use Case format with stick men and bubbles 15 years ago. There are many different ways to effectively express requirements. Sometimes requirements need to be specified in detail (when you have to meet regulatory compliance or comply with a standard or integrate with an existing system or implement a specific algorithm or…). Sometimes it’s better to work from a test case or a detailed use case scenario or a wire frame or some other kind of model, because somebody who knows what’s going on has already worked out the details for you. So pick the format and level of detail that works best and get to work. Relying on a Product Owner Relying on one person as the Product Owner, as the single solitary voice of the customer and the “one throat to choke” when the project fails, doesn't scale, doesn't last, and puts the team and the project and eventually the business at risk. It’s a naïve, dangerous approach to designing a product and to managing a development project, and it causes more problems than it solves. Many teams have realized this and are trying to work around the Product Owner idea because they have to. To succeed, a team needs real and sustained customer engagement at multiple levels, and they should take responsibility themselves for making sure that they get what they need, rather than relying on one person to do it all.
May 24, 2013
by Jim Bird
· 49,298 Views
article thumbnail
Azure Blob Storage - "The specified blob or block content is invalid"
If you’re uploading blobs by splitting blobs into blocks and you get the error – The specified blob or block content is invalid, then this post is for you. Short Version If you’re uploading blobs by splitting blobs into blocks and you get the above mentioned error, ensure that your block ids of your blocks are of same length. If the block ids of your blocks are of different length, you’ll get this error. Long Version Now for the longer version of this post . A few days back I was working with storage client library especially around uploading blobs in chunks and with one particular blob I was constantly getting the error – The specified blob or block content is invalid. I tried numerous combinations even resorting to REST API directly but to no avail. It only happened with just one blob. Furthermore if I uploaded the same blob without splitting it into blocks, all was well. I was at my wits’ end. Tried searching the Internet for this error but could not find a conclusive answer to my problem. After much trial and error, I was able to simulate the same problem on other blobs as well. Here’s how you can recreate it: Start uploading the blob by splitting it into blocks. For block id, let’s do a 7 character long string e.g. intValue.ToString(“d7”). This will ensure that my block ids would be “0000001”, “0000002”, …, ”0000010” ….. After one or two blocks are uploaded, cancel the operation. Now re-upload the blob by splitting it into blocks. However this time for block id, let’s do a 6 character long string e.g. intValue.ToString(“d6”). You’ll get the error as soon as you try to upload the 1st block. Possible Solutions Now that we know the root cause of this problem, let’s look at some of the possible solutions to solve this problem. Wait out One possible solution is to wait out. I know its lame but still a possible solution. We know that Windows Azure Blob Storage Service keeps all uncommitted blocks for a duration of 7 days and if within 7 days those uncommitted blocks are not committed, the storage service purges them. I wish storage service provided some mechanism to purge uncommitted blocks programmatically. Commit uncommitted blocks You could possibly commit the blocks which are in uncommitted state so that at least you get a blob (which would not be the blob we wanted to upload in the first place). You can then delete that blob and re-upload the blob by specifying block ids which are of same length. To fetch the list of uncommitted blocks, if you’re using REST API directly you can perform “Get Block List” operation and pass “blocklisttype=uncommitted” as one of the query string parameters. If you’re using storage client library (assuming you’re using the version 2.x of .Net storage client library), you can do something like the code below: private static List GetUncommittedBlockIds(CloudBlockBlob blob) { var sasUri = blob.GetSharedAccessSignature(new SharedAccessBlobPolicy() { SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(5), Permissions = SharedAccessBlobPermissions.Read, }); var blobUri = new Uri(string.Format("{0}{1}", blob.Uri, sasUri)); List uncommittedBlockIds = new List(); var request = BlobHttpWebRequestFactory.GetBlockList(blobUri, null, null, BlockListingFilter.Uncommitted, null, null); //request.Headers.Add("Authorization", using (var resp = (HttpWebResponse)request.GetResponse()) { using (var stream = resp.GetResponseStream()) { var getBlockListResponse = new GetBlockListResponse(stream); var blocks = getBlockListResponse.Blocks; foreach (var block in blocks.Where(b => !b.Committed)) { uncommittedBlockIds.Add(Encoding.UTF8.GetString(Convert.FromBase64String(block.Name))); } } } return uncommittedBlockIds; } A few things to keep in mind here: Microsoft.WindowsAzure.Storage.Blob namespace does not have the capability to get the list of uncommitted blocks. You would need to make use ofMicrosoft.WindowsAzure.Storage.Blob.Protocol namespace. Because we’re kind of invoking the REST API by executing an HttpWebRequest, I created a shared access signature on the blob so that I don’t have to create “Authorization” header. Fetch uncommitted blocks to see block id length You could fetch the list of uncommitted blocks just to find out the length of the block id used. You could then use that block id length for your new upload session and do the upload. Please see the code snippet above to find this information. Upload another blob with same name without splitting it into blocks You could also upload another blob with the same name without splitting it into blocks. It could very well be a zero byte blob. That way your uncommitted block list will be wiped clean. Then you could delete that dummy blob and re-upload the actual blob. A Few Words About Blocks Since we’re talking about blocks, I thought it might be useful to mention a few points about them: Blocks and block related operations are only applicable for “Block Blobs”. Duh!! You’ll get an error if you’re trying to do these operations on a “Page Blob”. For uploading large blobs, it is recommended that you split your blob into blocks. In fact if your blob size is more than 64 MB, then you have to split it into blocks. Minimum size of a block is 1 Byte and the maximum size of a block is 4 MB. It is recommended that you choose a block size based on your internet connectivity and number of parallel threads you want use to upload these blocks. A blob can be split into a maximum of 50000 blocks. It’s important to remember this limitation because you are reminded of this limit when you’re trying to upload 50001st block. The length of all the block ids must be same. So if you’re using an integer value to denote block id, you make sure that you pad that integer value with “0” so that you get same length. So you could do something likeint.ToString(“d6”). When passing the block id as a parameter, it must be Base64 encoded. While the order in which the blocks are uploaded is not important, the order is important when you commit the block list because that’s when the blob is constructed by the service. For example, let’s say you’re uploading a blob by splitting it into 5 blocks (with ids “000001”, “000002”, “000003”, “000004”, and “000005”). You could upload these blocks in any order – 000004, 000001, 000003, 000005, 000002 however when you commit the block list, ensure that the block ids are passed in proper order i.e. 000001, 000002, 000003, 000004, 000005. Summary That’s it for this post. I hope you’ve found this information useful. I spent considerable amount of time trying to fix this problem so I hope it will help some folks out. As always, if you find any issues with the post please let me know and I’ll fix it ASAP.
May 20, 2013
by Gaurav Mantri
· 11,089 Views
article thumbnail
Gradle Goodness: Running a Single Test
Learn how to run test code with Gradle using the test task that is added by the Java plugin.
May 17, 2013
by Hubert Klein Ikkink
· 116,964 Views · 1 Like
  • Previous
  • ...
  • 243
  • 244
  • 245
  • 246
  • 247
  • 248
  • 249
  • 250
  • 251
  • 252
  • ...
  • Next
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook
×