Scrapy, being based on Twisted, introduces an incredible host of obstacles to easily and efficiently writing self-contained unit tests: 1. You can't call reactor.run() multiple times 2. You can't stop the reactor multiple times, so you can't blindly call "crawler.signals.connect(reactor.stop, signal=signals.spider_closed)" 3. Reactor runs in its own thread, so your failed assertions won't make it to the main unittest thread, so test failures will be thrown as assertion errors but unittest doesn't know about them To get around these hurdles, I created a BaseScrapyTestCase class that uses tl.testing's ThreadAwareTestCase and the following workarounds. class BaseScrapyTestCase(ThreadAwareTestCase): in_suite = False def setUp(self): self.last_crawler = None self.settings = get_project_settings() def run_reactor(self, called_from_suite=False): if not called_from_suite and BaseScrapyTestCase.in_suite: return log.start() self.last_crawler.signals.connect(reactor.stop, signal=signals.spider_closed) reactor.run() def queue_spider(self, spider, callback): crawler = Crawler(self.settings) self.last_crawler = crawler crawler.signals.connect(callback, signal=signals.spider_closed) crawler.configure() crawler.crawl(spider) crawler.start() return crawler def wrap_asserts(self, fn): with ThreadJoiner(1): self.run_in_thread(fn) You'll use it like so: class SimpleScrapyTestCase(BaseScrapyTestCase): def test_suite(self): BaseScrapyTestCase.in_suite = True self.do_test_simple() self.run_reactor(True) def do_test_simple(self): spider = Spider("site.com") def _fn(): def __fn(): self.assertTrue(False) self.wrap_asserts(__fn) self.queue_spider(spider, _fn) self.run_reactor() 1. Call run_reactor() at the end of test method. 2. You have to place your assertions in its own function which gets called in a ThreadJoiner so that unittest knows about assertion failures. 3. If you're testing multiple spiders, just call queue_spider() for each, and run_reactor() at the end. 4. BaseScrapyTestCase keeps track of the crawlers created, and makes sure to only attach a reactor.stop signal to the last one. Let me know if you come up with a better/more elegant way of testing scrapy spiders!

this article represents top 6 popular angularjs topics that has been used most by the angularjs developer community to date. the inference is derived based on number of tagged discussions happening on stackoverflow . clearly, “directive” is the winner and attracts most of them all. the article presents my thoughts on why these topics have been most popular. please feel free to comment/suggest if i missed to mention one or more important points. also, sorry for the typos. following is the list of top 6 popular topics: directives scope object ng-repeat angular ui & bootstrap routing service following plot demonstrates the popularity of different feature/topics in relation with angularjs. angularjs topics popularity inference : some of the following could as well be inferred from the above data/plot. three features which have been most used by the developers and therefore, should be key reasons why you would also want to use angular in next project are following: directives routing ng-repeat one of the pain point (or shortcoming) that have been talked most by the angular developers is the ui widgets related support by angular. this is where most of them have jumped to angular ui and bootstrap. the topic/concept that has intrigued most to several developers is scope object. thoughts on why these topics may be most popular following are top 6 popular topics in angularjs discussed on forums such as stackoverflow: directives : this is, no doubt, the most popular and powerful feature of angularjs directives as also indicated by count of discussion threads posted on stackoverflow as of today. the power of directives lies in the following and this is why it is the most popular topic of angularjs. re-usability : once created a directive as part of a module, all that one need to do to use the directive is include the module as a dependency when defining new module and define the directives wherever required on the page. usability : owing to the fact that one could give intuitive names to directives, directive enhances the readability and understandability of code by a notch. greater adherence to dry principle : the aspect of templating makes directive a very attractive feature. it does reduce the duplication of code as same html template code could be used at several places without the need to write the code in html file. scope object : this is second most popular topic found based on the discussion count. rightfully expected as well! the whole notion of scope object and how it is key to dependency injection makes it one of the most powerful as well as tricky concept of angularjs. also, this is one of the topic which raised the barrier to entry for angularjs and contributed in making steep learning curve for developers. that said, scope is going to r.i.p in angular 2.0 which could be seen as a good sign for those who always struggled with scope object. ng-repeat : the ng-repeat feature brings power to angularjs from the fact that it is one of the feature that removed the need of server-side code required to repeat the html code over multiple iterations. with ng-repeat, one could easily repeat html code multiple times. angular ui & bootstrap : one of the shortcoming of angularjs for good or bad is its inability to be one and all solution to create some great ui along with powerful eventing feature. for creating fancy or great looking ui, one would still have to go to ui frameworks such as bootstrap, kendo-ui etc. this is where people have been looking for angularui and bootstrap. angularui comes with attractive feature set for enhanced routing, grid util, angularjs code editor plugins, bootstrap module etc. routing : routing feature is key to creating single page application. one of key reason why angularjs is very popular is the ease with which one could create single-page application using it. and, routing feature makes it all happen. no doubt, this is why many developers have been looking for it. service : service feature helps one to create reusable components in an angular module. these services could then be injected in another modules using dependency injection feature. the service could be injected in one of the following components: controllers services doing a quick recap, one may recall that for creating a service, one could use factory recipe method and define service that way. you could know details about creating a custom service on our another page dedicated on this.

CompletableFuture from Java 8 is an advanced abstraction over a promise that value of type T will be available in the future. Observable is quite similar, but it promises arbitrary number of items in the future, from 0 to infinity. These two representations of asynchronous results are quite similar to the point where Observable with just one item can be used instead of CompletableFuture and vice-versa. On the other hand CompletableFuture is more specialized and because it's now part of JDK, should become prevalent quite soon. Let's celebrate RxJava 1.0 release with a short article showing how to convert between the two, without loosing asynchronous and event-driven nature of them. From CompletableFuture to Observable CompletableFuture represents one value in the future, so turning it into Observable is rather simple. When Futurecompletes with some value, Observable will emit that value as well immediately and close stream: class FuturesTest extends Specification { public static final String MSG = "Don't panic" def 'should convert completed Future to completed Observable'() { given: CompletableFuture future = CompletableFuture.completedFuture("Abc") when: Observable observable = Futures.toObservable(future) then: observable.toBlocking().toIterable().toList() == ["Abc"] } def 'should convert failed Future into Observable with failure'() { given: CompletableFuture future = failedFuture(new IllegalStateException(MSG)) when: Observable observable = Futures.toObservable(future) then: observable .onErrorReturn({ th -> th.message } as Func1) .toBlocking() .toIterable() .toList() == [MSG] } CompletableFuture failedFuture(Exception error) { CompletableFuture future = new CompletableFuture() future.completeExceptionally(error) return future } } First test of not-yet-implemented Futures.toObservable() converts Future into Observable and makes sure value is propagated correctly. Second test created failed Future, replaces failure with exception's message and makes sure exception was propagated. The implementation is much shorter: public static Observable toObservable(CompletableFuture future) { return Observable.create(subscriber -> future.whenComplete((result, error) -> { if (error != null) { subscriber.onError(error); } else { subscriber.onNext(result); subscriber.onCompleted(); } })); } NB: Observable.fromFuture() exists, however we want to take full advantage of ComplatableFuture's asynchronous operators. From Observable toCompletableFuture> There are actually two ways to convert Observable to Future - creating CompletableFuture> orCompletableFuture (if we assume Observable has just one item). Let's start from the former case, described with the following test cases: def 'should convert Observable with many items to Future of list'() { given: Observable observable = Observable.just(1, 2, 3) when: CompletableFuture> future = Futures.fromObservable(observable) then: future.get() == [1, 2, 3] } def 'should return failed Future when after few items exception was emitted'() { given: Observable observable = Observable.just(1, 2, 3) .concatWith(Observable.error(new IllegalStateException(MSG))) when: Futures.fromObservable(observable) then: def e = thrown(Exception) e.message == MSG } Obviously Future doesn't complete until source Observable signals end of stream. Thus Observable.never() would never complete wrapping Future, rather then completing it with empty list. The implementation is much shorter and sweeter: public static CompletableFuture> fromObservable(Observable observable) { final CompletableFuture> future = new CompletableFuture<>(); observable .doOnError(future::completeExceptionally) .toList() .forEach(future::complete); return future; } The key is Observable.toList() that conveniently converts from Observable and Observable>. The latter emits one item of List type when source Observable finishes. From Observable to CompletableFuture Special case of the previous transformation happens when we know that CompletableFuture will return exactly one item. In that case we can convert it directly to CompletableFuture, rather than CompletableFuture>with one item only. Tests first: def 'should convert Observable with single item to Future'() { given: Observable observable = Observable.just(1) when: CompletableFuture future = Futures.fromSingleObservable(observable) then: future.get() == 1 } def 'should create failed Future when Observable fails'() { given: Observable observable = Observable. error(new IllegalStateException(MSG)) when: Futures.fromSingleObservable(observable) then: def e = thrown(Exception) e.message == MSG } def 'should fail when single Observable produces too many items'() { given: Observable observable = Observable.just(1, 2) when: Futures.fromSingleObservable(observable) then: def e = thrown(Exception) e.message.contains("too many elements") } Again the implementation is quite straightforward and almost identical: public static CompletableFuture> fromObservable(Observable observable) { final CompletableFuture> future = new CompletableFuture<>(); observable .doOnError(future::completeExceptionally) .toList() .forEach(future::complete); return future; } Helpers methods above aren't fully robust yet, but if you ever need to convert between JDK 8 and RxJava style of asynchronous computing, this article should be enough to get you started.

A long while ago a optimisation was added to the JVM so that if the same exception is thrown again and again and again a single instance of the Exception is created without the stack trace filled in in order to increase performance. This is an excellent idea unless you are trying to diagnose a problem and you have missed the original error. If you forgot about this optimisation you send the afternoon looking at the following log output and weeping slightly. (In my defence I have a little one in the house hence the fuzzy brain and lack of blogging action this year) java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException ... java.lang.Exception: Uncaught exceptions during test at oracle.jdevstudio-testware-tests.level0.testADFcMATS(/.../work/mw9111/jdeveloper/jdev/extensions/oracle.jdevstudio-testware-tests/abbot/common-adf/CreateNewCustomAppAndProjectWithName.xml:25) at oracle.jdevstudio-testware-tests.level0.testADFcMATS(/.../work/mw9111/jdeveloper/jdev/extensions/oracle.jdevstudio-testware-tests/abbot/level0/testADFcMATS.xml:94) at oracle.abbot.JDevScriptFixture.runTest(JDevScriptFixture.java:555) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at junit.textui.TestRunner.doRun(TestRunner.java:116) at junit.textui.TestRunner.doRun(TestRunner.java:109) at oracle.abbot.AbbotRunner.run(AbbotRunner.java:614) at oracle.abbot.AbbotAddin$IdeAbbotRunner.run(AbbotAddin.java:634) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ArrayIndexOutOfBoundsException [Crickets] It turns out that you can turn off this optimisation with a simple flag: java -XX:-OmitStackTraceInFastThrow .... In my particular case the actual exception causing this trouble was a NPE from the GlyphView code in JDK8. (One that is being caused by a glitch in hotspot it seems) But that in turn was causing the AIOOBE in some logging code clouding the issue even more. This in particular is a good flag to add by default when running your automated tests, particularly in combination with the stack trace length override I have talked about before.

today i'd like to introduce a new series of posts named untangling concepts . in the it business we are overloaded day by day with more and more information and sometimes it's difficult to handle all this knowledge without making mistakes. the main goal of these posts is to clarify the difference between concepts that sometimes are mixed together. in the first post i'll write about unit testing and acceptance testing. i'd like to emphasize that this post isn't about test driven development . tdd is a subject to other posts. unit tests when we talk about unit tests the first thing that should come to mind is the single responsibility principle . the name unit tests isn't a coincidence, you should test one small piece of code which does just one thing. i guarantee that if you code big classes with big methods which do a lot of stuff, you'll have trouble to write good and useful unit tests. high cohesion matters. so, what is the point in unit tests? imagine the design of a system. if you follow the object oriented software design good practices you will come up to a system where a lot of objects communicate with each other in order to achieve the program's goal. imagine the following situation: you have to implement an e-mail system. the program receives as input a list of e-mail addresses and have to send an e-mail to each one of the recipients. you might have something like this: in this case, someone will call the readrecipientlist method with an inputstream (e.g. a fileinputstream of a txt file with some e-mail addresses). after that, the method sendmails is called, returning the number of e-mails sent. but what should happen if the method sendmails is called before the method readrecipientlist is called? and what if i pass a fileinputstream of an empty file? or even a null inputstream object? how can i trust that the class mailsender is doing it's job right? those are the type of question that unit tests try to answer. when writing unit tests, our concern is to assert that our objects' internal structure is working as expected. it's like a puzzle... if i can assert that each piece is working as expected, i can trust that when the pieces are put together (in the right places... more about that in the next session), the puzzle will be completed correctly. and that's not everything. what if you have to change something inside one piece of code, or if you need to do some refactoring (you should do it a lot)? if you have tests to assert that this piece of code is working as expected you can run these tests after the changes and see if your modifications broke something. you can make changes more confidently. there are a lot to say about unit tests and how they can be beneficial to your code. here i just want that you understand the concept, but i'll put some links in the useful resources section so you can read a lot more about that. acceptance tests now let's talk about acceptance tests. put yourself in the place of a stakeholder. you know what you want the software to do but you know nothing about software development nor programming. how can you believe that the software is doing everything you want, in the way you want? unit tests are good to test small pieces of code and to assert that the code works as expected. but that isn't enough to be sure that the code is satisfying the stakeholders. acceptance tests are test cases written based on scenarios specified by the customer. usually each history will have at least one associated acceptance test. they work as black box tests , meaning that shouldn't be considered implementation details in those tests (one more reason to write them first...). it's like an input-output evaluation: " given some context, when something happen, then i expect the system to answer that (or to be in that state)." we use this given-when-then template to write good acceptance tests for a user history. let's try a little example. remember our e-mail system? which scenario do you think that our customer would like to satisfy? i believe that the most obvious is something like that: given that i am logged in the mail system, when i send an e-mail to [email protected] , then i expect that there is one new mail in someone's inbox. this is just one super basic example. it's not unusual to have acceptance tests with far more given and then clauses. the best way to write acceptance tests is using something that everyone can understand. but that's not easy. we can use, for example, selenium to capture tests based in user's input, but that depends in having at least a prototype interface where the user can navigate. in the last project i have been working on, we have successfully adopted acceptance tests written in gherkin , a language used by cucumber . for example, a more detailed version of our test of sending an e-mail would look like that in gherkin: feature: sending an e-mail to someone scenario: send an e-mail to an existing user given that i am authenticated in the mail system and that exists an account: [email protected] when i send an e-mail to [email protected] then someone's inbox must have one unread mail and my sent itens folder must have the sent mail as you can see, we are writing our tests in plain english. anyone can read them and understand what the system is supposed to do (the bold words have nothing to do with gherkin, it's a bug in the syntax highlight feature). i see two great benefits in writing acceptance tests before the development of a history. the first one is that we can be sure that the customer knows what he wants to be developed and that the developer knows what the customer wants . so we reduce the chance of misunderstandings, increasing customer satisfaction. the second is that we have an easy feedback when any of the user histories are broken during the development cycle. the more we run the acceptance tests, faster we can see if our system is not doing what it is supposed to do. and faster we fix it . conclusion in this article we learned that unit tests are the best tool that developers have to protect themselves from mistakes. the wider our test coverage is, more we can do refactoring with confidence, leading to better code. remember, code rots . acceptance tests is an excelent tool to help to make the gap between the customer desires and the developer understanding smaller. they help in the fast feedback loop, leading to software that makes our customer happier. remember that unit tests and acceptance tests complement each other. our goal is not just to build the right thing but make sure that we build the thing right . useful resources martin fowler's post: unit tests extremeprogramming.org site: page about unit tests extremeprogramming.org site: page about acceptance tests uncle bob's post: the truth about bdd

[This article was written by Barak Merimovich.] We have discussed OpenStack networking extensively in previous posts. In this post, I’d like to dive into a more advanced OpenStack networking scenario. Many cloud images are not configured to automatically bring up all network cards that are available. They will usually only have a single network card configured. To correctly set up a host in the cloud with multiple network cards, log on to the machine and bring up the additional interfaces. echo $'auto eth1\niface eth1 inet dhcp' | sudo tee /etc/network/interfaces.d/eth1.cfg > /dev/null sudo ifup eth1 Networks in the cloud A complex network architecture is a mainstay of modern IaaS clouds. Understanding how to configure your cloud-based networks, and hosts, is critical to getting your application working in the cloud. This is especially true with Cloudify, the open source cloud orchestration platform I work on. The cloud, like the world, used to be flat It was not that long a time ago that most IaaS providers only supported flat networks – all of your hosts were in one large network. Separation between services running in the cloud was enforced in software or with firewalls/security-groups. But technically, all of the hosts were connected to the same network and visible to each other. The flat network model is simple, and therefore easy to reason and understand. It was a good choice for the early days of the IaaS cloud and no doubt helped with getting applications into the cloud in the first place. It was one of the things that made EC2 so easy to use for anyone just starting out with the ‘cloud’. This model is in fact still available on Amazon Web Services under the title ‘EC2-Classic’. And for many applications, a flat network is good enough. But as cloud adoption increases, more complex applications are moving into the clouds, and issues like network separation, security, SLA and broadcast domains make more complex networks models a must. Software Defined Networks (SDN) fill that gap. They are now a staple of most major IaaS clouds. AWS has AWS-VPC, OpenStack has the Neutron project and there are many other implementations. Working with SDN requires knowing a bit more about how information moves around between your cloud resources. In this post I am going to discuss how to set up a host in the cloud so it will play nice with complex networks. I’ll be using OpenStack, but the concepts are similar for other cloud infrastructures. Openstack configuration I am going to start with an empty tenant, only the public network is available. First, lets set up out networks and router: neutron router-create demo-router neutron net-create demo-network-1 neutron net-create demo-network-2 neutron subnet-create --name demo-subnet-1 demo-network-1 10.0.0.0/24 neutron subnet-create --name demo-subnet-2 demo-network-2 10.0.1.0/24 neutron router-interface-add demo-router demo-subnet-1 neutron router-interface-add demo-router demo-subnet-2 neutron router-gateway-set demo-router public Note the network IDs: neutron net-list | id | name | subnets | | 2c33efe2-6204-4125-9716-3bc525630016 | demo-network-1 | 928dafa0-83ef-459c-b20d-71d8ea596fa2 10.0.0.0/24 | | aa30627e-c181-4a4b-89bf-5dd7c26c244e | demo-network-2 | 26d573f7-7953-4a54-825b-ed7bbc0661c7 10.0.1.0/24 | | e502de8d-929a-4ee0-bd18-efa297875cf6 | public | d40dab51-a729-452c-9ee6-b9ad08d10808 | We’ll start with a standard Ubuntu cloud image: glance image-create --name "Ubuntu 12.04 Standard" --location "http://uec-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img" --disk-format qcow2 --container-format bare Create the keypair and security group: nova keypair-add demo-keypair > demo-keypair.pem chmod 400 demo-keypair.pem nova secgroup-create demo-security-group "Security group for demo" nova secgroup-add-rule demo-security-group tcp 22 22 0.0.0.0/0 Let’s spin up an instance connected to both our networks: nova boot -flavor m1.small --image "Ubuntu 12.04 Standard" --nic net-id=2c33efe2-6204-4125-9716-3bc525630016 --nic net-id=aa30627e-c181-4a4b-89bf-5dd7c26c244e --security-groups demo-security-group --key-name demo-keypair demo-vm And set up floating IPs for the first network: nova list | ID | Name | Status | Task State | Power State | Networks | 2b17588b-8980-4489-9a04-6539a159dc3c | demo-vm | ACTIVE | None | Running | demo-network-1=10.0.0.2; demo-network-2=10.0.1.2 | neutron floatingip-create public neutron floatingip-list | id | fixed_ip_address | floating_ip_address | port_id | | 49c8b05e-bb8f-4b07-80ed-3155ab6ffc09 | | 192.168.15.42 | | neutron port-list | id | name | mac_address | fixed_ips | | 1ccfd334-7328-4b22-b93e-24a0888276ab | | fa:16:3e:14:39:39 | {"subnet_id": "94598487-c1fc-4f55-ac1f-ef2545d5cfeb", "ip_address": "10.0.1.3"} | | a482c4f6-fa74-476e-b1ce-cd8dd0c70815 | | fa:16:3e:18:92:79 | {"subnet_id": "94598487-c1fc-4f55-ac1f-ef2545d5cfeb", "ip_address": "10.0.1.2"} | | b23d7836-30c5-4bff-b873-15c87ba051f6 | | fa:16:3e:3a:28:40 | {"subnet_id": "dec6ec74-cfa9-4a08-8792-54900631b98e", "ip_address": "10.0.0.3"} | | d421b447-2adf-406f-876b-142238683344 | | fa:16:3e:9d:fc:7f | {"subnet_id": "dec6ec74-cfa9-4a08-8792-54900631b98e", "ip_address": "10.0.0.2"} | | dcf8696b-cc80-4b48-b09c-61c0f8ab02ac | | fa:16:3e:5b:39:fb | {"subnet_id": "94598487-c1fc-4f55-ac1f-ef2545d5cfeb", "ip_address": "10.0.1.1"} | | f6a1666e-495a-4d3f-afa3-754b3cb3cfc0 | | fa:16:3e:8a:1b:fb | {"subnet_id": "dec6ec74-cfa9-4a08-8792-54900631b98e", "ip_address": "10.0.0.1"} | neutron floatingip-associate 49c8b05e-bb8f-4b07-80ed-3155ab6ffc09 d421b447-2adf-406f-876b-142238683344 Note how we matched the VM’s IP to its port, and associated the floating IP to the port. I wish there was an easier way to do this from the CLI… If everything worked correctly, you should have the following setup: Let’s make sure ssh works correctly: ssh -i demo-keypair.pem [email protected] hostname demo-vm Cool, ssh works. Now, we should have two network cards, right? ssh -i demo-keypair.pem [email protected] hostname demo-vm Cool, ssh works. Now, we should have two network cards, right? ssh -i demo-keypair.pem [email protected] ifconfig eth0 Link encap:Ethernet HWaddr fa:16:3e:5f:a2:5f inet addr:10.0.0.4 Bcast:10.0.0.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe5f:a25f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:230 errors:0 dropped:0 overruns:0 frame:0 TX packets:224 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:46297 (46.2 KB) TX bytes:31130 (31.1 KB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) Huh?! The VM only has one working network interface! Where is my second NIC? Was there a configuration problem with the OpenStack network setup? The answer is here: ssh -i demo-keypair.pem [email protected] ifconfig -a eth0 Link encap:Ethernet HWaddr fa:16:3e:5f:a2:5f inet addr:10.0.0.4 Bcast:10.0.0.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe5f:a25f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:324 errors:0 dropped:0 overruns:0 frame:0 TX packets:332 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:69973 (69.9 KB) TX bytes:47218 (47.2 KB) eth1 Link encap:Ethernet HWaddr fa:16:3e:29:6d:22 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) The second NIC exists, but is not running. The issue is not with the OpenStack network configuration – it’s with the image. The image itself should be configured to work correctly with multiple NICs. All we have to do is bring up the NIC. So we ssh into the instance: ssh -i demo-keypair.pem [email protected] And run the following commands: echo $'auto eth1\niface eth1 inet dhcp' | sudo tee /etc/network/interfaces.d/eth1.cfg > /dev/null sudo ifup eth1 The second NIC should now be running: ifconfig eth1 eth1 Link encap:Ethernet HWaddr fa:16:3e:18:92:79 inet addr:10.0.1.2 Bcast:10.0.1.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe18:9279/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:81 errors:0 dropped:0 overruns:0 frame:0 TX packets:45 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:15376 (15.3 KB) TX bytes:3960 (3.9 KB) And there you go – your VM can access both networks. This issue can make life complicated when setting up a complex, or even a not very complex, application. When will this issue hurt you? Well, imagine a scenario where you have a web server and a database server. The web server is connected to both Network1 and Network2, and the database server is only connected to Network2. Network1 is connected to the external world over a router, and Network 2 is completely internal, adding another layer of security to the critical database server. So what happens if the web server only has one network card? If only the NIC for Network1 is up, the web server can’t access the database. If only the NIC for Network2 is up, the web server can’t be reached from the external world. Even worse, if this web server is accessed via a floating IP, this IP will also not work, so you won’t be able to access the web server and fix the issue. Tricky. In conclusion The above commands will bring up your additional network card. You will of-course need to repeat this process for each additional network card, and for each VM. You can use a start-up script (a.k.a. user-data script) or system service to run these commands, but there are better ways. I’ll discuss how to automate the network setup in a follow-up post. This was originally posted at Barak's blog Head in the Clouds, find it here.

For a long time JSF for me was just another web framework I didn’t care too much about. This changed. After being forced to use it for a couple of months now, I consider it a major project risk in almost all cases. Here I present the reasons for this verdict. Bad entanglement of UI and processing Logic The official tutorial claims the following about the benefits of JSF: One of the greatest advantages of Java Server Faces technology is that it offers a clean separation between behavior and presentation for web applications. The opposite is the case. Facelets, the preferred presentation technology of JSF looks at first sight like an ordinary templating technology like the good old JSP or Thyme Leaf. But if you look closer the horror becomes obvious. In the same place where you structure your HTML, you also place the logic what parts of the UI should get updated on an action. A clear violation of the separation of concerns principle in my book. Even better is the immediate attribute which changes the server side life cycle! And if this isn’t enough it does it in different waysdepending on what tag you use it on. You can’t make stuff like this up. It tries to abstract what you can not abstract. Except some weird edge cases clients and server of web application are located on rather different computers, separated by some kind of network. From this follows a simple fact: communication between client and server is slow and unreliable. JSF tries to abstract away the separation of client and server. It processes everything on the backend wildly communicating between client and server in a hard to control way. The result are all kind of failure scenarios just popping into existence because you use JSF. For me the most annoying one is this one: If you open a JSF page, let’s say a simple search page, wait an hour, then hit the submit button you will get an exception because the server side state expired. WAT? Why is there server state of any relevance for a trivial search page? (Yes I know you can change that behavior with the latest versions of JSF, but it is still the way JSF is designed to work) I though everybody learned since EJBs: If you want to abstract over the fact, if two parts of an application run on the same machine or not, you have to assume they don’t. Everything else is just hiding problems until they grow so large that they can eat your project for breakfast. Making stuff complex and complicated that was easy to start with.The architecture of the World Wide Web is a simple one. Simple meaning: It consists of a small set of concepts with limited interaction. This is what made it so widely successful. It also makes it not obvious for beginners how to use it to implement certain features. I’m sure most of us remember the first time they tried to implement something like a shopping cart without having session state. But the solutions for almost all these problems are well known and understood by know. And all you need is a little reading and what you gain is a strong conceptual understanding how to solve this kind of issue. And again, the basics are extremely simple: You send a request to an URL, with some headers and content using a HTTP verb. And you reply with some resource containing links and some headers. And you don’t have state in the server session. Making load balancing and fail over rather simple. Making bookmarkable URLs trivial. Making your site searchable for zero costs. Making your site cachable. Allowing the user to use their back buttons, history and tabs as they wish. Making it trivial to have nice URLs Compare that to the live cycle model of JSF: The page from which a user submitted a request will get synchronized with a model on the server side, then submitted values validated, converted, events generated and processed. As mentioned above the order in which things happen, and if they happen at all are controlled by XML Tags hidden away in a document camouflaged as markup. Apart from hardly anybody properly understanding all this (BalusC seems to be the only one available in the interwebs) it has the following effect on your application: The URLs become ugly. You’ll see the URL of the resource you came from instead of the one you are looking at, thus making bookmarking URLs as useful as a doorknob on your knee. Same for caching, fail over, load balancing and so on. Sure you can fix it with some convention here, and an additional library there. Which of course makes perfect sense when you are in the business of breaking stuff so people have to pay you for fixing it. I personally prefer helping to solve real problems. Hindering testability: I can’t speak for most frameworks but I can compare Spring MVC with JSF. Let me tell you this: If anybody is telling you JSF is nicely testable he probably doesn’t know automatic testing. With JSF you can test your backend beans using unit tests. You can test the whole UI, by deploying the application to a server and hitting it with Selenium. That’s basically it. Just in case you are wondering what else one should be able to test: Load a static version of a page in a browser and testing it with selenium, in order test your Client side UI behavior. Test your generated markup without starting a full blown application server. Test the mapping of attributes/parameter to bean methods. Test your generated markup without bootstrapping a complete application. All this is perfectly possible with Spring MVC and probably with many other sane server side frameworks, but not with JSF … Again: I’m aware there are fixes for many issues, but the simplest fix is> Don’t use JSF.

This article explains how errors are handled when using the messaging system with Spring Integration and how to handle route and redirect to specific channel.

This technical tip shows how to read Excel worksheet cells values in multiple threads simultaneously inside android applications. Often you need to read worksheet cells values in multiple threads simultaneously. To do so, set Worksheet.getCells().setMultiThreadReading() to true. If you do not set this property you might get the wrong cell values. Setting it to true, you always get the correct values. To achieve this task first create a workbook and adds a worksheet. Populates the worksheet with some string values then create two threads that simultaneously read values from random cells. If the values read are correct, then nothing happens. But if the values read are incorrect, then a message box shows up in the LogCat window. If you comment this line: testWorkbook.getWorksheets().get(0).getCells().setMultiThreadReading(true); the following message will show up in LogCat window: if (s.equals("R" + row + "C" + col)!=true) { System.out.println("This message box will show up when cells read values are incorrect."); } Otherwise, the program run without showing any message which means all values read from cells are correct. public class ThreadProc implements Runnable { boolean isRunning = true; Workbook testWorkbook; Random r = new Random(); public ThreadProc(Workbook workbook) { this.testWorkbook = workbook; } public int randomNext(int Low, int High) { int R = r.nextInt(High-Low) + Low; return R; } public void kill() { this.isRunning = false; } public void run(){ while(this.isRunning) { int row = randomNext(0, 100); int col = randomNext(0, 10); String s = testWorkbook.getWorksheets().get(0).getCells().get(row, col).getStringValue(); if (s.equals("R" + row + "C" + col)!=true) { System.out.println("This message box will show up when cells read values are incorrect."); } } } } //........MainActivity.java........ //................................. import java.io.File; import java.util.Random; import android.app.Activity; import android.os.Bundle; import android.os.Environment; import android.view.Menu; import com.aspose.cells.CellsHelper; import com.aspose.cells.IWarningCallback; import com.aspose.cells.WarningInfo; import com.aspose.cells.WarningType; import com.aspose.cells.Workbook; public class MainActivity extends Activity { @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_main); try{ Workbook testWorkbook = new Workbook(); testWorkbook.getWorksheets().clear(); testWorkbook.getWorksheets().add("Sheet1"); for (int row = 0; row < 100; row++) for (int col = 0; col < 10; col++) testWorkbook.getWorksheets().get(0).getCells().get(row, col).setValue("R" + row + "C" + col); //Commenting this line will show a pop-up message testWorkbook.getWorksheets().get(0).getCells().setMultiThreadReading(true); ThreadProc tp = new ThreadProc(testWorkbook); Thread myThread1 = new Thread(tp); myThread1.start(); Thread myThread2 = new Thread(tp); myThread2.start(); Thread.currentThread().sleep(5*1000); tp.kill(); } catch (Exception e) { e.printStackTrace(); } } ......... }

We use the Test task in Gradle to run tests. If we use the System.out.println or System.err.println methods in our test we don't see the output when we execute the tests. We can customize the test task to show any output send to standard out or error in the Gradle output. First we show our test class written with Spock, but it could also be a JUnit or TestNG test: // File: src/test/groovy/com/mrhaki/gradle/SampleSpec.groovy package com.mrhaki.gradle import spock.lang.* class SampleSpec extends Specification { def "check that Gradle is Gr8"() { when: def value = 'Gradle is great!' then: // Include a println statement, so // we have output to show. println "Value = [$value]" value == 'Gradle is great!' } } Now we write a simple Gradle build file which can execute our test: // File: build.gradle apply plugin: 'groovy' // Adds test task repositories.jcenter() dependencies { compile 'org.codehaus.groovy:groovy-all:2.3.7' testCompile 'org.spockframework:spock-core:0.7-groovy-2.0' } Let's run the test task from the command line and look at the output: $ gradle test :compileJava UP-TO-DATE :compileGroovy UP-TO-DATE :processResources UP-TO-DATE :classes UP-TO-DATE :compileTestJava :compileTestGroovy :processTestResources UP-TO-DATE :testClasses :test BUILD SUCCESSFUL Total time: 7.022 secs $ Well at least our test is successful, but we don't see the output of our println method invocation in the test. We customize the test task and add thetestLogging method with a configuration closure. In the closure we set the property showStandardStreams to the value true. Alternatively we can set the events property or use the events method with the values standard_out and standard_err to achieve the same result. In the next build file we use the showStandardStreams property: view sourceprint? 00.// File: build.gradle 01.apply plugin: 'groovy' // Adds test task 02. 03.repositories.jcenter() 04. 05.dependencies { 06.compile 'org.codehaus.groovy:groovy-all:2.3.7' 07.testCompile 'org.spockframework:spock-core:0.7-groovy-2.0' 08.} 09. 10.test { 11.testLogging { 12.// Make sure output from 13.// standard out or error is shown 14.// in Gradle output. 15.showStandardStreams = true 16. 17.// Or we use events method: 18.// events 'standard_out', 'standard_error' 19. 20.// Or set property events: 21.// events = ['standard_out', 'standard_error'] 22. 23.// Instead of string values we can 24.// use enum values: 25.// events org.gradle.api.tasks.testing.logging.TestLogEvent.STANDARD_OUT, 26.// org.gradle.api.tasks.testing.logging.TestLogEvent.STANDARD_ERROR, 27.} 28.} We re-run the test task from the command line and look at the output to see the result from the println method: $ gradle test :compileJava UP-TO-DATE :compileGroovy UP-TO-DATE :processResources UP-TO-DATE :classes UP-TO-DATE :compileTestJava :compileTestGroovy :processTestResources UP-TO-DATE :testClasses :test com.mrhaki.gradle.SampleSpec > check that Gradle is Gr8 STANDARD_OUT Value = [Gradle is great!] BUILD SUCCESSFUL Total time: 8.716 secs $ Written with Gradle 2.1.

It is possible to disable HTTP access on S3 bucket, limiting S3 traffic to only HTTPS requests. The documentation is scattered around the Amazon AWS documentation, but the solution is actually straightforward. All you need to do to block HTTP traffic on an S3 bucket is add a Condition in your bucket's policy. AWS supports a global condition for verifying SSL. So you can add a condition like this: "Condition": { "Bool": { "aws:SecureTransport": "true" } } Here's a complete example: { "Version": "2008-10-17", "Id": "some_policy", "Statement": [ { "Sid": "AddPerm", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "s3:GetObject", "Resource": "arn:aws:s3:::my_bucket/*", "Condition": { "Bool": { "aws:SecureTransport": "true" } } } ] } Now accessing the contents of my_bucket over HTTP will produce a 403 error, while using HTTPS will work fine.

This article shows how to mock your JMS infrastructure using MockRunner and test it using Spring.

I use Charles Proxy extensively for debugging all kinds of applications, and lately I've been using it more with mobile devices. One of the killer features of Charles is its ability to intercept SSL traffic. This is hard - and rightly so, it should be difficult to inspect SSL traffic! Charles handles this by using the server's SSL certificate for the connection from Charles to the remote server, and then using Charles' own SSL certificate for the "last mile" back to your browser or device. This means that the connection will be flagged as insecure; Charles' certificates aren't trusted by your browser or device - but we can easily change that. Get Charles' Certificate On Windows, you can grab the certificate from the Help menu in Charles, but for OS X or Linux you'll need to download it from their website. It's linked from this excellent documentation page about SSL:http://www.charlesproxy.com/documentation/using-charles/ssl-certificates/. Install it On Your Android Device Copy the ``.crt`` file into the root of your SD card (I found this didn't work in a subdirectory, although some documentation suggests it will). Under "Security" -> "Settings", choose "install from device storage" and it should pick up the certificate you put on the SD card. ... that's it :) Check you have enabled the destination for SSL proxying under "Proxy Settings" in Charles, and you're all set! Note, I have one device that refused to see the .crt file on the SD card, but when I emailed it to myself and then saved it from there, was added successfully. Thought I'd mention it! Further Reading Using Charles To Debug PHP SOAP Wireshark Capture on Remote Server View Only Headers with Curl

Property based testing is an alternative approach to testing, complementingexample based testing. The latter is what we've been doing all our lives: exercising production code against "examples" - inputs we think are representative. Picking these examples is an art on its own: "ordinary" inputs, edge cases, malformed inputs, etc. But why are we limiting ourselves to just few examples? Why not test hundreds, millions... ALL inputs? There are at least two difficulties with that approach: Scale. A pure function taking just one int input would require 4 billion tests. This means few hundred gigabytes of test source code and several months of execution time. Square it if a function takes two ints. For String it practically goes to infinity. Assume we have these tests, executed on a quantum computer or something. How do you know the expected result for each particular input? You either enter it by hand (good luck) or generate expected output. Bygenerate I mean write a program that produces expected value for every input. But aren't we testing such program already in the first place? Are we suppose to write better, error-free version of code under test just to test it? Also known as ugly mirror antipattern. So you understand testing every single input, although ideal, is just a mental experiment, impossible to implement. That being said property based testing tries to get as close as possible to this testing nirvana. Issue #1 is solved by slamming code under test with hundreds or thousands of random inputs. Not all of them, not even a fraction. But a good, random representation. Issue #2 is surprisingly harder. Property based testing can generate random arguments, but it can't figure out what should be the expected outcome for that random input. Thus we need a different mechanism, giving name to whole philosophy. We have to come up with properties (invariants, behaviours) that code under test exhibits no matter what the input is. This sounds very theoretically, but there are many such properties in various scenarios: Absolute value of any number should never be negative Encoding and decoding any string should yield the same String back for every symmetric encoding Optimized version of some old algorithm should produce the same result as the old one for any input Total money in a bank should remain the same after arbitrary number of intra-bank transactions in any order As you can see there are many properties we can think of that do not mention specific example inputs. This is not exhaustive and strict testing. It's more like sampling and making sure samples are "sane". There are many, many libraries supporting property based testing for virtually every language. In this article we will explore Spock and ScalaCheck later. Spock + custom data generators Spock does not support property based testing out-of-the-box. However with help from data driven testing and 3rd-party data generators we can go quite far. Data tables in Spock can be generalized into so-called data pipes: def 'absolute value of #value should not be negative'() { expect: value.abs() >= 0 where: value << randomInts(100) } private static def List randomInts(int count) { final Random random = new Random() (1..count).collect { random.nextInt() } } Code above will generate 100 random integers and make sure for all of them.abs() is non-negative. You might think this test is quite dumb, but to a great surprise it actually discovers one bug! But first let's kill some boilerplate code. Generating random inputs, especially more complex, is cumbersome and boring. I found two libraries that can help us. spock-genesis: import spock.genesis.Gen def 'absolute value of #value should not be negative'() { expect: value.abs() >= 0 where: value << Gen.int.take(100) } Looks great, but if you want to generate e.g. lists of random integers,net.java.quickcheck has nicer API and is not Groovy-specific: import static net.java.quickcheck.generator.CombinedGeneratorsIterables.someLists import static net.java.quickcheck.generator.PrimitiveGenerators.integers def 'sum of non-negative numbers from #list should not be negative'() { expect: list.findAll{it >= 0}.sum() >= 0 where: list << someLists(integers(), 100) } This test is interesting. It makes sure sum of non-negative numbers is never negative - by generating 100 lists of randoms ints. Sounds reasonable. However multiple tests are failing. First of all due to integer overflow sometimes two positiveints add up to a negative one. Duh! Another type of failure that was discovered is actually frightening. While [1,2,3].sum() is 6, obviously, [].sum() is... null(WAT?) As you can see even silliest and most basic property based tests can be useful in finding unusual corner cases in your data. But wait, I said testing absolute of intdiscovered one bug. Actually it didn't, because of poor (too "random") data generators, not returning known edge values in the first place. We will fix that in the next article.

If you use Java for programming, read on to learn about tools like Eclipse IDE, the Java Development Kit, and other must-know tools.

in “ semihosting with kinetis design studio ” i used printf() to exchange text and data between the target board and the host using the debug connection. kinetis design studio (kds) has that semihosting baked into its libraries. what about if using the gnu arm embedded (launchpad) tools and libraries (see “ switching arm gnu tool chain and libraries in kinetis design studio “)? actually it requires two more steps, but is very easy too. semihosting output there are three things to be in place to use semihosting with the gnu arm embedded (launchpad) libraries: option in the gnu linker settings enabling semihosting in the debugger settings initializing the gnu libraries linker option to enable semihosting for the gnu arm embedded ( launchpad ) libraries, i need to add --specs=rdimon.specs to the linker options: linker option to enable semihosting in case i’m using newlib-nano and want to use printf() and/or scanf() with floating point support, i need to pull in some symbols explicitly with the linker options ‘u': -u _scanf_float -u _printf_float debugger settings in the gnu arm eclipse plugins, i need to enable semihosting. segger j-link for segger j-link, i enable the console in the launch configuration: allocated semihosting console for segger additionally i enable semihosting options in the startup options of the debugger: enabled semihosting in the startup options for segger p&e multilink for p&e the following settings are used: semihosting settings for pne settings for openocd the following settings are used for openocd: openocd semihosting settings initializing the gnu libraries if you would now try to use semihosting with running the debugger, you probably will get error messages like this (e.g. from segger j-link): warning: semihosting command sys_flen failed. handle is 0. warning: semihosting command sys_write failed. handle is 0. warning: semihosting command sys_write failed. handle is 0. warning: semihosting command sys_write failed. handle is 0. the reason is that the semihosting needs to be enabled by the application. i need to call initialise_monitor_handles() before i’m using printf() : 1 2 3 4 5 6 7 8 extern void initialise_monitor_handles( void ); /* prototype */ int main( void ) { initialise_monitor_handles(); /* initialize handles */ for (;;) { printf ( "hello world!\r\n" ); } } with this, i can use printf() and scanf() through a debugger connection. semihosting printf output summary while i don’t like printf() for many reasons, sometimes it is useful to exchange data with the host. using semihosting no physical connection is required, as the communication goes through the debugger. it is somewhat intrusive, and adds code and data overhead, but the gnu arm embedded (launchpad) libraries (both newlib and newlib-nano) have semihosting built-in. it is a matter to enable it in the linker and debugger settings, and to initialize the handles in the application. happy semihosting :-)

Recently I had the opportunity to work on a project were I needed to create a custom SQL Server image for use with Azure VMs. The process was a little more challenging than I initially anticipated. I think this is mostly because I was not familiar with the process of preparing a SQL Server image. Perhaps this isn’t much of a challenge for an experienced SQL Server DBA or IT Pro. For me, it was a great learning experience. Why a Custom SQL Server Image? The Azure VM image gallery already contains a SQL Server image. It’s very easy to create a new SQL Server VM using this image. However, doing so has a few important trade-offs to consider: Unable to fully customize the base install of SQL Server. This is a template/image after all – you get a VM configured the way the image was configured. Unable to use your own SQL Server license. If your company has an Enterprise Agreement (EA) with Microsoft, it’s likely there is already some SQL Server licenses built into that agreement. Depending on the details, it may be significantly cheaper to use the licenses from the EA instead of paying the SQL Server VM image upcharge from Azure. The Basic Steps There are 6 basic steps to creating a custom SQL Server VM image for use in Azure. Provision a new base Windows Server VM Download the SQL Server installation media Run SQL Server setup to prepare an image Configure Windows to complete the installation of SQL Server Capture the image and add it to the Azure VM image gallery Create a new VM instance using the custom SQL Server image The basic idea here is to create a base VM, customize it with a SQL Server image, capture the VM to create an image, and then provision new VMs using that captured VM image. Let’s dive into each of these in a little more detail. Note: the terminology here can be a little confusing. When referring to the VM used to create the template/image, I’ll use the term “base VM”. When referring to the VM created from the base VM, I’ll use the term “VM instance”. 1. Provision a new base Windows Server VM There are multiple ways to create a Windows Server VM in Azure. Creating a VM via the Azure management portal and PowerShell are probably the two most popular options. Be sure to check out this tutorial to learn how to do so via the portal. For the purposes of this post, I’ll do so via PowerShell. $img = Get-AzureVMImage ` | where { ( $_.PublisherName -ilike "Microsoft*" -and $_.ImageFamily -ilike "Windows Server 2012 Datacenter" ) } ` | Sort-Object -Unique -Descending -Property ImageFamily ` | sort -Descending -Property PublishDate ` | select -First(1) $vmConfig = New-AzureVMConfig -Name "sql-1" -InstanceSize Small -ImageName $img.ImageName | Add-AzureProvisioningConfig -Windows -AdminUsername "[admin-username-here]" -Password "[admin-password-here]" New-AzureVM -ServiceName "SQLServerVMTemplate" -VMs $vmConfig -Location "East US" -WaitForBoot 2. Download the SQL Server installation media With the base Windows Server 2012 VM created, we can now get ready to prepare (sysprep) the SQL Server installation. To do that, we need to get the SQL Server installation media onto the machine. The easiest way I found to do this was to leverage Azure blob storage. Upload the SQL Server ISO file to Azure blob storage Remote Desktop (RDP) into the base VM From the VM, download the SQL Server ISO file to the local disk Mount the SQL Server ISO file to the VM Copy the ISO contents (not the ISO file itself) to the VM’s C:\ drive. For example, use C:\sql The SQL Server installation media files need to be copied to the local C: drive so it can be used later to complete the SQL Server installation (when provisioning the actual SQL Server VM instance). 3. Run SQL Server setup to prepare an image In order to prepare the (sysprep’d) SQL Server VM image (which we can use as a template for future VMs), we need to run the SQL Server installation and instruct it topreparean image – not run the full installation. An easy way to do this is with a SQL Server configuration file, an example of which I’ve included below. ConfigurationFile.ini ;SQL Server 2012 Configuration File [OPTIONS] ; Specifies a Setup workflow, like INSTALL, UNINSTALL, or UPGRADE. This is a required parameter. ACTION="PrepareImage" ; Detailed help for command line argument ENU has not been defined yet. ENU="True" ; Parameter that controls the user interface behavior. Valid values are Normal for the full UI, AutoAdvance for a simplified UI, and EnableUIOnServerCore for bypassing Server Core setup GUI block. ;UIMODE="Normal" ; Specifies setup not display any user interface. ;QUIET="False" ; Specifies setup to display progress only, without any user interaction. QUIETSIMPLE="True" ; Specifies whether SQL Server Setup should discover and include product updates. The valid values are True and False or 1 and 0. By default SQL Server Setup will include updates that are found. UpdateEnabled="True" ; Specifies features to install, uninstall, or upgrade. The list of top-level features include SQL, AS, RS, IS, MDS, and Tools. The SQL feature will install the Database Engine, Replication, Full-Text, and Data Quality Services (DQS) server. The Tools feature will install Management Tools, Books online components, SQL Server Data Tools, and other shared components. FEATURES=SQLENGINE ; Specifies the location where SQL Server Setup will obtain product updates. The valid values are "MU" to search Microsoft Update, a valid folder path, a relative path such as .\MyUpdates or a UNC share. By default SQL Server Setup will search Microsoft Update or a Windows Update service through the Window Server Update Services. UpdateSource="MU" ; Displays the command line parameters usage HELP="False" ; Specifies that the detailed Setup log should be piped to the console. INDICATEPROGRESS="False" ; Specifies that Setup should install into WOW64. This command line argument is not supported on an IA64 or a 32-bit system. X86="False" ; Specifies the root installation directory for shared components. This directory remains unchanged after shared components are already installed. INSTALLSHAREDDIR="C:\Program Files\Microsoft SQL Server" ; Specifies the root installation directory for the WOW64 shared components. This directory remains unchanged after WOW64 shared components are already installed. INSTALLSHAREDWOWDIR="C:\Program Files (x86)\Microsoft SQL Server" ; Specifies the Instance ID for the SQL Server features you have specified. SQL Server directory structure, registry structure, and service names will incorporate the instance ID of the SQL Server instance. INSTANCEID="MSSQLSERVER" ; Specifies the installation directory. INSTANCEDIR="C:\Program Files\Microsoft SQL Server" There are two steps in this process: Copy the ConfigurationFile.ini file (from your local PC) to the same location as the SQL Server installation media (i.e.c:\sql) on the base VM. Run SQL Server setup to prepare an image. From a command prompt (on the base VM), navigate to theC:\sqlfolder and then execute the following command: Setup.exe /ConfigurationFile=ConfigurationFile.ini /IAcceptSQLServerLicenseTerms=true 4. Configure Windows to complete the installation of SQL Server At this point the base VM should have an “installation” of SQL Server that is not fully completed. The SQL Server bits are in place, but they’re not configured for a full server install . . . at least not yet. The final configuration of SQL Server will take place when the VM instance (of which this template/image is the base) is provisioned and boots up for the first time. This is accomplished by using a CMD file with the following content: @ECHO OFF && SETLOCAL && SETLOCAL ENABLEDELAYEDEXPANSION && SETLOCAL ENABLEEXTENSIONS REM All commands will be executed during first Virtual Machine boot "C:\Program Files\Microsoft SQL Server\110\Setup Bootstrap\SQLServer2012\setup.exe" /QS /ACTION=CompleteImage /INSTANCEID=MSSQLSERVER /INSTANCENAME=MSSQLSERVER /IACCEPTSQLSERVERLICENSETERMS=1 /SQLSYSADMINACCOUNTS=%COMPUTERNAME%\Administrators /BROWSERSVCSTARTUPTYPE=AUTOMATIC /INDICATEPROGRESS /TCPENABLED=1 /PID="[YOUR-SQL-SERVER-PRODUCT-ID-HERE]" On your local PC, save the file as SetupComplete2.cmd RDP / log into the base VM Copy the SetupComplete2.cmd from your local PC file to the c:\Windows\OEM folder on the base VM Change the value for the SQLSYSADMINACCOUNTS value to be that of the administrative account created on the VM (or better yet – the local Administrators group account) If needed, supply the SQL Server product ID (PID) value. When Windows starts on the new VM instance for the first time, the SetupComplete2.cmd file should automatically run. It is invoked by the SetupComplete.cmd file already on the machine. 5. Capture the image and add it to the Azure VM image gallery At this point a base SQL Server VM has been created and the groundwork laid to complete the install. Now it is time to create the VM image from the base VM, and do to that you sysprep and capture the base VM. Please follow the guide on How to Capture a Windows Virtual Machine to Use as a Template. 6. Create a new VM using the custom SQL Server image With a new custom VM image template available in the VM image gallery, you can provision a new VM instance using that custom template. Upon first boot, the newly provisioned VM should complete the full SQL Server installation as laid out in your SetupComplete2.cmd file. Please follow the guide on How to Create a Custom Virtual Machine for more information on creating the VM from the template. Closing Thoughts One of the quirks I noticed when preparing the base SQL Server image is that it was not possible to prepare the image with SQL Server Management Studio (SSMS). I would have to do the install after the newly provisioned VM instance is created. Not hard, but time consuming (an annoying if doing this on multiple VM instances). I later learned that SQL Server 2012 Cumulative Update 1 does allow for preparing a SQL Server image with SSMS installed. I’ve included a link below that describes the process for creating a SQL Server image with CU1. In the end, this process really is not all that hard. Time consuming? Yes! The worst part (at least for me) was really just understanding how the SQL Server installation and sysprep process works. Once I wrapped my head around that, the process was a lot smoother. Helpful Resources While I was learning how to create a custom SQL Server VM image, the following resources were very helpful: How to: Create a Windows Azure Virtual Machine Operating System Image for Microsoft Dynamics NAV. This MSDN article provided the jumping off point on learning how to install SQL Server by using a sysprep image. Install SQL Server 2012 from the Command Prompt Install SQL Server 2012 Using a Configuration File Install SQL Server 2012 Using SysPrep How to create a slipstream SQL Server 2012 and Cumulative Update 1 image –http://sqlperformance.com/2012/12/system-configuration/sql-2012-slipstream I would like to thank Scott Klein for his assistance in verifying these steps. His help was extremely valuable to ensure I was doing this the right way.

JSF is not what we often think it is. It's also a framework that can be somewhat tricky to debug, especially when first encountered. In this post let's go over on why that is and provide some JSF debugging techniques. We will go through the following topics: JSF is not what we often think The difficulties of JSF debugging How to debug JSF systematically How JSF Works - The JSF lifecycle Debugging an Ajax request from browser to server and back Debugging the JSF frontend Javascript code Final thoughts - alternatives? (questions to the reader) JSF is not what we often think JSF looks on first look like an enterprise Java/XML frontend framework, but under the hood it really isn't. It's really a polyglot Java/Javascript framework, where the client Javascript part is non-neglectable and also important to understand it. It also has good support for direct HTML/CSS use. JSF developers are on ocasion already polyglot developers, whose primary language is Java but still need to use ocasionally Javascript. The difficulties of JSF debugging When comparing JSF to GWT and AngularJS in a previous post, I found that the (most often used) approach that the framework takes of abstracting HTML and CSS from the developer behind XML adds to the difficulty of debugging, because it creates an extra level of indirection. A more direct approach of using HTML/CSS directly is also possible, but it seems enterprise Java developers tend to stick to XML in most cases, because it's a more familiar technology. Also another problem is that the client side Javascript part of the framework/libraries is not very well documented, and it's often important to understand what is going on. The only way to debug JSF systematically When first encountering JSF, I first tried to approach it from a Java, XML and documentation only. While I could do a part of the work that way, there where frequent situations where that approach was really not sufficient. The conclusion that I got to is that in order to be able to debug JSF applications effectively, an understanding of the following is needed: HTML CSS Javascript HTTP Chrome Dev Tools, Firebug or equivalent The JSF Lifecycle This might sound surprising to developers that work mostly in Java/XML, but this web-centric approach to debugging JSF is the only way that I managed to tackle many requirements that needed some significant component customization, or to be able to fix certain bugs. Let’s start by understanding the inner workings of JSF, so that we can debug it better. The JSF take on MVC The way JSF approaches MVC is that the whole 3 components reside on the server side: The Model is a tree of plain Java objects The View is a server side template defined in XML that is read to build an in-memory view definition The Controller is a Java servlet, that receives each request and processes them through a series of steps The browser is assumed to be simply a rendering engine for the HTML generated at server side. Ajax is achieved by submitting parts of the page for server processing, and requesting a server to ‘repaint’ only portions of the screen, without navigating away from the page. The JSF Lifecycle Once an HTTP request reaches the backend, it gets caught by the JSF Controller that will then process it. The request goes through a series of phases known as the JSF lifecycle, which is essential to understand how JSF works: Design Goals of the JSF Lifecycle The whole point of the lifecycle is to manage MVC 100% on the server side, using the browser as a rendering platform only. The initial idea was to decouple the rendering platform from the server-side UI component model, in order to allow to replace HTML with alternative markup languages by swapping the Render Response phase. This was in the early 2000's when HTML could be soon replaced by XML-based alternatives (that never came to be), and then HTML5 came along. Also browsers where much more qwirkier than what they are today, and the idea of cross-browser Javascript libraries was not widespread. So let’s go through each phase and see how to debug it if needed, starting in the browser. Let's base ourselves in a simple example that uses an Ajax request. A JSF 2 Hello World Example The following is a minimal JSF 2 page, that receives an input text from the user, sends the text via an Ajax request to the backend and refreshes only an output label: JSF 2.2 Hello World Example The page looks like this: Following one Ajax request - to the server and back Let’s click submit in order to trigger the Ajax request, and use the Chrome Dev Tools Network tab (right click and inspect any element on the page).What goes over the wire? This is what we see in the Form Data section of the request: j_idt8:input: Hello World javax.faces.ViewState: -2798727343674530263:954565149304692491 javax.faces.source: j_idt8:j_idt9 javax.faces.partial.event: click javax.faces.partial.execute: j_idt8:j_idt9 j_idt8:input javax.faces.partial.render: j_idt8:output javax.faces.behavior.event: action javax.faces.partial.ajax:true This request says: The new value of the input field is "Hello World", send me a new value for the output field only, and don't navigate away from this page. Let's see how this can be read from the request. As we can see, the new values of the form are submitted to the server, namely the “Hello World” value. This is the meaning of the several entries: javax.faces.ViewState identifies the view from which the request was made. The request is an Ajax request, as indicated by the flag javax.faces.partial.ajax, The request was triggered by a click as defined in javax.faces.partial.event. But what are those j_ strings ? Those are space separated generated identifiers of HTML elements. For example this is how we can see what is the page element corresponding to j_idt8:input, using the Chrome Dev Tools: There are also 3 extra form parameters that use these identifiers, that are linked to UI components: javax.faces.source: The identifier of the HTML element that originated this request, in this case the Id of the submit button. javax.faces.execute: The list of identifiers of the elements whose values are sent to the server for processing, in this case the input text field. javax.faces.render: The list of identifiers of the sections of the page that are to be ‘repainted', in this case the output field only. But what happens when the request hits the server ? JSF lifecycle - Restore View Phase Once the request reaches the server, the JSF controller will inspect the javax.faces.ViewState and identify to which view it refers. It will then build or restore a Java representation of the view, that is somehow similar to the document definition in the browser side. The view will be attached to the request and used throughout. There is usually little need to debug this phase during application development. JSF Lifecycle - Apply Request Values The JSF Controller will then apply to the view widgets the new values received via the request. The values might be invalid at this point. Each JSF component gets a call to it’s decode method in this phase. This method will retrieve the submitted value for the widget in question from the HTTP request and store it on the widget itself. To debug this, let’s put a breakpoint in the decode method of the HtmlInputText class, to see the value “Hello World”: Notice the conditional breakpoint using the HTML clientId of the field we want. This would allow to quickly debug only the decoding of the component we want, even in a large page with many other similar widgets. Next after decoding is the validation phase. JSF Lifecycle - Process Validations In this phase, validations are applied and if the value is found to be in error (for example a date is invalid), then the request bypasses Invoke Application and goes directly to Render Response phase. To debug this phase, a similar breakpoint can be put on method processValidators, or in the validators themselves if you happen to know which ones or if they are custom. JSF Lifecycle - Update Model In this phase, we know all the submitted values where correct. JSF can now update the view model by applying the new values received in the requests to the plain Java objects in the view model. This phase can be debugged by putting a breakpoint in the processUpdates method of the component in question, eventually using a similar conditional breakpoint to break only on the component needed. JSF Lifecycle - Invoke Application This is the simplest phase to debug. The application now has an updated view model, and some logic can be applied on it. This is where the action listeners defined in the XML view definition (the 'action' properties and the listener tags) are executed. JSF Lifecycle - Render Response This is the phase that I end up debugging the most: why is the value not being displayed as we expect it, etc, it all can be found here. In this phase the view and the new model values will be transformed from Java objects into HTML, CSS and eventually Javascript and sent back over the wire to the browser. This phase can be debugged using breakpoints in the encodeBegin, encodeChildren and encodeEnd methods of the component in question. The components will either render themselves or delegate rendering to aRenderer class. Back in the browser It was a long trip, but we are back where we started! This is how the response generated by JSF looks once received in the browser: -8188482707773604502:6956126859616189525> What the Javascript part of the framework will do is to take the contents of the partial response, update by update. Using the Id of the update, the client side JSF callback will search for a component with that Id, delete it from the document and replace it with the new updated version. In this case, "Hello World" will show up on the label next to the Input text field! And so thats how JSF works under the hood. But what about if we need to debug the Javascript part of the framework? Debugging the JSF Javascript Code The Chrome Dev Tools can help debug the client part. For example let’s say that we want to halt the client when an Ajax request is triggered. We need to go to the sources tab, add an XHR (Ajax) breakpoint and trigger the browser action. The debugger will stop and the call stack can be examined: For some frameworks like Primefaces, the Javascript sources might be minified (non human-readable) because they are optimized for size. To solve this, download the source code of the library and do a non minified build of the jar. There are usually instructions for this, otherwise check the project poms. This will install in your Maven repository a jar with non minified sources for debugging. The UI Debug tag: The ui:debug tag allows to view a lot of debugging information using a keyboard shortcut, see here for further details. Final Thoughts JSF is very popular in the enterprise Java world, and it handles a lot of problems well, specially if the UI designers take into account the possibilities of the widget library being used. The problem is that there are usually feature requests that force us to dig deeper into the widgets internal implementation in order to customize them, and this requires HTML, CSS, Javascript and HTTP plus JSF lifecycle knowledge. Is polyglot an alternative? We can wonder that if developers have to know a fair amount about web technologies in order to be able to debug JSF effectively, then it would be simpler to build enterprise front ends (just the client part) using those technologies directly instead. It's possible that a polyglot approach of a Java backend plus a Javascript-only frontend could be proved effective in a nearby future, specially using some sort of a client side MVC framework like Angular. This would require learning more Javascript, (have a look at Javascript for Java developers post if curious), but this is already often necessary to do custom widget development in JSF anyway. Conclusions and some questions to the reader Thanks for reading, please take a moment to share your thoughts on these matters on the comments bellow: do you believe polyglot development (Java/Javascript) is a viable alternative in general, and in your workplace in particular? Did you find one of the GWT-based frameworks (plain GWT, Vaadin, Errai), or the Play Framework to be easier to use and of better productivity?

This article lists some of the best practices that would be useful for developers while they are coding with AngularJS. These are out of my own experiences while working on AngularJS and do not warranty the entire list. I am sure there can be more to this list and thus, request my readers to suggest/comment such that they could be added to the list below. Found some of the following pages which presents a set of good practices you would want to refer. Thanks to the readers for the valuable contribution. AngularJS Style Guide App Structure Best practices Initialization One should try and place the

we write service apis which cater to certain business logic. there are few cross-cutting concerns that cover all service apis like security, logging, auditing, measuring latencies and so on. this is a repetitive non-business code which can be reused among other methods. one way to reuse is to move these repetitive code into its own methods and invoke them in the service apis somethings like: public class myservice{ public servicemodel service1(){ isauthorized(); //execute business logic. } } public class myanotherservice{ public servicemodel service1(){ isauthorized(): //execute business logic. } } the above approach will work but not without creating code noise, mixing cross-cutting concerns with the business logic. there is another approach to solve the above requirements which is by using aspect and this approach is called aspect oriented programming (aop). there are a different ways you can make use of aop – by using spring aop, javaee aop. in this example i will try to use aop using cdi in java ee applications. to explain this i have picked a very simple example of building a web application to fetch few records from database and display in the browser. creating the data access layer the table structure is: create table people( id int not null auto_increment, name varchar(100) not null, place varchar(100), primary key(id)); lets create a model class to hold a person information package demo.model; public class person{ private string id; private string name; private string place; public string getid(){ return id; } public string setid(string id) { this.id = id;} public string getname(){ return name; } public string setname(string name) { this.name = name;} public string getplace(){ return place; } public string setplace(string place) { this.place = place;} } lets create a data access object which exposes two methods - to fetch the details of all the people to fetch the details of one person of given id package demo.dao; import demo.common.databaseconnectionmanager; import demo.model.person; import java.sql.connection; import java.sql.preparedstatement; import java.sql.resultset; import java.sql.sqlexception; import java.sql.statement; import java.util.arraylist; import java.util.list; public class peopledao { public list getallpeople() throws sqlexception { string sql = "select * from people"; connection conn = databaseconnectionmanager.getconnection(); list people = new arraylist<>(); try (statement statement = conn.createstatement(); resultset rs = statement.executequery(sql)) { while (rs.next()) { person person = new person(); person.setid(rs.getstring("id")); person.setname(rs.getstring("name")); person.setplace(rs.getstring("place")); people.add(person); } } return people; } public person getperson(string id) throws sqlexception { string sql = "select * from people where id = ?"; connection conn = databaseconnectionmanager.getconnection(); try (preparedstatement ps = conn.preparestatement(sql)) { ps.setstring(1, id); try (resultset rs = ps.executequery()) { if (rs.next()) { person person = new person(); person.setid(rs.getstring("id")); person.setname(rs.getstring("name")); person.setplace(rs.getstring("place")); return person; } } } return null; } } you can use your own approach to get a new connection. in the above code i have created a static utility that returns me the same connection. creating interceptors creating interceptors involves 2 steps: create interceptor binding which creates an annotation annotated with @interceptorbinding that is used to bind the interceptor code and the target code which needs to be intercepted. create a class annotated with @interceptor which contains the interceptor code. it would contain methods annotated with @aroundinvoke , different lifecycle annotations, @aroundtimeout and others. lets create an interceptor binding by name @latencylogger package demo; import java.lang.annotation.target; import java.lang.annotation.retention; import static java.lang.annotation.retentionpolicy.*; import static java.lang.annotation.elementtype.*; import javax.interceptor.interceptorbinding; @interceptorbinding 10 @retention(runtime) 11 @target({method, type}) public @interface latencylogger { } now we need to create the interceptor code which is annotated with @interceptor and also annotated with the interceptor binding we created above i.e @latencylogger : package demo; import java.io.serializable; import javax.interceptor.aroundinvoke; import javax.interceptor.interceptor; import javax.interceptor.invocationcontext; @interceptor @latencylogger public class latencyloggerinterceptor implements serializable{ @aroundinvoke public object computelatency(invocationcontext invocationctx) throws exception{ long starttime = system.currenttimemillis(); //execute the intercepted method and store the return value object returnvalue = invocationctx.proceed(); long endtime = system.currenttimemillis(); system.out.println("latency of " + invocationctx.getmethod().getname() +": " + (endtime-starttime)+"ms"); return returnvalue; } } there are two interesting things in the above code: use of @aroundinvoke parameter of type invocationcontext passed to the method @aroundinvoke designates the method as an interceptor method. an interceptor class can have only one method annotated with this annotation. when ever a target method is intercepted, its context is passed to the interceptor. using the invocationcontext one can get the method details, the parameters passed to the method. we need to declare the above interceptor in the web-inf/beans.xml file demo.latencyloggerinterceptor creating service apis annotated with interceptors we have already created the interceptor binding and the interceptor which gets executed. now lets create the service apis and then annotate them with the interceptor binding /* * to change this license header, choose license headers in project properties. * to change this template file, choose tools | templates * and open the template in the editor. */ package demo.service; import demo.latencylogger; import demo.dao.peopledao; import demo.model.person; import java.sql.sqlexception; import java.util.list; import javax.inject.inject; public class peopleservice { @inject peopledao peopledao; @latencylogger public list getallpeople() throws sqlexception { return peopledao.getallpeople(); } @latencylogger public person getperson(string id) throws sqlexception { return peopledao.getperson(id); } } we have annotated the service methods with the interceptor binding @latencylogger . the other way would be to annotate at the class level which would then apply the annotation to all the methods of the class. another thing to notice is the @inject annotation that injects the instance i.e injects the dependency into the class. next is to wire up the controller and view to show the data. the controller is the servlet and view is a plain jsp using jstl tags. /* * to change this license header, choose license headers in project properties. * to change this template file, choose tools | templates * and open the template in the editor. */ package demo; import demo.model.person; import demo.service.peopleservice; import java.io.ioexception; import java.sql.sqlexception; import java.util.list; import java.util.logging.level; import java.util.logging.logger; import javax.inject.inject; import javax.servlet.servletexception; import javax.servlet.annotation.webservlet; import javax.servlet.http.httpservlet; import javax.servlet.http.httpservletrequest; import javax.servlet.http.httpservletresponse; @webservlet(name = "aopdemo", urlpatterns = {"/aopdemo"}) public class aopdemoservlet extends httpservlet { @inject peopleservice peopleservice; @override public void doget(httpservletrequest request, httpservletresponse response) throws servletexception, ioexception { try { list people = peopleservice.getallpeople(); person person = peopleservice.getperson("2"); request.setattribute("people", people); request.setattribute("person", person); getservletcontext().getrequestdispatcher("/index.jsp").forward(request, response); } catch (sqlexception ex) { logger.getlogger(aopdemoservlet.class.getname()).log(level.severe, null, ex); } } } the above servlet is available at http://localhost:8080/ /aopdemo. it fetches the data and redirects to the view to display the same. note that the service has also been injected using @inject annotation. if the dependencies are not injected and instead created using new then the interceptors will not work. this is an important point which i realised while building this sample. the jsp to render the data would be hello world! idnameplace details for person with id=2 with this you would have built a very simple app using interceptors. thanks for reading and staying with me till this end. please share your queries/feedback as comments. and also share this article among your friends