Recently, there is a heaty debate regarding TDD which started by DHH when he claimed that TDD is dead.
This ongoing debate managed to capture the attention of developers world, including us.
Some mini debates have happened in our office regarding the right practices to do testing.
In this article, I will represent my own view.
How many kinds of tests have you seen?
From the time I joined industry, here are the kinds of tests that I have worked on:
- Unit Test
- System/Integration/Functional Test
- Regression Test
- Test Harness/Load Test
- Smoke Test/Spider Test
The above test categories are not necessarily mutually exclusive. For
example, you can crate a set of automated functional tests or Smoke
tests to be used as regression test. For the benefit of newbie, let do a
quick review for these old concepts.
Unit Test aim to test the functional of a unit of code/component. For
Java world, unit of code is the class and each Java class suppose to
have an unit test. The philosophy of Unit Test is simple. When all the
components are working, the system as a whole should work.
A component rarely work alone. Rather, it normally interacts with other
components. Therefore, in order to write Unit Test, developers need to
mock other components. This is the problem that DHH and James O Coplien
criticize Unit Test for huge effort that gain little benefit.
There is no concrete naming as people often use different terms to
describe similar things. Contradict to Unit Test, for functional test,
developers aim to test a system function as a whole, which may involve
Normally, for functional test, the data is retrieved and store to the
test database. Of course, there should be a pre-step to set-up test data
before running. DHH likes this kind of test. It helps developers test
all the functions of the system without huge effort to set-up mock
Functional test may involve asserting web output. In the past, it is
mostly done with htmlUnit but with recent improvement of Selenium Grid,
Selenium became the preferred choice.
In this industry, you may end up spend more time maintaining system than
developing new one. Software changes all the time and it is hard to
avoid risk whenever making changes. Regression Test supposes to capture
any defect that caused by changes.
In the past, software house did have one army of testers but the current
trend is automated testing. It means that developers will deliver
software with full set of tests that suppose to be broken whenever a
function is spoiled.
Whenever a bug is detected, a new test case should be added to cover new
bug. Developers create the test, let it fail, and fix the bug to make
it pass. This practice is called Test Driven Development.
Test Harness/Load Test
Normal test case does not capture system performance. Therefore, we need
to develop another set of tests for this purpose. In the simplest form,
we can set the time out for the functional test that run in continuous
integration server. The tricky part is this kind of test is very system
dependant and may fail if the system is overloaded.
The more popular solution is to run load test manually by using profiling tool like JMeter
or create our own load test app.
Smoke Test/Spider Test
Smoke Test and Spider Test are two special kinds of tests that may be more relevant to us. WDS provides KAAS (Knowledge as a Service
for wireless industry. Therefore, our applications are refreshed
everyday with data changes rather than business logic changes. It is
specific to us that system failure may come from data change rather than
Test are set of pre-defined test cases run on integration server with
production data. It helps us to find out any potential issues for the
daily LIVE deployment.
Similar to Smoke Test, Spider Test runs with real data but it work like a
crawler that randomly click on any link or button available. One of our
system contains so many combination of inputs that it is not possible
to be tested by human (closed to 100.000 combinations of inputs).
Our Smoke Test randomly choose some combination of data to test. If it
manage to run for a few hours without any defect, we will proceed with
our daily/weekly deployment.
The Test Culture in our environment
To make it short, WDS is a TDD temple. If you create the implementation
before writing test cases, better be quiet about it. If you look at WDS
self introduction, TDD is mentioned only after Agile and XP
"We are:- agile & XP,
deployment, Linux & AWS, Jeans & T-shirts, Tea & cake
Many high level executives in WDS start their career as developers. That
helps to fostering our culture as an engineering-oriented company.
Requesting resources to improve test coverage or infrastructure are
We do not have QA. In worst case, Product Owner or customers detect
bugs. In best case, we detect bugs by test cases or by team mates during
peer review stage.
Regarding Singapore office, most of our team members grow up absorbing
Ken Beck and Martin Fowler books and philosophy. That why most of them
are hardcore TDD worshippers. Even, one member of our team is Martin
The focus of testing in our working environment did bear fruits. WDS production defects rate is relatively low.
My own experience and personal view with testing
That is enough about self appraisal. Now, let me share my experience about testing.
Generally, Automated Testing works better than QA
Comparing the output of traditional software house that packed with an
army of QA with modern Agile team that deliver fully test coverage
products, the latter normally outperform in term of quality and even
cost effectiveness. Should QA jobs be extinct soon?
Over monitoring may hint lack of quality
It sounds strange but over the years, I developed insecure feeling
whenever I saw a project that have too many layer of monitoring. Over
monitoring may hint lack of confidence and in deed, these systems crash
very often with unknown reasons.
Writing test cases takes more time that developing features
DDH is definitely right on this. Writing Test Cases mean that you need
to mock input and assert lots of things. Unless you keep writing
spaghetti code, developing features take much less times compare to
You know it when you did it. Life is much better if you only need to
test Restful API or static html pages. Unfortunately, the trend of
side. For UI Testing, Asynchronous is evil.
you want to go with full control testing framework like htmlUnit or
using a more practical, generic one like Selenium, it will be a great
surprise for me if you never encounter random failures.
I guess every developer know the feeling of failing to get the build
pass at the end of the week due to random failure test cases.
Developers always over-estimate their software quality
It is applicable to me as well because I am an optimistic person. We
tend to think that our implementation is perfect until the tests failed
or someone help to point out a bug.
Sometimes, we change our code to make writing test cases easier
Want it or not, we must agree with DHH on this point. Pertaining to Java
world, I have seen people exposing internal variable, creating dummy
wrapper for framework object (like HttpSession, HttpRequest,...) so that
it is easier to write Unit Test. DHH find it so uncomfortable that he
chose to walk way from Unit Test.
On this part, I half agree and half disagree with him. From my own view,
altering design, implementation for the sake of testing is not
favourable. It is better if developers can write the code without any
concern of mocking input.
However, aborting Unit Testing for the sake of having a simple and
convenient life is too extreme. The right solution should be designing
the system is such a way that business logic is not so tight-coupling
with framework or infrastructure.
Domain Driven Design
For newbie, Domain Driven Design give us a system with following layers.
If you notice, the above diagram has more abstract layers than Rails or the Java adoption of Rails, Play framework
. I understand that creating more abstract layers can cause bloated system but for DDD, it is a reasonable compromise.
Let elaborate further on the content of each layer:
This layer is where you store your repository implementation or any
other environment specific concerns. For infrastructure, keep the API as
simple, dummy as possible and avoid having any business logic
For this layer, Unit Test is a joke. If there is any thing to write, it
should be integration test, which working with real database.
Domain layer is the most important layer. It contains all system
business logics without any framework, infrastructure, environment
concern. Your implementation should look like a direct translation of
user requirements. Any input, output, parameter are POJO
Domain layer should be the first layer to be implemented. To fully
complete the logic, you may need interface/API of the infrastructure
layer. It is best practice to keep the API in Domain Layer and concrete
implementation in Infrastructure layer.
The best kind of test cases for Domain layer is Unit Test as your
concern is not the system UI or environment. Therefore, it helps
developers to avoid doing dirty works of mocking framework object.
For mocking internal state of object, my preferred choice is using
Reflection utility to setup object rather than exposing internal
variables through setters.
Application Layer/User Interface
Application Layer is where you start thinking about how to represent
your business logic to customer. If the logic is complex or involving
many consecutive requests, it is possible to create Facades
Reaching this point, developers should think more about clients than the
system. The major concerns should be customer's devices, UI
responsiveness, load balance, stateless or stateful session, Restful
API. This is the place for developers to showcase framework talent and
For this layer, the better kind of test cases is functional/integration test.
Similar as above, try your best to avoid having any business logic in Application Layer.
Why it is hard to write Unit Test in Rails?
Now, if you look back to Rails or Play framework, there is no clear
separation of layers like above. The Controllers render inputs, outputs
and may contains business logic as well. Similar behaviours applied if
you use the ServletAPI without adding any additional layer.
The Domain object in Rails is an active record and has a tight-coupling with database schema.
Hence, for whatever unit of code that developers want to write test
cases, the inputs and output are nots POJO. This make writing Unit Test
We should not blame DHH for this design as he follow another philosophy
of software development with many benefits like simple design, low
development effort and quick feedback. However, I myself do not follow
and adopt all of his ideas for developing enterprise applications.
Some of his ideas like convention over configuration are great and did
cause a major mindset change in developers world but other ideas end up
as trade off. Being able to quickly bring up a website may later turn to
troubles implementing features that Rails/Play do not support.
- Unit Test is hard to write if you business logic is tight-coupling to framework.
- Focusing and developing business logic first may help you create better design.
- Each kinds of components suit different kinds of test cases.
This is my own view of Testing. If you have any other opinions, please feedback.