Testing – do we have to?

Reliability is the core of good testing. As we know our systems are made from many objects/components. Our system is as reliable as components on which it is build. It obvious that when our components are n percent reliable and we have k components our system is (n/100)^k percent reliable. We can quickly realize that if we have components that are not one hundred percent reliable, our system goes to zero percent as the number of components grow. From the other point of view if all of our components are one hundred percent reliable and we introduce to our system one component which is 50% it makes that our system is now 50% reliable. Of course it is theoretical, in practice we should provide weights for our components, and the system is rather non linear.

Now we agreed that reliability is good thing, but to talk about it we should measure it in some way. Of course to measure our system we should measure our low level components. To do this we must ensure that low level objects are reliable through unit testing. We must run this tests as often as we can (probably every commit), so the most important is that this tests must execute quickly (probably 2-3 second is maximum). And for such task the automation is needed. The answer is continuous integration tools, such as ( Atlassian Bamboo, TeamCity, Hudson, CruiseControl, etc.. ..). If you heard this idiom for the first time, you should read Martin Flower article about Continuous Integration.  BTW if you heard but don’t read this article you should read it, anyway. Even maven site goal may help here. In my company we use Atlassian Bamboo and it’s great tool, Atlassian has great license politics so if your projects is open source you can count on your free bamboo license ;). If your company didn’t has money to buy such software you should see Hudson.

Test Types :

Unit Tests – verify behavior of small elements (single class). Tests should be quick and run as often as we can. Every language has it’s own test libraries. Some of them : JUnit and TestNG (java), unittest (python), Test::Unit module (ruby).

Integration tests – verify behavior of portion of a system, or subsystem. Requires installed external dependencies such as databases, ldap, etc.. This test tests code via API which is rather not accessible to the clients, and this test are rather quickly to execute. There are also some tools which makes this testing simpler: DBUnit (java), fixtures (ruby), pdbseed (python).

System tests – verify behavior of complete system, so all the system part must be fully installed. This kind of test runs for the long period of time, so the good idea is schedule this tests for night activity. Some of frameworks allows to do this test easily some of them not. For example we use webwork, which action class has no dependency to HTTP objects, and we can easily can test action simulating user actions.

Functional tests – calls also as acceptance tests, provides testing from the viewpoint of client. The tools for helping us are rather language independent, they depends rather on technology which clients use, for example for web application Selenium which simulates browser behavior, but for fat client other tools are.

There are more types but they has different propose, maybe I write about it someday:

• UI testing (GUI testing, Usability testing, Accessibility testing)
• Speed testing (Performance testing, Load testing,   Scalability testing, Stress testing)
• Security testing
• Smoke testing
• Regression testing

and many others which I can’t remember now.
Summarize

• Categorize your tests, and run this test separately, do not trust yourself or other developer use for this task continuous integration tools or call it from cron or similar tool.
• Remember the cycle commit-test-result must be as quick as possible, there is no benefit from result which will came after developer goes to another task.
• Heavy tests run nightly and when developer come to work he had result from them. Remember to do this you must have this tests automated.
• And one important thing, if you ever find a defect, write test for it. The worse thing we can met is that our defect come back, so write test for defect first than resolve the problem. One point to mention is to write this test in such way that any new behavior is also tested (for example returning null when no data found)

Question : How about your tests, the speed, the quality, the schedule ?

Pedro