When to do written test cases?

Recently I've had several talks with various testers, whom I tutor, about use of written test cases. Since the subject has gained remarkable feedback lately so it looked well-cooked enough to publish.

Doing manual testing with written test cases tends to be one of the most expensive form of testing. Why so? Some of the reasons are

If doing the test and/or repeating it isn't either expensive nor extremly time-consuming then usually writing down in details what to test takes a lot longer than actually doing setup for test and running it. So from testers perspective it's not very wise choice of cost vs value. My personal worst (I'm not proud of it, but my that-time manager required it) was around 5 hours to write down the test case into really fine details while it took 5 minutes to run it. What makes it "my personal worst" - the test case was written for very testable functionality taking only 1 minute to setup and easy/cheap to run it again as many times as required. So in general I spend over five hours for a less than 5 minute easily repeatable test.

Someone with higher level of qualification should usually write the test case and review it several times since mistakes in test cases can cause various problems from false-negative bugs (bugs that aren't real and appear only because of the mistake(s) in test case) to false-positive bugs (bugs that are considered correct behavior because misinterpretation of test case instructions). That means that higher level specialist's time, what is usually more expensive, is used for this task. In addition it takes lot of time to think up and write down every possible combination how to test something since there are almost unlimited amount of possible test cases. And worst part is that usually there are too many unknowns about the product to reduce number of possible combinations to reasonable level.

If the Test Case is prewritten after some other document (Use case, requirement document), then often some areas are left uncovered in test case either because all possible combinations can't be thought at that time, some things are still uncertain at the time of writing the test case or some requirements have changed between writing the document and actually testing done. My experience I usually find majority of bugs from things not written down on test cases or in any other documentation.

If detailed test cases are written afterwards for someone else then it's usually good idea to find out for what purpose and what is done with them afterwards. If they are done for someone to review what's been done, then there are usually much cheaper ways to do it. For example to use screenrecording tool to record done tests. It's even to organize them like documents with versions etc. 1 GB of data storage in Hard Drives cost less than 0,1 USD. Or to write just test report describing test coverage at the end of each time-cycle or mission.

But then when to do written test cases?
Written test cases are the most formal ways to test and should be used when the highest possible level formality is required. The reason can be business decision or obligation from contract, but in any case the additional time required must be considered.

Also written test cases are useful if you can run tests only limited number of times and must make sure to cover every possible and required scenario with really limited reruns of tests. And the need for extra resources must be considered again.

Another reason for written test cases might be needed if something additional is done with the test cases. For example if certain test are needed to be automated for any number of reason. Then the test cases serve as use case documents for automated tests. In which case they need to be maintained and updated as any other product documentation along with automated tests in order to use them again in future properly.

My personal experience is that most software development projects doesn't have neither the need nor budget for fully formal testing to do all testing in written test case form.

How to hide problems in software behind business rules (and what happens afterwards). Case Study part 2

Few weeks have passed and after some very difficult red tape I succeeded getting the necessary information. It only took me around 6 emails and 3 phone calls to get answer to a question "How are after 23 o'clock made payments handled and why it takes so long to reach to third party system".
What I found out was little disturbing - Payments made after 23 o'clock are handled by persons during the next work day manually and then sent to Tax and Customs Board day after that. This of course explains why payment reaches to next systems over 24 hours later. The day-to-day human interaction are put in place to handle corner case problems. Also I found out that Tax and Customs Board is trying modify their own system to accept these corner cases by in their end. I personally don't have much faith in it. I see various problems like how to separate faulty data with enough certainty or if owner of the first part system (banks) aren't willing enough to give out all necessary information about their system behaviour to very high detail.

How to hide problems in software behind business rules (and what happens afterwards). Case Study

All started at the end of May 2010 when I found a strange bug (from my point of view) from self-service system of one of Estonian biggest banks.

I tried to pay my taxes to local Tax and Customs Board when I noticed the money wasn't transfered during the time I expected (later I found out took 2 banking days to arrive). So I started checking what's going on. I found that payments made after 23:00 local time got timestamp 24 hours to the future. For example payment order for payment made 25.05.2010 23:40 got timestamp 26.05.2010 23:40. So I enquired from bank. I got nice reply (translating and shorting the email):

"Thank you for your letter. According to our terms and conditions of account management we quarantee in-bank transfer within 1 hour though usually it takes less. So the money is transfered to another account but has timestamp for the next day of real transfer".

So I noted out that money wasn't transfered during next work day. After that I got reply pointing that "It is not a bug. According to our terms and conditions payments forwarded after 10 p.m. shall be excecuted immediatley but amount shall be expressed in the account statement of the customer under transactions of the following day".

Okey I thought. Bank says it's not a bug. So did I really make mistake this time... Doesn't feel right...

I started enquiring from Tax and Customs Board. They replied that current amount was logged into their system until 2 days later in the morning and according to their info the transfer didn't came through immediatly. If I tried to make payment during day the transfer went through on the same day so it couldn't have been issue from their side.

I forwarded current message to the bank but their reply didn't change much. They still refer to terms and conditions.

So I made my conclusion from information I posessed - Due to the fact that their banking day doesn't match astronomical day they couldn't fix it to make it match properly and applied business rule for allowing this bug to remain (now called a feature). However now that 3rd party systems are getting data from their system something goes wrong due to this feature. My best guess is that due to the wrong timestamp the information isn't forwarded at correct time, but instead it is forwarded in the time marked in the payment order. Hopefully I'll find out the real reason soon and can close this subject in my mind.

However I must admit. Nice idea to use business processes and rules to change bug to a feature. Have seen it before in other systems and working pretty well. But it might come back to haunt the system if another system might need correct value instead of value inserted by the feature.

PS. I also found mistake from Tax and Customs Board's system while looking for this specific mistake. It's getting slowly out of hand... Finding bugs from almost any system I use...

Bugged software checking bugged software for bugs

Some time ago I had nice conversation with Madis Jullinen about test automation. We starting wondering how business people understand automated testing. If they understand that any software has bugs (because they approve testing at all) then how come they still belive that automated tests don't have bugs in themselves? Let's forget all about theory of models ("All models are wrong, some are useful") and checking vs testing ("automated tests are only useful for checking") to simplify matters and only concentrate on the subject of automated tests as just another piece of software.

We assume that any software has unknown amount of bugs in them. Testers are there to find some of them (usually as many as possible within any given amount of time). Automated tests are another piece of software. Ergo automated tests have unknown amount of bugs in them and testers should test them as well and someone have to fix found bugs.

So I started approaching this subject mathematically (and financially)... To create automated tests you have to approach this as another development project. With it's own analysis, development, testing and support phase (plus some project management to schedule it to timetabels). You have to analysis what to automate and how to do it, develop necessary tests, test if tests works properly and support automated tests throughout project lifecycle (since project might change in time). I added up the numbers. It takes n hours to prepare what to test, m hours to develop it, p hours to test if it works, r hours to fix bugs in it and s hours to keep the tests up-to-date. So in general it takes n+m+p+r+s hours to prepare the tests and s hours every iteration to keep tests up-to-date. Sadly most of business people see test automation cost as only p hours total for developing tests.

Another bad trend I've seen lately is just the need to create automated tests because it is a good thing to do. Why is it a good thing to do? To spare tester some time from regression testing? How do you know it spares time? Have them done calculation of ROI to see whether or not it would be even useful to automate? So why to even bother to write test automation if you are not sure how much time and money they would actually save? The worst thing that could happen is when testers are required to create automated tests since some procedure dictates that n amount of tests have to be automated.

I won't say I'm against test automation. But everything must be done not because some procedure says it must be done, but because it must actually help to get better results in the long run. So instead of building one bugged system atop of another bugged system and hoping it would find some new bugs... Well... I think it would be wiser to spend that time either actually testing or calculating whether or not it's worth to automate these tests after all.

Having fun with computer terminal in a bank.

I heared this story from my tester Rasmus. He went to a bank to complain that their web application doesn't work properly when he tried logging in with Estonian ID-card. So the bank worker showed him a local computer terminal to try if it works here. He inserted his card into the reader and watched what happened. Program waited a little bit and opened a pop-up where you had to insert PIN1 of the card to log in. But... There was no OK or Cancel button... And when he tried to enter numbers to the field, that didn't work either.

So he thought to have little fun (read about that kind of fun from James Bach's blog here and watch here). He removed his card from a reader hoping something fun to happen. And... Computer started to shut down by itself... oops... the whole system started behaving really odd because of just removing ID card...

That made me think... I hope that they are not testing all their systems like they had tested this terminal... Who knows what else could he have stumbled upon there if that computer hadn't crashed completly...

Test Camp: puhkaeestis.ee

Another month has passed and again it was time to call for Test Camp. This time I tried to do little more. Add more people and add more different approaches. One of participants from previous time (Madis) recommended very intresting application to test - web application without database. Date of the event was agreed upon 17.04.2010.

I got the permission to use this application (www.puhkaeestis.ee) and now I started putting together participants list. Sadly Tanel from Fraktal couldn't come, though he wanted, but rest of people were Oliver and Rasmus from Celeg Hannas and Madis, Artur, Eero and Polina from Webmedia.

This time we got people with really different approaches and backgrounds to participate - Testers, QA manager, Test manager and even .Net programmer (Eero). So the results were also very various. Eero (as only participant who actually works as programmer) gave some nice results and they made great team with Rasmus. Both generating ideas and improving each other - both had programmer background, but currently doing different type of work, thus thinking differently.

Ideas were generated so fast that Madis had hard time keeping up the pace when everyone were asking additional questions. Again the average rate of issues found in an hour per participant was around 2. But none-the-less it was fun and Webmedia thinks of startingsimilar events in-house also. Glad to hear that my idea was well-recieved.

Hopefully everyone also learned some new tricks to try with our own projects. At least I know I did.

Test Camp: Edicy

Few weeks ago I got an idea - very dangerous, especially when you lack time. The idea was to create sort of camp out event for testers to practice our skills, learn new skills and get to know each other. After several days of thinking and discussing this idea with friends I trust (some are testers as well) I decided to organize an event called Test Camp. Idea was to spend 1-2 days in a row on one specific software and try to find as much bugs (or issues, since not all issues are bugs) as possible. Secondly I thought it would give participants excellent chance to see how other testers work and think. Thirdly the idea was to try new methods and techniques of software testing in order to see which ones work better. To add some coke and pizza to that event would make it like camping out with computers - kind of LAN party with the exception that instead of playing games, we would test.

So I start moving forward with my idea.
First I put up list of participants. Since it was a first time event I didn't want to make it too crowded. So first list contained 2 of my friends (and also ambitious testers) Rasmus Koorits (Tester from Celeg Hannas), Madis Jullinen (tester from WebMedia) and of course myself.
Then I chose website making tool called Edicy from company Fraktal as the first software to be Test Camped (term I invented to illustrate that the event is like sending software into boot camp). It has got some media coverage here and there and since my company decided to build our own website upon it, I thought it might be a good idea. So I talked with one of the creators (and owner) of that tool for permissions to do it. Not only did he agree with the idea, he also suggested that their tester should participate as well. Wow... That was surprising, but okey... so now there were 4 of us. So I named the event Test Camp: Edicy to show what software is being Test Camped. In addition I talked with Development Manager from Webmedia for feedback for the idea. Again another wow moment in that meeting... He also thought it was a good idea and suggested to add another tester from Webmedia... This time I had to decline, since 5 persons in an event I originally planned for 3 would have meant double the participation... I didn't had much time to prepare anyway so I figured I can't handle that many people at that moment.

So... List of participants was locked, software to test chosen, date and time for start of the event agreed upon and location fixed. We decided to start in the morning of saturday the March 13 in my company's office on the border of Old Town of Tartu. After 7+ hours we decided to end since all of us had rough workweeks behind us and wanted some rest. We were mentally exhausted and literally physically tired because of this event. But it was a success... Even for me though I couldn't try out not even as much as third of the new methology I wanted... We found total of 72 issues in just 7 hours, people got to know each other better and learned probably several nice new tricks how to generate better test ideas.

Test Camp's verdict for Edicy was: It works pretty well. So, Fraktal, keep up the good work!

I already have an idea for the next Test Camp as well and hopefully by then I can increase the amount of participants. Until next time and my thanks to Tanel, Madis and Rasmus. Without you, the event would have never been that much fun.

Testing exercises

Yesterday evening we had puzzle night and I got few good puzzle ideas for practicing testing. If you don't know the answers and want to solve them, don't google. instead contact me either by skype olivervilson or email oliver.vilson@hannas.ee and I'll help you solve them. Satisfaction of solving them is worth not googling for the answers.

1) A naked man is lying dead in the middle of the desert with a small wooden stick in his hand.

2) A man wakes up one night to get some water. After finishing his class of water, he turns off the light and goes back to sleep. The next morning he looks out the window, screams, and kills himself.

You have to find out what happened by only asking yes/no questions and I answer either yes, no or not relevant.

3) Businessman wearing expensive suit is on one side of the river. There is a trainstation on another side. He has to get to a train that leaves in 15 minutes. He lights up a cigarette and is not worried about missing his train. There are no bridges or boats nearby and he can't jump across the river. Why isn't he worried?

I posted them was because I consider that to evaluate whether or not person is smart, you can't use mathematical questions. Math is just one skill and if you posess this skill you can solve most of mathematical questions with ease. But real life doesn't tend to be filled with mathematical solutions. These puzzles demand you to ask questions and create more questions from answers you get. I see it as more closer to real life situations.

I consider testing as asking questions. Sometimes you get reasonable answer but often (especially if logs don't give enough answers) you might have to settle with yes/no replies (works/crashes/some data is wrong) from system. And then you have to know how to ask more questions in order to move forward.

So if you want to solve some puzzles (don't google ;)) then feel free to contact me.
Hopefully I find more puzzles to post here.

Secrets of a Buccaneer-Scholar review

I finished reading James Bach's book Secrets of a Buccaneer-Scholar and decided to share some of what I've learned. First of all - the book describes a way to become and expert through the eyes of James Bach. In the beggining James Bach makes very accurate point - it doesn't matter if you are a buccaneer-scholar or academic-scholar, you still have to learn. The difference is what you learn and how you learn it. I myself got about two dozen new ideas what to do soon and what to do not-so-soon. Anyway one good point was glowing beneath all those words and sentences. It is how to work you self-confidence as it seems to be the most important tool anyone uses. If you are self-confident that you can do something - then probably you can do it. Don't dare to challenge ideas, proccedures etc and don't be afraid to loose sometimes. I've noticed that as well in my career and it has helped me alot also. If you can generalize enough then this book is not only about how to become expert in Testing. This book shows a way to become expert almost in any area that provides some challenge to mind. I personally tagged some pages to review again when I have procrastinationed these ideas for awhile. Again another thing I realized I was already doing after reading the book. That sometimes putting things/issues/problesms on stand-by and not even thinking about them helps to solve them more efficently and more rapidly. If you know what you're doing.

Anyway for final suggestion - If you dare to challenge generally acknowledged ideas, people with authority and yourself, then this book might give additional ideas to become expert. If you don't dare, but would like to do it, then this book might give additional boost to your self-confidence to be able to dare. If you don't want and consider challenging ideas of other experts wrong, to trust completly what others say and follow the path that has been shown to you. Then this book might show you there is another way. If you are willing to read...

Why don't managers understand time issue in testing?

I read Michael Bolton's blog entry about "Why is testing taking so long?" and started thinking why don't managers understand it. So I came up with following aspects why managers won't tend to understand it completely.

a) They tend to think most of their days in rational aspects where is little room for unknowns and even little room for luck -> But testing itself consists of investigating unknowns since usually testers know very little what to expect from software they have to test. That's why it is hard to estimate time it would take (see "needles in haystack" aspect of testing I describe in previous posts). And every time we discover new unknown and investigate it, then it spends extra time and again has devastating effect on plan for testing. But if we wouldn't investigate unknowns then we wouldn't be doing much of testing. We would only do checking (see Michael Bolton's blog entry "Merely" Checking or "Merely" Testing. I won't say that checking isn't necessary but it is not testing.

b) They don't tend to understand that every time we discover a bug, we have to spend much more time for it - we have to investigate it, report it and check if it is fixed afterwards. So the point would be : "Good tester tend to generate more and more work for themselves by doing their job.". It might be hard to tell them that without provoking them to say : "why do you find so many bugs then?". It can be explained by this logic: since their tasklist tends to get shorter when they do their work efficently, it is hard for them to understand that ours tend to get longer when we do our work efficently.

c) Belief that written plans (and following them exactly) helps to solve or prevent most problems. But plan is a model. All models are wrong, but some are useful. Meaning that sometimes it helps us to plan a little ahead. But with so much variables as in testing we can hardly do it more than few days to maximum of few weeks forward and believe it is accurate and still useful after that time. Testers tend to make the plan usually quite short-term - "I see what happens after I try this combination" is a plan. Actually really good plan. If you have to use session based testing then of course planning is needed. But plan doesn't have to be 4 pages long document with detailed descriptions everything needed to be done. And our plans tend to fail... Outstandingly... This is acceptable. For us... We still learn... But for managers this is usually disaster. "What do you mean you are behind schedule more than 1 week after 2-3 weeks?"

d) Belief that preparing lots of documents which tell us what to do, increases the speed of our work. They put too much credits that every task can be prepared, measured and estimated before completing them. Since their work is based on planning (and it is required from them) they tend to think that they could expect the same from us. But if we know that there is a bug by planning to check it, then we ideally can fix (or report) it before it even gets into the software. It usually doesn't work that perfectly, but with certain reservation we can use it as a model. I personally have found about 60-80% of bugs from places not described in test cases or in any other documents. The issue is finding the bugs we don't expect. Again the unknowns are the involved.

So only advice (is it good or bad, you have to decide)I could give you is: If you don't understand what is expected from you, your manager annoys why testing takes so long, why do you find so many bugs, Why are you behind schedule etc then ask him/her - "Do you want me to check the software or do you want me to test the software?" (Don't forget to explain the difference between checking and testing).

Helping someone becoming tester...

At the begginning of June I had a chance to teach and guide someone with no IT background to become a Tester. At the beginning I couldn't give him direct guidelines due to fact that he was solving a test exercise for a job and I had to evaluate it. So I gave some books to read, articles to review, peoples blogs to check and presentations to learn from for him to understand what the job is about. But after the test exercise was submitted and evaluated, I had a chance to discuss his assignment, what he had learnt, what does he think about it and how he plans to use it. The greatest part about it wasn't the discussion, wasn't the work amount he had done, but to see true passion burning in his eyes after reading and looking and hearing all he did. To see that I helped to make him understand the fine art of testing. To see a new comet rising to horizon with great ideas and even greater passion for Testing. Anyway I hope in few years he himself becomes the guru for some. Madis Jullinen has it in him.

Last week I had similar experience. I was talking to one of my friends studying IT and he asked me what testing is all about. We had a discussion that lasted for over half an hour. After that he said to me something I never forget. He said he would have never imagined that testing could be so interesting, in some aspects more facinating than anything other Software Development has to offer.

I'm getting a strange feeling that I've created a small group of people who admire testing and found the passion in it with my help (at least I hope so). I have a feeling that I have created something much bigger than myself. Strange... Hopefully I have the time to make more out of it. To make the true community of Testers work. Not just number of people who happen to work on similar positions in Software Development.

Scripting vs Exploratory testin

Lately I was reading blogs of James Bach and Michael Bolton about Exploratory testing vs Scripted testing. It made me thinking about one of my first teaching sessions of testing where I showed this video at the beginning to demonstrate my idea of tester as proffession. I consider it a good example of describing the difference between scripted testing and exploratory testing. In scripted testing you do as is asked from you. In exploratory you learn how to notice things that might not be visible at first but are still there.

Another good example I recently discovered is this video. Again proof of concept for the need of sapient thinking.

Why these videos? Because I think they describe in someway software and testing software pretty well. The first video shows how scripted testing is done (counting passes) and how it should be done (paying attention to other aspect besides the obvious ones). The second one demonstrates for me how software changes over time and how testers should be aware of the context.

Perfect room

I read a magazine few weeks ago. In the introduction there was a sentence: "If you light every inch of a perfect room, then you will find a dead fly in some corner and an hungry spider eating it."

I started thinking about this sentence and realized something. Though this sentence came from social magazine, I realized that it can be implemented very well into Software testing. No matter how much effort you put into testing your software, there are still bugs in it. Sadly however I've heard quite too often phrases like "Why didn't you find them at time?", "Why didn't you do enough testing to find these bugs?" etc.

Project Managers, Analysts and Customers who don't understand the conception that software isn't just mathematical formula with small amount of limited results. They tend sometimes to blame Testers for being responsible for not finding all bugs in software. But we didn't put them in there! It is like throughing handful of needles into haystack, not saying how many you throw in and expecting someone with limited time to find them all.

Yes, during any given time we can tell with certain amount of confidence that it works with specific scenarios under specific conditions. Yes, with more resources (both in time, tools and skill) we can go through more scenarios and understand more and more about the software. But never can we check every last possible combination and be 100% confident that it works. I listened to Michael Bolton and James Bach conversation about "What to do when first step in test case is "Reboot your test system"" (Listen it here) and there was something said that called for my attention: "You can never run the same test under exactly the same conditions more than once. If you think you can, you don't have enough fantasy to understand.".

Of course people are upset that there are bugs within the system. I recommend to read this Michael Bolton posting about how a beesting can be just annoying or either it can be lethal. But if you give us enough time to use our skills, we will try to make sure that there are no such bees in the software that could kill it. But with limited time we can only find limited amount of such bees.

Note: Found the link, I have to apologize since the posting was in Michael Bolton's Blog instead. I've been trying to catch up with things I've missed with a year and with 5-6 blogs it gets a little out of hand. Will try better next time. Promise :)

The Beginning...

It was almost a year ago when I attended EuroSTAR 2008 and had a intresting conversation with Michael Bolton there, afterwhat he recommended me to start blogging. So it took only around a year to finally start with it. Loads of work that calmed down only recently couldn't allow me to start any sooner. Hopefully at least one person finds anything useful here. Then this blog has fulfilled it's purpose.

Today I was thinking about a testing and how to start this blog. I found a phrase that describes testing for me a little: Testing is like cooking. If you are new at it, you need cookbook texts helping you to cook. You need the recipes to know what raw materials to use, how to prepare them for cooking, how to cook them at what temperatures and for how long to make the food good. As you improve in the art of cooking you need the cookbook less and less. You rely more on yourself, on your experience, on your intuition, on your senses of smell and taste. Then one moment you start experimenting with the recipes. You start changing ingredients, substituting one thing for another seeing what happens.

So to draw a parallel then at the beginning of Your career as Tester you have to rely more on textbooks, prescribed descriptions of what to do and how. As time goes by You learn and gain more experience. Soon you can rely less and less on textbooks and more on your experience and intuition thus making yourself more flexible to the change of contexts of products, tests and enviroments.

I also started noticing one strange coincidence. As I improved in cooking I improved in testing and vice versa. The notice of details, possessing overview of large picture to understand context better, experimenting, exploring, use of other senses beside visual, how make it effectively, how to serve the results etc. The experience gained in one, showed up in another. For example I made fish in oven in mayonnaise sauce and after supper it I made a note to myself to try using different type of mayonnaise for the sauce. Reminder to use mayonnaise with higher quality to improve the overall quality of food and how it would affect the quality more specifically. Next time I was at work testing I suddenly understood that I was subconsciously using different type of test data for testing than before. Test data with higher quality for results. I got much more information out of the results with less time and effort. I realized only later that only thought that could have possible lead me to that decision was the one for the mayonnaise sauce and how the change would affect my cooking.