Few weeks ago I got an idea - very dangerous, especially when you lack time. The idea was to create sort of camp out event for testers to practice our skills, learn new skills and get to know each other. After several days of thinking and discussing this idea with friends I trust (some are testers as well) I decided to organize an event called Test Camp. Idea was to spend 1-2 days in a row on one specific software and try to find as much bugs (or issues, since not all issues are bugs) as possible. Secondly I thought it would give participants excellent chance to see how other testers work and think. Thirdly the idea was to try new methods and techniques of software testing in order to see which ones work better. To add some coke and pizza to that event would make it like camping out with computers - kind of LAN party with the exception that instead of playing games, we would test.
So I start moving forward with my idea.
First I put up list of participants. Since it was a first time event I didn't want to make it too crowded. So first list contained 2 of my friends (and also ambitious testers) Rasmus Koorits (Tester from Celeg Hannas), Madis Jullinen (tester from WebMedia) and of course myself.
Then I chose website making tool called Edicy from company Fraktal as the first software to be Test Camped (term I invented to illustrate that the event is like sending software into boot camp). It has got some media coverage here and there and since my company decided to build our own website upon it, I thought it might be a good idea. So I talked with one of the creators (and owner) of that tool for permissions to do it. Not only did he agree with the idea, he also suggested that their tester should participate as well. Wow... That was surprising, but okey... so now there were 4 of us. So I named the event Test Camp: Edicy to show what software is being Test Camped. In addition I talked with Development Manager from Webmedia for feedback for the idea. Again another wow moment in that meeting... He also thought it was a good idea and suggested to add another tester from Webmedia... This time I had to decline, since 5 persons in an event I originally planned for 3 would have meant double the participation... I didn't had much time to prepare anyway so I figured I can't handle that many people at that moment.
So... List of participants was locked, software to test chosen, date and time for start of the event agreed upon and location fixed. We decided to start in the morning of saturday the March 13 in my company's office on the border of Old Town of Tartu. After 7+ hours we decided to end since all of us had rough workweeks behind us and wanted some rest. We were mentally exhausted and literally physically tired because of this event. But it was a success... Even for me though I couldn't try out not even as much as third of the new methology I wanted... We found total of 72 issues in just 7 hours, people got to know each other better and learned probably several nice new tricks how to generate better test ideas.
Test Camp's verdict for Edicy was: It works pretty well. So, Fraktal, keep up the good work!
I already have an idea for the next Test Camp as well and hopefully by then I can increase the amount of participants. Until next time and my thanks to Tanel, Madis and Rasmus. Without you, the event would have never been that much fun.
Thursday, March 18, 2010
Monday, March 1, 2010
Testing exercises
Yesterday evening we had puzzle night and I got few good puzzle ideas for practicing testing. If you don't know the answers and want to solve them, don't google. instead contact me either by skype olivervilson or email oliver.vilson@hannas.ee and I'll help you solve them. Satisfaction of solving them is worth not googling for the answers.
1) A naked man is lying dead in the middle of the desert with a small wooden stick in his hand.
2) A man wakes up one night to get some water. After finishing his class of water, he turns off the light and goes back to sleep. The next morning he looks out the window, screams, and kills himself.
You have to find out what happened by only asking yes/no questions and I answer either yes, no or not relevant.
3) Businessman wearing expensive suit is on one side of the river. There is a trainstation on another side. He has to get to a train that leaves in 15 minutes. He lights up a cigarette and is not worried about missing his train. There are no bridges or boats nearby and he can't jump across the river. Why isn't he worried?
I posted them was because I consider that to evaluate whether or not person is smart, you can't use mathematical questions. Math is just one skill and if you posess this skill you can solve most of mathematical questions with ease. But real life doesn't tend to be filled with mathematical solutions. These puzzles demand you to ask questions and create more questions from answers you get. I see it as more closer to real life situations.
I consider testing as asking questions. Sometimes you get reasonable answer but often (especially if logs don't give enough answers) you might have to settle with yes/no replies (works/crashes/some data is wrong) from system. And then you have to know how to ask more questions in order to move forward.
So if you want to solve some puzzles (don't google ;)) then feel free to contact me.
Hopefully I find more puzzles to post here.
1) A naked man is lying dead in the middle of the desert with a small wooden stick in his hand.
2) A man wakes up one night to get some water. After finishing his class of water, he turns off the light and goes back to sleep. The next morning he looks out the window, screams, and kills himself.
You have to find out what happened by only asking yes/no questions and I answer either yes, no or not relevant.
3) Businessman wearing expensive suit is on one side of the river. There is a trainstation on another side. He has to get to a train that leaves in 15 minutes. He lights up a cigarette and is not worried about missing his train. There are no bridges or boats nearby and he can't jump across the river. Why isn't he worried?
I posted them was because I consider that to evaluate whether or not person is smart, you can't use mathematical questions. Math is just one skill and if you posess this skill you can solve most of mathematical questions with ease. But real life doesn't tend to be filled with mathematical solutions. These puzzles demand you to ask questions and create more questions from answers you get. I see it as more closer to real life situations.
I consider testing as asking questions. Sometimes you get reasonable answer but often (especially if logs don't give enough answers) you might have to settle with yes/no replies (works/crashes/some data is wrong) from system. And then you have to know how to ask more questions in order to move forward.
So if you want to solve some puzzles (don't google ;)) then feel free to contact me.
Hopefully I find more puzzles to post here.
Sunday, December 13, 2009
Secrets of a Buccaneer-Scholar review
I finished reading James Bach's book Secrets of a Buccaneer-Scholar and decided to share some of what I've learned. First of all - the book describes a way to become and expert through the eyes of James Bach. In the beggining James Bach makes very accurate point - it doesn't matter if you are a buccaneer-scholar or academic-scholar, you still have to learn. The difference is what you learn and how you learn it. I myself got about two dozen new ideas what to do soon and what to do not-so-soon. Anyway one good point was glowing beneath all those words and sentences. It is how to work you self-confidence as it seems to be the most important tool anyone uses. If you are self-confident that you can do something - then probably you can do it. Don't dare to challenge ideas, proccedures etc and don't be afraid to loose sometimes. I've noticed that as well in my career and it has helped me alot also. If you can generalize enough then this book is not only about how to become expert in Testing. This book shows a way to become expert almost in any area that provides some challenge to mind. I personally tagged some pages to review again when I have procrastinationed these ideas for awhile. Again another thing I realized I was already doing after reading the book. That sometimes putting things/issues/problesms on stand-by and not even thinking about them helps to solve them more efficently and more rapidly. If you know what you're doing.
Anyway for final suggestion - If you dare to challenge generally acknowledged ideas, people with authority and yourself, then this book might give additional ideas to become expert. If you don't dare, but would like to do it, then this book might give additional boost to your self-confidence to be able to dare. If you don't want and consider challenging ideas of other experts wrong, to trust completly what others say and follow the path that has been shown to you. Then this book might show you there is another way. If you are willing to read...
Anyway for final suggestion - If you dare to challenge generally acknowledged ideas, people with authority and yourself, then this book might give additional ideas to become expert. If you don't dare, but would like to do it, then this book might give additional boost to your self-confidence to be able to dare. If you don't want and consider challenging ideas of other experts wrong, to trust completly what others say and follow the path that has been shown to you. Then this book might show you there is another way. If you are willing to read...
Thursday, November 26, 2009
Why don't managers understand time issue in testing?
I read Michael Bolton's blog entry about "Why is testing taking so long?" and started thinking why don't managers understand it. So I came up with following aspects why managers won't tend to understand it completely.
a) They tend to think most of their days in rational aspects where is little room for unknowns and even little room for luck -> But testing itself consists of investigating unknowns since usually testers know very little what to expect from software they have to test. That's why it is hard to estimate time it would take (see "needles in haystack" aspect of testing I describe in previous posts). And every time we discover new unknown and investigate it, then it spends extra time and again has devastating effect on plan for testing. But if we wouldn't investigate unknowns then we wouldn't be doing much of testing. We would only do checking (see Michael Bolton's blog entry "Merely" Checking or "Merely" Testing. I won't say that checking isn't necessary but it is not testing.
b) They don't tend to understand that every time we discover a bug, we have to spend much more time for it - we have to investigate it, report it and check if it is fixed afterwards. So the point would be : "Good tester tend to generate more and more work for themselves by doing their job.". It might be hard to tell them that without provoking them to say : "why do you find so many bugs then?". It can be explained by this logic: since their tasklist tends to get shorter when they do their work efficently, it is hard for them to understand that ours tend to get longer when we do our work efficently.
c) Belief that written plans (and following them exactly) helps to solve or prevent most problems. But plan is a model. All models are wrong, but some are useful. Meaning that sometimes it helps us to plan a little ahead. But with so much variables as in testing we can hardly do it more than few days to maximum of few weeks forward and believe it is accurate and still useful after that time. Testers tend to make the plan usually quite short-term - "I see what happens after I try this combination" is a plan. Actually really good plan. If you have to use session based testing then of course planning is needed. But plan doesn't have to be 4 pages long document with detailed descriptions everything needed to be done. And our plans tend to fail... Outstandingly... This is acceptable. For us... We still learn... But for managers this is usually disaster. "What do you mean you are behind schedule more than 1 week after 2-3 weeks?"
d) Belief that preparing lots of documents which tell us what to do, increases the speed of our work. They put too much credits that every task can be prepared, measured and estimated before completing them. Since their work is based on planning (and it is required from them) they tend to think that they could expect the same from us. But if we know that there is a bug by planning to check it, then we ideally can fix (or report) it before it even gets into the software. It usually doesn't work that perfectly, but with certain reservation we can use it as a model. I personally have found about 60-80% of bugs from places not described in test cases or in any other documents. The issue is finding the bugs we don't expect. Again the unknowns are the involved.
So only advice (is it good or bad, you have to decide)I could give you is: If you don't understand what is expected from you, your manager annoys why testing takes so long, why do you find so many bugs, Why are you behind schedule etc then ask him/her - "Do you want me to check the software or do you want me to test the software?" (Don't forget to explain the difference between checking and testing).
a) They tend to think most of their days in rational aspects where is little room for unknowns and even little room for luck -> But testing itself consists of investigating unknowns since usually testers know very little what to expect from software they have to test. That's why it is hard to estimate time it would take (see "needles in haystack" aspect of testing I describe in previous posts). And every time we discover new unknown and investigate it, then it spends extra time and again has devastating effect on plan for testing. But if we wouldn't investigate unknowns then we wouldn't be doing much of testing. We would only do checking (see Michael Bolton's blog entry "Merely" Checking or "Merely" Testing. I won't say that checking isn't necessary but it is not testing.
b) They don't tend to understand that every time we discover a bug, we have to spend much more time for it - we have to investigate it, report it and check if it is fixed afterwards. So the point would be : "Good tester tend to generate more and more work for themselves by doing their job.". It might be hard to tell them that without provoking them to say : "why do you find so many bugs then?". It can be explained by this logic: since their tasklist tends to get shorter when they do their work efficently, it is hard for them to understand that ours tend to get longer when we do our work efficently.
c) Belief that written plans (and following them exactly) helps to solve or prevent most problems. But plan is a model. All models are wrong, but some are useful. Meaning that sometimes it helps us to plan a little ahead. But with so much variables as in testing we can hardly do it more than few days to maximum of few weeks forward and believe it is accurate and still useful after that time. Testers tend to make the plan usually quite short-term - "I see what happens after I try this combination" is a plan. Actually really good plan. If you have to use session based testing then of course planning is needed. But plan doesn't have to be 4 pages long document with detailed descriptions everything needed to be done. And our plans tend to fail... Outstandingly... This is acceptable. For us... We still learn... But for managers this is usually disaster. "What do you mean you are behind schedule more than 1 week after 2-3 weeks?"
d) Belief that preparing lots of documents which tell us what to do, increases the speed of our work. They put too much credits that every task can be prepared, measured and estimated before completing them. Since their work is based on planning (and it is required from them) they tend to think that they could expect the same from us. But if we know that there is a bug by planning to check it, then we ideally can fix (or report) it before it even gets into the software. It usually doesn't work that perfectly, but with certain reservation we can use it as a model. I personally have found about 60-80% of bugs from places not described in test cases or in any other documents. The issue is finding the bugs we don't expect. Again the unknowns are the involved.
So only advice (is it good or bad, you have to decide)I could give you is: If you don't understand what is expected from you, your manager annoys why testing takes so long, why do you find so many bugs, Why are you behind schedule etc then ask him/her - "Do you want me to check the software or do you want me to test the software?" (Don't forget to explain the difference between checking and testing).
Tuesday, November 24, 2009
Helping someone becoming tester...
At the begginning of June I had a chance to teach and guide someone with no IT background to become a Tester. At the beginning I couldn't give him direct guidelines due to fact that he was solving a test exercise for a job and I had to evaluate it. So I gave some books to read, articles to review, peoples blogs to check and presentations to learn from for him to understand what the job is about. But after the test exercise was submitted and evaluated, I had a chance to discuss his assignment, what he had learnt, what does he think about it and how he plans to use it. The greatest part about it wasn't the discussion, wasn't the work amount he had done, but to see true passion burning in his eyes after reading and looking and hearing all he did. To see that I helped to make him understand the fine art of testing. To see a new comet rising to horizon with great ideas and even greater passion for Testing. Anyway I hope in few years he himself becomes the guru for some. Madis Jullinen has it in him.
Last week I had similar experience. I was talking to one of my friends studying IT and he asked me what testing is all about. We had a discussion that lasted for over half an hour. After that he said to me something I never forget. He said he would have never imagined that testing could be so interesting, in some aspects more facinating than anything other Software Development has to offer.
I'm getting a strange feeling that I've created a small group of people who admire testing and found the passion in it with my help (at least I hope so). I have a feeling that I have created something much bigger than myself. Strange... Hopefully I have the time to make more out of it. To make the true community of Testers work. Not just number of people who happen to work on similar positions in Software Development.
Last week I had similar experience. I was talking to one of my friends studying IT and he asked me what testing is all about. We had a discussion that lasted for over half an hour. After that he said to me something I never forget. He said he would have never imagined that testing could be so interesting, in some aspects more facinating than anything other Software Development has to offer.
I'm getting a strange feeling that I've created a small group of people who admire testing and found the passion in it with my help (at least I hope so). I have a feeling that I have created something much bigger than myself. Strange... Hopefully I have the time to make more out of it. To make the true community of Testers work. Not just number of people who happen to work on similar positions in Software Development.
Tuesday, November 17, 2009
Scripting vs Exploratory testin
Lately I was reading blogs of James Bach and Michael Bolton about Exploratory testing vs Scripted testing. It made me thinking about one of my first teaching sessions of testing where I showed this video at the beginning to demonstrate my idea of tester as proffession. I consider it a good example of describing the difference between scripted testing and exploratory testing. In scripted testing you do as is asked from you. In exploratory you learn how to notice things that might not be visible at first but are still there.
Another good example I recently discovered is this video. Again proof of concept for the need of sapient thinking.
Why these videos? Because I think they describe in someway software and testing software pretty well. The first video shows how scripted testing is done (counting passes) and how it should be done (paying attention to other aspect besides the obvious ones). The second one demonstrates for me how software changes over time and how testers should be aware of the context.
Another good example I recently discovered is this video. Again proof of concept for the need of sapient thinking.
Why these videos? Because I think they describe in someway software and testing software pretty well. The first video shows how scripted testing is done (counting passes) and how it should be done (paying attention to other aspect besides the obvious ones). The second one demonstrates for me how software changes over time and how testers should be aware of the context.
Perfect room
I read a magazine few weeks ago. In the introduction there was a sentence: "If you light every inch of a perfect room, then you will find a dead fly in some corner and an hungry spider eating it."
I started thinking about this sentence and realized something. Though this sentence came from social magazine, I realized that it can be implemented very well into Software testing. No matter how much effort you put into testing your software, there are still bugs in it. Sadly however I've heard quite too often phrases like "Why didn't you find them at time?", "Why didn't you do enough testing to find these bugs?" etc.
Project Managers, Analysts and Customers who don't understand the conception that software isn't just mathematical formula with small amount of limited results. They tend sometimes to blame Testers for being responsible for not finding all bugs in software. But we didn't put them in there! It is like throughing handful of needles into haystack, not saying how many you throw in and expecting someone with limited time to find them all.
Yes, during any given time we can tell with certain amount of confidence that it works with specific scenarios under specific conditions. Yes, with more resources (both in time, tools and skill) we can go through more scenarios and understand more and more about the software. But never can we check every last possible combination and be 100% confident that it works. I listened to Michael Bolton and James Bach conversation about "What to do when first step in test case is "Reboot your test system"" (Listen it here) and there was something said that called for my attention: "You can never run the same test under exactly the same conditions more than once. If you think you can, you don't have enough fantasy to understand.".
Of course people are upset that there are bugs within the system. I recommend to read this Michael Bolton posting about how a beesting can be just annoying or either it can be lethal. But if you give us enough time to use our skills, we will try to make sure that there are no such bees in the software that could kill it. But with limited time we can only find limited amount of such bees.
Note: Found the link, I have to apologize since the posting was in Michael Bolton's Blog instead. I've been trying to catch up with things I've missed with a year and with 5-6 blogs it gets a little out of hand. Will try better next time. Promise :)
I started thinking about this sentence and realized something. Though this sentence came from social magazine, I realized that it can be implemented very well into Software testing. No matter how much effort you put into testing your software, there are still bugs in it. Sadly however I've heard quite too often phrases like "Why didn't you find them at time?", "Why didn't you do enough testing to find these bugs?" etc.
Project Managers, Analysts and Customers who don't understand the conception that software isn't just mathematical formula with small amount of limited results. They tend sometimes to blame Testers for being responsible for not finding all bugs in software. But we didn't put them in there! It is like throughing handful of needles into haystack, not saying how many you throw in and expecting someone with limited time to find them all.
Yes, during any given time we can tell with certain amount of confidence that it works with specific scenarios under specific conditions. Yes, with more resources (both in time, tools and skill) we can go through more scenarios and understand more and more about the software. But never can we check every last possible combination and be 100% confident that it works. I listened to Michael Bolton and James Bach conversation about "What to do when first step in test case is "Reboot your test system"" (Listen it here) and there was something said that called for my attention: "You can never run the same test under exactly the same conditions more than once. If you think you can, you don't have enough fantasy to understand.".
Of course people are upset that there are bugs within the system. I recommend to read this Michael Bolton posting about how a beesting can be just annoying or either it can be lethal. But if you give us enough time to use our skills, we will try to make sure that there are no such bees in the software that could kill it. But with limited time we can only find limited amount of such bees.
Note: Found the link, I have to apologize since the posting was in Michael Bolton's Blog instead. I've been trying to catch up with things I've missed with a year and with 5-6 blogs it gets a little out of hand. Will try better next time. Promise :)
Subscribe to:
Posts (Atom)