PEST 4.5: Workshop on Visualization

Apparently, both Kristjan Uba and me were toying with the idea of visualization and wondering how to use it communicate different aspects of testing and its results. Therefore, we quickly came to the agreement that the topic of the next peer conference should be visualization. We also agreed that this should be a practical workshop where something could be tried out instead of coming with a prepared presentation. The reason behind it was that well… I have done little of it and do it selectively; pretty much the same applies to Kristjan.

I hosted the event at my office where we gathered on the morning of April 26. Participants at PEST 4.5:

  • Kristjan Uba
  • Oliver Vilson
  • Rudolf Elbrecht
  • Kadri-Annagret Petersen (newcomer!)
  • me

We had some last minute cancellations as well.

Kristjan did the necessary introductions and explained the four tasks he wanted solutions for in visual form:

  1. Feature/area vs bugs/severity (how to show you have major and minor bugs in different areas of the product/system under test)
  2. Test coverage in different areas throughout iterations
  3. New features vs risks/bugs
  4. Status of the project/product overall

For bugs, he specified the following kinds:

  • “obviously major” bugs
  • important but can wait until next release to be fixed
  • “stupid bugs” that happen once every full moon
  • a lot of small things

We also had an imaginary project: electric bicycle for Dakar rally with all sorts of awesome features (just to prime our brains and have something tangible to start with).

Everybody picked a topic, grabbed the coloring pens and papers and got to work. Afterwards everyone explained what they did, then questions were asked, comments made. You know, the regular peer conference stuff. Since the group was so small, we didn’t use the cards to moderate discussions.

I picked the first task. Here’s how I went about it.

Attempt 1

I imagined boxes. Structure. Here’s the result:

Helena Jeret-Mäe_1


There were four areas of importance to my electric bicycle, so I each of the large quadrants is an area. Each of them is divided in 4 by severity of the bugs. There’s color coding: each severity has a corresponding color and my idea was that visualizing the clusters of bugs would be helped by colors. The letters are just placeholders: in real life there could by bug titles as hyperlinks or some kind of tags or other useful names.

For whatever reason, it seemed like it makes sense to have the critical bugs in the centre to make sure their clustering would catch everyone’s eye. But then again… what about the clustering of minor issues? This wouldn’t be as clear… which is why I changed tactics in the second attempt.

Attempt 2

I still went for a central cluster but thought of it more in terms of a bull’s eye.

Helena Jeret-Mäe_2

You may throw arrows at it and fix the bugs you pierce through 😛

Anyway, now I got some concentric rings. The good thing is that it’s easier to compare the bugs and their amount between areas. But then again you’d need to zoom in to the critical area because there may be too many and it won’t show up well. Also, the minor bugs at the edges may get less attention which is why Oliver suggested that in this model you could reverse the order and cluster the small ones in the centre to get attention.

You may notice some bugs have special marking: a circle around them. I used the severities I use daily: critical, major, medium, minor. But then I remembered Kristjan’s “stupid bugs” (see above) and decided that it may be worth marking those special cases while keeping their severity as is.

Attempt 3

Then I felt there was too little color… What about making it stronger?

Helena Jeret-Mäe_3

I used the same model but changed the means. Now color was in the background but bugs were spots. When I was working on it, new ideas spawned (or dawned on me):

  • and area could be broken down into sub-areas if there are some that are worth paying attention to. So the dotted lines represents breakdowns of an area.
  • I accidentally drew some bugs where I drew the dotted line but then it dawned on me that oh, I may have to show bugs that are between areas. Of course, there is a difficult limitation: you can only show “integration” bugs between areas that are side by side in your model already. But as we found out during the workshop, such relationships are hard to show.
  • You notice some spots are bigger than others… which is where I started playing with a new idea which I realized in attempt 4.

During the discussion, Kristjan pointed out that one good thing in this model is that we can place the release criteria into it. That’s pretty much what he meant:

Helena Jeret-Mäe_3 - Copy

The release criterion is simple: if there is anything inside the orange circle, we won’t release. Stuff outside is something we can live with. A clear goal emerges: let’s clean up the bull’s eye and its neighbouring areas.

Attempt 4

Helena Jeret-Mäe_4

I guess I was getting more abstract now… There were jokes about who can spot a constellation 🙂

I did return to my original structure but modified it. This visualization of bugs in 4 areas is meant to experiment with the following ideas:

  • size of the dot as the indication of severity
  • different sizes color coded vs without color coding
  • 4 areas divided into sub-areas
  • each sub-area gets its bug count but the whole status can be deduced by looking at this area (and comparing it with others if needed).

Somehow I avoided numbers and counts of bugs in my models. Others did use numbers and I think it is alright. But my brain was wired to create pictures of bugs, their whereabouts and importance for quick assessment. The details would have to be discussed if there is a need. But if you just want to know what’s going on… you don’t details at first.

I had the annoying thought of coverage that become harder and harder to suppress. Because hey… it is important if I found critical showstoppers in the first few tests which means the rest of the area is not covered or if I found them after having covered some ground already. So I just drew two slices of cake (I may have been getting hungry).

Helena Jeret-Mäe_5

The slice on the left is a sketch of showing what kind of test techniques were used. Maybe I used a lot of scenarios but didn’t focus on the data as much.

The slice on the right does a bad job but it was meant to communicate a hazy idea: the thickness of the slice would indicate how much of something was covered. The problem is obvious: the size of the slice itself is fixed and I don’t know how big exactly the “whole slice” is anyway.

Afterwards I realized that my visualizations were quite abstract, more like proofs of concept. But that’s okay: I have been thinking that I should try something for a loooong time but still haven’t. So this workshop was great for me to actually get visualizing and thinking about the problems. I got some good ideas from other participants but I’ll comment on those once they’ve summarized their ideas (hopefully, it will be soon-ish).

Before we wrapped up, Kristjan showed how he has tried to do some visualization and add colors to help with understanding. I definitely want to steal some ideas from how he has worked with test results to report them. He said that well, it’s not much visualization but some of the things he had I hadn’t really thought about even. Awesome!

All in all, PEST 4.5 was short and sweet and very useful. Looking forward to the next one!

PEST3: Kickstart learning by teaching! Vol 2

Here are my notes  and thoughts about the rest of the presentations at PEST3.

Raimond Sinivee talked about how testing is perceived without teaching. His example was fairly simple but effective: have someone who doesn’t know much about testing set up a Selenium script. If they don’t know much about testing, then the idea of simply recording and playing back the script sounds very alluring (as we know…). Yet spending a little time on trying to set it up and then play it back with appropriate comments and feedback may teach more than all the hallway talks.

As Raimond showed, it is easy to come across the scenarios that should fail but don’t. This should prompt the teacher to talk about the weaknesses but also strengths of tools, about how, where and why to use or not to use tools. It is the teacher’s job to explain those possible contexts. I think he made an excellent point about keeping in mind that teaching about tools shouldn’t vacillate between “tools are great!” and “tools are useless!”. It’s better to start a conversation rather than remain binary.

Harles Paesüld talked about his teaching experience at the universities and what he has learned from this (he also touched upon his experience with the BBST course and other AST courses). The main learning outcomes for him were the following: have specific goals for learners; prefer narrower scope when choosing material and topics for teaching; make a quick and solid connection between the concept you explain and a practical exercise. Concepts may be easy to lecture on but it is the immediate practical application that will help the learners remember something.

Harles also pointed out that peer evaluation has been a big part of his own learning. He expects constructive criticism and refuses evaluations that simply say “very good” 🙂 Also, evaluating someone else helps you to digest the content of the course.

Petteri talked about the testing dojos. Boy, did this dojo thing get me thinking 😀 You could check out Petteri’s blog where he has described the dojos held in Helsinki (

In a nutshell (and in my interpretation), a testing dojo is an informal gathering of people who get to work in teams testing something for a limited amount of time. As I understood there may be unexpected scenarios tossed at the participants (this is the fun part, right?). Importantly, the participants don’t have to be testers per se. I think it’s actually more interesting if there are non-testers involved… (by non-testers I mean those that are not testers by their job description).

The main benefit seems to be the collaboration between people with different backgrounds and the possibility of non-testers getting their feet wet. I can imagine a lot of interesting conversations to be sparked if different perspectives are smashed together like this. So I’ll be looking forward to a testing dojo in Tartu 🙂 I want to experience one and then I have other ideas what to do with that experience 🙂

Kristjan Uba, our content holder and facilitator, also briefly talked about his teaching experience. His main point was that an integral part of doing a testing exercise is a debrief. This is because it will summarize the experience, reinforce learning, and help relate the experience and the testing skills this particular exercise was supposed to teach. Kristjan also pointed out that finding examples that anyone can easily relate to (some sort of real life situations) helps to explain concepts. Even though you may be teaching testing, you don’t have to relate EVERYTHING to testing.

Another method (that I myself have used as well) is that you have your student rephrase what they heard (OR they think they heard). I’d like to point out that this should be done carefully. I have some bad experiences with people who try to make me rephrase something TOO obvious (yes, it’s subjective). This didn’t help with cooperation and made me angry or insulated. Like, hey, do you think I didn’t get it? It wasn’t the fact that I was asked to rephrase. It was HOW I was asked to rephrase something. So here goes for the manners 🙂


PEST3: Kickstart Learning by Teaching! vol1

Last weekend I had the pleasure of presenting at PEST3 (third Estonian Context-driven Testers’ Peer Conference). The theme for this peer conference was focused on teaching testing. In addition to the presentation, each participant had to present an exercise for teaching testing. This “added bonus” turned out to be pretty fun for me at least.

So what was covered? Here’s the first part of my summary of the presentations.

Risko Ruus talked about how he taught management that things can be different and how he had to do his share of learning in the process. In his case, it appeared that the motivation to learn more came about when he was asked to work on his idea of a test tool to be used for smoke tests. As he admitted, his motivation had been sinking as his duties at work very quite routine and tedious most of the time. So when the opportunity was there, he took it and started teaching himself about programming to create the tool. Once the proof of concept was ready, he successfully sold the idea to management. I must say he has some mad salesman skills or the management happened to be able to grasp the idea without much difficulty 🙂 In any case, he could proceed working on his tool.

What he pointed out was that he always kept in mind the fact that he knew he was going to give the tool to other people to use. This forced him to think about teaching the future users and keeping everything simple and modular.

Eventually, he other team who got their hands on the tool got a bit carried away. Risko explained that they wanted to immediately broaden the scope and start using the tool for things that… Risko hadn’t planned. So here goes another point for teaching using a tool: be very clear and firm about the scope of your teaching. Listen to what your students say but don’t hatch a new plan or solution along the way. Managing the expectations is also a big part of successfully teaching someone.

Risko had a fun testing exercise, namely, juggling. So what does juggling teach you? Consistency, breaking down a complex activity. Risko also said he uses it for defocusing: it’s a good incentive to use in order to drag your butt up from the chair, move around, and give your mind a break. So you’re doing something physical and your mind is concentrated on juggling. But we all know how this works: you’re dealing with something on the surface while your mind solves some issues in the background.  This is how the heureka! moments are born.

Aare Nurm talked about coaching and SBTM. I really was one big ear because I’m interested in hearing about the experiences with implementing SBTM. I’ve been fidgeting with the idea and we occasionally apply this but I haven’t commited to SBTM just yet.

Fortunately, a lot of what Aare was talking about had a familiar ring to it. Important points I took note of:

  • he uses the debriefing session for a different function depending on context. For novice testers, it’s a coaching session. For experienced testers, it’s a learning session where he studies their minds and their test results.
  • getting the size of the session notes right is a difficult task at times. How much is too much or too little? His criteria were that the session notes are sufficient if all of it can be covered during debriefing (if something is left out – too much; if something needs to be checked – too little) and one has to be able to tell what they tested in a couple of weeks’ time.
  • Debriefing may turn into a brainstorming session and this is OK. It is my personal experience that these sessions easily become productive discussions. I haven’t tried to hold it back because I see a lot of good coming from this. I was glad to hear that someone has a similar experience.

I still need to figure out a bunch of administrative things before employing SBTM but I got a few simple but solid ideas that can help me.

Aare’s testing exercise was a sudoku. I haven’t really solved sudokus… I just haven’t picked them up. Aare, of course, gave us just a couple of minutes to solve this. Aaarghh… Well, I got a bunch of numbers right at least. I spent some time trying to figure out the strategies to use: I looked at number of digits in a row/column; how many numbers were in one 3×3 box and which number would most definitely match etc). Well, I guess if I practiced and found more strategies, I’d also become a better… sudoku solver 🙂 Though when talking about teaching testing, sudoku may be good for recognizing patterns, learning to use a proper strategy quickly (especially if you put yourself under a time constraint).

PEST2: Bug Chase

One of the things I mentioned in my presentation at PEST was the “bug chase” idea. I would categorize it under “fun things to do with your team to get the adrenaline running”.

Back in February somebody IMd me about a strange bug he had seen but couldn’t really reproduce. Since the effects of the bug seemed important and it affected critical business flows, it was logical for me that we as a team should act swiftly.

I felt that it would be a good idea to shake things up a bit and pose a challenge for a couple of testers. Depending on the workload, I allow or then curb distractions, shake-ups, or any other interventions to our work. If we have a good pace and things get done quickly, it’s good to pop in something fun. If there are a lot of things going on and tasks are pouring in, it is still a good idea to pop something in. It just might be something less time consuming.

In this case, I came up with a bug chase which ended up being a duel. If you want to include more testers, go ahead. There shouldn’t be any limits.

So… both got the same amount of limited second-hand information from me, both were familiar with this business flow.

If I remember correctly, I limited the time to 1 hour.

Ready, set, go! Who will be able to reproduce the bug first? Infinite glory will be yours.

A bit later one of them cheered and let us know that he got the steps. The other tester found another pretty interesting bug that is difficult to find (a kind of edge case but it was in an area that is at the heart of the product).

I was really pleased and excited because, for one, a certain important area got a lot of attention and depth in testing. Also, the feedback from them was positive and they liked this friendly competition.

Obviously, as a team lead you don’t want to put two people up against each other who don’t get along. Luckily, I don’t have this problem and the two testers get a long very well working together, sharing ideas, and supporting each other.

This exercise also gave me the confidence that I can approach them like this. Of course, I can’t crash the party when somebody has thoroughly focused on something.

So, if the atmosphere and the timing are right, the relationships are good, and the people understand that they will not be evaluated, the bug chase can contribute to team work and learning (what better way to compare the approaches when debriefing afterwards!).

PEST 2: Taking One for the Team – First Impressions

I kind of want to say “PEST was awesome as usual” but then again…. it was only the second event we held.

I’m glad Kristjan took it upon himself to organize the event (and the catering :)). And thanks to Oliver for doing the labour-intensive job of a facilitator.

As always, I was a bit hesitant when I saw the topic (Taking One for the Team) and the questions to be answered. Somehow the stupid voice in my head said “well… I don’t have anything to say”. Despite this I did some thinking and the story I came up with was about creating my first team. Unexpectedly, I received a lot of questions and heard that some of my ideas were useful for others as well. I will blog about the “hot topics” of my presentation and expand on them.

Oh, I’ll be sure to think of the feedback when I have my moments of doubt. It really makes me feel there are people out there who listen, think along, and give support. This is probably the most awesome outcome of PEST besides the influx of new ideas.

The overall highlights for me:

  • Harles’s presentation about fighting against something without appearing to be up in arms. This one really rang a bell and I need to think about this more. It is very difficult to not reject the orders from above when they’re presented in that top-down manner which doesn’t really consider you as a thinking human being. It really is about the power relationships. I have been thinking about what to do because I know that my reputation among some of my foreign colleagues is not top notch precisely because I can’t be easily bent into doing things (that I don’t really agree with). They probably can’t deny what I’ve done but maybe I can be more pleasant (and then also more effective with people). It’s a fine balance, though. Retaining my integrity is extremely important for me but maybe I can find ways to react better and change people’s minds without saying “no” to them off the bat.
  • Aare’s presentation about a long-term project and the troubles with it was a topic up my alley as well. The danger of regression testing becoming unbearable is something that I actively try to deal with. My team is going to face a growing amount of regression testing so I try to monitor the boredom levels and also the effectiveness of testing.
  • Ülar’s contemplation of being a one-man-army… oh yes, sounds familiar. This situation reminds of the importance of motivation. Or rather, finding out and learning about the mechanisms that drive you. Really, know yourself.
  • Raimond’s approach to ordering a tool (or any other piece of software you need) didn’t seem to click with my work until yesterday. Then I learned that my team will have a developer assigned to assist us. So this is where I can work off Raimond’s approach. Describe it. Break it down. Review it. I think this will prove useful for me too. However, the teamwork and communication part made me think of ways how to facilitate better communication. Maybe, one fine day, I’ll be doing video conferences as well.
  • Ervin and the effectiveness of the work he does really makes me want to look into my team’s and my own work  processes to see if I can make it more effective. Automating something immediately is not possible but I will be asking for a lot more testability to be built into the product.
  • Not that I didn’t know how Rasmus works before… but now I definitely know better. And I have been wondering how we could employ this “jump in to impress” tactics better. Obviously, we don’t have to and sometimes can’t do it… but I’d like to have this competence in my team.

Overall, I think it was a great conference. It’s a very intriguing mind game to imagine if we all would work in the same company…. Kristjan thought that then we’d feel too good about ourselves and would really have to go out there to look for failures and problems to understand what we could do better.

Truly, to get thoroughly motivated, PEST will get you worked up!

See also: and

Positive Peer Pressure at PEST

A little alliteration doesn’t hurt, right?

The three day weekend which consisted of Rapid Software Testing Management course and the first peer conference of context-driven testers  in Estonia (in other words: Peers of Estonian Software Testing) was awesome. Literally. I went to work today and I was happy and energetic. Despite the fact that the days were intensive, I felt as if I had just come back from vacation. That’s what being around great people can do to you!

Also, everything that I experienced gave me the relevant kick-start to actually post to the blog I had created a while back. So I get to “blame” Oliver Vilson for inviting me to that event and convincing me that I have something to talk about, and James Bach for telling me he wants to hear more from me. The sarcastic part of me asks if this blog post is all about vanity then but it is not so. I’m sincere and I mean every word here.

What does it take to create the magic? Get great and thoughtful people together and have them share their experience with others about something that really matters to everyone involved. I truly hope the magic will last and I hope I can contribute to it as well.

The most important gains for me were

  • the people – I got to know a number of human beings I didn’t know anything about. And I feel lucky now that I do know them.
  • the revitalized self-belief – I had been bogged down by the routine of everyday work; I’d got tired and didn’t have the strength to rear my head and be hopeful. Don’t mock me if I say I feel like I’m breathing again
  • the stories and the ideas – I can’t wait to do great things with my team!
  • the community – if I get tired again or I feel overwhelmed, I know where to go. I am thankful for this.

Special thanks goes out to Rasmus Koorits and Indrek Kõnnussaar for making me rack my brain. It felt great and I realized I had also been losing some of my edge (similar to Rasmus’s experience story except it happened to me at work – so beware!). I also learned something about how the wheels in my head turn.

Eventually, one of the most important thoughts I took with me was about my background and its value. But that’ll be another blog post.