Reporting Risks to Business: First Steps from Trying to Delivering (The Better Late Than Never Edition)

In June 2013 I wrote about how I tried to do a better job talking to the business people and providing them with useful information found in testing. This is the blog post you should read or else this one may not make much sense.

I kept updating a draft of this post but never got around to publishing it… Stuff kept happening, I kept experimenting, and every now and then I felt like what I try to do is stupid anyway… Naive as I am, I hoped I had found some sort of magic recipe for getting testing to be considered an integral and acknowledged part of how we developed software. Looking back I think that changing the way I reported test results was just one ingredient in informing decisions. It was an important one because it kind of felt like I’d kicked in a door when I started reporting risks. But it didn’t change as much as I personally would’ve liked or expected. Yes, I have high expectations for myself and others… The way the system reacted or didn’t react taught me more about how the system worked and its inertia.

What Changed After I Started Reporting About Risks?

I – Getting Things Moving

I started to send out the report a couple of times a week to communicate the progress my team had made in testing and the information that was found in testing. Later I changed it to once a week.

All showstoppers that I initially reported were either resolved or I could lower their risk level because something was fixed or we got the information we needed. In other words, I got the attention of people who could solve problems and this helped the testing team along. Getting people informed of what was really going on in the product got some conversations going and relevant meetings set up to get problems resolved. Because really… most of the information we provided about the product was about failures in areas that were said to be important. It’s just people in decision making position tended to be preoccupied with things going on right now.

A note about the development process: the ideas for features were realized in code quickly; some meetings were held, no written specs or prototypes – prototyping was done right in the code… If you’re building on an existing product, it’s not too hard to do. Then when the developer and someone on the business side felt it was kind of done, it was given to the testing team. And then developers and the business moved on to the next things… which caused some problems as you may imagine. Even though I could write at length about the kind of problems such a process brings about, I’m leaving this for another time.

II – Making The Report a Staple of a Meeting

My boss proposed that I present the report at the weekly meeting with product owners and other folks (it’s a meeting where there are product, owners, a dev team lead and other people, including the CEO).

The first time I presented it, it sure got my adrenaline running!

I explained the purpose of the risk report, what each of the sections meant and covered the issues as briefly as I saw fit, highlighting the most important aspects of each issue. They could see the report on my screen as well (we had the “phone into a conference line and share your screen” type of meetings most of the time). I forgot to time myself but I don’t think I spent more than 5 minutes.

I got some additional input and requests for 2 meetings: 1 to discuss the risks I talked about (we found that a feature only worked when following a happy path) and 1 to discuss the risk assignment and perspectives on my report. I was OK with discussing the reasons I assigned certain risk levels and the reasons why someone else may disagree as long as we get to talk about it.

III – Maintaining the Focus of Decision Makers

Later on I noticed that some tasks I had put on the report I couldn’t take off for weeks because there were fixes that weren’t delivered yet (or we kept finding issues when retesting the fixes; in other words, the fault-feedback-ratio was high). So I left those items on the report as a FYI  item – the functionality was almost done and tested but we couldn’t sign it off yet. I commented on those briefly, stating that the status of these items hasn’t changed.

There were features we tested that appeared to be “good enough” for showing to the customers to get their feedback as they were going to be the early adopters. However, I noticed that the feedback never arrived… So I still kept those items on my report and said that those we are still waiting feedback on.

One might ask what’s the point and why did I keep coming back and talking about those features. I don’t want to go into lengthy explanations but the reason lies in the context: I noticed a pattern where things don’t get followed through properly and this had consequences for customers in the end. I cared enough to risk a little neutral-toned nagging to make sure the whole team keeps their eye on the ball (or rather, the sea of balls…). For me it was  sad to see if someone drops the ball and doesn’t carry things out to the end because oh hey, there’s something shiny elsewhere…

However, after an extended period of time when even I was starting to feel embarrased, I simply said that I’m not going to report on this item anymore as there is no interest whatseoever in finalizing it, so you have been notified of this particular risk.

My comment about this situation is that it was a characteristic of a system, a trait of the company culture, a way of going about things. Even though there was a lot of talk about being accountable and keeping customers’ interests in mind, some still got left behind even though they were declared important at first.

IV – Contrasting Testing Tasks, Highlighting Problems in Workload

In addition to the 2 reports I already mentioned in my previous blog post, I included a simple Excel spreadsheet which contained a list of low priority tasks that we weren’t be able to test before the release. Each of those items was a fairly small one but combined, it was a couple of weeks’ worth of work.

My idea behind this was to create a  contrast between the few important items that have more details about them. And then a long list of things that won’t be covered at all. I had no intention of postponing the release or using this list to manipulate anyone. I just wanted to SHOW how things are at this point without complaining and give the decision makers the opportunity to jump in if they want to. If not, it’s OK.

Later I also introduced a breakdown of the workload my team has: how much new stuff, how much old pending stuff and how much of it we’re getting done. This I did in an attempt to figure out good ways for transparency about what and how much is going on in general. I dropped the list of low priority stuff fairly quickly as I didn’t think it was or was actually useful.

V – Starting Point for Further Experimentation

I also kept experimenting with the written report and the oral reporting based on the text during the meetings. I kept the breakdown of high and medium priority for a while, then described the issues, risks and possible mitigation. At some point I felt like I couldn’t always provide anything reasonable for this three-tier breakdown. So I decided to ditch it. Instead, I wanted to work on describing the issues and risks we’ve found in a concise but not too detailed way.

I noticed that when I included an issue that seemed interesting or important then it ended up being too context-specific to explain quickly. My interpretation of it was that I had not properly thought through the “why”, and, therefore, wasn’t able to link the issues to the big picture well enough.

VI – Changing How Testers Report the Results

Because I changed how I provide information to others, I had to change the way I obtained information myself. I explained to my team how I needed to get input from them: a well-summarized nutshell and I’d ask for details if I needed them. I encouraged them to use the same mode when they were on meetings where I wasn’t present. I observed that more focus on clarity did help make the meetings more efficient.

As far as oral reporting is concerned, I tried to observe myself while speaking. I’ve noticed that I to try to gloss over or fumble too much from time to time and I take it as a heuristic for not having thought things through.

I tried to start out with outlining the major testing activities or areas concerned in order to get folks oriented and then go into particular sections of the report. I also tried to give a lead in for each part of the report to make the flow of information better.

Reflections On Some Lessons Learned

Unwanted Attention

The risks and problems I reported about weren’t anonymous. They belonged to an area a programmer was responsible for which meant that if a lot of problems came about the programmer didn’t look particularly good. Therefore, after the report had become a staple, I informed the programmers beforehand that I was going to include something that concerned them on the report. Since I took quite a lot of “project management by stealth” responsibilities on my shoulders,  it meant that when I pointed out the problems I’d also try to negotiate resources for solving them. For example, trying to get hold of someone in the US to answer the questions testers had that the programmer also wanted to know about was something I arranged instead of dumping it onto the programmer or tester to take care of.

There also was a bit of an anecdotal story a tester shared with me about the “power of the risk report” (or then how I can be really scary :)). My tester asked a programmer about some issues that had been outstanding for quite a while. The programmer shrugged the questions off saying he didn’t have the time to deal with the issues. My tester said, “OK, I’ll let Helena know.” The issues were resolved in a few hours.

Keeping It Sharp and Positive Takes Practice

That’s the long and short of it. When talking about risks and problems, it’s so easy to sound as if you were complaining. So I typically tried to highlight some positive progress both testers and programmers had been making. Trying to keep it sharp and to the point  takes some practice and focus because you want to avoid your report being a sleeping pill…

Undercover Influence

As much as I would’ve appreciated direct feedback on how the report helped people, I had to accept the second-hand account by my boss who told me that the CEO referred to the reports in several meetings where I wasn’t present. Well, that definitely says something about the relationship between me and the CEO 🙂 I’ve got a few good stories about that but I’ll save them for the future. I think I was reasonably happy with what I got, though. It’s nice to get publicly recognized but I’d take indirect recognition over no recognition anytime.

Heavy on Text, Light on Visuals – Not the Way to Go

Looking back, I think my reports had too much text and too little visualization. I reckon that I’m comfortable with doing a lot of reading (an English major speaking here) and extracting information from text. Obviously, if you want to report efficiently, you have to learn how the receivers of the report process information best. I also reckon that remaining in my comfort zone while making first attempts at reporting risks isn’t something I should beat myself up about. I can’t jump into perfection right away, can I?

Also, being able to nicely visualize the information in graphs and such requires that the tools you use for collecting information (bug tracking tool, for instance) are geared towards outputting it in a way you can manipulate well later on.

After the PEST 4.5 workshop on visualization and after seeing some great examples by Kristjan Uba that he uses in his work, I’ve realized how much better my reports could’ve been. It would still require a lot of work to make the reports useful even if you have a few good examples to begin with. The additional factors of complexity in my context were the size of the product under test (humongous) and the change of direction (frequent). The product was being grown in several directions at once and it felt like a very daunting task to try to create a picture of it. Now I think I could’ve done that had I taken a few steps back. Mindmaps for test coverage such as proposed by Katrina Clokie could have been subject to experimentation as we were using mindmaps in our team anyway.

It’s Hard to Report Well on Your Testing if You’re Not Sufficiently Clear on Why You Were Testing in the First Place

The subtitle for this section sounds harsh – almost as if me and my team didn’t know what we were doing. The problem wasn’t so much in not knowing but not articulating our testing missions. The problem was that business, development and testing were hardly on the same page or working intensely on the same thing. The testing team was also filling in for roles that didn’t exist but were necessary regardless (a bit of project management, a bit of business and system analysis). We were testing the new features to find out about the big issues because otherwise the customers would get them. We were testing the existing features in older branches to find out if the quality had remained the same or had got worse. We were testing to provide information about how optimizations and changes “agreed with” the product and to discover risks and issues that weren’t paid attention to when the code was written.

As you can see, it’s quite a lot and it dragged us in different directions. I can write about the why and how some other time… But the bottom line for me is that it was sometimes hard to connect the results to the mission of testing because it tended to blur into “trying to keep things under control” as programmers and business charged forward. The business and the product had competing missions, so it’s no wonder testing did too. But testing mission is what provides you the foundation for any kind of test reporting you do.

PEST 4.5: Workshop on Visualization

Apparently, both Kristjan Uba and me were toying with the idea of visualization and wondering how to use it communicate different aspects of testing and its results. Therefore, we quickly came to the agreement that the topic of the next peer conference should be visualization. We also agreed that this should be a practical workshop where something could be tried out instead of coming with a prepared presentation. The reason behind it was that well… I have done little of it and do it selectively; pretty much the same applies to Kristjan.

I hosted the event at my office where we gathered on the morning of April 26. Participants at PEST 4.5:

  • Kristjan Uba
  • Oliver Vilson
  • Rudolf Elbrecht
  • Kadri-Annagret Petersen (newcomer!)
  • me

We had some last minute cancellations as well.

Kristjan did the necessary introductions and explained the four tasks he wanted solutions for in visual form:

  1. Feature/area vs bugs/severity (how to show you have major and minor bugs in different areas of the product/system under test)
  2. Test coverage in different areas throughout iterations
  3. New features vs risks/bugs
  4. Status of the project/product overall

For bugs, he specified the following kinds:

  • “obviously major” bugs
  • important but can wait until next release to be fixed
  • “stupid bugs” that happen once every full moon
  • a lot of small things

We also had an imaginary project: electric bicycle for Dakar rally with all sorts of awesome features (just to prime our brains and have something tangible to start with).

Everybody picked a topic, grabbed the coloring pens and papers and got to work. Afterwards everyone explained what they did, then questions were asked, comments made. You know, the regular peer conference stuff. Since the group was so small, we didn’t use the cards to moderate discussions.

I picked the first task. Here’s how I went about it.

Attempt 1

I imagined boxes. Structure. Here’s the result:

Helena Jeret-Mäe_1


There were four areas of importance to my electric bicycle, so I each of the large quadrants is an area. Each of them is divided in 4 by severity of the bugs. There’s color coding: each severity has a corresponding color and my idea was that visualizing the clusters of bugs would be helped by colors. The letters are just placeholders: in real life there could by bug titles as hyperlinks or some kind of tags or other useful names.

For whatever reason, it seemed like it makes sense to have the critical bugs in the centre to make sure their clustering would catch everyone’s eye. But then again… what about the clustering of minor issues? This wouldn’t be as clear… which is why I changed tactics in the second attempt.

Attempt 2

I still went for a central cluster but thought of it more in terms of a bull’s eye.

Helena Jeret-Mäe_2

You may throw arrows at it and fix the bugs you pierce through 😛

Anyway, now I got some concentric rings. The good thing is that it’s easier to compare the bugs and their amount between areas. But then again you’d need to zoom in to the critical area because there may be too many and it won’t show up well. Also, the minor bugs at the edges may get less attention which is why Oliver suggested that in this model you could reverse the order and cluster the small ones in the centre to get attention.

You may notice some bugs have special marking: a circle around them. I used the severities I use daily: critical, major, medium, minor. But then I remembered Kristjan’s “stupid bugs” (see above) and decided that it may be worth marking those special cases while keeping their severity as is.

Attempt 3

Then I felt there was too little color… What about making it stronger?

Helena Jeret-Mäe_3

I used the same model but changed the means. Now color was in the background but bugs were spots. When I was working on it, new ideas spawned (or dawned on me):

  • and area could be broken down into sub-areas if there are some that are worth paying attention to. So the dotted lines represents breakdowns of an area.
  • I accidentally drew some bugs where I drew the dotted line but then it dawned on me that oh, I may have to show bugs that are between areas. Of course, there is a difficult limitation: you can only show “integration” bugs between areas that are side by side in your model already. But as we found out during the workshop, such relationships are hard to show.
  • You notice some spots are bigger than others… which is where I started playing with a new idea which I realized in attempt 4.

During the discussion, Kristjan pointed out that one good thing in this model is that we can place the release criteria into it. That’s pretty much what he meant:

Helena Jeret-Mäe_3 - Copy

The release criterion is simple: if there is anything inside the orange circle, we won’t release. Stuff outside is something we can live with. A clear goal emerges: let’s clean up the bull’s eye and its neighbouring areas.

Attempt 4

Helena Jeret-Mäe_4

I guess I was getting more abstract now… There were jokes about who can spot a constellation 🙂

I did return to my original structure but modified it. This visualization of bugs in 4 areas is meant to experiment with the following ideas:

  • size of the dot as the indication of severity
  • different sizes color coded vs without color coding
  • 4 areas divided into sub-areas
  • each sub-area gets its bug count but the whole status can be deduced by looking at this area (and comparing it with others if needed).

Somehow I avoided numbers and counts of bugs in my models. Others did use numbers and I think it is alright. But my brain was wired to create pictures of bugs, their whereabouts and importance for quick assessment. The details would have to be discussed if there is a need. But if you just want to know what’s going on… you don’t details at first.

I had the annoying thought of coverage that become harder and harder to suppress. Because hey… it is important if I found critical showstoppers in the first few tests which means the rest of the area is not covered or if I found them after having covered some ground already. So I just drew two slices of cake (I may have been getting hungry).

Helena Jeret-Mäe_5

The slice on the left is a sketch of showing what kind of test techniques were used. Maybe I used a lot of scenarios but didn’t focus on the data as much.

The slice on the right does a bad job but it was meant to communicate a hazy idea: the thickness of the slice would indicate how much of something was covered. The problem is obvious: the size of the slice itself is fixed and I don’t know how big exactly the “whole slice” is anyway.

Afterwards I realized that my visualizations were quite abstract, more like proofs of concept. But that’s okay: I have been thinking that I should try something for a loooong time but still haven’t. So this workshop was great for me to actually get visualizing and thinking about the problems. I got some good ideas from other participants but I’ll comment on those once they’ve summarized their ideas (hopefully, it will be soon-ish).

Before we wrapped up, Kristjan showed how he has tried to do some visualization and add colors to help with understanding. I definitely want to steal some ideas from how he has worked with test results to report them. He said that well, it’s not much visualization but some of the things he had I hadn’t really thought about even. Awesome!

All in all, PEST 4.5 was short and sweet and very useful. Looking forward to the next one!

Walk the Line, Draw the Line

Borders are amusing. I’ve always been intrigued by the imaginary line that cuts through an area and separates countries, people, ideas… I think it’s an abstraction that irrevocably alters the state of the world but the logic of the border sometimes escapes me. When I have stood on the border between two countries (easy to do here – just drive down to Latvia, for instance), I get a ticklish feeling. It’s so puzzling that I have to imagine that a step forward will take me to another country. It’s the same road that continues over there… it’s the same forest, the same grassland, it’s the same freakin’ air on both sides.

But it’s not really the same because of the border.

Instead of getting into a discussion of the nature of borders, I actually wanted to meditate on the border between the responsibilities of testers and engineers.

The (supposed) Problem

A while back, testers in my team reported a number of issues related to the GUI: typos, missing words, and confusing and/or ungrammatical sentences in system messages, typos in label text, other inconsistencies in GUI (different terms are used to refer to the same thing).

Since these issues were reported quite often, it stuck out. It stuck out that testers report a number of typos and other “minor” issues.

Usually, these problems are fixed quickly. There have been very few occasions were the report is sent back and forth because the engineer didn’t get it right the first time.

Soon enough the proposition was made that maybe testers should fix these issues themselves. The line of reasoning was that these are such minor issues that engineers shouldn’t spend time on fixing them.

The Implications

In my opinion, the aforementioned reasoning has a number of implications:

  • the value of testers’ time versus the value of engineers’ time
  • the severity of the problem motivates redrawing the lines between different roles
  • different perceptions of the value of information
  • empathy for end users
  • assumptions about the level of knowledge
From the technical perspective, it is true that those issues are not difficult to fix with the exception of the system messages which can be tricky in our context. However, it seemed to me that I’d step into an unintended trap if I did agree. After having thought about this issue, I listed these implications and found that it doesn’t really make that much sense. My final line of reasoning is much more mundane than the given list. But discussion first.
The implication about the difference in the cost of engineer’s time and tester’s time is nothing new. Also, I don’t think it’s something to get worked up about anyway since the past year has proven that testing and testers are a useful and productive part of the team. Since the problem I’ve described is (claimed to be) a borderline case, it’s my job as a team lead to explain why I think the border should be drawn differently.
I think the second item in the list is the most interesting one. If a serious bug is found, it is always clear and never debated who should fix it. It’s the engineer. Now minor UI issues seem to motivate the argument for shift in responsibilites and spark the border dispute. All of a sudden, the world (as we know it) must be remapped.
The source of the problem (engineers making mistakes which IS a normal part of their job) remains the same, yet it is argued that it makes more sense that someone else fixed it. This implies that the roles of testers and engineers are not clearly and universally defined.
The voiced concern is that the engineers have so many other issues to fix in the code and that “it would take just five minutes” for the testers to do it themselves. The unvoiced assumption is that engineers wouldn’t want to spend time on it either. Well, that’s a matter of taste. I know engineers who think fixing the UI is a nice snack in between serious code-rich meals. Some have even said that it’s nice to have some quick and easy tasks to be done. I think this is human. Of course, if one gets bombarded with such reports, it can get annoying too.
Still, these problems come from somewhere and it’s obvious that the engineers haven’t double-checked the UI and proofread the messages. Being asked to do so, doesn’t feel that great but it’s probably due to their main focus being on the code (which makes sense; and no, we don’t have an in-house GUI specialist around). It’s not that testers try to rub these reports in the engineers faces. I closely monitor the tone of the reports, so it’s not a problem as far as I’m able to decide. But it’s simply the underlying idea that these mistakes don’t seem to be worth engineer’s second look.

Personally, I have always been a bit of a GUI and terminology “nazi” (pun intended)… well, that’s another blog post. The point is that I tend to have an eye for the inconsistencies and I also pick up on messy terminology and word usage because of my background.

I have instilled the same attitude in my testers that it IS important to look at these things. It’s not that these GUI problems are showstoppers or that we wouldn’t look for critical issues and risky scenarios.

Checking the consistency in GUI is an important aspect of testing in my context (software for medical clinics and practices). I often find that the mistakes in GUI tend to be an unnecessary source of confusion not only for testers but also for technical writers. I also know that if the testers wouldn’t catch these mistakes, the technical writers will and their work process will suffer from this.

The value of information about GUI seems to only become very clear when one confronts a system message that just doesn’t make any sense or when you’re trying to filter a list but the filter fields don’t match the list columns… The program must speak the user’s language, be consistent in terminology and be clear about what is communicated. In the end, the showstoppers tend to be the stars among the bug reports.

Yet the GUI problems are like drawing pins – they look deceptively harmless until you happen to sit on one.

Herewith I’d like to take the opportunity to refer to Kristjan Uba’s blog post ( which tackles a similar situation. I agree with his thoughts on small things being huge but it’s a different cup of tea when you actually have to explain someone why it’s important to pay attention.

Empathy for end users is an important implication because the motivation for fixing the GUI issues (and trying to avoid them in the first place) shouldn’t come from the fear that a tester will send another annoying report. It should be something one cares about. However, walking in the end user’s shoes is not easy to do and it’s not a simple task to explain why it’s necessary to do so if the perspectives on usability are different. Especially when the perspectives on what actually constitutes an issue are very different. Oh yes, it’s very easy to get into a debate about whether the inconsistencies in GUI actually are issues. I haven’t tried yet (and I think I should) but I could make an attempt to explain the problem using the concept that quality is value to some people, then tie it with the end user’s perspective.

Another unvoiced assumption is that the tester and the engineer have the same level of knowledge which is necessary for fixing the UI issues. Sure, a typo or a missing word in a sentence is a no-brainer (most of the time). However, changing the text in a system message or deciding which term is the correct one is a different deal.

Often the meaning that seems to be conveyed in the system message text can be a bit different from what was intended. Sometimes engineers ask me what would be the best wording, it can sometimes take me (even me!) 15-30 minutes to figure out the best wording (oh, that’s another blog post…), especially if the message needs to ask the user something and do it with few simple words in a simple syntax.

So the tester should ask the engineer a few questions anyway. This won’t save much time, would it?

Secondly, deciding which term is correct to use in the UI is also not a tester’s decision. Sometimes it can seem very clear that it should be one not the other. But I still check. Because I can’t know for sure if there isn’t some hidden underlying logic somewhere… If one misjudges the situation, they may have made the problem worse. Also, sometimes messages seem to be similar so it’s easy to jump to the conclusion that they should be the same. When I’ve checked, it’s not the case. Another mistake avoided.

I usually escalate the terminology issues myself if I see the consistency issue is not localized in one part of the program but is actually a widespread problem (or has the potential for becoming one). Also, the engineer is more likely to know if the change in the UI should trigger other changes (which the tester may not have found out about).

Coming back to the “five-minute-fix”, then of course it’s not just five minutes. You fix it. You check if you fixed it correctly. You check the file(s) in. You merge it to other versions. you compile the package. You download the packaged code. You check it again. Then you fix the next issue. Five minutes? Nope. Plus you have the “opportunity” to mess something else up. Also, I simply find it a disruption which is not necessary in tester’s work.

For me the severity of the issue doesn’t justify nudging the border. It’s a sure thing that testers should make fixing the GUI issues easy (provide the correct wording, make clear screenshots of the issue, etc). But that’s the far end and that’s the border I generally wouldn’t like to cross. It’s not because I’m too stubborn but it’s mostly because I’m aware how the stealthy shifts in responsibilities can have other interesting implications.

I also discussed this topic with my team. At first, fixing the GUI issues didn’t seem too bad an idea to them but when they started considering the risks and possible complications… they also thought that it might not be a good idea. In some cases (when a window’s GUI is seriously deficient), it may be a good idea to make the changes instead of describing them but then again, the changes should be reviewed by the engineer.

All in all, I think defining and discussing the borders between testers’ and engineers’ domains is fruitful and important. Instead of arguing if the borders are correct, though, I thing it’s far more intriguing to look into why such discussions arise at all. Does it imply anything else about the mentality in the company? Where does it place on the “normality” scale (if we employ one)? Are there any unvoiced assumptions that drive these discussions? Is it just the matter of egos?

Anyway, borders are there. It’s important to know why they are where they are. And you should know if you need a visa when you plan to cross one.