Reporting Risks to Business: First Steps from Trying to Delivering (The Better Late Than Never Edition)

In June 2013 I wrote about how I tried to do a better job talking to the business people and providing them with useful information found in testing. This is the blog post you should read or else this one may not make much sense.

I kept updating a draft of this post but never got around to publishing it… Stuff kept happening, I kept experimenting, and every now and then I felt like what I try to do is stupid anyway… Naive as I am, I hoped I had found some sort of magic recipe for getting testing to be considered an integral and acknowledged part of how we developed software. Looking back I think that changing the way I reported test results was just one ingredient in informing decisions. It was an important one because it kind of felt like I’d kicked in a door when I started reporting risks. But it didn’t change as much as I personally would’ve liked or expected. Yes, I have high expectations for myself and others… The way the system reacted or didn’t react taught me more about how the system worked and its inertia.

What Changed After I Started Reporting About Risks?

I – Getting Things Moving

I started to send out the report a couple of times a week to communicate the progress my team had made in testing and the information that was found in testing. Later I changed it to once a week.

All showstoppers that I initially reported were either resolved or I could lower their risk level because something was fixed or we got the information we needed. In other words, I got the attention of people who could solve problems and this helped the testing team along. Getting people informed of what was really going on in the product got some conversations going and relevant meetings set up to get problems resolved. Because really… most of the information we provided about the product was about failures in areas that were said to be important. It’s just people in decision making position tended to be preoccupied with things going on right now.

A note about the development process: the ideas for features were realized in code quickly; some meetings were held, no written specs or prototypes – prototyping was done right in the code… If you’re building on an existing product, it’s not too hard to do. Then when the developer and someone on the business side felt it was kind of done, it was given to the testing team. And then developers and the business moved on to the next things… which caused some problems as you may imagine. Even though I could write at length about the kind of problems such a process brings about, I’m leaving this for another time.

II – Making The Report a Staple of a Meeting

My boss proposed that I present the report at the weekly meeting with product owners and other folks (it’s a meeting where there are product, owners, a dev team lead and other people, including the CEO).

The first time I presented it, it sure got my adrenaline running!

I explained the purpose of the risk report, what each of the sections meant and covered the issues as briefly as I saw fit, highlighting the most important aspects of each issue. They could see the report on my screen as well (we had the “phone into a conference line and share your screen” type of meetings most of the time). I forgot to time myself but I don’t think I spent more than 5 minutes.

I got some additional input and requests for 2 meetings: 1 to discuss the risks I talked about (we found that a feature only worked when following a happy path) and 1 to discuss the risk assignment and perspectives on my report. I was OK with discussing the reasons I assigned certain risk levels and the reasons why someone else may disagree as long as we get to talk about it.

III – Maintaining the Focus of Decision Makers

Later on I noticed that some tasks I had put on the report I couldn’t take off for weeks because there were fixes that weren’t delivered yet (or we kept finding issues when retesting the fixes; in other words, the fault-feedback-ratio was high). So I left those items on the report as a FYI  item – the functionality was almost done and tested but we couldn’t sign it off yet. I commented on those briefly, stating that the status of these items hasn’t changed.

There were features we tested that appeared to be “good enough” for showing to the customers to get their feedback as they were going to be the early adopters. However, I noticed that the feedback never arrived… So I still kept those items on my report and said that those we are still waiting feedback on.

One might ask what’s the point and why did I keep coming back and talking about those features. I don’t want to go into lengthy explanations but the reason lies in the context: I noticed a pattern where things don’t get followed through properly and this had consequences for customers in the end. I cared enough to risk a little neutral-toned nagging to make sure the whole team keeps their eye on the ball (or rather, the sea of balls…). For me it was  sad to see if someone drops the ball and doesn’t carry things out to the end because oh hey, there’s something shiny elsewhere…

However, after an extended period of time when even I was starting to feel embarrased, I simply said that I’m not going to report on this item anymore as there is no interest whatseoever in finalizing it, so you have been notified of this particular risk.

My comment about this situation is that it was a characteristic of a system, a trait of the company culture, a way of going about things. Even though there was a lot of talk about being accountable and keeping customers’ interests in mind, some still got left behind even though they were declared important at first.

IV – Contrasting Testing Tasks, Highlighting Problems in Workload

In addition to the 2 reports I already mentioned in my previous blog post, I included a simple Excel spreadsheet which contained a list of low priority tasks that we weren’t be able to test before the release. Each of those items was a fairly small one but combined, it was a couple of weeks’ worth of work.

My idea behind this was to create a  contrast between the few important items that have more details about them. And then a long list of things that won’t be covered at all. I had no intention of postponing the release or using this list to manipulate anyone. I just wanted to SHOW how things are at this point without complaining and give the decision makers the opportunity to jump in if they want to. If not, it’s OK.

Later I also introduced a breakdown of the workload my team has: how much new stuff, how much old pending stuff and how much of it we’re getting done. This I did in an attempt to figure out good ways for transparency about what and how much is going on in general. I dropped the list of low priority stuff fairly quickly as I didn’t think it was or was actually useful.

V – Starting Point for Further Experimentation

I also kept experimenting with the written report and the oral reporting based on the text during the meetings. I kept the breakdown of high and medium priority for a while, then described the issues, risks and possible mitigation. At some point I felt like I couldn’t always provide anything reasonable for this three-tier breakdown. So I decided to ditch it. Instead, I wanted to work on describing the issues and risks we’ve found in a concise but not too detailed way.

I noticed that when I included an issue that seemed interesting or important then it ended up being too context-specific to explain quickly. My interpretation of it was that I had not properly thought through the “why”, and, therefore, wasn’t able to link the issues to the big picture well enough.

VI – Changing How Testers Report the Results

Because I changed how I provide information to others, I had to change the way I obtained information myself. I explained to my team how I needed to get input from them: a well-summarized nutshell and I’d ask for details if I needed them. I encouraged them to use the same mode when they were on meetings where I wasn’t present. I observed that more focus on clarity did help make the meetings more efficient.

As far as oral reporting is concerned, I tried to observe myself while speaking. I’ve noticed that I to try to gloss over or fumble too much from time to time and I take it as a heuristic for not having thought things through.

I tried to start out with outlining the major testing activities or areas concerned in order to get folks oriented and then go into particular sections of the report. I also tried to give a lead in for each part of the report to make the flow of information better.

Reflections On Some Lessons Learned

Unwanted Attention

The risks and problems I reported about weren’t anonymous. They belonged to an area a programmer was responsible for which meant that if a lot of problems came about the programmer didn’t look particularly good. Therefore, after the report had become a staple, I informed the programmers beforehand that I was going to include something that concerned them on the report. Since I took quite a lot of “project management by stealth” responsibilities on my shoulders,  it meant that when I pointed out the problems I’d also try to negotiate resources for solving them. For example, trying to get hold of someone in the US to answer the questions testers had that the programmer also wanted to know about was something I arranged instead of dumping it onto the programmer or tester to take care of.

There also was a bit of an anecdotal story a tester shared with me about the “power of the risk report” (or then how I can be really scary :)). My tester asked a programmer about some issues that had been outstanding for quite a while. The programmer shrugged the questions off saying he didn’t have the time to deal with the issues. My tester said, “OK, I’ll let Helena know.” The issues were resolved in a few hours.

Keeping It Sharp and Positive Takes Practice

That’s the long and short of it. When talking about risks and problems, it’s so easy to sound as if you were complaining. So I typically tried to highlight some positive progress both testers and programmers had been making. Trying to keep it sharp and to the point  takes some practice and focus because you want to avoid your report being a sleeping pill…

Undercover Influence

As much as I would’ve appreciated direct feedback on how the report helped people, I had to accept the second-hand account by my boss who told me that the CEO referred to the reports in several meetings where I wasn’t present. Well, that definitely says something about the relationship between me and the CEO 🙂 I’ve got a few good stories about that but I’ll save them for the future. I think I was reasonably happy with what I got, though. It’s nice to get publicly recognized but I’d take indirect recognition over no recognition anytime.

Heavy on Text, Light on Visuals – Not the Way to Go

Looking back, I think my reports had too much text and too little visualization. I reckon that I’m comfortable with doing a lot of reading (an English major speaking here) and extracting information from text. Obviously, if you want to report efficiently, you have to learn how the receivers of the report process information best. I also reckon that remaining in my comfort zone while making first attempts at reporting risks isn’t something I should beat myself up about. I can’t jump into perfection right away, can I?

Also, being able to nicely visualize the information in graphs and such requires that the tools you use for collecting information (bug tracking tool, for instance) are geared towards outputting it in a way you can manipulate well later on.

After the PEST 4.5 workshop on visualization and after seeing some great examples by Kristjan Uba that he uses in his work, I’ve realized how much better my reports could’ve been. It would still require a lot of work to make the reports useful even if you have a few good examples to begin with. The additional factors of complexity in my context were the size of the product under test (humongous) and the change of direction (frequent). The product was being grown in several directions at once and it felt like a very daunting task to try to create a picture of it. Now I think I could’ve done that had I taken a few steps back. Mindmaps for test coverage such as proposed by Katrina Clokie could have been subject to experimentation as we were using mindmaps in our team anyway.

It’s Hard to Report Well on Your Testing if You’re Not Sufficiently Clear on Why You Were Testing in the First Place

The subtitle for this section sounds harsh – almost as if me and my team didn’t know what we were doing. The problem wasn’t so much in not knowing but not articulating our testing missions. The problem was that business, development and testing were hardly on the same page or working intensely on the same thing. The testing team was also filling in for roles that didn’t exist but were necessary regardless (a bit of project management, a bit of business and system analysis). We were testing the new features to find out about the big issues because otherwise the customers would get them. We were testing the existing features in older branches to find out if the quality had remained the same or had got worse. We were testing to provide information about how optimizations and changes “agreed with” the product and to discover risks and issues that weren’t paid attention to when the code was written.

As you can see, it’s quite a lot and it dragged us in different directions. I can write about the why and how some other time… But the bottom line for me is that it was sometimes hard to connect the results to the mission of testing because it tended to blur into “trying to keep things under control” as programmers and business charged forward. The business and the product had competing missions, so it’s no wonder testing did too. But testing mission is what provides you the foundation for any kind of test reporting you do.

Devil in the Details: How to Connect Ideas in Text

Ilari wrote about indexicality a while back.

Then Duncan Nisbet wrote a blog post about how he tries to improve/reduce his use of deixis.

I ended up ranting a bit in the comments section about it because this topic is very familiar to me: I’ve been drilled and trained and what not about deixis at the university. And there’s another topic similar to that of deixis where the devil is in the details: linking words.

I had learned about the linking words back in school, of course, but after entering the university, I quickly learned I hadn’t paid sufficient attention and respect to them. All those comments and strikethroughs in red pen… My supervisor for my BA thesis gave me the ultimate lessons in using linking words by asking in a very ironic voice “Why do you use “because” here? Do you really think there is a causal relationship between these sentences?”

What are linking words?

Linking words and phrases are sentence connectors that are used  to link one idea to another. For example, “because”, “as”, “so”, “firstly”, etc are linking words.

Each of them is used to create a specific link between ideas. You would use “because” if there is a causal relationship: “He was tired because he had stayed up late”.

Why are linking words important?

They are important because using them right makes the text easily readable. If misused, the text may become fairly difficult to understand.

In my experience people don’t often pay attention to linking words and sprinkle them in the text incorrectly. As a result, the text becomes less clear or the text suggests connections between ideas that are not there. I often find the linking words for causality and contrast are misused. Then it’s the burden of the reader to figure out why two sentences that seem to go together are contrasted or why causal relationship is suggested even though the content of the sentences doesn’t suggest it.

As a (frequent) editor of different kind of texts, I have learned that the most difficult texts to crack aren’t necessarily those where the terms are used incorrectly or where the grammar or syntax isn’t clear. The biggest source of misunderstanding for me comes from disjointed ideas and poor connection between the ideas.

I have looked out for the use of linking words in my testers’ bug reports and have explained to them how to use linking words and what to pay attention to.

The reason for this is that if linking words are used effectively, the text is easier to read, so the reader doesn’t have to make a big effort. The text will flow much more nicely if you connect your ideas properly. Also, since I try to make my bug reports crisp and to the point, I want to make sure I have used linking words efficiently.

Clarity is important when dealing with programmers who often skim read the reports (because they’re busy… what did you think?). Importantly, I think that getting such details right is a fundamental skill in both written and oral texts.

Therefore, I suggest it’s worthwhile to scrutinize your reports and find out (if? and) how you use linking words in them.

There is plenty of material out there but here’s a fairly decent collection of linking words. Print the list out and stick it on the wall!

Reporting Risks to Business: Truly Trying

There were 4 more or less recent events that pushed me out of my comfort zone and that made me see how stupid and futile were my attempts to hide behind “I don’t know how” and “they don’t care” in the context of reporting potential and actual risks in the product by talking business to decision makers.

I

Scott Barber’s keynote  (slides) at Let’s Test blew me away because of how he managed to connect the value of testing to speaking the business language for me. It’s not like I *didn’t* know that information from testing should be used for informing business decisions. But I’d say my recognition of it was shallow and I didn’t fully realize it or act upon this knowledge. It isn’t that me and my team *don’t* give out any information but I realized that we could do it so much better than before by focusing less on giving information to programmers.

Scott’s keynote really pushed me to think seriously about what I’ve done so far. Somehow his presentation of it ended up being so liberating for me that I could take a very long and hard look in the mirror without trying to rationalize the mirror away by saying that I’m somewhat of a victim to my context.

II

Kristjan Uba’s workshop at Nordic Testing Days (brief description and slides) dealt with test reporting and also tied it to mission of testing which, in turn, was tied to the business needs. Briefly: teams got a brief from the customer and then had an hours to test the product, then they had to report their results in 4 minutes. I saw different examples in reporting and recognized the ones I’d like to sample. It became even clearer to me that I need to try and do things differently.

III

Over the past weekend I read “Perfect Software” and I felt as if I had been tweaked by my ear. Sure, a lot of what is in this book  is what I know from other sources or that relates to something I’ve experienced. It wasn’t really new information for me. However, reading it made me more decisive about having to try to do better because I felt like Gerald was hammering me for my mistakes… I haven’t met him in person but well, if he’s so effective via a book, then… 😀

IV

This is probably the most important one but it is also the most painful one. It is the original source to my discontent with how I’ve been giving information obtained from testing. And also a great lesson. This happened a couple of months ago when I ended up between a rock and a hard place, and boy… did I hate this feeling!

The details won’t matter (and I can’t really share them) but here’s what happened: I resisted agreeing to giving beta status to certain important parts of the product by claiming that we haven’t been able to test those parts. Then I was asked “but do you know of any important problems?”

SLAP!

I only had my gut feeling (which turned out to be right…) that certain things in the product are going to cause problems. But what does the business have to do with my gut feeling if they really-really want to release AND I don’t have any better information to give them? It also doesn’t really matter why I didn’t have that information at the time (objectively, the plan I knew about was different and I was supposed to have time to test to find that information, etc). The fact is that I didn’t have it  when it was needed.

So for the next release I wanted to do things differently. And I did it today.

Full of determination but without an exact plan I sat down yesterday and started putting together a risk report based on information my team had gathered while testing. I wanted it to be very simple to read (ideally at a glance), fairly short, containing teasers not full details. I thought of my audience (CEO, a few managers directly below CEO, COO…) and remembered how they have got entangled in details that I have brought to their attention previously. I started putting it together so that it would be fairly high level and easy to read for myself, too.

Here’s the sample: Risk report from testing

The total length was 2.5 pages and I almost didn’t want this to run longer than 2 pages. I should note that this report didn’t aspire to be a full report on even every risk I am vaguely aware of. I just focused on the most important stuff right in front of me. I can start experimenting later which is going to be the most interesting part…

You probably noticed I don’t have the “low risk” category at all. I excluded it to avoid diluting the report or losing sharpness because I feel like this is what is needed right now.

I also had a another document about areas that I think are high and medium risk but that we haven’t started testing yet. I wanted to highlight those areas in case anybody wanted to change the risk assessment (nobody did). This document should potentially include more information about what are the important things we may not be able to test on time (shuffling priorities, unexpected hiccups…). I also asked if there is anything they know of that I should add here or if there is anything in the product they want more information about. Lo and behold, I got 3 more items on my list (plus a few hints).

Maybe if I look at it tomorrow, I will think it looks ugly or maybe I want to cave out my eyes because of how I have used formatting.

But today it looks great enough.

Don’t get me wrong: I have tried to bring different risks to decision makers’ attention but I’ve never presented it in such a way. Also, it’s not like the business side never asks for information from us. But I think they should be better at it and realize that they could ask for and get more and better information. I can help them discover this to some extent.

So I sent it out and I saw a fairly quick effect on some showstoppers (tasks were reprioritized or reassigned…). Today I went through this report at a meeting and well, all those items got sorted out (fixes got higher priority, one feature got pulled out of the release, etc) without any hassle.

My boss said, “Very impressive!”

As for me, I am pleasantly surprised 🙂 And happy because it went better than expected. And even more happy because I learned that I can report to business in a way that is heard, that I broke the vicious circle in my head. I hope I can use this experience to tackle those other circles…

What’s next? I want to keep experimenting with the content, structure and formatting to see if I can find something better (testing reactions…). I may end up changing it around completely because I don’t intend this to by my Template of Final Destination in Reporting. Maybe it’s a good idea to keep people on their toes about how the risk report looks like. 🙂

I expect to run into issues and challenges as I keep going, too but I know I can and will tackle them.

Feedback and criticism is welcome as always.