Reporting Risks to Business: First Steps from Trying to Delivering (The Better Late Than Never Edition)

In June 2013 I wrote about how I tried to do a better job talking to the business people and providing them with useful information found in testing. This is the blog post you should read or else this one may not make much sense.

I kept updating a draft of this post but never got around to publishing it… Stuff kept happening, I kept experimenting, and every now and then I felt like what I try to do is stupid anyway… Naive as I am, I hoped I had found some sort of magic recipe for getting testing to be considered an integral and acknowledged part of how we developed software. Looking back I think that changing the way I reported test results was just one ingredient in informing decisions. It was an important one because it kind of felt like I’d kicked in a door when I started reporting risks. But it didn’t change as much as I personally would’ve liked or expected. Yes, I have high expectations for myself and others… The way the system reacted or didn’t react taught me more about how the system worked and its inertia.

What Changed After I Started Reporting About Risks?

I – Getting Things Moving

I started to send out the report a couple of times a week to communicate the progress my team had made in testing and the information that was found in testing. Later I changed it to once a week.

All showstoppers that I initially reported were either resolved or I could lower their risk level because something was fixed or we got the information we needed. In other words, I got the attention of people who could solve problems and this helped the testing team along. Getting people informed of what was really going on in the product got some conversations going and relevant meetings set up to get problems resolved. Because really… most of the information we provided about the product was about failures in areas that were said to be important. It’s just people in decision making position tended to be preoccupied with things going on right now.

A note about the development process: the ideas for features were realized in code quickly; some meetings were held, no written specs or prototypes – prototyping was done right in the code… If you’re building on an existing product, it’s not too hard to do. Then when the developer and someone on the business side felt it was kind of done, it was given to the testing team. And then developers and the business moved on to the next things… which caused some problems as you may imagine. Even though I could write at length about the kind of problems such a process brings about, I’m leaving this for another time.

II – Making The Report a Staple of a Meeting

My boss proposed that I present the report at the weekly meeting with product owners and other folks (it’s a meeting where there are product, owners, a dev team lead and other people, including the CEO).

The first time I presented it, it sure got my adrenaline running!

I explained the purpose of the risk report, what each of the sections meant and covered the issues as briefly as I saw fit, highlighting the most important aspects of each issue. They could see the report on my screen as well (we had the “phone into a conference line and share your screen” type of meetings most of the time). I forgot to time myself but I don’t think I spent more than 5 minutes.

I got some additional input and requests for 2 meetings: 1 to discuss the risks I talked about (we found that a feature only worked when following a happy path) and 1 to discuss the risk assignment and perspectives on my report. I was OK with discussing the reasons I assigned certain risk levels and the reasons why someone else may disagree as long as we get to talk about it.

III – Maintaining the Focus of Decision Makers

Later on I noticed that some tasks I had put on the report I couldn’t take off for weeks because there were fixes that weren’t delivered yet (or we kept finding issues when retesting the fixes; in other words, the fault-feedback-ratio was high). So I left those items on the report as a FYI  item – the functionality was almost done and tested but we couldn’t sign it off yet. I commented on those briefly, stating that the status of these items hasn’t changed.

There were features we tested that appeared to be “good enough” for showing to the customers to get their feedback as they were going to be the early adopters. However, I noticed that the feedback never arrived… So I still kept those items on my report and said that those we are still waiting feedback on.

One might ask what’s the point and why did I keep coming back and talking about those features. I don’t want to go into lengthy explanations but the reason lies in the context: I noticed a pattern where things don’t get followed through properly and this had consequences for customers in the end. I cared enough to risk a little neutral-toned nagging to make sure the whole team keeps their eye on the ball (or rather, the sea of balls…). For me it was  sad to see if someone drops the ball and doesn’t carry things out to the end because oh hey, there’s something shiny elsewhere…

However, after an extended period of time when even I was starting to feel embarrased, I simply said that I’m not going to report on this item anymore as there is no interest whatseoever in finalizing it, so you have been notified of this particular risk.

My comment about this situation is that it was a characteristic of a system, a trait of the company culture, a way of going about things. Even though there was a lot of talk about being accountable and keeping customers’ interests in mind, some still got left behind even though they were declared important at first.

IV – Contrasting Testing Tasks, Highlighting Problems in Workload

In addition to the 2 reports I already mentioned in my previous blog post, I included a simple Excel spreadsheet which contained a list of low priority tasks that we weren’t be able to test before the release. Each of those items was a fairly small one but combined, it was a couple of weeks’ worth of work.

My idea behind this was to create a  contrast between the few important items that have more details about them. And then a long list of things that won’t be covered at all. I had no intention of postponing the release or using this list to manipulate anyone. I just wanted to SHOW how things are at this point without complaining and give the decision makers the opportunity to jump in if they want to. If not, it’s OK.

Later I also introduced a breakdown of the workload my team has: how much new stuff, how much old pending stuff and how much of it we’re getting done. This I did in an attempt to figure out good ways for transparency about what and how much is going on in general. I dropped the list of low priority stuff fairly quickly as I didn’t think it was or was actually useful.

V – Starting Point for Further Experimentation

I also kept experimenting with the written report and the oral reporting based on the text during the meetings. I kept the breakdown of high and medium priority for a while, then described the issues, risks and possible mitigation. At some point I felt like I couldn’t always provide anything reasonable for this three-tier breakdown. So I decided to ditch it. Instead, I wanted to work on describing the issues and risks we’ve found in a concise but not too detailed way.

I noticed that when I included an issue that seemed interesting or important then it ended up being too context-specific to explain quickly. My interpretation of it was that I had not properly thought through the “why”, and, therefore, wasn’t able to link the issues to the big picture well enough.

VI – Changing How Testers Report the Results

Because I changed how I provide information to others, I had to change the way I obtained information myself. I explained to my team how I needed to get input from them: a well-summarized nutshell and I’d ask for details if I needed them. I encouraged them to use the same mode when they were on meetings where I wasn’t present. I observed that more focus on clarity did help make the meetings more efficient.

As far as oral reporting is concerned, I tried to observe myself while speaking. I’ve noticed that I to try to gloss over or fumble too much from time to time and I take it as a heuristic for not having thought things through.

I tried to start out with outlining the major testing activities or areas concerned in order to get folks oriented and then go into particular sections of the report. I also tried to give a lead in for each part of the report to make the flow of information better.

Reflections On Some Lessons Learned

Unwanted Attention

The risks and problems I reported about weren’t anonymous. They belonged to an area a programmer was responsible for which meant that if a lot of problems came about the programmer didn’t look particularly good. Therefore, after the report had become a staple, I informed the programmers beforehand that I was going to include something that concerned them on the report. Since I took quite a lot of “project management by stealth” responsibilities on my shoulders,  it meant that when I pointed out the problems I’d also try to negotiate resources for solving them. For example, trying to get hold of someone in the US to answer the questions testers had that the programmer also wanted to know about was something I arranged instead of dumping it onto the programmer or tester to take care of.

There also was a bit of an anecdotal story a tester shared with me about the “power of the risk report” (or then how I can be really scary :)). My tester asked a programmer about some issues that had been outstanding for quite a while. The programmer shrugged the questions off saying he didn’t have the time to deal with the issues. My tester said, “OK, I’ll let Helena know.” The issues were resolved in a few hours.

Keeping It Sharp and Positive Takes Practice

That’s the long and short of it. When talking about risks and problems, it’s so easy to sound as if you were complaining. So I typically tried to highlight some positive progress both testers and programmers had been making. Trying to keep it sharp and to the point  takes some practice and focus because you want to avoid your report being a sleeping pill…

Undercover Influence

As much as I would’ve appreciated direct feedback on how the report helped people, I had to accept the second-hand account by my boss who told me that the CEO referred to the reports in several meetings where I wasn’t present. Well, that definitely says something about the relationship between me and the CEO 🙂 I’ve got a few good stories about that but I’ll save them for the future. I think I was reasonably happy with what I got, though. It’s nice to get publicly recognized but I’d take indirect recognition over no recognition anytime.

Heavy on Text, Light on Visuals – Not the Way to Go

Looking back, I think my reports had too much text and too little visualization. I reckon that I’m comfortable with doing a lot of reading (an English major speaking here) and extracting information from text. Obviously, if you want to report efficiently, you have to learn how the receivers of the report process information best. I also reckon that remaining in my comfort zone while making first attempts at reporting risks isn’t something I should beat myself up about. I can’t jump into perfection right away, can I?

Also, being able to nicely visualize the information in graphs and such requires that the tools you use for collecting information (bug tracking tool, for instance) are geared towards outputting it in a way you can manipulate well later on.

After the PEST 4.5 workshop on visualization and after seeing some great examples by Kristjan Uba that he uses in his work, I’ve realized how much better my reports could’ve been. It would still require a lot of work to make the reports useful even if you have a few good examples to begin with. The additional factors of complexity in my context were the size of the product under test (humongous) and the change of direction (frequent). The product was being grown in several directions at once and it felt like a very daunting task to try to create a picture of it. Now I think I could’ve done that had I taken a few steps back. Mindmaps for test coverage such as proposed by Katrina Clokie could have been subject to experimentation as we were using mindmaps in our team anyway.

It’s Hard to Report Well on Your Testing if You’re Not Sufficiently Clear on Why You Were Testing in the First Place

The subtitle for this section sounds harsh – almost as if me and my team didn’t know what we were doing. The problem wasn’t so much in not knowing but not articulating our testing missions. The problem was that business, development and testing were hardly on the same page or working intensely on the same thing. The testing team was also filling in for roles that didn’t exist but were necessary regardless (a bit of project management, a bit of business and system analysis). We were testing the new features to find out about the big issues because otherwise the customers would get them. We were testing the existing features in older branches to find out if the quality had remained the same or had got worse. We were testing to provide information about how optimizations and changes “agreed with” the product and to discover risks and issues that weren’t paid attention to when the code was written.

As you can see, it’s quite a lot and it dragged us in different directions. I can write about the why and how some other time… But the bottom line for me is that it was sometimes hard to connect the results to the mission of testing because it tended to blur into “trying to keep things under control” as programmers and business charged forward. The business and the product had competing missions, so it’s no wonder testing did too. But testing mission is what provides you the foundation for any kind of test reporting you do.

A Workshop on Testing for Programmers

An idea had been twirling around my brain for a while. Then it became more and more insistent on breaking into the realm of reality. In the context of continuously learning about testing, and in the context of recognizing the ongoing challenges of programmers and testers working together, I realized that I need to do more about it. Quick conversations over lunch or in the hallway about one or other aspect about testing just didn’t seem to be enough. There needed to be something that’s a full package, coherent,  concentrated and experiential. There needed to be something that would renew the foundation for future conversations between testers and programmers.

Mission: create a full-day workshop on testing that would be tailored and targeted at the programmers I work with.

I consider this workshop as one of the great outcomes of Transpection Tuesdays. Without the work I’ve done with Erik Brickarp and without his support and encouragement, it would’ve taken me much longer to muster up the confidence to create and carry out the workshop.

The keyword here is “tailored”. Fun and exercises aside, I focused on tying together the general introduction into testing, the way testing had been done by my team, and whatever else was relevant in the company’s context in relation to testing. I wanted to frame some of the specific problems in our company as testing problems and have a discussion about them.

Goals:

• give a concentrated introduction to testing so that fragments of what programmers already knew were tied into a whole
• introduce and explain testing problems to  programmers so that they could be understood and discussed better
• immerse programmers in some testing problems so that they would see and experience them up close (followed by a discussion and explanations)
• explain what testers do and tie it into their experiences of working with testers so that programmers can make sense of their experience and forge even better relationships with testers
• explain the tester’s mindset and vocabulary so that further conversations between testers and programmers could be more productive

Why I’m Sharing This

I’m sharing this just because I believe from my experience that creating such a workshop can be useful and educational in many ways, and there are surely people out there who need just a bit of nudging to do something similar where they work.

Slides and some accompanying materials can be found here.

It doesn’t contain all handouts because well… what you think you need to provide to people you work with will really depend on the kind of workshop you create. I also didn’t include my notes for slides because they have quite a lot of context-specific stuff in them and it didn’t make sense to try to clean it up.

You can grab them and run but it would be so much more useful for you if you read the following as well 🙂 Then you will know WHY it is the way it is.

Planning: content and structure

Preliminary notes

One of my favourite ways of getting some mental work done is to go to a long lunch alone, take my notebook with me and just write my ideas down. I did quite a bit of work on the workshop in that manner, clarifying my initial ideas about the structure of the workshop, taking notes about what kind of exercises to use etc.

Mindmapping

• I created a mindmap using Xmind to sketch the initial structure of the workshop. The mindmap I’m sharing with you isn’t perfect and it looks sketchy. I just used it as a starting point.
• I mindmapped the big sections and also drafted ideas for subtopics to be covered in sections
• I added labels to each chunk estimating the time I’d be spending on it. Then I summed them up and the result was the time I was going to spend on each section. And from there I could calculate the time I’d spend on the whole workshop.
• Estimating the time spent helped me balance the sections and decide whether to include or exclude content. It also kept me in check and guarded against adding too much.

Slides and notes

• Based on the mindmap structure, I started drafting my PowerPoint slides, taking notes about the content and gathering relevant references.
• I kept track of references in a separate document to make sure I can hand that list to the attendees later.
• Working with the mindmap and slides I did a lot of gauging of the flow and used internal monologue or tried explaining some concepts to myself. The purpose was to make sure the flow of ideas made sense and that each section built upon the previous one. In that process I made structural changes to the mindmap which provided me with a good backbone and overview of the workshop.
• For each section, I added a separate title slide and a recap slide. The idea was that the day will be so packed with information that it’s better to add some points where I can concisely summarize what we just talked about. All those recaps would then go on one sheet of paper. If that was all they ever kept from the workshop… they would have the essential ideas captured.

Exercises

General idea behind beer

• I wanted to have a “thin red line” that would run through the workshop. There would be so much ground covered in terms of concepts that I felt like falling back on a single constant thread would help pull us all back in.
• I found that thin red line in the workshop by Louise Perold on test planning that I attended at Copenhagen Context this January. She used bottles of beer as the product in the workshop and it worked really well for a test planning exercise. Sure, it’s not software but… beer is fun, fairly well understood and doesn’t need much explanation. So I discussed this with Louise and she was fine with me picking up the idea and using it myself. Many thanks for that! 🙂

Kick-off exercise

• Testing the beer was to be the thin red line and the kick-off exercise.  Each team of 3-4 people got two different bottles of beer. The only brief I would give to testing beer at 10am in the morning was to “test the beer and explore it” in 20 minutes. I didn’t give them the bottle opener as I wanted to see if any would ask for it. I gave them a sheet of paper and colorful markers to write down whatever they wanted about the beer they tested. I made observations and took notes about what they said in their interactions while they were testing the beer – I’d use these notes for debriefing and for tying the discussed topics with the rest of the workshop. For instance, if I’d ask how they knew the beer was OK, they could give me answers like “from my personal experience” and later when getting to oracles, we could discuss their initial test run in the light of heuristics and oracles (or anything else we covered during the workshop).

Follow-up exercises with beer

• Testing mission – since very few asked why they were testing the beer and what was supposed to be accomplished, I used this as an example of how not to approach testing… and then asked them to figure out a few testing missions for beer that would frame the testing activities. I also brought in the project lifecycle to show how different things can be tested for over time.
• Oracles – since it wasn’t that easy to find actual issues with the beer, it gave me a good segue into asking why couldn’t they spot some problems. And from there we could get into oracles and how they’re useful for testing. Later I used as another recap of the kick-off exercise when trying to come with some oracles for testing the beer and pointing out the ones they had actually used but didn’t realize it.
• Test strategy – I gave them a brief, then asked to come up with a simple description of a strategy. I mostly meant this one as the “dip into this but don’t get too deep” kind of an exercise. The point was to teach about test strategy per se but to make them feel how difficult it may be to word it. Also, I gave them the whole HTSM on paper but limited what they had to use. It was probably a bit too much…. in terms of balancing time and materials. But I feel like they got the sense of complexity out of it at least…
• Creating a script for opening the beer. This was a pretty fun one… The idea was to create an experience where you’re supposed to do something simple – just follow the instructions. But then describing how to open a beer in a specific manner isn’t THAT easy. So on the one hand, people could get creative (some drew up pictures to illustrate the steps) but then when executing the steps, I limited them in their interpretation. One of the coolest instances was when the script was followed precisely but the beer bottle ended up with its cap on not off  🙂 There was a very interesting contextual cue that a programmer decided to “misinterpret”…
• Testability – when discussing testability, we returned to the beer trying figure out what would make it easier to test beer.

Other logistics

• I divided programmers into 3 groups of 9 which meant I ran this workshop 3 times. This seemed like a manageable amount of people for me.
• Since I knew them all well (I’ve worked with them for years), I tried to balance out the groups so that programmers from different teams could interact as well.
• I tried to make the first group the most difficult for me in order to make sure I will find any issues in how I’ve set up the workshop and fix them before the next one. That one I got right – they were active, wanted to argue (not just with me… even more amongst themselves), wanted to discuss problems and potential solutions to them. I was able to figure out some issues in the order of sections which was a good thing to fix.
• I had snacks ordered for the breaks and pizza for lunch – just to make it easier for everyone.

Lessons learned

•  Oh my god… was I tired after the first workshop! I felt completely exhausted. It’s the type of feeling when everything is in slow motion… like reaaaaaallllyy slllooowwww…. But I still felt great about how it all turned out. Luckily, the 2nd and 3rd runs weren’t as exhausting.
• I learned how much time actually goes into putting a workshop together. I think I may have spent at least 100 hours when preparing for it.
• I learned that I enjoy working on a workshop like this. It helped me to freshen up what I know (well, it’s fairly well known that teaching someone teaches yourself a lot, too). I also enjoyed delivering workshop even though I got tired and some people started arguing a lot 🙂 It was fun to observe how programmers from the same team started arguing how to test better and what they could or couldn’t do on their end. I felt like getting such a conversation going was an accomplishment on its own.
• I wasn’t very good at using all of the information I picked up from eavesdropping on the conversations taking place during the exercises. I guess this is a matter of practice and a few other things… like having a much better command of the material. I feel like I could’ve done this part a bit better.
• The second half of the workshop could’ve used more actively engaging exercises. For some reason I figured people could be too tired for them… but I guess there should’ve been something that would shake people up 🙂
• After the first run, I did change the order of topics in part two: I figured I should introduce the terminology of oracles and heuristics before starting to talk about heuristic test strategy model. Else I would introduce words I would only explain later.
• Good feedback from participants was along the lines of “I had no idea testing is so complex” and “I didn’t know how much more there was to it”. Someone even said that “I figured testers had to be smarter than programmers” 🙂 Since I left the company about a month after wrapping up the workshop series, I can’t tell what kind of overall influence the workshop had. But a few programmers I’ve talked to afterwards have told me they’ve thought about the workshop and what they took from it, so… I guess that’s a good thing. Even though I didn’t mean it this way, the workshops turned out to be my gift to my colleagues before I left.
• It was also pointed out the using the image of a house helped to pull together some loose ends. Talking about the point of a house (it’s where people live in) nicely summed up the reason we ever develop something – there is a purpose for some human beings.
• Would I change anything next time? Yes. I would probably pick a piece of software or prepare it myself (you know, learning challenges for me too…). I’m not sure I’d pick a different one for each exercise or not but I’d consider it.

 

Am I happy with my first attempt at putting together a workshop? Yes, quite 🙂

 

Cheers!