You sense a problem with your software testing. But you don’t really know if the problem is the people, the skills, the process, or the practices. Use the questions below to get a high-level view of problem areas so that you point your improvement dollars and time in the right direction.
Software testing is rarely the first place anyone looks to start improving the development organization. And there is wisdom in ensuring your project and development practices are in good order to avoid problems getting through to testing. But a review of the quality assurance (QA) side of the house cannot be ignored.
There are many symptoms that may be causing you to believe that there is a problem with your testing:
- The final product delivered to customers is not meeting customer needs or is delivered with bugs.
- The process and product exposed to customers during the project has too many bugs and is impacting customer confidence.
- The internal work is taking too long or costing too much.
- Testing estimates never seem to be accurate.
- Your testing people are complaining that they don’t know what the product is supposed to do and they can’t test properly.
- Your development people are complaining that the testers report bugs that aren’t bugs.
- Your project managers set the testing estimates or immediately cut back on the estimates provided by the testers.
You don’t need to be in trouble to justify a closer look at the QA process. Even if you haven’t seen these symptoms, perhaps you are growing your business and expanding the kinds of projects and products you take on. You wonder if your testing people and practices are sufficient to grow with you. Perhaps you’ve already spent time improving your development processes or your project management processes and are surprised you are seeing some symptoms of things still not going well.
A Good Assessment Takes a Holistic View
Test your testing using the following perspectives:
- Test Environment: Process, Practice, Tools
- QA/Testing Staff: Skills, Behaviors, Attitudes
- Team and SDLC environment: Relationship to other roles on teams, fit and integration of testing into the project, and development processes
This first pass assessment is not meant to be a scientific study. Its most important part is the interviews you hold and the questions you ask. Reviewing current templates and deliverables and tools is helpful, but be careful not to fall so far down in the details you miss gathering that holistic view.
Interview a representative set of your QA/testing folks, at least a couple of the lead developers, a couple project managers and business analysts. The assessment questions below are mainly written for the testing people. For the other roles, instead of asking, “When do you...” ask, “When does the tester...” Not only are you gaining information on the three perspectives above, you’ll discover if there are differences in understanding across your key development roles.
The Test Environment
Answers to these questions give you a good sense of the current processes and practices. They often also highlight skills issues or issues with team dynamics. To assess the results of these questions, you do need someone with good QA/Test knowledge who can determine the most appropriate practices for your organization (current and future) and identify any gaps in your current state.
- How do you create your estimates? What approaches or techniques do you use?
- How do you use requirements in your testing process? Are test cases created from requirements? Is this working well? Why or why not?
- How do you determine what to test and what not to test? Do you do risk-based testing? How do you determine risk?
- What kinds of testing do you do? Probe to see if destructive testing is part of functional testing. Find out what kinds if structural testing is done (e.g. performance, stress, concurrency). Find out if security testing, regression testing and integration testing is done.
- How do you decide what type of testing is needed?
- How do you communicate your test approach, plan, or strategy to the rest of the team?
- What tool(s) do you use for your testing?
- For capturing test cases and scripts?
- For running or re-running tests?
- For identifying and documenting bugs and other issues?
- What kinds of classification do you use for bugs and issues, and why?
- How do you communicate issues back to the developers? What do you use for that communication? How do you ensure they get all the messages (urgency, priority, etc.)?
- What kind of templates are available to you (e.g. test strategy/test approach/test plan, standard reports)?
- Do you meet with the other testers to discuss lessons learned, or share practices or ideas across projects? If so, is that knowledge captured anywhere?
- If there was one testing practice you would like to improve or add, what would it be? Why?
An organization can and should support someone’s professional development. But no organization can make someone into a professional practitioner unless the aptitude and desire are there.
These questions provide the current skill baseline and help you determine motivation. I have recommended letting someone go on the basis of these questions alone. If the current skills are very low and the motivation to learn or ability to learn are also low, the person is not a good fit for the QA/Test career. Keeping them demotivates the rest of the team and ultimately stops that person from going out and finding what is a good fit for them.
- What resources do you use (either provided by the organization or found on your own) for learning about testing?
- How would you rate your skill level in the following areas:
- Assessing scope of testing
- Assessing requirements testability
- Assessing the technology needed to execute testing
- Communicating test results
- Are you involved in any external/industry QA and testing organizations or associations?
- What kind of process/practice/skill support would you like to get?
- Where do you see yourself in 1 year? In 3 years?
The Team & SDLC Environment
It is highly unlikely that your testing issues are isolated to test practices and testing skills. Effective development is the result of a well-choreographed dance with all the roles involved. Everyone needs to know his part and enough of each others’ parts to not trip over one another. And the choreography is a properly integrated view of all the activities, tasks, and deliverables across the roles. The best dance teams respect one another, support one another, and focus on the teams shared goals. The best development teams are no different.
- Do you feel the time allocated for your job is too little, too much, about right? Why?
- When do you typically get involved in a project? Are you satisfied with the timing? Why or why not?
- How do you understand what a project is about and get a handle on what you will be testing? Ideally, what kind of information would you like to see and when? Are you getting the project detail you need?
- How are you involved in identification and estimation of your testing tasks for a project?
- When do you see the requirements? Is the timing working or not?
- Are your QA/testing activities a standard part of project plans?
- How are you kept up to date on changing requirements? What about other design or significant code changes?
- When and how do you raise bugs, issues, and concerns to the project manager for tracking as a project issue?
- Are you involved in daily or weekly team meetings? How? What works, what doesn’t?
- Are you involved in project debriefs or project closure meetings? What testing or QA concerns have you raised in the past? Were they addressed?
- Describe your typical relationships with project teams. Are you satisfied with the relationships? If there was one thing you could change, what would it be? What works and what doesn’t?
- Describe your relationship with each of the roles on the project teams. Are you satisfied? Anything you would change?
Along with all the questions above are two key questions to ask everyone you interview. I suggest you ask these questions last. The first one helps you sort through all the information the person gave you to find the key issue. The second ensures future changes don’t undo something that ought not be undone, and it ends the interview on a positive note.
- What do you see as the biggest QA/testing issue? Why?
- What do you see as something that really works well and needs to be leveraged going forward?
OK, Now What?
The goal of this assessment is to develop a roadmap to follow.
As with any assessment, you will end up with an understanding of the current state and a list of problems or gaps that must be addressed to improve performance. Some of those will not be directly related to testing. For example, your questions may uncover that the real issue is poorly documented or communicated requirements, or that the project managers don’t involve testers early enough for them to properly plan their work, or that project initiation activities don’t include any assessment of the nature of the project and what kind of testing should even be included in the client contract or project plan. Or even that the individual roles on the projects are simply not communicating with each other.
The next step is to go back to the three initial perspectives and do a quick assessment on the impact of the issue, and look at solution options including the time and effort required. From that you’ll have a list of things you can do in the short-term, the mid-term, and the long-term. You may also have a list of a few areas where more detailed problem analysis is required. That work should also be included in the roadmap.
Now you know the kind of questions that should be asked and can imagine the kinds of issues that could be surfaced.
Here’s the list of questions I ask of the leaders I work with so that I have a good sense of their thinking and can have good discussions with them on the interim and end findings and the final roadmap. You want the recommendations and actions to make sense for your environment and your budget. You can be a step ahead and provide this information to whoever helps you get started.
- How would you define your problem or concern? What specific symptoms are you seeing?
- What triggered you to look at this problem now?
- What do you think may be the underlying issue(s)?
- What does the end or target state look like for you? Is it simply the opposite of the problem or something more?
- What is it worth to you to get rid of the problems and achieve the target state? What are you willing to spend or invest?
- What are you willing to do or not do? If necessary, are you willing to change processes, how teams work together, implement new practices, rework performance management, etc.?
Look at your in-house skills for interviewing and for QA/test knowledge. Decide whether to hire out the whole assessment and roadmap or whether you can do it all (or in part) in-house. For a smaller development shop, say 25 people overall, expect to do 4-6 interviews and have work take about 35-45 hours from start to roadmap. Larger shops require more interviews to get a representative set or may include interviews and some surveys. Typically the roadmap is also more complex. For any organization, though, this should be kept as a light and agile approach meant to get a high-level view that inevitably provides some obvious improvement areas that can be addressed very quickly.
The End Result
You’ve sensed there is a problem with your testing. You will now know if the problem is the people, the skills, the process or the practices. Your improvement time and dollars are pointed in the right direction balanced across short to long-term scale so that benefits start to accrue as soon as possible.