Trends & Insights into Dev Collaboration
Learn More About Collaborator
Code review remains the number one method for improving code quality. Yet, different organizations ─ or different teams within organizations ─ vary in their methods for implementing code review. In the second section of the State of Code Review 2017 report, we look at how teams are using code review and the challenges that can often get in the way.
“Over the shoulder” code review and meeting-based code review climbed by 3 and 7 percentage points, respectively, over last year. Yet, for the most part, both are performed only monthly or less and lag far behind tool-based code review in daily or weekly reviews.
Ninety percent of respondents say that code quality is the best reason for doing code review.
Beyond the quality of code, reviews are also viewed as a primary method for improving the ways teams collaborate, including team-wide knowledge sharing, mentoring less experienced developers, and increasing the frequency of collaboration.
Ad-hoc, or “over the shoulder” code review is the most commonly used type of code review within organizations, with three-quarters of respondents participating in ad-hoc reviews throughout the year.
The majority of respondents who are doing ad-hoc code review do it at least once per month, with a majority of respondents conducting ad-hoc code review on a weekly basis.
57% of respondents are doing meeting-based code review; 44% are doing it at least once a month.
Meeting-based code review is the least commonly used method for code review, with a little more than half of organizations participating in meeting-based reviews.
Six out of every ten individuals surveyed say that their organization are using a tool to help with code review. The 60% number represents a 3-point drop from last year, which we attribute to a larger number of this year’s survey respondents coming from smaller development teams, specifically those with 10 or fewer members. Such teams find it easy and convenient to perform ad-hoc reviews, as they are often seated together in a single physical location. For many small teams, combining a collaboration tool (e.g. Slack) with ad-hoc reviews is sufficient.
Nevertheless, organizations that are doing tool-based code review are doing it on a more consistent basis, with 49% doing it at least weekly versus 45% in last year’s survey. It is also evident from the survey data that having a tool enables more frequent code review. Twenty-one percent of respondents perform tool-based code reviews daily, which is 30% higher (or more) when compared with other review methods.
While teams understand the importance of code review, they often face obstacles when it comes to implementing it into their team or organization. Workload is the number one obstacle facing their teams when it comes to doing code review. However, only 55% of respondents cited workload as an obstacle this year ─ a significant drop from last year’s 66%. This is good news for development teams. It demonstrates how companies have stepped up hiring for their software development teams and are lowering the work burden placed on team members.
Similarly, time restraints and a lack of manpower are also looked at as primary roadblocks as teams struggle to allocate the necessary resources to conduct code reviews. Yet, more positive developments were revealed around manpower shortages in this year’s survey. The surge in hiring has led to a drop of nearly 15% in those citing a lack of manpower from 2016 to 2017.
The data from sections 1 and 2 reveal that most organizations can benefit from using a tool to assist with code review. There are several business drivers that determine the need for a code review tool, and choosing the right tool will depend on your specific software team’s needs. In section 3 of the State of Code Review report, we take a closer look at how teams are choosing code review tools and the factors that go into selecting a tool.
An overwhelming majority of respondents say that “improving code quality” is the number one business driver that determines their need for a code review tool. In fact, improved code quality outpaced the #2 driver ─ “internal audits” ─ by nearly three times. This is a significant jump from the 69% of respondents who cited code quality as the biggest business driver for using a tool and further underscores the widespread acceptance of code quality tools.
Rounding out the top four business drivers for using a tool were “external audits” and “finding bugs and product outages.”
Among companies with fewer than 11 developers, finding bugs and product outages took the #2 spot, while internal audits were the #2 driver among development teams with 11 or more members.
The State of Code Review 2017 report is an industry-wide survey that was not limited to SmartBear customers. We are proud to see that Collaborator, our code review tool, was the most commonly used tool among organizations that are doing tool-assisted code review with a focused tool.
For those organizations that do not need best-of-breed code review functionality, they can now look within their TFS toolset for baseline code review features. This expanded
use within TFS is the main driver behind the tool’s code review usage climbing from 17.6% last year, to 24% this year.
In addition, we also found that 42% of teams are not using a tool for code reviews.
These findings reflect the fact that larger teams are performing more regular audits with a formal review process while smaller teams perform more ad-hoc reviews via informal collaboration.
With approximately 40% of respondents not currently using a tool for code review, we wanted to better understand the factors that keep teams from adopting a code review tool. Beyond the common “workload constraints” answer, we delved deeper into the other obstacles that people found noteworthy. Respondents cited lack of budget (33%), management buy-in (25.5%), and manpower (24.7%) as the top blockers that prevent teams from adopting code review tools.
The most commonly reviewed artifact type is requirements documents (65.5%), followed closely by test cases (60%). User stories (53.3%), documentation (49.8%), and Design documents (48.4%) round out the top 5 most-reviewed artifact types
While just 35% of respondents use a tool to review their artifacts, only 4.9% stated that they are in searching for a tool to use for artifact review.
This survey was designed to establish benchmarks for the software industry concerning the ways in which organizations are developing and maintaining the quality of software in 2017. The structure of this year’s report closely mirrors the 2016 report structure so that we may provide meaningful year-over-year comparisons and comment on significant trends around code quality. This report covers the following topics:
To ensure the 2017 report data is relevant SmartBear Software conducted a global online survey over the course of four weeks during the months of December 2016 and January 2017. The findings presented are based upon the aggregated responses from more than 550 software developers, testers, IT/operations professionals, and business leaders representing more than 30 different industries. Participants in the survey work at companies of all sizes, from fewer than 25 employees to over 10,000. Similarly, their software teams range in size from fewer than 5 to more than 50 team members.
With software code driving competitive advantage in numerous products and processes across industries, organizations of all types and sizes view code quality as a top priority. But, are these organizations meeting the high code standards they set for themselves? How are they ensuring that they maintain the quality of their software? In the first section of the State of Code Review 2017 report, we look at how organizations are maintaining code quality in 2017.
Overall, we found that respondents felt positively about the quality of the software they help build (64.9%). Of those who didn’t either agree or strongly agree, just 13% said they felt negatively about the software they help develop. The 13% figure represents a 30% increase over the 10% who disagreed or strongly disagreed in 2016. Based on our observations across industries, we see several potential factors that may give rise to the higher rate of dissatisfaction. These include new regulations in certain industries, increasing pressure on teams to ship new code faster, or breakdowns in the development and QA processes.
The overall positive sentiment was felt across industries and roles within software teams. One area where negative sentiment was higher was within software teams of 50 people or more.
Two-thirds of respondents say they are regularly able to get releases out on time.
Development teams with 6-10 members were the most positive about their ability to get releases out on time, with 76.6% reporting that they either agree or strongly agree. That’s a change from 2016, when teams with 21-50 members were the most positive (78.2%). That category dropped to just 55.4% in 2017 and drove the increasing rates of dissatisfaction.
When asked what the number one way to improve code quality is, 51.9% of respondents said that code review is the best way. That is nearly double the 27.4% of respondents favoring code review last year. We attribute most of the jump to increased spend on software quality. The growth in popularity of repository management solutions has provided more teams with basic tool based code review functionality.
Another dramatic increase occurred in relation to unit testing, which 51.2% of respondents cited as the best way to improve code quality in 2017, up from just 18.7% in
2016. This change reflects the overall shift in awareness and focus around quality which we saw increase greatly the last 12 months. Development teams and management
recognized that code quality needed to improve. With the increased focus on continuous delivery and deployment, unit testing has become more critical due to condensed delivery time lines. The automated testing process allows teams to receive rapid feedback on the success or failure of changes before they hit production. Overall, testing is considered a critical method for improving code quality ─ 51.2% say unit testing, 39.8% say functional testing, and 37.8% say integration testing are primary ways to improve code quality.
The State of Code Review 2017 report includes responses from software developers, testers, IT/operations professionals, and business leaders representing more than 30 different industries. Participants in the survey work on software teams of all sizes, from less than 5 employees, to up to more than 51 employees. They also work for companies ranging from small businesses, with less than 25 employees, to enterprise organizations, with 10,000 employees or more.
Looking at how teams are distributed, we found that a majority of them are now working across multiple locations. Of those multi-location teams, a majority are collaborating across two or more countries.
Just 15% of organizations have teams that work in more than one US location.
The State of Code Review 2017 report includes responses from software professionals working in small, medium, and enterprise organizations. Where last year 45% of survey respondents worked at companies with more than 10,00 employees, this year’s pool of respondents had 23% from such large companies. By having all company sizes reflected in a more balanced survey pool, this year’s data provides a more broadly representative data set.
Respondents in the State of Code Review 2017 survey work primarily on smaller development teams — with nearly 3 in 10 on teams having fewer than 5 employees. As companies pursue more agile approaches to software development, team sizes are shrinking (even at large companies).1 This explains why only 17% of respondents work on development teams of 21 or more members.
1 Schwaber, Ken and Sutherland, Jeff (2014). “The Scrum Guide,” Scrum.org and Scruminc. Copyright 2014. Referenced online April 5, 2017 from the following source: http://www.scrumguides.org/docs/scrumguide/v1/Scrum-Guide-US.pdf
Computer software and financial services are the industries with the largest representation in the State of Code Review 2017 report, followed by consulting/systems integration, retail/ecommerce, and healthcare.
Development and QA combined for 79% of the roles represented in the State of Code Review 2017 report.
Try Collaborator For Free
Code review can be counted among numerous critical processes for developing and maintaining software quality. As such, software teams rely upon an array of tools and systems during software development. In section 4 of the State of Code Review 2017 report, we examine several trends within the software industry related to the adoption of software tools and systems.
Fifty-seven percent of respondents cited Git as their choice for an SCM tool, which represents a 14-point increase over 2016. We attribute the increase to GitHub staffing up and making a significant sales push. Strong growth was also experienced by BitBucket and GitLab. Overall, growth in software configuration management tool purchases was significant, as evidenced by GitHub’s annual recurring revenue from large clients increasing from $25 million to $70 million at one point in 2016.
Almost one-third of development teams are using Subversion, and nearly one quarter are using Microsoft TFS.
26% of respondents are not using any of the well-known repository management tools. Of those who are using a repository management tool, more are using GitHub than any other tool. Users cited GitHub at a 3-4 times higher rate than the second-most cited tool, Bitbucket Server.
Of those respondents that are planning on introducing a repository management system, a majority are planning on implementing GitHub or GitHub Enterprise.
A vast majority of respondents are using an integrated development environment (92%) — with half using Visual Studio and nearly half using Eclipse. The use of these tools is definitively on the rise, as Visual Studio (up 26%) and Eclipse (up 125%) rose far above last year’s numbers. More than 1 in 5 respondents are using IntelliJ.
JIRA outpaced all other bug tracking tools by at least a 4:1 margin. JIRA’s numbers jumped 16 percentage points from 2016 to 2017, and we attribute this to Atlassian going public in late 2015, providing the company with the resources to perform marketing, sales, and service at greater scale.
For organizations that aren’t using JIRA, there is a lot of variation in how they are tracking bugs, with no clear runner up to JIRA.
Similar to what we saw with bug tracking tools, JIRA dominates and saw a percentage jump of 16 points between the 2016 and 2017 surveys. There is also a good amount of variation in the tools organizations are using for requirements management. While JIRA is being used by half of organizations, there were no other clear standout tools for requirements management.
31% of respondents cited a tool other than the 8 most popular tools on the market for requirements management.
Table of Contents