How does Google achieve test coverage of 1 billion lines of code?

The line number of the colored logo (highlighted with a red rectangle) is used to visually view the code coverage. If the line number is green, it is overwritten by the test;  If it is orange, it is not overwritten. Or if it is white that it is not detected.

In software development, a common metric called code coverage Standard. It is used to measure the percentage of system code covered by the tests performed before deploying the code. Code coverage is usually measured automatically by a separate software program or some code coverage tools can be manually launched from the command line. The test results accurately show what code was executed when the test suite was run. And can reveal which lines of code may require further testing.

Ideally, the goal of the software development team is 100% Code coverage. However , this rarely happens in reality because some code blocks go through many different execution paths and  various edge cases. These edge cases should (or should not be considered) based on system requirements.

Metric code coverage has become a common practice for software development and test teams but whether this practice really improves the quality of the code is still controversial.

Some people think that developers may focus on quantity but not quality. And create tests just to satisfy code coverage. Instead of being robust enough to identify high-risk or critical code areas. Others are concerned about its cost-effectiveness – it requires valuable developer time to review the results and does not necessarily improve the quality of the test.

For large organizations, like Google (with 10 Measuring billions of lines that receiving tens of thousands of submissions per day and supporting seven programming languages) It is especially difficult to measure code coverage.

Google's AI researcher Marko Ivanković and Goran A recent study by Petrović provides a little-known case of the technology giant's code coverage infrastructure, which consists of four core layers. The bottom layer combines a code coverage library designed for each programming language, and the middle layer integrates the code coverage automation process into the company's development and Built in the workflow. The top level uses a code editor and other custom tools to visualize the code coverage information.

As part of this study, Ivanković and Petrović analyzed the adoption rate of code coverage in five years. They found that although Google did not mandate code coverage, adoption rates have been growing steadily since 2014. In the first quarter of the year in 2018, more than 90% of projects used automated code coverage tools.

The researchers also collected 512 questionnaires on the usefulness of code coverage from 3,000 randomly selected Google developers and other non-engineers. Among the respondents, only 45% of people use code coverage testing frequently when code changes and 40% use code coverage regularly when reviewing code.

Google's survey participants rated the usefulness of code coverage when code modification (red), checking code changes (blue), and browsing code changes (green).

Ivanković to IEEE Spectrum Speaking of their research and the role of code coverage in software development and testing. (For clarity, the following interview records have been edited and condensed.)

IEEE Spectrum: Why do you think code coverage is most important?

Marko Ivanković: Many people may expect us to say, "Good coverage can reduce bugs. quantity. "This is certainly one of the reasons, but as we found before that even more surprising is that even if coverage is not directly a quality signal, it is still worth calculating."

Coverage may not be directly helpful to the person viewing the code but it is still helpful for the tool . For example, analysis A tool for relationship. For example, if code A declares that it depends on code B but the test for code A can never touch the code B, then this dependency may not be real and automated tools can try to remove it to simplify the code base.

Of course, the actual implementation is much more complicated. We have discovered many of these tools that can use their coverage information provided by our infrastructure to improve their own functionality. For many of these use cases, the correlation between code coverage and code quality is not important at all.

IEEE Spectrum: What inspired you to research Google's code coverage?

Ivanković: Because we are facing a problem. During the code evaluation, we spent time figuring out that the test really tested the code for these evaluations. At that time, the code concept The system supports coverage calculations, but you must manually activate it and manually overlay the coverage results against the code you are reviewing. One day, we said to ourselves, “We must find an automated method.” A week later, we had the first prototype that could be run. After other engineers saw it Ask if they can have such a prototype. We want to make sure we provide them with the best experience, so we started to study this issue.

IEEE Spectrum: What is your most surprised part in your research?

Ivanković: We were amazed at the fact that the code coverage method was initially suspected, but it ended up being very useful. Some of the people we surveyed were basically against coverage but they still admitted that they sometimes used the coverage method and found it useful.

IEEE Spectrum: What is the biggest challenge that you face in research? How did you overcome it?

Ivanković: On the surface, code coverage seems to be a simple concept: whether a line of code is covered by a test. But it turns out that when it is implemented on a large scale. It is full of various corners and unexpected situations. It took us a few years to fix all of the tested infrastructure Lost situation.

We encountered similar challenges when conducting research. Most of the engineers we surveyed had the same overall concept of coverage. However, when asked about the details, their responses were quite different. We must conduct a few surveys on a small group of people in order to get the correct answer.

IEEE Spectrum: What are the advantages that do you think Google's code covers infrastructure? Do you have any suggestions to improved it?

Ivankovic: We work hard to ensure that infrastructure resources are effectively utilized and can run on, such a large code size as Google. Showing people that it is work, that maybe the biggest contribution of our research.

We design our infrastructure to make experiments easier and do A/B Test and evaluate hypotheses. We also update all the data in an accessible format so that coverage information can be visualized, which helps the team maintain healthy code and is ready to fix the incident.

When we were investigating engineers, some of them gave us suggestions for improvement, some of which were built. It is worth exploring. One of the more interesting methods is that if the code coverage test results are too good. Don't show them, so the engineers won't be overconfident.

IEEE Spectrum: Do you have any suggestions for software development and test teams who want to deploy code coverage or improve existing code coverage?

Ivanković: I think the most important suggestion that I can give is to focus on their workflow. Don't just deploy coverage testing, also make sure that you integrate it into the correct location of the developer workflow and only at this location and coverage results are the most useful. In our experience, code evaluation is the cornerstone of code health.

IEEE Spectrum: What will happen to the future of Google's code coverage infrastructure?

Ivankovic: At the moment, we are taking a closer look at the data usage and developer opinions and the brakes better understand how to use coverage. For example, we are researching that the difference between usefulness and actual usefulness of people. “Does the display coverage actually speed up the evaluation process during the code evaluation?” The results of this study will determine how our next infrastructure will improve.

The above article is from PASSHOT, PASSHOT can help you pass CCIE WRITTEN EXAM and CCIE R&S EXAM once and help you get your CCIE certificate. If you need to reprint, please indicate the source!