Code Coverage Tools in C#: Your Guide to Choosing - NDepend Blog (2024)

January 11, 2024 9 minutes read

In this post, we’ll go through 8 different .NET code coverage tools and list their features to help you make a decision.

But first, I want to offer a quick disclaimer and warning.

If you’re here because you’re looking for a way to let management track the team’s code coverage progress, please stop and reconsider your actions.

I’ve written in the past about how code coverage should not be a management concern, and that holds as true now as ever. Code coverage is a tool that developers should use — not something that should be donetothem. Full stop.

With that out of the way, let’s get back to helpingdeveloperspick a tool to keep an eye on code coverage.

Introduction

What Is Code Coverage and Why Do It?

Okay, with that out of the way, let’s talk extremely briefly about what code coverage is.

And take note that this is going to be averysimple explanation, glossing over the fact that there are actually (slightly) different ways to measure coverage. But the short form is this. Code coverage measurements tell you which lines of code your test suiteexecutes and which lines it doesn’t.

Let’s look at the simplest imaginable example. Here’s a pretty dubious implementation of division:

Let’s say this was the only method in our codebase. Let’s also say that we wrote a single unit test, from which we invoked Divide(2, 1) and asserted that we should get back 2.

We would then have 67% code coverage. Why?

Well, this test would cause the runtime to test the conditional and then to execute the return x/y statement, thus executing two-thirds of the method. Absent any other tests, no automated test ever executes that last line.

So in the grossest of terms, that’s the “what.” As for the “why,” developers can use code coverage analysis to see holes in their testing efforts. For instance, code coverage analysis would tell us here, “Hey, you need to test the case where you pass a 0 to the method for y.”

What Are Code Coverage Tools?

What, then, are code coverage tools?

Well, as you can imagine, computing code coverage the way I just did would get pretty labor-intensive.

So developers have done what developers do. They’ve automated code coverage detection.

In just about any language and tech stack imaginable, you have ways to detect how thoroughly the unit test suite covers your codebase. Don’t confuse code coverage with an assessment of the quality of your tests, though. It’s just a measure of the amount of code exercised by tests. It is not correlated with the amount of output actually asserted within test code.

The result is an array of tools to choose from that help you see how well your test suite covers your codebase. And that can become somewhat confusing. So today, I’ll take you through some of your options in the .NET ecosystem.

Let’s take a look at those options.

Visual Studio Code Coverage

Before you gear up to start downloading things or breaking out your wallet, understand that you mayalreadyhave this capability. It depends on your version of Visual Studio. If you have theenterprise version, you can take advantage of this capability out of the box.

And it’s as simple as this screenshot.

You can read about this in more detail at the Microsoft documentation site. But this is a nice, comprehensive option that requires very little additional work, assuming you have the correct version. And it has cool features:

• It reports coverage percentages at various granularities (e.g., assembly, class, etc.).
• You can select all of your tests or subsets of them.
• You can have it paint your IDE, meaning it lets you actually visualize the coverage as you look at your code.
• It comes from Microsoft, so you can expect plenty of maintenance and support if you invest in its usage.

Rider’s and ReSharper’s dotCover

If you have a Visual Studio Enterprise license, your life is good in a lot of ways. But if you don’t, shelling out $3,000 per year is probably going to be a bit of a deal-breaker just to see code coverage.

And itshould be a deal breaker. Visual Studio Enterprise is an AWESOME tool, but if the only reason you’re buying it is to see code coverage, you have significantly cheaper options.

Take JetBrains dotCover, for instance. This comes with Jetbrains offers, which costs only $349 to $779 per year. So you get coverage analysis and you also get Rider and ReSharper, which are really cool tools.

With dotCover, you get plenty of features as well. Some highlights include

• Detailed reporting.
• Test runners. Tests can be run also with the JetBrains dotTrace profiler to measure performance.
• Both Visual Studio and CI integration of coverage measurements.
• Navigate to covering tests.
• A cool “hotspots” view that calls out risky methods.
• IDE green/red statements painting.

NCrunch

I’m going to switch gears a little now, having given treatment to some major players in the .NET code coverage tools space. So far, I’ve listed what I think of as “traditional” code coverage tools.

NCrunch is different.

I mean, itis a code coverage tool, and itwill give you code coverage data. It will also give you the IDE painting that some of the other tools give you, as well. (If you look at the screenshot of my environment above, you’ll see red and green dots — those are NCrunch telling me that tests are failing and passing, respectively).

But it gives you a killer feature as well.

NCrunch constantly runs your tests, in real-time, as you type. This is Live Unit Testing. With NCrunch, you don’t need to run your tests, or even compile, to get feedback on whether your code changes are breaking anything. The dots will changeas you type. Below the green passing-test indicators along the left side of the IDE show the results of live unit testing:

Note that the NCrunch pricing is very reasonable, starting at $159 to $289 per seat for an individual developer. Of course, NCrunch has other features besides that killer one, as well:

• Small and customizable memory footprint.
• Inline exception details and easy VS-integrated debugging.
• Mature product that has been in development for 15 years.
• Safe multi-process parallel execution of tests, able to scale out over a grid of remote machines for more capacity.
• Passive collection of both inline code coverage and performance metrics.
• Intelligent execution able to target only the tests that are impacted by changes.
• Very fast wall times on test execution and able to work with very large test suites.

AltCover (formerly OpenCover)

For a few years now, the free and OSS tool AltCover replaced the well-known OpenCover tool in maintenance mode.

AltCover is a .NET code coverage tool that distinguishes itself through its open-source nature and focuses on simplicity. Developed in F# by Steve Gilham, AltCover is designed to be a lightweight and easy-to-use alternative for developers seeking a straightforward solution for code coverage analysis. It features:

1. A command-line tool designed to capture .NET code coverage. The .NET Core, .NET Framework and Mono runtimes are supported.
2. It seamlessly integrates into MSBuild tasks, offering robust support for driving the tool, including seamless integration with dotnet test functionality.
3. AltCover provides a versatile API that encompasses its core functionality. This API is extendable and integrates smoothly with popular build automation tools like Fake and Cake.
4. AltCover extends its compatibility to PowerShell environments by offering a dedicated module compatible with both PowerShell 5.1 and PowerShell Core 6+. This module includes a cmdlet for executing the AltCover tool and additional cmdlets for manipulating coverage reports.
5. AltCover includes a specialized coverage visualizer tool, based on the cross-platform AvaloniaUI toolkit.

Coverlet

Coverlet is a versatile code coverage framework designed for .NET, offering support for line, branch, and method coverage. It is compatible with .NET Framework on Windows and .NET Core across all supported platforms. It features:

• Lightweight integration: Coverlet can be easily integrated into your .NET projects without imposing a heavy burden on your system.
• Free and Open-source: Coverlet is an open-source framework, allowing for community contributions and ongoing development.
• Flexible configuration: Users can customize and configure Coverlet to suit their specific code coverage requirements and preferences.
• Robust analysis: Coverlet delivers reliable and detailed insights into your code, aiding in the identification of areas that require testing and improvement.

Let’s mention that if you are developing with Visual Studio Code, Coverlet is a great tool as explained here.

Fine Code Coverage

Fine Code Coverage is new in town. It is a free Visual Studio extension designed to provide coverage data within the IDE. It works with all Visual Studio editions including the Community one.

With this extension, you’ll see which line is covered or not within the Visual Studio editor.

Fine Code Coverage includes a dedicated Windows pane. This pane features tabs that offer valuable insights into the extent of code coverage, providing detailed information on the covered lines of code.

(Fine Coverage screenshots obtained from this Matthew Regis’ blog post).

NCover (put in maintenance mode)

So far, we’ve looked at a couple of code coverage tools that come bundled with other products. So, while they integrate nicely into your development environment, they aren’t exclusively focused on the subject of code coverage.

NCover, however,issolely focused on that. By specializing, NCover provides extremely detailed information, not only supporting measurements of coveragebut also integrating it for the whole team. It tracks trends in coverage and provides detailed reporting.

NCover is a comprehensive tool for this purpose. Some features include

• Detailed and centralized data about coverage.
• Extensive documentation and user support.
• 32- and 64-bit support, plus memory consumption optimization.
• IDE painting.

Unfortunately by now (2023 / 2024) NCover is in Maintenance mode so don’t expect any new features.

Beyond Raw Coverage Data with NDepend

I’ll close out (not surprisingly, on the NDepend blog) by mentioning NDepend.

While NDepend doesn’t directly measure code coverage, it happily imports the data from other coverage toolsand lets you do somereally powerful things with it.

NDepend gives you tons of metrics out of the box, and when you combine those with the imported coverage data, you can do important assessments of your codebase, such as assessing method risk. You can also make use of NDepend’s heat map capabilities to get powerful visuals of where you have good coverage and where you don’t.

Ruling Code Coverage

But my favorite part, personally, is how you can integrate this data with CQLinq to create your own rules and measurements as they relate to coverage. Many rules are provided by default like:

• Code should be tested: This rule pinpoints untested code. It also offers a smart estimation of the effort to write tests to cover untested code.
• New Types and Methods should be tested: This rule spots new but untested code added since the baseline. Clearly, new classes should deserve special testing focus. A similar rule exists for untested refactored code.
• Methods should have a low C.R.A.P score: This rule computes a Change Risk Analyzer and Predictor (CRAP) index. This metric helps spot code that is both complex and untested

A Lightweight Report with All Coverage Data

The NDepend report also comes with a comprehensive overview of the coverage of your application. You can easily browse code coverage by source files.

For each source file, you can dig into which statement is covered or not:

The sky is the limit when it comes to incorporating test coverage data with other metrics about your code. Some heat-map can be generated for you as well:

And that brings us back around to my initial warning in the beginning.

Code coverage isn’t something management should use to take developers to task. Rather, it’s something that developers should use to make sure they’re satisfied with their codebases. Use one or more of these tools to measure your coverage, and then use NDepend to create seriously actionable information based on coverage.

Then you’ll have the sort of clean, reliable codebase that management doesn’t ever worry about.

Bonus: Our approach to Coverage at NDepend

Edit: 11th of January 2024: Patrick Smacchia CTO at NDepend added this section.

Which tool do we use to obtain coverage data?

We have used DotCover for many years. Here are the reasons why:

• The DotCover test runners and test profiler are great tools. They are fast enough which is essential given the size of our code base (roughly 5.000 classes).
• It is easy to integrate into our CI/CD (although a bit slow).
• It is easy to gather coverage files that NDepend can use to generate reports. We dogfood a lot, more in the next section.
• It is more polished than the free alternatives presented in this post. Also, it is cheaper than the Visual Studio’s paid versions.
• NCrunch is a fantastic and mature tool. I know many developers who are addicted to its lightning-fast Live Unit Testing feature. But to achieve such performance NCrunch requires certain constraints on the tests. As a result, to adapt the NDepend test suite (15.000+ tests) we would have to make many changes. This proved to be impracticable, unfortunately.

How do we use coverage?

Now that you have a good idea of the paid and free .NET coverage tools let’s explain how we approach testing and coverage at NDepend. For us coverage is an essential development tool that we continuously use. It is one of our 4x Lines of Defense Toward Bug-Free Code.

The heatmap shown in the screenshot above helps a lot in pinpointing which code needs more tests. But our preferred usage of coverage is the actionable rule: Types that used to be 100% covered by tests should still be 100% covered. We have thousands of classes 100% covered. We believe in 100% coverage for many reasons:

• When refactoring a class that initially boasted 100% test coverage, any discovered gaps in coverage are treated with utmost seriousness. In many instances, these gaps unveil regression bugs. At the very least, highlight scenarios that were not anticipated during the refactoring process.
• When a class achieves 95%+ coverage, the presence of the remaining 5% of hard-to-test code suggests design flaws, since design and testability work hand in hand. Thus, these gaps are more susceptible to errors. In such cases, it is often the code that merits thorough testing.
• If a bug is discovered in code 100% tested, it is often easy to reproduce with new tests. The new test is a much better approach than manually starting the application, do this and do that to finally reproduce the bug. Also, once the bug is fixed, tests to reproduce it are integrated into the test suite to make sure the bug won’t happen again.
• 100% coverage is part of our definition of done. It is an objective fact that can be verified with the coverage rules aforementioned.
• Also, we use a [FullCoveradeAttribute] and regularly check that the classes it tags are 100% covered. This way the full coverage intent is explicit in the class source code.

Conclusion

Coverage is not enough. It is easy to write an integration test that exercises hundreds of classes without actually checking any results. It works for us because our code contains thousands of assertions that are challenged more than 200 million times when running our 15K+ tests. Assertions are not just for test code, application code should contain many as well.

There are debates around 100% Code Coverage. Not all classes deserve 100% coverage, especially the ones bound to hard-to-test concerns like UI or network. Not all development shops agree to spend resources on achieving 100% coverage and they do well. I genuinely explained what we do and why we do it this way.

Now it’s your turn to decide which coverage tool to use and how you will make the most of your coverage data!