Per Erik Strandberg /cv /kurser /blog

Background

This is part three in my series on Testing In Visual Studio. Having worked with testing in other environments for some years now I really wanted to dive deep into the Visual Studio way of testing. In Testing In Visual Studio Part 1 we took a good look at static code analysis, unit testing and code coverage. In Testing In Visual Studio Part 2 we looked at coded user interface tests and also data-driven coded user interface tests.

Today we will take a closer look at Continuous Integration in Visual Studio - or rather Team Foundation Server (that can interface to Visual Studio) - and discover some really awesome features that will help your agile team produce better quality while having more fun.

What is Continuous Integration?

Before we look at how Visual Studio and Team Foundation Server deals with continuous integration we need to understand a bit more about the concept of continuous integration.

The short explanation

Continuous integration is a practice to - in an automated way - compile, run tests and provide feedback on every code change.

The long explanation

In the extreme programming community circa 1999-2000 Martin Fowler and Kent Beck proposed the concept of continuous integration. On his homepage (see [1]) Martin Fowler summarizes the concept as:

Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly...

See also Wikipedia: [2]

What is Team Foundation Server?

Team Foundation Server is a server solution for developers in Microsoft environments including source code control and build service. Wikipedia summarizes Team Foundation Server in [3] as

Team Foundation Server (commonly abbreviated to TFS) is a Microsoft product offering source control, data collection, reporting, and project tracking, and is intended for collaborative software development projects. It is available either as stand-alone software, or as the server side back end platform for Visual Studio Team System (VSTS)...

Installing Team Foundation Server

Since I only have an old Windows Vista sacrifice system I cannot get the latest Team Foundation Server (but the Team Foundation Server 2012 Express edition looks really interesting [4]).

I am installing this on the same system as I a developing and I guess that is not the intended use case - I assume installing it on a dedicated server is a much better solution.

I select the advanced wizard during my installation:
http://www.pererikstrandberg.se/blog/testing-visual-studio/41-advanced-wizard.png

Connecting Visual Studio to Team Foundation Server and adding code

Once the install finished I open up Visual Studio and connect to Team Foundation Server using the button in the top left on the start page.
http://www.pererikstrandberg.se/blog/testing-visual-studio/44-connect.png

There is now, of course, a Team Explorer view in Visual Studio - and I use it to add a new team project.
http://www.pererikstrandberg.se/blog/testing-visual-studio/46-new-team-project.png

I follow the appearing wizard and select the Agile Template.
http://www.pererikstrandberg.se/blog/testing-visual-studio/48-agile-template.png

Once that is finished I add my solution to Source Control using the typical (and probably patented) Visual Studio Right Click of the Mouse on the solution and select Add Solution to Source Control...
http://www.pererikstrandberg.se/blog/testing-visual-studio/49-add-solution-to-source-control.png

Upon check-in there is a user-friendly little window appearing asking you to comment on this commit. It also shows you the files you are about to check-in.
http://www.pererikstrandberg.se/blog/testing-visual-studio/54-checkin.png

Once you get the hang of it a good idea is to always check the differences you are about to check in before checking them in. Right click -> Compare on the item of interest. If you have never seen this diff view before it is about time you saw it now. There are a number of visual diff's out there but the original terminal based ones are still awesome. According to Wikipedia the first diff was written in the early 1970's (see [5]). Any way - this is what a diff in Visual Studio looks like:
http://www.pererikstrandberg.se/blog/testing-visual-studio/51-modify-commit.png

Making a build

Build Configuration

First you need to configure Team Foundation Server to have a Build Service by following a configuration wizard. I had already installed the ability to perform builds - but not configured it properly:
http://www.pererikstrandberg.se/blog/testing-visual-studio/53-build-configuration.png

Here is what my configuration looked like just before finishing the configuration wizard.
http://www.pererikstrandberg.se/blog/testing-visual-studio/54-confirm-configuration-wizard.png

Adding a build

Adding a build is done with the typical right click in the right place - in this case in the Team Explorer view:
http://www.pererikstrandberg.se/blog/testing-visual-studio/55-new-build-definition.png

Configuring this build can be a real pain in the neck - there are a number of pitfalls. I'll try to walk you through them.

In the General "tab" I just gave my build a unique name:
http://www.pererikstrandberg.se/blog/testing-visual-studio/55-new-build-1.png

The first really interesting part is under the Trigger "tab". Here you can select continuous integration (or Gated Check-in). Just this feature can be the reason for throwing away what ever build system you have and using Visual Studio + Team Foundation Server instead. This is awesome!
http://www.pererikstrandberg.se/blog/testing-visual-studio/56-new-build-continuous-integration.png

Under Build Defaults you should add a drop folder as a shared on some server - in my case C:\tfs-drop-folder. (I of course had to make this folder shared by doing the magic right click on it while browsing my files and folders.)
http://www.pererikstrandberg.se/blog/testing-visual-studio/57-new-build-drop-folder.png

A good practice is to force your developers (and yourself) to take action if anyone checks in code (or tests) that break the build or if tests fail. Do this by setting "Create Work Item on Failure" to False.
http://www.pererikstrandberg.se/blog/testing-visual-studio/58-new-build-create-on-failure.png

A problem I had that I found easier to dodge than to solve was the case where my test project and my application had different platforms - in my case x86 vs Any CPU. This caused the build to break since it was somehow unable to build both the tests and the executable at the same time. The workaround was to use a better architecture in my solution and projects: I separated a lot of the code to libraries (dll's) instead and wrote tests to target them. And then I wrote a kind of a front-end on top of the libraries. This did not solve my problem with Team Foundation Server but it was good enough for today. Here is what to look out for if you get a similar problem:
http://www.pererikstrandberg.se/blog/testing-visual-studio/58-conflicting-target-platform.png

To change the configurations to build you have the possibility to do so under 1. Required -> Items to Build -> Configurations to Build. I believe you might need to add at least one configuration here - something like Any CPU|Release - like I did.
http://www.pererikstrandberg.se/blog/testing-visual-studio/60-build-configuration-any-cpu.png

Automatic Code Coverage in the build report

The next problem I had (and a loot of you people out there) was that I did not get any Code Coverage in the build. The problem was that I bad to both activate Code Coverage and then select what build items to analyze.

I did this in four steps. First I double clicked Local.testsettings under Solution Items in the Solution Explorer to get the Test Settings window to appear. Second I activated code coverage under the Data and Diagnostics tab. As you can see in this screen shot there is also a button called Configure in the middle of the screen that is activated once Code Coverage is selected.
http://www.pererikstrandberg.se/blog/testing-visual-studio/53-configure-code-metrics.png

Third: once I clicked the Configure button I select the libraries to do code coverage analysis on. I ignore the front ends and the tests.
http://www.pererikstrandberg.se/blog/testing-visual-studio/54-select-what-to-analyze.png

Once that is done and I have a build I verify that the code coverage looks ok by inspecting some random source code file:
http://www.pererikstrandberg.se/blog/testing-visual-studio/54-verify-code-coverage.png

We have Continuous Integration!

Now you should have a situation where every code check-in triggers a build, some test runs and code coverage analysis. In my case I would get something like this:
http://www.pererikstrandberg.se/blog/testing-visual-studio/71-tsf2010-gated-checkins-continuous-integration-and-code-coverage.png

The next step: Gated check-ins

You should activate the gated check-in feature by the magi right-click on the build and selecting trigger -> Gated Check-in.
http://www.pererikstrandberg.se/blog/testing-visual-studio/80-gated-check-in.png

Now on each commit Visual Studio will let you know that You need to build your changes for validation before they can be committed to the Team Foundation Server. Your changes will also be put on a shelf - they are "shelved" (I guess the git-speak term stashed was not professional enough for Microsoft). This is the alert you will get on each check-in now:
http://www.pererikstrandberg.se/blog/testing-visual-studio/81-build-for-validation.png

And what I did next was to deliberately break the build by making a test fail at all times. Once the build had finished I was informed: Gated Check-in Rejected. Also: You may want to unshelve your changes to correct the problem.
http://www.pererikstrandberg.se/blog/testing-visual-studio/82-check-in-rejected.png

Summary

Today I tried setting up an instance of Team Foundation Server in order to try out Continuous Integration and Continuous Integration with Gated Check-Ins. This took me less than a work day on an old laptop running an old version of Windows OS, but I did some research yesterday:

In short what we did was:

  1. Install Visual Studio (already done)
  2. Install Team Foundation Server and configure it
  3. Hook up Visual Studio to Team Foundation Server
  4. Configure a build service
  5. Add some existing projects to Team Foundation Server
  6. Creating and configuring builds
    • Continuous Integration does a build and performs tests on check-in
    • Gated Check-Ins performs the tests before check-in, in order to verify that your commit is ok.

Belongs in Kategori Programmering
Belongs in Kategori Test
See also Testing In Visual Studio
See also Testing In Visual Studio Part 1 where we look at Static Code Analysis, Unit Testing and Code Coverage.
See also Testing In Visual Studio Part 2 where we look at coded user interface tests