Test steps and results in TFS 2010

All SSRS reports you get out of the box with TFS 2010 are trend reports and they don’t show any details. I was recently asked to build a custom report listing all test cases in a test plan and for each test case every step should be listed with action, expected result ant test outcome. Basically the same kind of info you can get for a single test case when you view the test results in Microsoft Test Manager 2010. This turned out to be quite a bit trickier than expected but after getting some much needed info from a developer at Microsoft (thank you Sriram) I was able to get it all together.

Getting test steps and results from Tfs_DefaultCollection

Test steps are stored as XML and are found in table WorkItemLongTexts. You can use this query to get them.

declare @fldIdSteps int = (Select top 1 fldid from Fields where ReferenceName = ‘Microsoft.VSTS.TCM.Steps’)
select * from WorkItemLongTexts where FldID = @fldIdSteps

Test results for each step are found in table tbl_TestActionResult. You can use this query to get them.

select * from tbl_TestActionResult

Now, you’ll notice there are a couple of issues popping up. You’ll need to join a XML result with a table result, there might be several revisions of the test steps and there are probably many results saved for each and every test case. But most of all, there is no obvious way to link steps to results. Every test step has an ID but tbl_TestActionResult does not contain a “test step ID” column.

ActionPath explained

This is where the column ActionPath in tbl_TestActionResult comes in. This column will typically contain an empty string, 8 chars string or 16 chars string. For a specific test result in a test run there will be one line with the empty string (this line is the over all test result) and then one line for each step, containing 8 or 16 chars. These ActionPath chars are hierarchical hexadecimal representations of test step ids. The first 8 chars is the step id and the next 8 chars (if they exist) is a shared step id.

And now you have the knowledge needed to pair up test results with corresponding test steps!

Some issues to consider…

There are a few more issues you’ll have to tackle as well.

  • Step ID != Sequence number. The test step id is not the same as the numbers you see when viewing a test case in a GUI. The numbers you see in the GUI are generated in the GUI to clarify the step sequence. The step id is never shown and the sequence number is not stored in the database. You’ll have to rely on the order of the <step> elements in the XML to figure out the step sequence.
  • Revisions and result. There can be several revisions of the test steps stored in table WorkItemLongTexts and there can be several test results stored for each revision. You’ll have to compare timestamps for revisions and results when joining them to avoid errors.
  • Performance. The Database Tfs_DefaultCollection is the production database and you should always take performance hits into consideration before deploying reports that read directly from your production database.
More details

When I first tried to figure out how this all worked I put a question up at the MSDN forums. For SQL and code samples you can go to my post on MSDN and get some more details.

Advertisements

Moving from VS2005 to VS2008

This is actually really easy as long as every developer on your project makes the move simultaneously. If not, well, there are a couple of things to keep in mind.

  • VS2008 will convert your solution and project files even when you stay in .Net 2.0. Converted solution files can not be opened by VS2005. Project files can still be opened (you’ll get a warning but that is all). This means you’ll have to work with two versions of your .sln files, one for VS2008 and one for VS2005.

  • If you’ve used Microsoft’s Test Project template for writing unit test etc, the reference to ”Microsoft.VisualStudio.QualityTools.UnitTestFramework” will have changed in the conversion, breaking the project for VS2005. This reference will have to be manually changed back and forth depending on what VS version developers use.

That’s it so far, but I’m sure there are some other pitfalls out there waiting for me 😛

Disastrous inclination towards complexity and ingenuity

Found a very good quote at Worse Than Failure today…

“Programmers… often take refuge in an understandable, but disastrous, inclination towards complexity and ingenuity in their work. Forbidden to design anything larger than a program, they respond by making that program intricate enough to challenge their professional skill.”
Michael A JacksonPrinciples of Program Design, 1975

Must have been what happened when the program I’m currently maintaining was originally designed… not that I would ever do something like that… promise…

Enterprise Library Pitfall

I was recently asked to bugfix an application. The bug was eaasy enough to fix, just add two lines of code, no big deal. In order to build the application I had to install Enterprise Library 3.1 and re-reference a few .dll files in it. Also no big deal, done it before with previous versions of Enterprise Library… but it turns out i stumbled right into a configuration pitfall!

Short version for those who dont want to read Tom Hollander‘s blog about Avoiding configuration pitfalls with incompatible copies of Enterprise Library:
If you install Enterprise Library 3.x a go with all default options/choices, you end up with two sets of .dll files. If you have to re-reference these files in a project and use the wrong .dll your code will compile but throws an exception in runtime… referencing the other set of dll files will solve the problem (but might cause new problems as well).

Conversion Issues

Objective:
Get a VS2003 ASP.NET web application project from SourceSafe, migrate to VS2005 and add to TFS source control. All done on a PC running Windows Vista. As VS2003 will not run on Vista opening the solution in VS2003 and disconnect it from source control is not possible… might be possible to do from SourceSafe, but I decided to give it a go and just open it in VS2005 straight away.

Problem:
VS2005 conversion wizard fails to convert the web application. Depending on if I let the .sln file point to localhost or changed the projekt path to “C:\myAppPath\myWeb.vbproj” I get two different errors:

Error 1 (keeping localhost/myWeb/myWeb.vbproj as project path):
Conversion Issues – myWeb.vbproj
Thats it… No clue as to what is wrong.

Error 2 (changing to C:\myPath\myWeb.vbproj as project path):
Conversion Issues – myWeb.vbproj:
Unable to open the Web ‘C:\myPath\myWeb.vbproj’. The Web ‘C:\myPath\myWeb.vbproj’ does not exist.
Looks a bit more helpful, but since the file DOES exist… No, no more helpful than what I got in Error 1.

Solution:
As it turned out there wasn’t really anything wrong with the myWeb.vbproj file. Except that it was write protected… Checking the “make writable” checkbox when getting the code from SourceSafe (or just removing the write protection from the file) made all the difference. VS2005 disconnects the code from SourceSafe source control and converts everything Cool

Now I just need to solve all the compilation errors and the code will be ready to check into our TFS… Looking back, I might have been able to avoid the entire problem by following Microsoft’s instructions for migration. But why learn to swim when it is soooo simple to just jump into the deep end of the pool? Anyway, hope this is helpful to someone else…