Updated: Fixed a few formatting and badly positioned graphics.

No tool can ever possibly meet every team’s needs out of the box. It’s crucial to select tools that are extendable or customizable to handle things specific to you and your team.

Test Studio gives you the ability to customize a number of things under the hood. One great feature is tying into events for test lists in order to handle setup or teardown steps. This lets you do things like load baseline datasets, clear out test-generated data, or perform configuration actions.

The post below is written by Ivaylo Angelov, a QA engineer on Telerik’s great Sitefinity product. Ivaylo and his team have customized their test list process to provide reports via mail—a better process for them.

I hope you enjoy this post, and learn something useful from it!

-- Jim Holmes

Extending your automation to bring it closer to you

Recently, I’ve been working on improving my existence automation.

Of course, this included all the basic stuff like code refactoring, performance optimizations, creating new tests and fixing existing ones, etc.

But I really wanted to do something more and help myself in a way that I have more clear vision on what is happening after my test lists have run, I wanted to know briefly what is the overall result and I needed this information in a fast and easy to get manner.

Why?

Well, using mostly the framework to manage my UI tests, I had the opportunity to observe results either by visiting an internal web page where they were uploaded once run or by exploring the result file after the list has finished.

All that is great and substantial enough when it comes to my day-to-day duties. But these methods turned out to be impractical when I am out of the office or I don’t have access to a computer from which I can establish a remote connection to do my job. That’s why I needed something simple to let me know quickly what happens when I am on a vacation for a couple of days or we are facing a release where bugs and failures are not welcome at the last moment and ‘the night before the show’.

What was the approach?

I decided to use something very simple and which has been around for about 30 years. Yes, that is right, we are talking about mailing.

My idea was to send myself a message once the test list has finished running on the build machine. I wanted the body of the email to contain:

· Overall information: Number of passed and total tests, overall result, execution time, etc.

· Number of failed tests (if any) and details about them

I was aware that something similar exists as a feature in Test Studio if one uses the Scheduling. However, this was not my case. I use primarily MSBuild to trigger and manage the execution of my tests. So I decided to go on and do it by myself.

Implementation

I had a couple of options going in my head but all seemed to be hard to integrate or needed a lot of handwork and code replication here and there.

And then I decided to ask my colleagues from Test Studio what would be the best way to do that. They were very kind and helped me in an amazing way by directing me to this article in their documentation: Execution extensions

After I read what is written there, I realized I can very easily do whatever I want once the test list has completed. And this extensibility made my life much easier.

I created my own class library and named it ExecutionResultsMailReporting.

What I did was to simply write some methods to format and send my email. The body of the message was supposed to have information about the number of passed tests, the total number of tests in the list, overall result of the run and execution time.
I have also decided to list the names of all failed tests (if any) and attach to the message a PDF file holding the logs and details for the failures.
I decided to use PDF because of the better formatting and the ability to view the file on every device I use. However, I needed a 3^rd party library to get the job done. I used iTextSharp which is a C# port of the iText library to create my PDF file. The library is open source and it could be downloaded for free from here - http://itextpdf.com/download.php

The assembly itextsharp.dll also exists in the Reference Assemblies folder of my project. All this stuff together with the solution could be downloaded from here: ExecutionResultsMailReporting

In more details

If you have a closer look at my implementation, you will see it’s rather simple. My class needed to implement the IExecutionExtension interface. The method we need to provide with our custom logic is the OnAfterTestListCompleted(RunResult result). Others could be left empty.

The content of my email is based on the result file – the one and only parameter of the implemented method mentioned above.

The private method ProcessMailSending takes the argument provided and does the entire trick. I catch all possible exceptions at top level and if something bad happens I log details about it in a local text file.

The method GetMailContent creates the body of the message.

Using a variety of properties from the result object, you will be able to build your content. For example, the AllCount property returns the number of all tests run. The boolean PassedResult gives you the overall outcome. There is much more additional information like StartTime, EndTime and many more interesting and fancy stuff to get from the result.

To get the failed tests, one can build a very simple expression:

var failedTests = result.TestResults.Where(t => t.Result.ToString().Equals("Fail")).ToList();

Then based on the collection and whether it is empty or not, the structure of the main message content is left in your hands. You can also play around with the html used for the message to style it the way you want and show the summarized information you find most useful. I will let the code snippets speak for themselves.

Apart from the body content, based on whether we have or not failed tests, we create an attachment for the email. Again, we have a collection with the failed tests. What we further do is to delete the older attachment created on the file system, create the new document and fill it with content.

We iterate through all the failed tests we have. For each of them we add the test name and its message. The property Message returns all execution details and failure information – the same you can see in the typical Test Studio Log window. At the end, we close the document and return the created attachment object. You can use the official documentation (link here) of iTextSharp to find out how you can further enrich your attachment in terms of formatting and style.

The forwarding of the message itself is done by the SmtpClient class included in the System.Net.Mail namespace. It is important to specify that the body is HTML. Then simply add all the recipients for the email and send it.

And guess what, that’s it. There is nothing more special about that.

Note: If you use the code in my project, have in mind the constant variables in the region at the end of the file ExecutionResultsMailReporting.cs. You will need to provide your own values for them to make the mail sending possible.

Resolve any missing or corrupted Test Studio assemblies. And also, if you use the itextsharp assembly, copy it together with the DLL file generated from the build of your project in the Plugins folder of Test Studio (on the machine where tests will be executed).

The result

After I’ve added my extension to the Plugins folder of Test Studio and waited for my test list to finish running, I got the following in my inbox:

And this is the way the mail looks like when we have failed tests:

And the details PDF file:

A few ending words

To summarize it all, Test Studio gives you some great flexibility. The option to put your own piece of magic before, during or after the execution of your tests is something that can really enhance your QA process. Now when you know you can extend your automation in so many different ways, go and give your imagination a try.

About the author

Ivaylo Angelov

Ivaylo Angelov works as a QA Engineer in Telerik’s Sitefinity team. He is passionate about testing and all possibilities of automating the whole process. A big fan of music, literature and sports. More information could be found in his blog (in Bulgarian only).