I'm evaluating WebUI TestStudio and have several questions:
How I could create common global methods for all my tests (not as script step for specific test)?
Is it possible to make all project tests not in keyword steps but strictly in code?
Does TestStudio support global project variables or built-in data store that could be used for all tests?
Is there some ability to collaborate between tests, pass arguments to them and run set of tests without using test lists?
Could I make my login, logout, verification, navigate, etc tests as separated functionality that couldn't be run as tests?
Is it possible to develop custom framework for project using resources of TestStudio?
I'm looking forward for your feedbacks.
Thanks.
18 Answers, 1 is accepted
Thank you for taking the time to evaluate Test Studio. I am confident you will find it a valuable tool in your test automation efforts.
We like to use the term "Test Modularization" which I believe is the goal you're after. Certainly Test Studio supports this and we very much encourage implementing modularization! There are two main methods of accomplishing modularization.
1) Using our Test-as-step feature - This is what is most commonly used by our customers. You write a test to be a subtest called by some parent test. A "Login" subtest is very common. We do not limit how deep you can go with subtests (though it becomes unmanageable if you go too deep). I know of a couple of customers going 3 subtests deep.
It is also common to create one (or more) folders in the test project in order to better organize and group the main tests and subtests together.
2) Using code - In this approach you are using our coded steps feature and from the coded step you are calling your own coded functions. Those coded functions can exist in one of three places:
a) Within the same code file that test is using (each test may have one code file associated with it)
b) Within your own code file. Any code file that is placed in the projects folder will automatically be included in the compile process. Thus any coded step can refer to the class (or classes) defined with this file. You could even have multiple code files of your own creation. But you must write them using some tool outside of Test Studio (e.g. Notepad, Visual Studio, etc.).
c) Within a third party .dll. You can add a reference to some external .dll, either of your own creation or from some third party vendor, then call the functions contained in that .dll from the coded step.
On the question about globally accessible data, the closest built-in feature we currently have is by using data driven testing and having all your subtests inherit the data from the parent test. Many of our customers like this feature because they can essentially parameterize subtests by binding the parent test to a different set of data.
The next best solution is to implement it in code. Define your global variables/data in code and use it in code. You also have the option of setting the value of a variable in code such that a non-coded test step can use it like a data bound setting.
I hope that answers your questions and you can see the power and flexibility of Test Studio. Please do not hesitate if you have further questions.
Cody
the Telerik team
Rank 2
Cody,
I think that was well written, please note the best approach is to break down the tests. I am trying to separate the navigation from the actual verification point. For example if I where to need to navigate or to prepping before getting to the actual point of test. If your UI flow changes then you need to change navigation scripts. Here are some examples.
1) Testing to validate that the scores entered are correct.
a) The login is on to its own navigation and data driven from the parent.
b) The navigate too and prep are broken down (at least one or more test scripts). Make know I would have the prep changes as data driven ie the login user name and password.
c) Then the Validation of whatever screens you have (if need be you might want to break those out among steps and verification points).
2) Please note that if you data drive something not to wind up in an infinite loop. I sometimes test for a point that might not come out. Just let it time out. My favorite example is Dr. Who, The Doctor goes in trades, he leaves; don’t bother for that trip to end. *GRIN* be careful on what you base your actual test steps on.
Just my two cents,
John
Rank 1
I still misunderstand how to create common functions that could be called from all my tests. So adding own code file or third party .dll is only solution? So is it possible to develop custom framework for project using resources of TestStudio? For ex. methods for making some actions for all my spec. fields (edit field, drop-down, check box, etc.).
For about project properties, as I understand there is no ability to use common global property file, like .config or .ini file for all project items (only DDT storage for single test supported)?
Another question related to DataGrid contained in Silverlight application. How I can find and open last item under this DataGrid (this item is not visible and could be visible only after scrolling to it)?
My current problem is developing common custom function! I have page with links. I need global function that will be accessible for all my tests. This function should receive link string as argument, find this on the page and return page object to the caller:
Test1 \ Page navigate(String url) {
Test2 - /* find link by provided url on the current page; click on the link;
Test3 / wait while page loaded; return page object */ }
So any test (Test1, Test2 and Test3) should have ability to call this unique function with parameter (url) and receive result (page object).
I apologize for the delay in getting back to you on this.
"So adding own code file or third party .dll is only solution?"
No. Our Test-as-step feature is what is most commonly used to modularize tests and provide the global functions I think you are seeking. If you're worried about being able to pass parameters to it, you accomplish this via one coded step in the parent test to set an Extracted Value which works like a global variable. The sub-test consumes this just like a data driven test. The only other thing you need to set is to turn on Inherit Parent Data Source in the subtest.
Sure you can write your own code in a code file or create your own .dll. I view this as the hard approach. Test-as-step usually is much easier.
Let's take your specific example - create a global function that will:
a) Accept a URL as a parameter.
b) Scan the current web page for a link matching that URL.
c) Click on the link.
d) Return the resultant page.
Stepping back for a minute and looking at this from a high level, if we know the URL to navigate to, why don't we just use a NavigateTo step to go directly to that page? Why spend the effort finding a matching link, clicking on it and waiting for the page to load?
Having said that, it is true steps a-c need to be done in code because at present data driven find expressions can only be done in code. The sequence would go like this:
1) Parent test uses a coded step to set a variable containing the URL.
2) Parent test calls subtest, subtest has Inherit Parent Data Source set to True.
3) The first step of the subtest will be a coded step that:
a) Scans the page for the matching link.
b) Clicks on it.
c) Waits for the page to finish loading.
4) The subtest is finished (unless there's something more you want it to do) and returns to the parent test.
5) The parent test will automatically start interacting with the newly loaded web page.
This is by design in Test Studio. Every test step will always work with "the active browser" which is defined as the last browser window that was opened. This is how we work with popup windows. If a link opens a popup, all test steps following the open popup step will immediately interact with that popup window... until it is closed at which point the follow on test steps automatically start working with the parent window.
Now any test in your test project can use this subtest via test-as-step. You can even have multiple tests using the same subtest and/or call the same subtest multiple times from the same parent test.
If you're still not sure how to proceed, I suggest we meet online via GoToMeeting where we can discuss this in more detail and you can ask more questions and we can refine how to accomplish your goals.
Cody
the Telerik team
Rank 1
There is no need to take your time. I have one more question:
Does TestStudio support running different tests from Test List on the same WPF application, like "Recycle Browser" in the Web applications? Because I need to run all my tests in the same WPF instance without interruption.
Unfortunately we do not have the equivalent of Recycle Browser for WPF app testing. I filed a feature request here for this.
What you can do in the meantime is to create another "master test" that is nothing more than using our Test-as-step feature for executing all of your real tests in the same WPF instance.
Cody
the Telerik team
Rank 1
I've a couple of questions:
First question related to testing of WPF applications.
How I could access or find necessary WpfWindow (not ActiveApplication.MainWindow)? Like in the same way as in the web applications:
Pages.<My Page>.<Page Elements>
Also I would like to know, how I could implement Wait/Verify conditions for waiting or verifying appearance or disappearance of necessary window (e.g. waiting while progress bar Wpf window disappeared)?
The next question related to Continuous Integration. As I verified test could be run via TestAgent as a service. Chose Your Service->Properties->Log On tab->check Local System account and Allow service to interact with desktop.
In this way all applications under test will be run as 'invisible' processes. The problem that when I minimize my Remote Desktop Connection window test couldn't find some target elements and fail. Even when I run TestAgent from Console. Is it normal? Previously i read that just enough this one: "Leave the machine running and logged into the account but locked".
Also maybe there is some possibility to avoid using absolute paths in the .tstest and config files. My .tstest tests failed on start and not executed when an absolute path provided in the TestAssemblyPath node :(
Another question related to reporting. Is it possible to customize report in own format manually or by code? For example exporting it as
VS Result file using code? When i run test via MSTest is it possible to attach captured screenshot or application log file into this report?
How to continue test execution without fail in the result of test if some step in this test fails(Not ContinueOnFail option that anyway will fail common result)? How to continue parent test if his "test as step" has only one failed step with ContinueOnFail option?
The last question related to Storyboard. Is it possible during tests execution to recapture Storyboard, exporting this and attaching recaptured images into my report automatically after test execution even if my test contains only step as test items?
Thanks.
"How I could access or find necessary WpfWindow (not ActiveApplication.MainWindow)?"
Use either:
Manager.ActiveApplication.GetWindow("window caption here");
or
Manager.ActiveApplication.WaitForWindow("window caption here", 10000);
"I would like to know, how I could implement Wait/Verify conditions for waiting or verifying appearance or disappearance of necessary window"
There's the WaitForWindow which I already spoke about. Unfortunately we don't have a good mechanism to wait for a window to go away. WaitForWindow will throw the generic Exception if it cannot find the window. GetWindow will return null if it cannot find the window. You could sit in a loop, with some delay, monitoring the output from GetWindow until it returns null.
There's also the:
Manager.ActiveApplication.Windows;
This returns a list of known windows. Here again you could sit in a loop waiting for a specific window to no longer appear in the list.
I do want to caution you against waiting for a window to no longer exists. Since we have to poll for it, if it appears and disappears very quickly we might miss it all together.
"when I minimize my Remote Desktop Connection window test couldn't find some target elements and fail"
Yes this is expected for any test step that needs to use a real mouse or keyboard action. The only type of test that will work that way is a pure DOM test, i.e. one that has no Mouse actions, no Keyboard actions, no Simulate Real User actions. Unfortunately not many tests are able to meet this criteria. It is not possible for tests against a Silverlight application to meet this requirement simply because most actions against Silverlight require real mouse or keyboard interaction.
I think for the storyboard question, what you really want is our Browser Capture feature. You can also do this in code if needed.
Cody
the Telerik team
Rank 1
Are there some way for accessing or waiting Dialog windows (like this: Manager.ActiveApplication.WaitForWindow("window caption here", 10000))? As I understand TestStudio doesn't have good support for WinForm applications. Maybe there are some workarounds for setting values and clicking buttons on WinForm application?
How to prevent in TestStudio "Simulate Real User actions" setting in my test steps. Because when I converted such steps (e.g. type value into filed) into code, "Simulate Real User action" option set as "true" by default.
Also I have problem with handling Generic Dialog window. This dialog window appears after launching application, but TestStudio couldn't find this:
'11/24/2011 9:24:20 PM' - LOG: Unexpected dialog encountered. Taking no action.
'11/24/2011 8:42:50 PM' - 'Fail' : 10. Handle 'Generic' dialog.
------------------------------------------------------------
Failure Information:
~~~~~~~~~~~~~~~
Object reference not set to an instance of an object.
InnerException:
System.NullReferenceException: Object reference not set to an instance of an object.
at ArtOfTest.WebAii.Design.IntrinsicTranslators.Descriptors.DialogHandlerDescriptor.Execute(Browser browser)
at ArtOfTest.WebAii.Design.IntrinsicTranslators.Descriptors.DialogHandlerDescriptor.Execute(IAutomationHost browser)
at ArtOfTest.WebAii.Design.Execution.ExecutionEngine.ExecuteStep(Int32 order)
------------------------------------------------------------
In the OnBeforeTestStarted method I set these settings:
Settings.Current.UnexpectedDialogAction = UnexpectedDialogAction.DoNotHandle;
Settings.Current.ExecutionDelay = 10;
...and then Handle 'Generic' Dialog step. This step was recorded and is valid(!), but TestStudio couldn't handle it. I tried different ways and found only one working but not suitable solution via Desktop action:
Window dialog = WindowManager.FindWindowRecursively(IntPtr.Zero, "window caption here", true, 0);
dialog.SetFocus();
Manager.Desktop.KeyBoard.KeyPress(System.Windows.Forms.Keys.Enter);
I really need to resolve this issue.
Happy Thanksgiving.
You are correct in that Test Studio cannot be used to automate a WinForms type of application. We have limited support for dialog handling. What we have is documented here.
Most actions in a WPF application require the "Simulate Real User". Are you having success turning this off anywhere?
For the System.NullReferenceException problem, I need a way of reproducing this problem on my machine so I can study it and figure out what's causing it. Can you send me a sample WPF application I can use to reproduce this problem?
Cody
the Telerik team
Rank 1
I would like to use the test-as-step feature in the way described in this post. I need to call a subtest but use a global variable to set a number for the subtest. However, I is there a way of setting the value manually as I do not want to use the extract method.
For example:
MainTest
set value "num" = 20
Call SubTest using test-as-step
.... continue other steps
End MainTest
SubTest
Use the value $(num) as input to a text field
.... continue other steps
End SubTest
Many thanks,
Ghandi
If you would like to create global variables or functions in Test Studio Standalone version that are accessible from all the tests within the test project, you can create a utility class. This class will contain the intended functions or variables.
Please follow this article in order to get detailed information on how to set/get variables in code.
Hope this helps.
Boyan Boev
the Telerik team
Test Studio Trainings
Rank 1
2. Either way, once the variables are created, how can I set their values on the fly?
Example, if I have a variable called x:
MainTest
Set x = 5
Call SubTest
Set x = 10
Call SubTest
End MainTest
SubTest
Set the value of a text box with x
Other commands
End SubTest
Thanks,
Ghandi
1. You can also define a variable in test Y and access it in test X.
2. You need do define it as a public static variable in the class of the test Y (not in a coded step).
I recorded a short video as a demonstration.
Let me know if you need further assistance.
Boyan Boev
the Telerik team
Test Studio Trainings
Rank 1
I have tried the above but while it compiles okay, it doesn't seem to execute okay.
Test_pagesize_argument (Main test)
Test Steps
[SetPageSize05] : Custom code
Test as step: Page Sizing
[SetPageSize010] : Custom code
Test as step: Page Sizing
End
// Custom variables
public static string PageSize = "";
// Add your test methods here...
[CodedStep(@"New Coded Step")] public void SetPageSize05() { PageSize = "5"; } [CodedStep(@"New Coded Step")] public void SetPageSize10() { PageSize = "10"; }
End Test_pagesize_argument
Page_Sizing (Sub test)
Test Steps
Page_Sizing_CodedStep] : Custom code
Page_Sizing_CodedStep1] : Custom code
Page_Sizing_CodedStep2] : Custom code
End
// Add your test methods here...
[CodedStep(@"RadInput('MainContentDgAssetsChangePageSizeTextBoxWrapperSpan'): page size variable")] public void Page_Sizing_CodedStep() { // RadInput('ctl00_MainContent_dgAssets_ctl00_ctl02_ctl00_ChangePageSizeTextBox_wrapper'): value '15' entered. Pages.Data_Viewer.MainContentDgAssetsChangePageSizeTextBoxWrapperSpan.TypeText(Test_pagesize_argument.PageSize, 50, 50, f }
[CodedStep(@"RadGridPager('TableRow'): table 'dgAssets' change page size to page size variable.")] public void Page_Sizing_CodedStep1() { // RadGridPager('ctl00_MainContent_dgAssets_ctl00'): table 'dgAssets' change page size to '15'. Pages.Data_Viewer.TableRow.ChangePageSize(); }
[CodedStep(@"RadGridPager('TableRow'): info text 'Contains' 'Displaying 1 to page size'")] public void Page_Sizing_CodedStep2() { // RadGridPager('ctl00_MainContent_dgAssets_ctl00'): info text 'Contains' 'Displaying 1 to 15' Pages.Data_Viewer.TableRow.ControlAssert().StringValue("InfoText", "Displaying 1 to " + Test_pagesize_argument.PageSize, ArtOfTest.Common.StringCompareType.Contain }
My sincere apologies for the mistake.
I recorded another video according to your case.
It is not very clear what exactly is happening in your tests. In order to assist you best please provide us with a copy of your test and access to your application. If it is not possible to grant us access to your application, please take a Fiddler trace using FiddlerCap and send it to us in a zip file.
If you feel any of this information is sensitive, you can submit a support ticket which is confidential, unlike this forum.
A Jing video demonstrating the issue may also help us to better understand what is happening.
Thanks for providing the information we need to best assist you.
Boyan Boev
the Telerik team
Test Studio Trainings
Rank 1
I fixed the problem by disabling the "InheritParentDataSource" on the child sub-test.
Many thanks for your help,
Ghandi
I am glad to hear you have solved the problem.
Please contact us again if you have further problems.
Kind regards,
Boyan Boev
the Telerik team
Test Studio Trainings