Test List 'skips over' execution code

[Nigel]

Hi,

I have an issue with Test Lists.  I am running Test Studio version 2011.2.1413.0

I have a script that iterates through 7 rows of data in a data file.  When I run the script directly from the Record tab everything works perfectly.  When I run the script from a Test List the script fails at the first iteration.  There is no error reported from the Test List.

The faiure happens at the first IF...ELSE statement in the script.  The IF statement test is:
IF (Verify 'InnerText' 'StartsWith' 'Showing 0' on 'TableUsersInfoDiv') THEN

On the first pass the IF clause should report TRUE, and it does do that when the script is run from the Record window.  Below is an extract from the (verbose) log generated by the Test List execution (and please note that all conditions prior to execution through the Reporting screen and the Test List are identical in every instance):

Trace] : Wait for 'InnerText' 'StartsWith' 'User List' on 'HtmlTag'
[Trace] : Click 'AllRadio'
[Trace] : Click:([Element: 'input:3' (id=all)])
[Trace] : Enter text 'test' in 'Text' - DataDriven: [$(UserName)]
[Trace] : SetText:([Element: 'input:1']) - Text:
[Trace] : ScrollToVisible:([Element: 'input:1']) - ElementTopAtWindowTop
[Trace] : MouseClick:LeftDown (1108,454)
[Trace] : MouseClick:LeftUp (1108,454)
[Trace] : MouseClick:LeftClick (1108,454)
[Trace] : IF (Verify 'InnerText' 'StartsWith' 'Showing 0' on 'TableUsersInfoDiv') THEN
[Trace] : Comment: User ID already exists so just logout
[Trace] : Execute test 'g_Logout'
[Trace] : Comment: Script: g_GenericLogout; Author: Nigel Edwards; date: 02/03/12
[Trace] : Execute test 'g_LogoutWaitForButton'
[Trace] : Comment: Script: g_WaitForLogoutButton; Author: Nigel Edwards; date: 02/03/12

The COMMENT line reported above is actually within the ELSE clause.  This shows that a swathe of code (over 30 lines) within the IF statement is not being executed when run from the Test List.  In fact the 'InnerText' does start with 'Showing 0'.  I've attached the full test script to this posting, but in brief, what should happen after the IF test is a further verification line immediately followed by a button-click.  Instead the Test List version skips to the ELSE clause, which is a Logout command.
As I say, the script functions 100% correctly when run directly from the Record window.

Thanks,
Nigel Edwards, Transition Computing.

[Nigel]

A little bit more information for you, here is the .aiilist file from the last time I tried to execute my script through a Test List:

<TestList Type="ArtOfTest.WebAii.Design.Execution.TestList" Version="2011.2.1413.0">
  <ID Type="System.String">2ac18a53-d0af-4a41-8ed9-d9dee6f17704</ID>
  <TestListName Type="System.String">temp list</TestListName>
  <Date Type="System.DateTime">2012-03-26 12:38:16Z</Date>
  <Settings Type="ArtOfTest.WebAii.Core.Settings" Version="2011.2.1413.0">
    <WebSettings Type="ArtOfTest.WebAii.Core.Settings+WebSettings" Version="2011.2.1413.0">
      <DefaultBrowser Type="ArtOfTest.WebAii.Core.BrowserType">InternetExplorer</DefaultBrowser>
      <WebAppPhysicalPath Type="System.String">
      </WebAppPhysicalPath>
      <EnableScriptLogging Type="System.Boolean">True</EnableScriptLogging>
      <EnableUILessRequestViewing Type="System.Boolean">False</EnableUILessRequestViewing>
      <LocalWebServer Type="ArtOfTest.WebAii.Core.LocalWebServerType">None</LocalWebServer>
      <BaseUrl Type="System.String">
      </BaseUrl>
      <AspNetDevServerPort Type="System.Int32">-1</AspNetDevServerPort>
      <KillBrowserProcessOnClose Type="System.Boolean">False</KillBrowserProcessOnClose>
      <RecycleBrowser Type="System.Boolean">True</RecycleBrowser>
      <UseHttpProxy Type="System.Boolean">False</UseHttpProxy>
      <EnableSilverlight Type="System.Boolean">False</EnableSilverlight>
      <VerboseHttpProxy Type="System.Boolean">False</VerboseHttpProxy>
      <SilverlightConnectTimeout Type="System.Int32">30000</SilverlightConnectTimeout>
      <MultiBrowsersExecution />
      <UseMultiBrowsers Type="System.Boolean">False</UseMultiBrowsers>
      <IsProfilingExecution Type="System.Boolean">False</IsProfilingExecution>
      <ProfilerConfigurations />
    </WebSettings>
    <WpfSettings Type="ArtOfTest.WebAii.Core.Settings+WpfSettings" Version="2011.2.1413.0" />
    <ClientReadyTimeout Type="System.Int32">30000</ClientReadyTimeout>
    <ExecuteCommandTimeout Type="System.Int32">20000</ExecuteCommandTimeout>
    <LogLocation Type="System.String">C:\WebAiiLog\</LogLocation>
    <QueryEventLogErrorsOnExit Type="System.Boolean">False</QueryEventLogErrorsOnExit>
    <ExecutionDelay Type="System.Int32">0</ExecutionDelay>
    <AnnotateExecution Type="System.Boolean">True</AnnotateExecution>
    <SimulatedMouseMoveSpeed Type="System.Single">0.3</SimulatedMouseMoveSpeed>
    <WaitCheckInterval Type="System.Int32">500</WaitCheckInterval>
    <AnnotationMode Type="ArtOfTest.WebAii.Core.AnnotationMode">All</AnnotationMode>
    <LogAnnotations Type="System.Boolean">True</LogAnnotations>
    <IsStressRecordingMode Type="System.Boolean">False</IsStressRecordingMode>
    <IsUserInteractiveMode Type="System.Boolean">False</IsUserInteractiveMode>
    <CreateLogFile Type="System.Boolean">True</CreateLogFile>
    <XMultiMgr Type="System.Boolean">True</XMultiMgr>
    <UnexpectedDialogAction Type="ArtOfTest.WebAii.Core.UnexpectedDialogAction">HandleAndFailTest</UnexpectedDialogAction>
    <ElementWaitTimeout Type="System.Int32">10000</ElementWaitTimeout>
    <DisableDialogMonitoring Type="System.Boolean">False</DisableDialogMonitoring>
  </Settings>
  <Tests>
    <Item Type="ArtOfTest.WebAii.Design.Execution.TestInfo" Version="2011.2.1413.0">
      <ID Type="System.String">a4ece2e6-629e-4997-b575-5abf386e7ed7</ID>
      <Path Type="System.String">SystemAdmin\Users\sa_UsersSetupAllRequired.tstest</Path>
    </Item>
  </Tests>
  <Filter Type="ArtOfTest.WebAii.Design.Execution.Filter" Version="2011.2.1413.0">
    <Keys />
    <Comparisons />
    <Values />
  </Filter>
  <TestListType Type="ArtOfTest.WebAii.Design.TestListType">Automated</TestListType>
</TestList>


Thanks,
Nigel.

[Nigel]

...also, I have created a stripped down script that also fails in exactly the same way, so it is not just one script.  Here's the Test List execution log:

[Trace] : Execute test 'g_NavigateToURL'
[Trace] : Comment: Script: g_NavigateToURL; Author: Nigel Edwards; Date: 02/03/12
[Trace] : Comment: To consistently navigate to the URL and open the LogOn screen
[Trace] : Comment: Script uses Local Data
[Trace] : Navigate to Project - DataDriven: [$(URL)]
[Trace] : NavigateTo: http://vs-demo/BaseProjectQA
[Trace] : Wait for 'InnerText' 'StartsWith' 'Base Project - Log On' on 'HtmlTag' - DataDriven: [$(LogonScreenTitle)]
[Trace] : Extract 'TextContent' on 'VersionTagDiv' into DataBindVariable $(VersionNumber)
[Trace] : *** Version Number is:   v 1.9.0.11 ***
[Trace] : Comment: * End of g_NavigateToURL test script *
[Trace] : Enter text 'user' in 'TextBoxUserIdText'
[Trace] : SetText:([Element: 'input:0' (id=textBoxUserId)]) - Text:user
[Trace] : Enter text 'password' in 'PasswordPassword'
[Trace] : SetText:([Element: 'input:1' (id=Password)]) - Text:password
[Trace] : Click 'ButtonLogOnSubmit'
[Trace] : Click:([Element: 'input:2' (id=ButtonLogOn)])
[Trace] : Click 'SystemAdminLinkLink'
[Trace] : Click:([Element: 'a:7' (id=SystemAdminLink)])
[Trace] : Click 'UsersSpan'
[Trace] : Click:([Element: 'span:34'])
[Trace] : Wait for Exists 'CreateNewSpan'
[Trace] : Enter text 'abc' in 'Text'
[Trace] : SetText:([Element: 'input:1']) - Text:
[Trace] : ScrollToVisible:([Element: 'input:1']) - ElementTopAtWindowTop
[Trace] : MouseClick:LeftDown (1083,394)
[Trace] : MouseClick:LeftUp (1083,394)
[Trace] : MouseClick:LeftClick (1083,394)
[Trace] : IF (Verify 'InnerText' 'StartsWith' 'Showing 0' on 'TableUsersInfoDiv') THEN
[Trace] : Comment: else do nothing
[Trace] : Closing Window

[Ivaylo]

Hello Nigel,

Please excuse me on the delayed reply.

I have reviewed your case but I cannot find any reason of the failure. The only difference between the Quick Execution and the TestList execution are the TestList settings. The settings are taken into account when running as part of a TestList and they can lead to some unexpected behavior:
http://www.telerik.com/automated-testing-tools/support/documentation/user-guide/test-execution/test-list-settings.aspx

However, I can't think on any particular setting that will cause this type of behavior. Can you confirm that the option StopTestListOnFailure is not checked (see screenshot).  We will need more info on this, please provide us the log from test results so we can investigate further.

Looking forward to hear from you.

Greetings,
Ivaylo
the Telerik team

[Nigel]

Hi Ivaylo,

Apologies for my own late response.  I was on vacation.
I can conform that the StopTestListOnFailure function was not checked.  As to test logs, I've already posted these above (the Test Execution log, and also the contents of the .aiilist file), as well as the actual test script as a zip.  Is there something else you need to see?

Thanks,
Nigel.

[Ivaylo]

Hello Nigel,

I hope you enjoyed your vacation.
Unfortunately we never had the Log file and by looking at your test we cannot see anything that might be causing the failure. Please kindly provide us with the log so we can see the actual error. Please check this short video in order to see what we need as files. It would be also helpful to record a short video for us with execution of the test list so we can see the actual moment where the test fails. You can use Jing for creating the video for us. I also want you to test something else. Could you please turn the annotations on for the test list and try to execute it again in order to see if that would pass. Please refer to the screenshot attached to see how to enable annotations for the test list.

Looking forward to hear from you.

Kind regards,
Ivaylo
the Telerik team

[Nigel]

Hi Ivaylo,

Okay, I'll do as you ask, though at the moment I'm firefighting another project, so it may be a while until I get back to you.

Thanks,
Nigel.

[Ivaylo]

Hello Nigel,

Sure, take your time. When possible for you please send us the log, and also try the annotation thing, this might slow the execution of the test list and pass the execution.

Looking forward to hear from you.

Greetings,
Ivaylo
the Telerik team

[Nigel]

Hi Ivaylo,

I've managed to squeeze a bit of time to follow this up.  I've attached a zip containing:

 - a screenshot of the Anotation setting for Test Lists
 - the project script
 -  the log file you requested
 - the swf file
Note that only a single file was created in the Results folder (there was no folder created, as was shown in the short video link you posted in your earlier reply).

You can also see the Jing video at this address:
     http://www.screencast.com/t/WGHWSmw2h

Hopefully this will allow you to see the issue clearly.  Let me know if you need anything else.

Thanks,
Nigel.

PS - I note that ScreenCast / Jing doesn't replay in IE9, but I checked that the video replays correctly in FireFox and Chrome.

[Ivaylo]

Hello Nigel,

Thank you for providing the files, I am able to see the problem in the video you have provided, however I cannot have it reproduced on our side so we cannot investigate further. Is it possible for you to create a new simple project with data driven test and if the issue is still present please provide the entire project so we can have a local repro.

Looking forward to hear from you.

Regards,
Ivaylo
the Telerik team

[Nigel]

Hi Ivaylo,

I fully understand about not being able to fix something you cannot reproduce.  I've seen my share of those.

I'll be in receipt of a new build to resolve a memory leak I raised a ticket against (515438) in a couple of weeks.  I'll wait until I get that build and see if the problem is still present.  If it is I'll do as you suggest and get back to you.

Thanks,
Nigel.

[Ivaylo]

Hello Nigel,

Thank you. Looking forward to hear from you.

All the best,
Ivaylo
the Telerik team