This is a migrated thread and some comments may be shown as answers.

Actual Time is Much Less Than That Captured by Test Studio!

4 Answers 66 Views
General Discussions
This is a migrated thread and some comments may be shown as answers.
Kruba
Top achievements
Rank 1
Kruba asked on 22 Dec 2011, 02:15 PM
Hi,

I'm currently evaluating Test Studio - to use it as performance test tool in our company.

The performance test captured time for opening a page as 9.564sec but when I tried the same step manually, the page opened up in less than a 2seconds.
It is not the case with only this page but I see the tool adding some delay to all steps.
This highly impacts the performance report that is seen as a tool to improve the applications performance.

Also, I saw the CPU usage to spike up to 89% when using Test Studio for executing a test step.
But the same step when executing manually doesn't have much impact on CPU usage.

Kindly advice.

Thanks,
Kruba

Note: Attached config details of the machine on which Test Studio is run.

4 Answers, 1 is accepted

Sort by
0
Anthony
Telerik team
answered on 27 Dec 2011, 05:19 PM
Hello Kruba,

I believe this is a misconception on how Test Studio reports step times in Performance runs. Say you have the following test:

  1. Navigate to page
  2. Click button

The Navigate step occurs almost immediately, however this does not necessarily mean that the page is fully loaded. For that information, Test Studio relies on the Find Element time for the next step. That time is included in step one's time. So, in our example above, if the Navigation takes 2 seconds and the button takes 10 seconds to be found, then step 1 will report a time of 12 seconds.

We'll need more information to troubleshoot your second point:
  • What Test Studio process spikes?
  • Is it standard test execution or a Performance run?
  • What type of application is under test (HTML, Silverlight, etc.)?
  • Is it a specific step or the entire test?

Ideally we'd like to reproduce the issue locally. Please provide reliable reproduction steps against a publicly accessible site so we can see the behavior first-hand.

Greetings,
Anthony
the Telerik team
Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
0
Kruba
Top achievements
Rank 1
answered on 29 Dec 2011, 10:35 AM
Hi Anthony,

Thanks for your reply!

Please find below, my answers.
  • What Test Studio process spikes? - When the Performance test was run, we saw CPU usage go up, but not much when we ran through the same steps manually.
  • Is it standard test execution or a Performance run? - Performance Run
  • What type of application is under test (HTML, Silverlight, etc.)? - Silverlight
  • Is it a specific step or the entire test? - The time delay is across the test

  • I've attched the screenshot of the results and would require few inputs from you.
    1. How to calculate the Time Taken?

    If you look at Step 4: Time for Login shows 26.638s and 1.298s for Step 5 where it waits for the page to load and thus find an element.
    But, according to your reply the time for step should also be calculated as part of Step 4 - please correct me if I misunderstood.

    2. If possible, can you please do a manual run of the steps and compare the time taken - time delay will be very much evident.

    3. The size column displays the total data transfer during the step execution - is there anyway we can get a break down wrt Server and client?

    4. If I'm running a Performance Test List, is there any way I can have the test to continue from one test to another without closing the browser?
    For example: If I break the attached step into 2 parts namely Login and page verification. Is there a way where in I add these two as part of a list and run it continuously? The first test(Login) used for login and second test(page verification) continues from there in the same browser.

    Thanks for your time!
    Do let me know if you would require more information.

    Regards,
    Kruba
    Note: I've attahced all the required information.

    0
    Anthony
    Telerik team
    answered on 29 Dec 2011, 06:59 PM
    Hello Kruba,

    Thank you for the files.

    1. My previous explanation was in regards to an action step, where a built-in "wait for exists" always occurs first. In this case, step 5 is a Wait step itself and not an action step. The time taken to find the LogOutGrid element is added to step 4, and the 1.298 seconds reported in step 5 is the time between when it was found and when it became visible.

    Keep in mind that Waits and Verifications don't really have a place in Performance testing, as explained here, and it's hard to interpret those types of steps in this context.

    2. I did not see a dramatic difference in times between running your test with and without Test Studio. 

    I first ran through the steps in IE, outside of Test Studio, and saw the CPU usage spike as high as 80% at certain times. When run through Test Studio the average CPU usage was slightly higher, which I expected due to the additional automation overhead. The maximum usage never exceeded what was seen when run manually, however. 
     
    3. The Size column represents the incoming data size received from the server. No client information is included.

    4. Set RecycleBrowser to True on the Web tab of Test List Settings.

    Greetings,
    Anthony
    the Telerik team
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
    0
    Kruba
    Top achievements
    Rank 1
    answered on 04 Jan 2012, 04:58 AM
    Hi Anthony,

    Happy New Year!
    And thanks for your reply!

    Things look more clear to me now, than it was before.
    Just a few test runs for a week or two and I should be in a state to complete my POC.

    Thanks for your continuous support!

    Regards,
    Kruba
    Tags
    General Discussions
    Asked by
    Kruba
    Top achievements
    Rank 1
    Answers by
    Anthony
    Telerik team
    Kruba
    Top achievements
    Rank 1
    Share this question
    or