Remote execution server - read from dll

[Jerne]

Hello,

I am having a problem running my test lists on remote execution server. Let me say first that tests work correctly and without problems localy, but the server is unable to execute those test that include or have included custom coded steps. (my observation)

It seems like test studio cant access AllHoursTest.dll. 

I am getting an error, which is posted bellow.
------------------------------------------------------------
'31.7.2014 10:26:51' - Executing test: '4.4. Team and employee filters are working', path: 'Whole Tests\4. Presence View\4.4. Team and employee filters are working.tstest.'
'31.7.2014 10:26:51' - Using .Net Runtime version: '4.0.30319.18449' for test execution. Build version is '2014.2.618.0'.
'31.7.2014 10:26:51' - Starting execution....
'31.7.2014 10:26:57' - Detected custom code in test. Locating test assembly: AllHoursTest.dll.
'31.7.2014 10:26:57' - Assembly Found: C:\Users\Administrator\AppData\Local\Temp\2\Projects\edac5f4b-18d8-4838-89df-56a91199366d\Test.Studio.AllHours.Tests\bin\AllHoursTest.dll
'31.7.2014 10:26:57' - Loading code class: 'AllHoursTest.__4__Team_and_employee_filters_are_working'.
'31.7.2014 10:26:57' - Failure detected during execution. Details:
------------------------------------------------------------
'31.7.2014 10:26:57' - System.ArgumentException: Unable to find the test class type 'AllHoursTest.__4__Team_and_employee_filters_are_working' in your test assembly. Please make sure your test code behind is compiled with your latest changes.
   at ArtOfTest.WebAii.Design.Execution.ExecutionUtils.EnsureTypeExists(Assembly assm, String typeName)
   at ArtOfTest.WebAii.Design.Execution.ExecutionUtils.CreateCodedTestInstance(Test test, TestResult result, String binariesFolder)
   at ArtOfTest.WebAii.Design.Execution.ExecutionEngine.InternalExecuteTest(Test test, TestResult initializationResult)
   at ArtOfTest.WebAii.Design.Execution.TestExecuteProxy.ExecuteTest(ExecuteTestCommand command)
------------------------------------------------------------
'31.7.2014 10:26:57' - Test completed!

[Boyan Boev]

Hi Jerne,

If you use a custom dll please copy this it to the root folder of the project and reference it from that location. Note this will work only if you put the dll in the root directory as it is described at the bottom of this article. Test Studio will not find references to a DLL in the subfolders of the project folder. These include: bin, Properties, obj, Data, Backup, TestResults, Results, TestLists, and Profiler Configurations.

Let me know if this helps.

Regards,
Boyan Boev
Telerik

 

[Jerne]

Thank you for your answer, Boyan.

While your post was helpful and informative, it wasnt the solution. The problem wasnt even in test studio.

Regards,
Jernej

[Konstantin Petkov]

Hello Jernej,

Since this is a forum thread can you please share how you managed to solve the problem so that others can benefit from that as well? Thanks in advance!

Regards,
Konstantin Petkov
Telerik

 

[Jerne]

Hello, Konstantin,

I noticed the problem in test names. My tests were named with numbers, for example : (1.1, 1.2, 1.2.1, etc). Somehow either in test studio remote exectuion or when uploading  on SVN names seemed to get messed up, and replaced first numbers as _. This lead to some test names beeing the same as others, and some references not found.
My solution was to rename tests without leading numbers.

Regards,
Jernej

[Konstantin Petkov]

Hello,

Thanks for sharing that!

Regards,
Konstantin Petkov
Telerik

 

[Muhammad]

Hi,
I am facing same issue when running test lists remotly , is there any other solution insted of changing script name?

[Jerne]

Hey Muhamman, 

Me and my colleague were trying several things, but only renaming seemed to solve the problems. We removed numbers from test names and that solved it.
It only happened to us when we were trying to execute some of the test that included coded steps. 

[Cody]

Hello,

When Test Studio creates a coded step, it has to create a method name for that coded step. It uses the name of the test as part of the method name. For example, a test named "Login" will generate coded steps like:

Login_CodedStep
Login_CodedStep1

And so on. In code, method names are not allowed to start with numbers, nor are they allowed to contain special characters like ".". As a result Test Studio has to do something when you have a test named "1.1". It will generate a method name "_11_CodedStep".

Thus renaming your tests is really the only solution if you want your coded steps to work right. The easy thing to do is to add some constant character to beginning of every test name e.g. "T1.1" or "Test 1.1". Also I recommend you replace all "." characters with "_" characters e.g. "T1_1" or "Test 1_1". This will eliminate the illegal characters in method names for coded steps.

Regards,
Cody
Telerik

 

[Jerne]

Hello Cody,

Thanks for the informative reply. I still find it very weird that all the tests actually worked untill trying remote execution. Even if ran locally, the test generate coded steps, so it should fail even locally. Or am i missing something?

Also, were using Jenkins as our test scheduler and builder. The problem was also if we didn't first run the test remotely with test studio on our server and then with Jenkins. If we did it the other way, tests with coded steps failed and the only way we could get it to work was by renaming coded steps. This seemed to cause Test studio to realise it needs to upload the whole test again.

[Cody]

Hi Jerne,

You're right we would expect the test to fail locally and in the same way as remotely.

Question for you however: If you're using Jenkins, where are you "uploading" your files to? I'm referring to this statement: "This seemed to cause Test studio to realise it needs to upload the whole test again". I don't understand where Test Studio would be uploading the tests to in a Jenkins build environment?

Regards,
Cody
Telerik

 

[Jerne]

Hey Cody,

Perhaps uploading was not a correct word to use. All project is already saved on server on which jenkins is used to run windows scripts that run tests. I have the project opened on my PC and edit it there, but all the changes are saved to the server. 
What i wanted to say with this "uploading" word is just me guessing that test studio somehow figures there has been a change in project when renaming test steps or projects, and then it updates the database. This change in DB is what i referred to with "uploading". Sorry for the misunderstanding.

There was another thing i found out today, is that not only does T.Studio remote execution not allow test names starting with numbers. For example, Test3_test is the same as Test_test and gives an error. 

The problem with this error report is that i'm still trying to figure out why and how this issue appears. For now, all i can do is give you reports on what i've found to solve some of the problems. If i figure out how exactly to recreate my problem, i will be sure to post it.

Regards,
Jernej

[Cody]

Hi Jerne,

test studio somehow figures there has been a change in project when renaming test steps or projects, and then it updates the database.

I'm still trying to figure out what you're referring to here. There is no DB (as in a SQL DB) that Test Studio uses. There is only a large set of files:

• .tstest files - these hold the actual tests, including the element definitions used by that test
• .cs/.vb files - these hold the code used by any coded steps the tests may contain
• .resx files - these hold the images you see in the storyboard view
• Pages.g.cs/vb - this is a generated file. It contains element definitions of all elements used by all tests. Coded steps may reference the elements it contains rather than manually hand coding finding of elements on a page. This file is generated by scanning and pulling all element definitions contained in all tests contained in the entire test project.
• Settings.aiis - this file contains project wide settings

Given the above, can you clarify what "DB" you are talking about?

For example, Test3_test is the same as Test_test and gives an error.

I just tested this but do not find a problem:
http://screencast.com/t/L1eyH6gq89Q

Am I doing something different? Please let me know.

Regards,
Cody
Telerik

 

[Jerne]

Hello Cody,

The problem i am having is when running tests on remote execution server. Not locally. Also, for the naming issue, its not names for projects, it is when naming tests inside projects that include coded steps. Those tests sometimes have issues running on remote execution server.

As for the DataBase, i was just speculating what happens. Are you sure that telerik isn't running a DB? For example in .\Telerik\Test Studio\StorageService, you have a config file where you can set DB name. .\Telerik\Test Studio\StorageService\logs also shows changes made to SQL DB.

Regards,
Jernej

[Jerne]

Hello, 

I didn't find the option to edit the upper post. please note that the steps were correct, only thing is that all tests work locally and have troubles working REMOTE. 

Check the upper post as well.

[Cody]

Hi Jerne,

Because of this statement "Also, were using Jenkins as our test scheduler and builder" it led me to believe that you are not using the Test Studio Scheduler or remote Executor. Is this true? Basically you can use either our Scheduler or use Jenkins. The two are incompatible and cannot be used together.

You are correct that our Storage server (which our Scheduler server and remote Executor rely on) does connect to and store tests into a SQL DB. As I just stated above, a Jenkins CI setup will not use this DB.

So please confirm, are you using Jenkins or our Scheduling server?

for the naming issue, its not names for projects, it is when naming tests inside projects that include coded steps.

Understand and that's exactly the scenario my video demonstrated. Once you confirm which mechanism you are using for running tests remotely, I'll test that out here as well to try and reproduce the problem.

Regards,
Cody
Telerik

 

[Jerne]

Hello Cody,

I'm sorry for my bad explanation. We are using Jenkins for scheduling and remote executor i believe. No studio scheduler. Steps in your video were OK.

I will post out actual setup as soon as our server administrator gets to work, should be in couple of days. I  think we are using 2 servers, one that is running Jenkins and connects to other one which runs remote executor. I have difficulties checking our actual architecture being a student at the company and not having privlliges to access the server and check it out myself.

Regards, 
Jernej

[Cody]

Hello Jernej,

Please do post your setup. That will help me understand how your system is put together and I may be able to then understand how/why you're running into the naming problems you describe. Thanks!

Regards,
Cody
Telerik