I am evaluating the studio to check whether it fits my needs. I have found a *lot* of issues, let me try to sum them up here:

Test runner

1. Test runner on different machine *must* use run as administrator (I get an error), still it continues to run. Quit the app if it must be executed as administrator. Also, *force* it to be started as administrator.
2. Configure as scheduler must also be executed as administrator.

Scheduler

1. The create database in the scheduler only works half of the time. In fact, I had to create an empty database and then it could be created. The server was not listed automatically, though other servers (that MS SQL Manager does not recognize) were found.
2. Start execution server at startup does not work (read: is useless). It must be executed as administrator.

Load test services

1. Load test services has no password field, it just shows the password there in clear text.
2. Load test services has different button sizes. I know this is low prio, but it gives me the feeling the product is not developed with care.

Test studio

1. The navigation inside the app is really hard to understand.
2. Why run as administrator?

ScheduleService API
We are trying to automate the scheduler service. What we do is create a client for the web service and schedule a new test for the next minute. Then we wait and we are able to see whether a test succeeds or fails. However, we cannot find any documentation on this service. Also, we cannot find any way how we can automate the retrieval of the log. In case of a failing UI test, we want to create a JIRA issue (we will automate this ourselves because it lacks JIRA support), but we do need the test log.

Can you please provide info on how we can retrieve the test log for a test that is executed on a remote machine? Just like the test studio does.

Thanks.

Hi Geert,
thank you for choosing to evaluate the Test Studio. We always appreciate user feedback and suggestions on how to make our product better so let me go through this stuff one by one:

Test runner

1. Test runner on different machine *must* use run as administrator (I get an error), still it continues to run. Quit the app if it must be executed as administrator. Also, *force* it to be started as administrator. - I don't entirely understand what you mean here. Is this in the context of Scheduling? What are you referring to when you say "Test Runner" - would you please give us some details. If you're referring to the ArtOfTest.Runner.exe - then yes - you will need to be logged in as an administrator.

1. Configure as scheduler must also be executed as administrator - yes, this is correct. Why do you feel that both of these are a problem? We've very interested in some details on this.

Scheduler

1. The create database in the scheduler only works half of the time. In fact, I had to create an empty database and then it could be created. The server was not listed automatically, though other servers (that MS SQL Manager does not recognize) were found.

Would you elaborate on this. In my experience creating the database is not complex. 90% of the time people usually install an SQL server with all the default settings. This means that 90% of the time you just have to type in
(local)\SQLEXPRESS
and check "integrated security" and you will be able do create the database without problems. I say that as someone who closely monitors Test Studio's tech support correspondence and then kind of problems our customers face.

1. Start execution server at startup does not work (read: is useless). It must be executed as administrator. - I was not able to reproduce this i.e. it worked when I tried it in my environment. Would you please give me the necessary steps to reproduce the problem. And also the environment and the Test Studio version you're using.

Load test services

1. Load test services has no password field, it just shows the password there in clear text. - are you referring to the Services Configuration menu and specifically to the SQL Authentication credentials? Excellent suggestion! I've added this as a bug: http://www.telerik.com/support/pits.aspx#/public/test-studio/13685

500 Telerik points added to your account.

1. Load test services has different button sizes. I know this is low prio, but it gives me the feeling the product is not developed with care. - could you attach a screenshot that highlights the buttons in question. Jing is my favorite software for this but a simple Windows screencapture + Paint will work too.

Test studio

1. The navigation inside the app is really hard to understand. - would you please elaborate on this. We honestly think our product is very intuitive so we're very interested in hear what you didn't like and how you feel it can be improved.

1. Why run as administrator?- Well for us to be able to hook into and control the browser (IE specifically) requires Admin level access. For us to be able to hook into and control the browser, (IE specifically I think) requires Admin level access. So if you want to be able to record or playback your test in IE - Admin security level is required. It's not required for anything else. And it's worth noting that we are evaluating the effort required to record in alternate browsers. Admin level access is required for the Visual Studio plugin.

ScheduleService API
We are trying to automate the scheduler service. What we do is create a client for the web service and schedule a new test for the next minute. Then we wait and we are able to see whether a test succeeds or fails. However, we cannot find any documentation on this service. Also, we cannot find any way how we can automate the retrieval of the log. In case of a failing UI test, we want to create a JIRA issue (we will automate this ourselves because it lacks JIRA support), but we do need the test log. - we have the Scheduling feature which will allow you to do all of this stuff without the need to access the Scheduling services yourself in code. When you schedule a TestList (through Test Studio's GUI) to run  on remote machine the results will be automatically fed back into your calendar view after the run has completed. So I don't see why would manually need to write code to get them. 

But if you absolutely want to do it - you will have to get the results from the Remote machine. The results are first generated in the Output folder on the remote machine and only then are fed back into the Calendar. The default location for the Output folder is C:\Temp\Output

Bug tracking is supported out of the box for TFS. But you can make it work with Jira too by writing a custom plugin as described here:
http://www.telerik.com/automated-testing-tools/support/documentation/user-guide/extensions/bug-tracking/custom-plugin.aspx

I hope this helps. When submitting support tickets/forum posts in the future please try to break them down a bit more. What I mean is people don't group together different problems under the same ticket/post. This messes with out ticketing system and virtually guarantees that you'll get a slower response then you would have otherwise.

Thank you for your feedback!

All the best,
Stoich
the Telerik team

Thanks for the reply. Next time I will split the post, but there were so many things, I didn't want to create 9 new posts.

Test runner
1) Well, the issue here is that you can run it not elevated. This means that it is possible to run the app without administrator rights. This however immediately results in an error (you must run as administrator), and another one (cannot read/write registry value). But, still it starts which is confusing. For now, I have fixed it by right clicking => properties => compatibility => run as administrator.

2) The issue with this is that we have several test machines (regular pc's) which we log on automatically. However, automatically starting the service is not possible (not running as admin). If we use the compatibility fix, we always have to press "yes" in the UAC dialog. A workaround is turning off UAC, but that is no option for me.

Scheduler
1) I agree creating a database shouldn't be hard. However, Create Database gives me the feeling that telerik will take care of creating the database. This is not the case (although I did create the user). Also, the first 2 or 3 attempts failed. The 4th time, it did work.

2) Same as Test runner - 2. We cannot get this to start without UAC intervention. I have added screenshots for this.

Load test services
1) Solved

2) See screenshot.

Test studio
1) Maybe, if other issues are solved, I will do this.

2) Ok, sounds fair. Maybe it's possible to start running as admin when we start recording? I think that's a better UX, but that's just my opinion.

Scheduler API
We don't want / need your UI to create issues. We have a CI server and we build a new nightly build. As soon as the build is done, we schedule a new UI test (via the web service). Then we wait for it to complete and we want to check the results (we know when a test passes or fails). However, we want to get the actual log. I will try to load it directly from the remote machine.

Hello Geert,
2) It's a known fact that UAC can get in the way of test automation with Test Studio. Is it possible for you to talk with your system administrators about this so you can set aside a few machine intended for test execution (i,,e with the UAC turned off).

Scheduler
1) I agree creating a database shouldn't be hard. However, Create Database gives me the feeling that telerik will take care of creating the database. This is not the case (although I did create the user). Also, the first 2 or 3 attempts failed. The 4th time, it did work. - would you please give me additional details here. Maybe you can record a video demonstrating the issue. Test Studio should create the database for you. You shouldn't need to manually create the database.


Load test services

2) See screenshot. - I'm still not sure what you're referring to unfortunately. Are we talking about the size of the OK/Cancel/Apply buttons

Scheduler API
You might be able to get the Execution server to store its results in a shared location. This will make it easier to get the results. You can configure the Output folder which is where the results are stored:
http://www.telerik.com/automated-testing-tools/support/documentation/user-guide/scheduling-test-runs/create-an-execution-server.aspx

Let's jump in a call and we can talk about the problems you're facing and what's the best way to address them. Let me know if this is an option for you.

Kind regards,
Stoich
the Telerik team

Hi,

Password visibility problem reminds me another issue I faced with.

I configured smtp notification for scheduling server with my own mail username/password. 

In DB [smtp_settings_db] table, my password is seen without any encryption. It would be nice if it is encrypted because DB admin may not be the same person who configured smtp server.

Thanks

Ercan

Hello Geert,
thank you for this suggestion!

I've logged this as a Feature Request:
http://www.telerik.com/support/pits.aspx#/public/test-studio/13774

500 Telerik points added to your account.

Regards,
Stoich
the Telerik team