Managing Environmental Settings

[ColinBowern]

I'm looking at the workflow between developer, tester and CI server to come up with an approach for maintaining environmental specific settings (local dev box, dev/test environment, qa environment, production) such as:

• Base URL
• Username / Password by Actor / Role
• Key Data Values 

I'm wondering how others are tackling this issue?  I see a number of disconnected techniques such as the VS test settings file, data sources, and test list settings.  There seems to be a lack of a universal "project file" across the tool sets which doesn't help either.  What I would like to be able to do is:

• Developer - selects a "settings/parameters file" in Test Studio Express with environment parameters
• Tester - selects a "settings/parameters file" in Test Studio Standalone with environment parameters
• Build - execute runner with a specific "settings/parameters file" that contains environment parameters

Within the test then I can say "Navigate to {BaseUrl}/Reports", "Login as {Regular User}", etc... Without having to have three different types of files per environment what is the best strategy for managing this type of information?

UPDATE: I've added an item to the feedback portal to capture this gap in functionality.

[Cody]

Hello Colin,

My recommendation for your case is to take the Data Driven approach. You can data drive the NavigateTo steps, all the user name and password input, plus your key value pairs (the key being the column name and the value being the data in the row). Put all your environment settings into a spreadsheet, but use different spreadsheets in your different environments. All 3 environments would look for the same named spreadsheet, but give it different content in each environment. Does this approach sound workable to you?

All the best,
Cody
the Telerik team

[ColinBowern]

Correct me if I'm wrong - the consequence I see with that approach is that I need to connect each test to the data source. With only one data source per test I'm left with a giant spreadsheet containing every possible combination of values if I ever want to use data driven testing properly.  Also this means I have to remember to manage that association across my entire test suite. I understand that it can be done but it needs to scale across a real world test suite not just a hypothetical demo example.  Maybe I'm missing something in how to structure our test suite? (We're new to Test Studio :))

[Cody]

Hello Colin,

I apologize for the delay getting back to you on this.

I admit it's difficult to say how hard it would be to implement my suggestion not knowing any real details about your environment and how many variables/settings you need. I can tell you that multiple tests can be bound to the same data source at the same time e.g. you have an excel file "Environment.xlsx" and tests A, B, C can all be bound to this same file all concurrently. Granted you have to put all environment variables into one spread sheet. Usually this is just one row with around 10-20 columns/values.

Another option would be to use code and read the Windows environment variables and make the test use those. Of course this assumes you set those environment variables in Windows prior to running the test.

Regards,
Cody
the Telerik team

[ColinBowern]

Cody - here's what I'm thinking...

ArtOfTestRunner.exe list=MyTestSet.aiilist variables=LocalDevEnvironment.xml
ArtOfTestRunner.exe list=MyTestSet.aiilist variables=TestEnvironment.xml

The variables document is a key-value dictionary made available to all tests.  Maybe there is a global variables properties piece that allows you to define default values to make it easier to run in the IDE?  Or maybe the variable set can be chosen within the IDE (similar to how your platform configuration is configurable in Visual Studio - Debug, Release, etc...).

[Cody]

Hello Colin,

We do already support a "settings=" parameter from the command line. It was meant to load the settings that our runner recognizes and would configure the test to use. I am imagining that you could add your own values to the same file, using code to access the data and get it that way.

Following your approach "variables=LocalDevEnvironment.xml" how is this significantly different than "With only one data source per test I'm left with a giant spreadsheet containing every possible combination of values"?

Regards,
Cody
the Telerik team

[ColinBowern]

I'll try adding a few values to that file.  How would that translate to the tester working in the IDE though?

Using the data source would prevent me from using data-driven testing features which are useful in some of the test scenarios. To me environment parameters which are applied globally to all tests and data sources for data driven testing are two different scenarios.  Yes you could use the data-driven testing features to manage environmental parameters, but it would also increase the maintenance burden when testers use the DDT features.

[Cody]

Hello Colin,

How would that translate to the tester working in the IDE though?

I admit it wouldn't. It's a possible method to use given the features included in the product today.

To me environment parameters which are applied globally to all tests and data sources for data driven testing are two different scenarios.

Very good point. I've created a new feature request around this concept. Let me know if I missed anything in the description.

Kind regards,
Cody
the Telerik team

[Ray]

I posted this on the feedback item linked also. I would be able to work around the issues caused here if I could access environment variables from code. These would probably be useful:

BASE_URL
WEB_USERNAME
WEB_PASSWORD
CURRENT_TEST_LIST - the name or NULL, if not running a test list.
CURRENT_TEST - the name
CURRENT_STEP - the name or NULL, if not in a step.
BROWSER - the name
BROWSER_VERSION - the name and the version.
OS_VERSION - the name and the version.
HOST - the name
HOST_IP - the active network interface

 

[Elena]

Hello Ray,

Since the post and request you refer to date back six years ago, I guess that the desired scenario can be covered with the functionalities in Test Studio nowadays. 

Here is what I have in mind - you can prepare a standalone code file in the Test Studio project, which will be accessible for all tests in the current project. This file can contain any type of custom code, which you can then reuse in coded steps in each test. You also can include reference to any external dlls in the project and use these in the coded files. 

Then you can reuse these global variables from the Helper standalone class file across the tests - the only limitation is that you cannot use them in the UI, but need to get the variables in code. Still, if you would like to keep the visual representation in the UI, you can use Get and Set ExtravtedVariable() methods in coded step, and then assign the variable in the UI. 

Moreover, during the test execution you can access the ExecutionContext class in a coded step and take some of the data in real time. Along with the Manager class you can have access to all of the listed information. 

Then, to cover any possible scenarios, you also have the ability to design an execution extension if you need to customize some actions before and after test or test list run. 

I hope these shared details will be helpful for you. However, if I have missed anything in the scenario you need to cover, or have misread any of the requirements, do not hesitate to get back to me with further details on the case and we can continue the discussion. 

Thank you in advance for your cooperation and understanding. 

Regards,
Elena
Progress Telerik

 

[Ray]

Thank you for your response, Elena.

I think that your information will help me. If I was more expert in C# and Windows, I might have known some of this.

I think that all of the things I asked about are accessible, except for the current test Test List. There are methods and fields for steps and the the test and those are helpful.

I will see if I can see something in the environment that tells me about a test list.

Thanks, Elena, for your time.

- ray

[Elena]

Hi Ray,

I am glad to know that provided resources will help you to get familiar with the C# coding in Windows - probably it will take you some time to switch to the Windows environment and its features, but it's never late to gain new experience. 

When you get to this, you can review the available methods in the execution extension, which you can override. The OnBeforeTestListStarted() method uses the test list as an argument and this could be what you are looking for. 

Of course, if you encounter any scenario that you have difficulties with, you can contact us back with details on the requirements and we can find a solution together. 

Thanks for your continuous cooperation in our discussions. 

Regards,
Elena
Progress Telerik

 

[Ray]

It will probably seem odd for me to say this, but I do not have access to VS, which would be required to build a Execution Extension.

Indeed, it seems that the OnBeforeTestListStarted()method would be helpful. I will see what I can do.

thanx - ray

[Nikolai]

Hi Ray,

I know that this might not be as easy as writing extensions in classic VisualStuido, but you can use VisualStudio Code to write and compile C# code.
The tool is free and works on all platforms. 
Here is how to do it:
1. Download VisualStudio Code
2. Download MsBuild - Build Tools for Visual Studio 2019
3. Open the attached project in Visual Studio Code
4. Run the 'build' task -> if the task fails because 'msbuild' command is not recognized you'll need to add it as an Environment variable.


We will log your suggestion as a 'Feature Request', so 'OnBeforeTestListStarted' and 'AfterTestListCompleted' methods can be over-ridden inside TestStudio itself.


Regards,
Nikolai
Progress Telerik