I setup the Executive Dashboard for our environment and have been looking at it over the last few days and I have a few thoughts and suggestions.
1. This could be a bug but, if not the documentation should be updated to include that the Scheduling Server Settings needs to be set to Run Remotely for this to work. If you are running locally you set it to Run Remotely and put in the server name of Localhost. I tried for a couple of hours to get it to work while I had it set to Run Locally and did not get anything in the Executive Dashboard, it was only once I changed it to Run Remotely that I started getting information in the Executive Dashboard.
2. It would be nice to have the Executive Dashboard run under Https.
3. On the Test Run List page (Where it lists each of the test runs) it would be nice to have a listing of how many tests ran, How many Passed and How many Failed. I could have a test run of 1000 scripts and 1 failed and another that ran 1000 scripts and 999 failed, from this view I would not see a difference in the 2 runs.
4. On the Test Run Details (Where it lists the actual tests) this needs to be configurable as to how many items are view able at a time. 5 is not sufficient for our needs, it just adds a whole bunch of pages that no one is going to click through. A selection to view 5, 10, 25, All would be handy.
5. On the Test Run Details (Where it lists the actual tests) a filter would be nice to be able to filter to see only the failed, only the tests that rerun, only passed, etc... would be nice. That way you do not have to weed through everything to find the failures or reruns to see what the problem was.
Overall this is a great addition to be able to show our testing effort. With a few tweaks this would knock it out of the park!
1. This could be a bug but, if not the documentation should be updated to include that the Scheduling Server Settings needs to be set to Run Remotely for this to work. If you are running locally you set it to Run Remotely and put in the server name of Localhost. I tried for a couple of hours to get it to work while I had it set to Run Locally and did not get anything in the Executive Dashboard, it was only once I changed it to Run Remotely that I started getting information in the Executive Dashboard.
2. It would be nice to have the Executive Dashboard run under Https.
3. On the Test Run List page (Where it lists each of the test runs) it would be nice to have a listing of how many tests ran, How many Passed and How many Failed. I could have a test run of 1000 scripts and 1 failed and another that ran 1000 scripts and 999 failed, from this view I would not see a difference in the 2 runs.
4. On the Test Run Details (Where it lists the actual tests) this needs to be configurable as to how many items are view able at a time. 5 is not sufficient for our needs, it just adds a whole bunch of pages that no one is going to click through. A selection to view 5, 10, 25, All would be handy.
5. On the Test Run Details (Where it lists the actual tests) a filter would be nice to be able to filter to see only the failed, only the tests that rerun, only passed, etc... would be nice. That way you do not have to weed through everything to find the failures or reruns to see what the problem was.
Overall this is a great addition to be able to show our testing effort. With a few tweaks this would knock it out of the park!
1 Answer, 1 is accepted
0
Hello Kris,
Thank you for sharing your feedback on the new feature - Executive Dashboard. I highly value your input and we will try to implement all of those suggestions in our future releases. Please find more details about our plans and how you can check the progress for each feature request.
- I added some additional information to make it clear that local executed test lists will not be published to the storage server automatically. You can, however, publish them to the server manually from the Results tab. The documentation will be updated on Tuesday/Wednesday next week with some additional details and features as well.
- We also think that this can be a good improvement to the Executive Dashboard and I created a public feature request on your behalf. We will try to include them in our first major release of 2020, but I can not promise that for sure. Please keep an eye on the feature request and follow it for more details.
- You are right that it will be very useful to know what portion of the tests failed in each test list. This will require a bigger change and we need to make sure that we don't break the existing results. Our engineering team will look into it more closely and try to find the best solution. There is a separate feature request in our public portal that you can also follow and vote fore.
- This was actually requested by another user and we already did some changes to allow the choice of 5, 10, 15 and 20 tests per page. This selection will be saved per browser and will apply for all test lists and projects. Our major concern to limit the number of tests per page is that it might take a lot of time to load and become unusable. You will be able to test it next week after we release the service pack on Tuesday/Wednesday and I will be happy to hear your feedback again.
- You are right that there is no filter on that page. I will advise you to use the current sorting options that allow you to sort by multiple criteria at the same time. For example, sort by Status to see the Fail/Pass on top and add sort by Pass/Total in addition to that for more precise overview. Please let me know if that will be enough to help you filter the results.
I am happy to know that you find the Executive Dashboard helpful and I hope that we will make it even better with time. Thank you once again for your feedback.
Regards,
Plamen Mitrev
Progress Telerik