Frankly I doubt Test Studio has ever been tested on Windows 98. Even Microsoft support for it ended in 2006.
We recommend a minimum of Windows XP. Here is our page describing how to add dialog handlers.
All the best,
the Telerik team
Do you want to have your say when we set our development plans?
Do you want to know when a feature you care about is added or when a bug fixed?
Telerik Public Issue Tracking
system and vote to affect the priority of the items