Thank you for contacting us.
I tested the described scenario fully but was able to reproduce it only in our online demos, when the "Generate New Image" is clicked too soon after the "Set" button of the configurator. If this is the scenario followed by you, the reason for this issue is that clicking over the "Generate New Image" button cancels the ajax requesting updating the Captcha with the passed character set, resulting in not applied custom character set at all.
Testing it on a separate project, though, when the custom character set were applied on the server, I was not able to reproduce the reported problem. I am attaching my test page to this rely so you can examine it on your side and see whether it differs somehow from your own configuration.
Looking forward to receiving a reply from you,
Do you want to have your say when we set our development plans?
Do you want to know when a feature you care about is added or when a bug fixed?
Telerik Feedback Portal
and vote to affect the priority of the items