The described behavior is expected. Through the options there is only the meta data of the custom action existence. However, after setOptions, widget is re-rendered. Therefore, any events bound and classes added are removed after the DOM operation. This is a normal behavior when DOM elements are recreated.
The solution is to process the custom action in the same way as initially done:
"Custom action button clicked"
'Not working after setOptions. Icon is also changed.'
Telerik by Progress