Hi Antony Selvaraj,
Yes, you would need a theme to make customizations related to the color states of RadDocking tabs. However, I assume you already have such since you are able to modify the mentioned brushes.
You have troubles with pinned an unpinned color states most likely because you have not updated all needed brushes.
As you mention in your post, please send us a sample app with your modifications and we can verify the proper functioning of the tab states.
the Telerik team
Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking
system and vote to affect the priority of the items