The described Menu behavior of opening a hovered item if there already is an opened item is by design. It has been discussed previously on the forums, you can see Kamen's post in this
thread for instance, which explains the behavior in different scenarios.
The openOnClick option prevents an item to be opened on hover only if all items are closed. Currently there is no alternative to the openOnClick configuration option that would allow you to prevent the items opening on hover when there is one opened. If you want to have the ability of customizing this behavior we can suggest logging this as a feature request in our public Feedback Portal
Telerik by Progress