In order to force the diagram to use a custom connection object you can override its GetConnectionContainerForItemOverride() method and return the custom connection. Here is an example for this.
CustomDiagram : RadDiagram
Please give this approach a try and let me know if it works for you.
Do you want to have your say when we set our development plans?
Do you want to know when a feature you care about is added or when a bug fixed?
Telerik Feedback Portal
and vote to affect the priority of the items