In order to call Silverlight method from Ajax, you'll need to make Html to Silverlight bridge, which is well documented on msdn here
About the second problem you may use RadWindow for Silverlight and won't have this layout problem. You can customize it as well to like normal ChildWindow as well.
the Telerik team
Do you want to have your say when we set our development plans?
Do you want to know when a feature you care about is added or when a bug fixed?
Telerik Public Issue Tracking
system and vote to affect the priority of the items.