Well now. It seems that I'd been a little careless as to what went into the assembly.
Reverting back to silverlight 3 did not resolve anything, it just moved the problem elsewhere.
I stumpled across this article:
While I was looking another way, some silverlight 3 dll's had snuck their way into my GAC. And this was the root of my agony.
Removed all dll's with version 126.96.36.199 and everything is back to normal again.