Check if your application has the <xhtmlConformance /> element in the web.Config file. If it is there and its mode is set to "Legacy", then you need to remove it in order for the ASP.NET AJAX controls to work properly.
If you do not have a <xhtmlConformance /> element in the web.config, then you will need to use a HTTP debugging proxy (Fiddler for IE or Firebug for Firefox) and check what is happening to the script HTTP requests when you press the Spell Check button. The files you need to monitor are WebResource.axd and ScriptResource.axd. If you see 404 (not found) or 500 (server error) for these requests, then there is a problem with the ASP.NET AJAX configuration in your application. Check if you have all the needed handlers and module definitions at the correct place in your web.config file.
the Telerik team