Exception when calling ORM through WCF (as of 2012 Q1 SP1)

[Kevin]

Ever since I updated to 2012 Q1 SP1, perhaps co-incidentally, I have been getting the following exception in my production environment (not in dev):

--------------------

System.ServiceModel.FaultException<MyApp.CompanySvc.UnhandledServiceFault> was caught
  Message=An exception occured during the execution of '
Extent<MyDAL.Companies.Company>().Where(c => (c.OrganizationId == value(MyServices.CompanyService+<>c__DisplayClass0).organizationId))'.
Failure: Type is enhanced and registered, but not available from the database class meta data. This can be caused by a wrong connection id or configuration.
Parameter name: src
Actual value was MyDAL.Companies.Company.

  Source=mscorlib
  StackTrace:
    Server stack trace:
       at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
       at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
       at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
       at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
    Exception rethrown at [0]:
       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
       at MyApp.CompanySvc.ICompanyService.RetrieveCompanies(Int32 organizationId)
       at MyApp.CompanySvc.CompanyServiceClient.RetrieveCompanies(Int32 organizationId) in C:\MyApp\Service References\CompanySvc\Reference.cs:line 1635
       at MyApp.MainWindow.OnGetCompanies(Object sender, RoutedEventArgs e) in C:\MyApp\MainWindow.xaml.cs:line 23
  InnerException:
--------------------

I am calling OpenAccess from my business layer, which is called by my UI through a WCF service layer over TCP. When I run it all on my dev box, everything is fine but I get the exception after I deploy to production. I've checked the configuration in a multitude of different ways and don't see any striking difference.

I have read all the posts related to this exception but none seem to fit my situation. I have only one domain model and there is only one executable calling it. I ran iisreset on the server but that did not fix the problem. Most recently, when I finally got it to work, I had reset IIS directly from the IIS Management console, but I've had it work before only to be thwarted by the exception the next time. Besides, I have not had to go to such lengths before I started deploying with SP1.

Can anyone suggest what is going on here?

Thanks,
=Kevin=

[Thomas]

Hello Kevin,

this could be because an older version of the OpenAccess runtime is in the GAC of the deployment machine; check that you deploy all assemblies (Telerik.OpenAccess.dll, Telerik.OpenAccess.Runtime.dll, Telerik.OpenAccess.35.Extensions, ...).
The exception essentially says: I've found a type that you were looking for, but it is not enhanced, so not correctly build. The question is how did this unenhanced assembly get deployed?

All the best,
Thomas
the Telerik team

[Kevin]

Thanks for your reply, Thomas. I checked that all the DLLs being deployed were correct (I deploy them with the executable and don't put them in the GAC on the production machine).

It turns out that something in the installation was corrupted. I didn't see anyone else having problems with this, so I decided to roll back:

• first, I uninstalled SP1;
• then re-installed the original 2012 Q1 version;
• then re-installed SP1

Now everything works as expected!

=Kevin=

[Ivailo]

Hello Kevin,

What was the OpenAccess version you had installed before the Q1 SP1 initially? I am interested because this would point us the direction to research what might have happened so that we prevent such issues in the future.

Thanks in advance for your feedback.

Regards,
Ivailo
the Telerik team

[Kevin]

Hi Ivailo,

First of all, the problem itself was not resolved--when we started up today, we started seeing the same exceptions as I reported before. :o(

My next step is to roll back to the previous version, which is 2012.1.214.1, and re-deploy.

We are not ruling out the possibility that it's a failure on our part, but we were running the first Q1 release without problem. I will let you know how that turns out.

Thanks,
=Kevin=

[Ivailo]

Hello Kevin,

How is your system behaving after re-deploying it with the Q1 release? Are there any issues or the problem has disappeared? We are going to assist you in resolving this problem if you still want to upgrade - there are some useful bug fixes in the service pack you might want to take advantage of.

Greetings,
Ivailo
the Telerik team

[Kevin]

Hi Ivailo,

Since rolling back to 2012.1.214.1, the problem does seem to have disappeared.

What can be done to help us upgrade to SP1?

Thanks,
=Kevin=

[Ivailo]

Hello Kevin,

First, there are a couple more possibilities to explore:

1. When deploying with OpenAccess Q1 again, have you replaced any configuration files, or just the recompiled DLLs? I just want to make sure that only the difference in OpenAccess DLLs is causing the problem.
2. Can you verify that the Q1 SP1 release you have downloaded and installed is the full version of OpenAccess or a trial release?  I am thinking in this directions since the trial stops working if you haven't recompiled your data access assemblies for some time which looks like it can have something to do with your experience with the service pack.

In general, there are two ways to proceed and analyze the issue deeper:
A) Is it possible for you to send us the WCF service project and the OpenAccess class library? We can try to deploy it on a test environment here and check if we can reproduce the issue this way. If you can send us the projects, please attach them to a new, private support ticket so that they will not be visible for other people.

B) You can download our latest SDK browser, find a sample using the same kind of user interface and WCF service type (let us know if there isn't any matching sample), try to deploy on the same target server and let us know if any issue is reproducible this way.

I am looking forward to your feedback.

All the best,
Ivailo
the Telerik team

[Jakkie Esschert van den]

We actually ran into the same issue on our production machine and test machine. But I can't find a way to reproduce this consistently.

We upgraded from 2011Q3 to the latest 2012Q1 (2012.1.301.2). After upgrading the production machine we got this error on the website and the WCF service. I then touched the web.config to make the website restart and the error went away. This morning however it was back after the AppPool recycled. Again touching the web.config made it go away.

The version deployed is build on our build server which doesn't have OpenAccess installed but uses files from a library directory (which is fully updated to the latest version). I also deployed this version on our test server where I didn't see this error yesterday, although I did find it in the log files as the result of WCF calls from an external service.

The stack trace logged looks like this:

System.InvalidOperationException: An exception occured during the execution of '
Extent<Webservice.Model.ScheduleTask>()'.
Failure: Type is enhanced and registered, but not available from the database class meta data. This can be caused by a wrong connection id or configuration.
Parameter name: src
Actual value was Webservice.Model.ScheduleTask.
---> System.ArgumentOutOfRangeException: Type is enhanced and registered, but not available from the database class meta data. This can be caused by a wrong connection id or configuration.
Parameter name: src
Actual value was Webservice.Model.ScheduleTask.
at OpenAccessRuntime.QueryBuilderImp..ctor(ObjectScope sc, Type src, Type trgt, String lang, Object exp)
at OpenAccessRuntime.ObjectScope.GetQueryBuilder(Type src, Type dest, String lang, Object exp)
at Telerik.OpenAccess.Query.ExpressionCompiler.PerformDatabaseQueryImpl(Type type, Int32 elementAt, Object[] groupResolutionParamValues, Boolean single, Boolean checkOid)
  at Telerik.OpenAccess.Query.ExpressionCompiler.PerformDatabaseQuery(Type type, Int32 elementAt, Object[] groupResolutionParamValues, Boolean single, Boolean checkOid)

I got the errors on different objects, but the stacktrace is always basically the same.

I'm trying to find a way to reproduce this, but I haven't found it yet. It does seem to be related to the initialization of the application though.  We do use a multi-tenant setup where we use the same model on separate databases for each tenant, so I could be the Database instances getting mixed up. 

[Jakkie Esschert van den]

It just went down again, but without anything restarting. It looks like a call which came in through the net.tcp endpoint triggered the error. So it could be something specific to net.tcp. I am going to revert to the previous version of OpenAccess on the production machine, but I'll leave this version running on the test box to see if that stays up properly.

[Ivailo]

Hi Jakkie,

Since it does sound like a specific issue for a particular setup, I would kindly ask you the same - if you can either send us a stripped down version of the failing service in a private support ticket for investigation, or alternatively try a sample WCF service from our SDK (so that we have it here already) and let us know if it leads to the same exception.

Additionally, if you believe that the issue can have something to do with the application initialization  you can also send us the initialization code and the handling of your OpenAccess context so that we can verify them and try to reproduce the possible issues on our test environments.

Regards,
Ivailo
the Telerik team

[Jakkie Esschert van den]

I'll try to create a small project which has the same issue. This could prove to be rather hard as the issue doesn't occur very often, but when I succeed I'll create an issue for it.

I did figure a few things out though, which I'll try to explain.

To initialize the model we have a factory method in our context class which create a new context based using the connectionstring associated with the current user like this:

public static SpaasModel CreateNew()
{
    BackendConfiguration backend = new BackendConfiguration();
    // Backendtype is always mssql.
    backend.Backend = backendType;
    // Get the connectionstring for the current tenant.
    string connection = SecurityManager.ActiveTenantConnection();
    SpaasModel result = new SpaasModel(connection, backend, metadataSource);
    // Attach audit events to the scope.
    Audit.AddObjectScopeToAudit(result.GetScope());
    return result;
}

After the application pool recycled we had a scenario where one tenant we got 'Type is enhanced and registered, but not available from the database class meta data' errors on all database actions while all other tenants worked correctly. 

We run a windows service on the same machine which connects over net.tcp. When this service looses it's connection to the WCF service it tries to reconnect immediately and starts reloading the data it needs. This means this service is bound to be the first to run into this issue when it occurs, so I can't be sure it's related to net.tcp.

The service is multi-threaded it will connect and loads data for different tenants concurrently (over separate connections), which means the initialization code above is likely to run in parallel for different connection strings when the service reconnects, which might trigger this issue as well. Perhaps the wrong Database instance gets associated with a connectionstring when databases are initialized in parallel. 

If I find a way to reproduce this consistently I'll let you know.

[Jakkie Esschert van den]

I've been able to reproduce the issue in a test project. I've created ticket #528831 and attached the project. I've also uploaded the project in case anyone else is interested.

The error is indeed triggered when the same model is being initialized using different connection strings in parallel. When the context creation is done sequentially it doesn't fail, which might provide a workaround for some people. 

I would like to see a fix for this though, because in our situation I can never guarantee there will never be two clients connecting at the same time.

[Ivailo]

Hello Jakkie,

We are going to analyze the ticket you have sent and I am going to respond in this forum thread for further reference once the issue is clarified and we have some outcomes.

All the best,
Ivailo
the Telerik team

[Thomas]

We have identified an issue that is present when two or more threads concurrently use the same metadata source. We will fix this for the next version (which is not scheduled yet).
In the meantime, the workaround is to force the generation of the complete metadata information (metadata container) from a single-threaded place; I've used the static ctor of the EntitiesModel for this:

public partial class EntitiesModel
    {
        private static bool ok = metadataSource.GetModel().PersistentTypes.Count == 0 ? false : true;

        public static EntitiesModel CreateNew(string connection)
        {
            if (!ok)
                throw new Exception("No Persistent Types found.");
            BackendConfiguration backend = new BackendConfiguration();
            backend.Backend = "mssql";
            EntitiesModel result = new EntitiesModel(connection, backend, metadataSource);
            return result;
        }
    }

The ok static member is calculated when the EnitiesModel type is used the first time, under synchronization of the CLR runtime. After this, the metadata is correctly prepared and can be just used by multiple threads.

Regards,
Thomas
the Telerik team

[Dan]

Thomas-

Has this issue been resolved in the latest 2013 Q1 release?  I think I may be having the same issue and just wanted to check on this before going down other routes.

[Thomas]

The issue was already solved in the last release. Which problems occur on your side?
Do you use the same connection string for multiple different databases?

All the best,
Thomas
the Telerik team

[Dan]

Thomas-

Upon further investigation to give you details about the error I found the problem and it was unrelated.  Thanks.