Connection problems?

[Hessner]

Hi,

I have developed a CMS system and are beginning to experience that the system "hangs up" occasionally.
An application pool recycle will fix it for a while, but this solution are like no solution.

Today I got(for the first time) this message:
"Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached"

So now I want to run my OpenAccess code by you to comment on:

public void Init(HttpApplication app)
{
  app.BeginRequest += ContextBeginRequest;
  app.EndRequest += ContextEndRequest;
}

private static void ContextBeginRequest(object sender, EventArgs e)
{
  var app = (HttpApplication)sender;
  app.Context.Items["ObjectScopeProvider"] = Database.Get("ORMConnection").GetObjectScope();
}

private static void ContextEndRequest(object sender, EventArgs e)
{
  var app = (HttpApplication)sender;
  var dataObj = (IObjectScope)app.Context.Items["ObjectScopeProvider"];
  if (dataObj == null)
    {
    return;
    }
  dataObj.Dispose();
  app.Context.Items.Remove("ObjectScopeProvider");
}

This code execute on every aspx page request.

Now I have direct data access inside all my aspx/ascx file,- like this:

var os = (IObjectScope)Context.Items["ObjectScopeProvider"];

var getModule = (from o in os.Extent<Module1>()
where o.ModuleID == moduleid && o.LockedBy == userid
select o).FirstOrDefault();

if (getModule != null)
  {
  os.Transaction.Begin();
  getModule.LockedBy = null;
  os.Transaction.Commit();

  and so on....

Do you see any "deadlock", connection or performance problems here?

Regards,
Hessner

[IT-Als]

Hi Hessner,

OK, I don't know about this for sure, but this might be the problem:

In your object scope handling on each request you do a Database.Get("ORMConnection") which essentially gets an abstraction of the storage (the database) currently used. It also reads the mapping information from the app.config.
Best practice is to only create the Database instance once and re-use it everytime you want to get an ObjectScope instance.

It is better (like in the ObjectScopeProvider source delivered with OA) to do Database.Get("ORMConnection") and store the resulting instance (a Database class instance) in a place where you can get hold of it again (application context, static class, whatever). Say, you have stored the instance in a static class called MyDatabase, which has a property called MyStoredDatabase which returns the stored Database instance:

On each request, you use the stored instance to get a new object scope like:

MyDatabase.MyStoredDatabase.GetObjectScope()

And then you can go on like you do in your source code..

Does it make sense?

Regards

Henrik

[Hessner]

Hi Henrik,

Thanks for your input, I will try moving the "Database.Get" to a static class and use this ref. instead.

 

Can anyone, from Telerik perhaps, confirm Henrik's suggestion?

 

Regards,

Hessner

[Jan Blessenohl]

Hello Hessner,
In case of web server this is not a good idea. You would use the same ObjectScope in all your threads, which means the local transaction handling will not work. Your implementation to have one scope per page is better.

The exception seems to come from the mysql driver, can you send me the complete stack trace?

That the application is hanging might have to do with the amount of connections in the pool. How high is your load? Our pool implementation has a setting 'block if empty' which lets the request wait until a connection is available in the pool. Maybe that causes the problem. You may want to play with the max connection count in the pool by using the backend configuration dialog from the OpenAccess menu.

Regards,
Jan Blessenohl
the Telerik team

Instantly find answers to your questions on the new Telerik Support Portal.
Watch a video on how to optimize your support resource searches and check out more tips on the blogs.

[IT-Als]

Hello Jan (and Hessner)

Just to clarify:
I am not asking Hessner to only have one object scope. I am asking him to have one Database class instance and pull of ObjectScope instances from that (Database)instance per thread.
We do this in our system (WCF services hosted on IIS) and it works like a charm. I know our scenario is not straighforward compared to that of Hessner, but the same principles still apply.

Anyway, I was thinking:
Are you (Hessner) using Ajax on your pages? I think I remember seeing an article (on Telerik.com) regarding handling of the object scope in ajaxified webapps.. there was some special handling ïn that case.

Basically every request (ajax, "normal" request) should start off by aquiring the scope and end by disposing the scope

Just my few cents..

[Hessner]

Hi Jan,

I am using SQL Server 2005 64 bit.

Years back, when I only used SP, I had a situation with deadlocks - the solution was to add NOLOCK to every query.

In the mean time I have added LINQ2SQL, and have not experienced any lock problems.

Now I am in the process of replacing LINQ2SQL with OpenAccess (still await support for Join in Linq :-)).

1. I therefore suspect the supplied code to create much more connections than I had before, and I would like you to especially review the "ContextEndRequest" code supplied in my first post - and please tell me when a connection are actually taken/released.

2. Inside SQL Server I see a lot of connections(via sp_who2) with the status: "Sleeping" and expect this to be ok?

I will leave my setup as is, and raise the number of connections allowed inside the pool.

[Hessner]

Hi Henrik,

Yes, I am using Ajax "all over". Was expecting that the ref inside "Context" could be used everywhere in my web app and ajax content dosent seem to have any problems, but I will look into that aspect also. Thank you.

[IT-Als]

Hi Hessner,

Ok found the article:

http://www.telerik.com/support/kb/orm/general/best-practices-two.aspx

This code sample seems to deal with exactly your scenario, where you have a page, that is possible built up by multiple controls. Both the page itself and the controls use the same object scope.

And regarding your orginal code:

You actually put the retrieved object scope into the Context (app context) items collection under the same key, meaning two simultaneusly executing requests can conflict.
In the example link above a unique key is used for each retrieved object scope that are put on in the items collection.

Hope to help.

Regards

Henrik

[Jan Blessenohl]

Hi Henrik,
The Database instance is not a problem, we have our own cache below and by specifying the same alias name you will get the same database instance.

Hi Bo,
the code looks good to me. The only question is: can the same sender make several asynchronous requests in parallel. In this case, because the scope is not thread safe by default, you might get errors. Those error are usually exceptions and not a blocking app.

Do you see the complete app hanging or are there hanging requests?

Our connection handling is as follows:
If the scope has to execute a server request is gets an connection from the pool. If the result of the request is read completely the connection goes back into the pool. There are some cases where the connection is not immediately given back to the pool.
1. If you have a query result with more then 50 rows and you do not read the result completely. The result is a cursor on the server and as long as you do not read the result completely or dispose the result we have to keep the connection at the scope.
2. If you use AUTOINC keys and ask for the key before the new object is inserted. Here we have to flush the object to the server to get the key. This means we have to keep the server transaction open until the scope transaction is committed.
3. If you call transaction.flush

All the best,
Jan Blessenohl
the Telerik team

Instantly find answers to your questions on the new Telerik Support Portal.
Watch a video on how to optimize your support resource searches and check out more tips on the blogs.

[IT-Als]

Hi you both

Jan:
Thanks for pointing the issue with the Database.Get method out. I wasn't aware of this.

But again, isn't it a problem by using a key in the Items collection that is not related to the request? And putting different ObjectScopes in the collection by that same key per request?

Scenario (the problem as I see it):

Request A comes in -> Object scope is retrieved and put in the items collection using the key "ObjectScopeProvider"
Before Request A finishes Request B comes in -> Object scope is retrieved and put in the items collection using the key "ObjectScopeProvider" (Conflict->request B overwrites the objectscope that request A put in the items collection)

When it comes to disposing the scope upon ending the request only the last written (the request B) object scope id disposed, while the request A object scope is never disposed and left dangling around..

However, when using a key that is request specific... like the hash generated in the example in the link above..  you can put the scope in by that unique key, and get it out by that unique key. Thus, no two or more simultaneuos request can overwrite each others object scopes since they have unique keys.

Regards

Henrik

[Hessner]

Hi,

But are the Context.Items collection not a per-request information store?
 
If yes, then it will never be a problem, if no - you are right, off course.

[IT-Als]

Must be the heat..

Of cause, you're right. HttpContext.Current IS the request.

[Hessner]

Hi Henrik,

My code are implemented in a httpmodule - and I use: var app = (HttpApplication)sender;  

So maybe you are on to something, I will look into this (also :-)).

Regards,
Bo Hessner

Updated:

Peter Bromberg say:
"When an incoming request arrives, the HttpRuntime class takes an HttpApplication instance from the pool and puts it to work on the request. So each separate incoming request is actually associated with a single HttpApplication object. That should provide some insight!"

[IT-Als]

Hi Hessner,

Thanks for the update.

What made me think was: In Teleriks own example (posted earlier in the thread) they use Context.Current.Items and they insist that the key should be unique (something about a hash + the current thread context id) as I remeber

And the fact that I have had similar "funny things" going on in our system, because we did not dispose the scope (or to be precise: the correct scope)...and left scopes dangling around. Funny things were: Sequence generator starting to take out ids that were already used, connection stall, etc...

In our solution we also use a unique key that can be identified upon start and end of the (WCF in my case) request.

Regards

Henrik

[Hessner]

Hi Henrik,

"connection stall" describes perfectly what I am experiencing, from time to time.

For your information - I have been advised to turn on "the secund level cache".

Other than that I have turned on OpenAccess logging (in my test environment) and are working on a logging mechanism for production.