OpenAccess hanging

[Wolfgang]

Hi,

I have a problem with Telerik Openaccess 2012.3 1209 from december with SQLAnywhere 12.

I have a call to an IObjectScope -> objectScope.GetObjectById(objectId);

Normally the call returns an object. But sometimes the following problem occurs:

If a connection to a database was succesful, and everything worked, and afterwards the connectino is removed from the other site (e.g. Database ist down), the next call does not return.

In parallel tasks I can see the following:

ObjectScope.GetObjectID
SynchronizedPMProxy.getObjectId
OpenAccessPersistenceManagerImp.getObjectById
OpenAccessPersistenceManagerImp.getStateMan
OpenAccessPersistenceManagerImp.getStateJdoConnection
LoggingStorageManager.fetch
OpenAccessPersistenceManagerImp.getStateMan
OpenAccessPersistenceManagerImp.getStateMan
RelationalStorageManager.fetch
RelationalStorageManager.getState
RelationalStorageManager.getStateImp
RelationalStorageManager.obtainConnection
RelationalConnectionPool.getConnection
RelationalConnectionPool.GetManagedConnection
RelationalConnectionPool.createPooledConnection
RelationalConnectionPool.createRealConWithRetry
RelationalConnectionPool.createRealCon
DBDRiver.connect
IADOFactory.CreateConnection
Intellitrace.OpenConnection
SAConnection.Open
SAConnectionPoolManager.AllocateConnection
->Monitor.Enter

Even after several minutes nothing seems to happen.

Do you see any way to tell the objectScope to stop trying to do something after a defined time?

(Or even a better, working solution?)

[Ady]

Hello Wolfgang,

 You can try specifying a larger connection timeout in the connection string. Specify a larger value for the 'Connection Timeout' property in the connection string.

Do let me know if that helps.

Kind regards,
Ady
the Telerik team

[Wolfgang]

Hi Ady,

unfortuanaly no - but why should it helped?

If i look at the SA connection it says it is "open" ( IAnywhere.Data.SQLAnywhere.ConnectionState State is shown as Open), but
database is down for sure (it is shut down - this is mostly to try out somethign what happened over network, when db was no longer available).

Next command seems to be triggered on the "open" connection which should be "closed/faulted/aborted/whatever"..


This next command is the GetObjectID, and it seems to just run forever without returning.

Regards
Wolfgang

[Wolfgang]

Hi Ady,

I had an additional hint from sybase forums.
If the sybase connection is set manually and Pooling is set to disabled in conenction string it should work.

I tried it, but it only seems to work if the connection is closed programmatically, not in the case where the other side is just down.

Best regards,
Wolfgang

[Ady]

Hello Wolfgang,

This seems to be a problem in the SqlAnywhere ADO driver. Ideally if a query is executed on a 'broken' connection the command should timout after the default command timeout (30 seconds).
 Can you try to switch off OpenAccess connection pooling completely and rely on  the ADO connection pool? This can be done by specifying the <poolType>2</poolType> entry in the backendconfiguration section.


Greetings,
Ady
the Telerik team

[Wolfgang]

Hi,

Now:
The ADO pooling is NOT disabled in connection string, the Telerik pooling is set to use only ADO pooling with <poolType>2</poolType>

The following happens:
- if i close the remote database and then try to access the remote database again, i get the connection state as "closed", if i try to reopen it (with closed db) i get nearly immediately the correct exception that the db is closed/not accessible or that the TransactionScope is not active.

This seems to be the expected behaviour. Some warning that something occured with the connection (even it would be better to have the same warning and not different ones) and no deadlock with the Monitor.Enter.

 

Before the problem was occuring every time if i was just closing the database or

disconnecting the network on another computer and then directly using Telerik to try to get an object by identifier

objectScope.GetObjectById(objectId)

I don't know if this is a stable state yet, or if the problem can reoccur by some strange race condition.

What are the drawbacks of the <poolType>2</poolType> setting?

Best regards,
Wolfgang


P.S.

From sybase i have the following information:

> SAConnectionPoolManager can potentially have thread-safety issues if

> multiple threads are trying to allocate/close multiple connection pools.

> This investigation has been opened as CR #730642, but has not been

> currently resolved.

[Ady]

Hello Wolfgang,

 By turning off OpenAccess connection pooling you are using the standard ADO.NET connection pooling. OpenAccess does not pool any connections and does not perform any house keeping tasks such as closing long active connections, validating connections on 'Open' (if an initial SQL is specified) etc.
I would suggest using the ADO pooling if that avoids the infinite wait on GetObjectById.

Regards,
Ady
the Telerik team

[Wolfgang]

Hi Ady,

for the moment i have left the ADO-connection "solution". 

Also I still fell a little bit unhappy because I don't know what happens there, and why ADO helps in the situation...

Best regards,
Wolfgang

[Ady]

Hello Wolfgang,

 As mentioned before by using the ADO pooling you are using the standard ADO.NET connection pooling. This is perfectly fine and you need not worry about something 'fishy' happening. We will try and reproduce the problem with the hanging during GetObejctById and investigate the problem.

All the best,
Ady
the Telerik team

[Wolfgang]

Thanks

[Wolfgang]

Drawback of the solution:

On some dialogs wth some properties the following error occurs:

System.NullReferenceException: Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt.

   bei Telerik.OpenAccess.RT.Adonet2Generic.Impl.DBDriver.connect(ConnectionString connectionString, IDictionary driverProps, ConnectionPoolType poolType, LogEventStore pes)
   bei OpenAccessRuntime.Relational.conn.RelationalConnectionPool.createRealCon()
   bei OpenAccessRuntime.Relational.conn.RelationalConnectionPool.CreatePlainAdoConnection()
   bei OpenAccessRuntime.Relational.conn.RelationalConnectionPool.getConnection(Boolean highPriority, Boolean autoCommit, Int32 smId)
   bei OpenAccessRuntime.Relational.RelationalStorageManager.obtainConnection(Boolean forWriting)

- with pool = 0 the error does not occur, but the original error occurs
- with pool = 1 both errors occur
- with pool = 2 this error occurs, but the original one is gone.

So either it hangs forever (GetObjectById) or some properties are not accessable. 

Any further help?

[Ady]

Hi Wolfgang,

 Do you get this exception randomly or one first access to the database i.e  is it the first object that is being read from the database? Can you consistently reproduce this exception? Looking at the source code I do not see where the null pointer exception could occur? Is there any sort of parallel activity going on?

Greetings,
Ady
the Telerik team

[Wolfgang]

Hi,

- with pool = 0 (connection is not recognized as aborted)
-> occurs every time if 
1. a succesful connection was established
2. Later the connection is removed (e.g. remote PC is shut down, network error (e.g. cable is removed),
    remote database is shut down,...) 
- with pool = 2
-> occurs every time in specified scenarion:

In DevExpress XtraGrid there is something called MasterDetail data presentation, where you can show additional details in a grid (e.g. trucks for a delivery, product details for an invoice ( see  http://documentation.devexpress.com/#WindowsForms/CustomDocument783 for example). They have two flags: OptionsDetail.EnableMasterViewMode and OptionsView.ShowDetailButtons which can be set to true to enable the detail views. We use this on multiple WinForms. If we have configured detail views and set the flags to true, with pool = 0 everything works as expected. With pool = 2 the error occurs, and the XtraGrid becomes non responsive (a.k.a. Red cross of Death).

In this case no threads are involved. Sadly the DevExpress XtraGrid is can not be used threadsafe in our environment, workarounds with BeginDataUpdate only seems to work with WPF, we are using WinForms (even with locks & invokes, some people seam to have the same problem http://www.devexpress.com/Support/Center/p/CB4246.aspx ). So because of this there should be no parallel activity.

[Ady]

Hello Wolfgang,

 I've run out of ideas as to why this could happen. Can you prepare a sample application that helps reproduce this exception by stopping the database? This would be really helpful in trying to resolve this matter finally.
I'm sorry for the inconvenience caused.

All the best,
Ady
the Telerik team

[CAM]

I also have the same issue with the latest telerik ORM.  All that has occurred in the software is updating the orm to the latest release from the last release.  Now I have connections never closing (as seen in the notes below, and the app pools escalating out of control memory-wise under load).  I put the apppool in web garden mode and recycle every 30 minutes to control this.

Something has changed in the behaviors of the ORM. I am currently attempting to back out the current release.

Exception information: 
    Exception type: InvalidOperationException 
    Exception message: Prepare requires an open and available Connection. The connection's current state is open.
   at Telerik.OpenAccess.RT.Adonet2Generic.Impl.PreparedStatementImp.executeQuery(Nullable`1 commandTimeout)
   at OpenAccessRuntime.Relational.conn.PooledPreparedStatement.executeQuery(Nullable`1 commandTimeout)
   at OpenAccessRuntime.Relational.fetch.FetchResultImp.Execute()

[Ady]

Hello CAM,

 Do you see a pattern when this happens? What operations are you performing when the application hangs ? Is it always a read operation (as the call stack suggests) or is also during CUD operations?
Are you using plain ADO.NET connection pooling?

Regards,
Ady
Telerik

[CAM]

I am not using CUD ops.  I use views for lists and manage the updates with writes in the MVC post operation.

Turning off telerik connection pool handing fixes this (going back to basic ADO).

This was during operations where an SP was being called that reads the database.  I have two that continuously get stuck.  One does a check with ANY() to see if the database is online (aspnet applications table, 1 record), the other builds a list of facets/profiles for the user (SP that runs in sub 1 sec times for my largest user). The SP contains nolock hints, and when profiling, there are no waits in the database...the ORM is just "stuck" in the app pool.  Recycling the app pool fixes this...it is like the connection pool gets lost and corrupts the app pool.

[Ady]

Hello CAM,

 That is indeed strange. It seems that a combination of the code and the Data Access connection pool settings cause this to happen.
Since the plain ADO.NET connection pooling is working for you I would suggest that you stick to that but nevertheless can you send us the 'BackendConfiguration' settings that are configured for the OpenAccessContext derived class? Maybe that throws some light on what could be happening with the connection pool.

Regards,
Ady
Telerik