Why WebLogic Server hangs while writing JTA logs?

WebLogic Server hangs while writing JTA logs. The client software cannot connect to WLS. There is no error in the application log. We collected a thread dump (kill -3) of the server java process and found the following key messages:

“ExecuteThread: ’5′ for queue: ‘weblogic.kernel.Default’”:
waiting to lock monitor 01c96328 (object 2e4cdbf0, a weblogic.transaction.internal.TransactionLoggerImpl$IOBuffer),
which is held by “ExecuteThread: ’17′ for queue: ‘weblogic.kernel.Default’”
“ExecuteThread: ’17′ for queue: ‘weblogic.kernel.Default’”:
waiting to lock monitor 01c96528 (object 2e4cdc08, a weblogic.transaction.internal.TransactionLoggerImpl$IOBuffer),
which is held by “ExecuteThread: ’5′ for queue: ‘weblogic.kernel.Default’”

JTA has transaction exceptions and is busy writing them to the JTA log and the server stops responding.

“ExecuteThread: ’5′ for queue: ‘weblogic.kernel.Default’”:
at weblogic.transaction.internal.TransactionLoggerImpl$LogDisk.release(TransactionLoggerImpl.java:1324)
- waiting to lock (a weblogic.transaction.internal.TransactionLoggerImpl$IOBuffer)
- locked (a weblogic.transaction.internal.TransactionLoggerImpl$IOBuffer)
at weblogic.transaction.internal.TransactionLoggerImpl.release(TransactionLoggerImpl.java:389)
at weblogic.transaction.internal.ServerTransactionImpl.releaseLog(ServerTransactionImpl.java:2767)
at weblogic.transaction.internal.ServerTransactionManagerImpl.remove(ServerTransactionManagerImpl.java:1466)
at weblogic.transaction.internal.ServerTransactionImpl.setRolledBack(ServerTransactionImpl.java:2597)
at weblogic.transaction.internal.ServerTransactionImpl.globalRetryRollback(ServerTransactionImpl.java:2816)
at weblogic.transaction.internal.ServerTransactionImpl.globalRollback(ServerTransactionImpl.java:2576)
at weblogic.transaction.internal.ServerTransactionImpl.internalRollback(ServerTransactionImpl.java:371)
at weblogic.transaction.internal.TransactionManagerImpl$TMXAResource.rollback(TransactionManagerImpl.java:1035)
at weblogic.wtc.gwt.OatmialUnknownXidHandler.execute(WTCService.java:3402)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
“ExecuteThread: ’17′ for queue: ‘weblogic.kernel.Default’”:
at weblogic.transaction.internal.TransactionLoggerImpl$LogDisk.release(TransactionLoggerImpl.java:1323)
- waiting to lock (a weblogic.transaction.internal.TransactionLoggerImpl$IOBuffer)
at weblogic.transaction.internal.TransactionLoggerImpl.release(TransactionLoggerImpl.java:389)
at weblogic.wtc.gwt.TuxedoLoggable.onDisk(TuxedoLoggable.java:347)
at weblogic.transaction.internal.TransactionLoggerImpl$LogDisk.write(TransactionLoggerImpl.java:1252)
- locked (a weblogic.transaction.internal.TransactionLoggerImpl$IOBuffer)
- locked (a weblogic.transaction.internal.TransactionLoggerImpl$LogDisk)
at weblogic.transaction.internal.TransactionLoggerImpl.flushLog(TransactionLoggerImpl.java:614)
at weblogic.transaction.internal.TransactionLoggerImpl.store(TransactionLoggerImpl.java:305)
at weblogic.wtc.gwt.OatmialUnknownXidHandler.execute(WTCService.java:3320)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

Reason

The problem has occurred due to lack of resources in the server. The server’s resources must be reviewed and tuned for production runtime.

Solution

Following tuning should help to resolve the issue:

1. Decrease the CheckpointIntervalSeconds.
2. For WLS 8.1 and prior releases, move the tlogs to a directory where it will not run out of space.
   For 9.x and above, move the default persistence store folder to a place where it will not run out of space.