DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Generative AI has transformed nearly every industry. How can you leverage GenAI to improve your productivity and efficiency?

SBOMs are essential to circumventing software supply chain attacks, and they provide visibility into various software components.

Related

  • MongoDB Change Streams and Go
  • Serverless vs Containers: Choosing the Right Architecture for Your Application
  • Before You Microservice Everything, Read This
  • Power BI Embedded Analytics — Part 3: Power BI Embedded Demo

Trending

  • Deploying LLMs Across Hybrid Cloud-Fog Topologies Using Progressive Model Pruning
  • Modernize Your IAM Into Identity Fabric Powered by Connectors
  • Parallel Data Conflict Resolution in Enterprise Workflows: Pessimistic vs. Optimistic Locking at Scale
  • Rust: The Must-Adopt Language for Modern Software Development

Dealing with Stuck Threads on WebLogic

By 
Spyros Doulgeridis user avatar
Spyros Doulgeridis
·
May. 09, 12 · Interview
Likes (2)
Comment
Save
Tweet
Share
140.2K Views

Join the DZone community and get the full member experience.

Join For Free
WebLogic Server diagnoses a thread as stuck if it is continually working (not idle) for a set periodof time.

You can tune a server's thread detection behavior by changing the length of time before a thread is diagnosed as stuck (Stuck Thread Max Time), and by changing the frequency with which the server checks for stuck threads. Check here to see how to change the Stuck Thread Max Time.

The problem or Why are Stuck Threads evil?

WebLogic Server automatically detects when a thread in an execute queue becomes "stuck." Because a stuck thread cannot complete its current work or accept new work, the server logs a message each time it diagnoses a stuck thread. If all threads in an execute queue become stuck, the server changes its health state to either "warning" or "critical" depending on the execute queue:
  • If all threads in the default queue become stuck, the server changes its health state to "critical." (You can set up the Node Manager application to automatically shut down and restart servers in the critical health state. For more information, see "Node Manager Capabilities" in Configuring and Managing WebLogic Server.)
  • If all threads in weblogic.admin.HTTP, weblogic.admin.RMI, or a user-defined execute queue become stuck, the server changes its health state to "warning."
So practically, a couple of Stuck Threads might not crash your server preventing it from serving request, but it is a bad sign. Usually, the number of stuck threads will increase and your server  will eventually crash.

What you can do to avoid your application completely fail?
WebLogic Server checks for stuck threads periodically (this is the Stuck Thread Timer Interval and you can adjust it here). If all application threads are stuck, a server instance marks itself failed, if configured to do so, exits. You can configure Node Manager or a third-party high-availability solution to restart the server instance for automatic failure recovery.You can configure these actions to occur when not all threads are stuck, but the number of stuck threads have exceeded a configured threshold:Shut down the Work Manager if it has stuck threads. A Work Manager that is shut down will refuse new work and reject existing work in the queue by sending a rejection message. In a cluster, clustered clients will fail over to another cluster member.
  • Shut down the application if there are stuck threads in the application. The application is shutdown by bringing it into admin mode. All Work Managers belonging to the application are shut down, and behave as described above.
  • Mark the server instance as failed and shut it down it down if there are stuck threads in the server. In a cluster, clustered clients that are connected or attempting to connect will fail over to another cluster member.
How to identify the problem?
The most recommended way is to check the thread dumps. Check Sending Email Alert For Stuck Threads With Thread Dumps post of Middleware magic, to have Thread Dumps mailed to you automatically  when they occur.
Tools to help you with analyzing the Thread Dumps can be:
  • TDA - Thread Dump Analyzer
  • Samurai
How to workaround the problem?
After you have identify the code that causes the Stuck Thread, that is the code which execution takes more than the Stack Thread Max Time, you can use Work Manager to execute your code. Work Managers have a Ignore Stuck Thread options that gives the ability to execute long running jobs. See below:
Below are some posts on how to create a Work Manager

  • https://blogs.oracle.com/jamesbayer/entry/work_manager_leash_for_slow_js
  • http://jdeveloperfaq.blogspot.com/2011/05/faq-34-using-weblogic-work-managers-to.html

Test: How to create a Stuck Thread?
How to create a Stuck Thread in order to test your weblogic settings? Put a breakpoint in a  backing bean or model method that is called with you request. If you wait in the breakpoint for  Stuck Max Thread Time, you notice a Stuck Thread trace will be shown in servers log:
<16 =Ύί 2011 12:28:22 ΉΉ EET> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 
    'weblogic.kernel.Default (self-tuning)' has been busy for "134" seconds working on the 
request "weblogic.servlet.internal.ServletRequestImpl@6e6f4718[
GET /---/---/----/---/days.xhtml HTTP/1.1
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.120 Safari/535.2
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-GB,en-US;q=0.8,en;q=0.6
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: JSESSIONID=DYG5TDTZSnKLTFw5CMMdLCD9sPsZS4Jqlmxj9wdGNyt1BnPcfNrR!-1520792836


]", which is more than the configured time (StuckThreadMaxTime) of "60" seconds. Stack trace:
        --------------------------------------------(--------------------.java:83)
        javax.faces.component.UIComponentBase.encodeBegin(UIComponentBase.java:823)
        com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:285)
        com.sun.faces.renderkit.html_basic.GridRenderer.renderRow(GridRenderer.java:185)
        com.sun.faces.renderkit.html_basic.GridRenderer.encodeChildren(GridRenderer.java:129)
        javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:848)
        org.primefaces.renderkit.CoreRenderer.renderChild(CoreRenderer.java:55)
        org.primefaces.renderkit.CoreRenderer.renderChildren(CoreRenderer.java:43)
        org.primefaces.component.fieldset.FieldsetRenderer.encodeContent(FieldsetRenderer.java:95)
        org.primefaces.component.fieldset.FieldsetRenderer.encodeMarkup(FieldsetRenderer.java:76)
        org.primefaces.component.fieldset.FieldsetRenderer.encodeEnd(FieldsetRenderer.java:53)
        javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:878)
        javax.faces.component.UIComponent.encodeAll(UIComponent.java:1620)
        javax.faces.render.Renderer.encodeChildren(Renderer.java:168)
        javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:848)
        org.primefaces.renderkit.CoreRenderer.renderChild(CoreRenderer.java:55)
        org.primefaces.renderkit.CoreRenderer.renderChildren(CoreRenderer.java:43)
        org.primefaces.component.panel.PanelRenderer.encodeContent(PanelRenderer.java:229)
        org.primefaces.component.panel.PanelRenderer.encodeMarkup(PanelRenderer.java:152) 
More digging:
  • Excellent post by Frank Munz: WebLogic Stuck Threads: Creating, Understanding and Dealing with them. Updated for Weblogic 12c. Includes sample app for creating Stuck Thread too.
  • http://stackoverflow.com/questions/2709410/weblogic-stuck-thread-protection
src:
  • Maxence Button excellent post: http://m-button.blogspot.com/2008/07/using-wlst-to-perform-regular.html
  • http://download.oracle.com/docs/cd/E13222_01/wls/docs81/perform/WLSTuning.html#1125714
  • http://download.oracle.com/docs/cd/E21764_01/web.1111/e13701/overload.htm
  • http://java.sys-con.com/node/358060?page=0,0
application

Published at DZone with permission of Spyros Doulgeridis, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • MongoDB Change Streams and Go
  • Serverless vs Containers: Choosing the Right Architecture for Your Application
  • Before You Microservice Everything, Read This
  • Power BI Embedded Analytics — Part 3: Power BI Embedded Demo

Partner Resources

×

Comments

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • [email protected]

Let's be friends: