AWT causes Tomcat to hang on shutdown

I have spent the last couple of work days tracking down a bug that can cause our current Code Collaborator 5.0 beta to not shut down properly. The root cause lies at the intersection of at least two bugs — neither of which were ours directly, but the effect is still that for some users, Code Collaborator is broken. Note to Code Collaborator beta users, most of the rest of this blog entry will be about the underlying technical issues. If you just want to get to the punchline and get the workaround, skip down to Workaround.

Symptoms

The first, and most obvious, symptom was that when I hit Ctrl-C to kill my development version of Tomcat, it hung. The second symptom was the following stack trace in the console window:
Exception in thread "AWT-Windows" java.lang.IllegalStateException: 
Shutdown in progress
at java.lang.Shutdown.add(Unknown Source)
at java.lang.Runtime.addShutdownHook(Unknown Source)
at sun.awt.windows.WToolkit.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

So, it looks like something in AWT is going haywire. I confirmed the scope of the issue. It seems limited to Java 5 on Windows. Other platforms and Java 6 (later update versions) are not affected. A search of Google and the Java Bug database turned up lots of reports of the bug, but no fixes or workarounds (that I could find).

Diagnosis


Suspecting a possible deadlock, I checked the Thread Dump. A lot of uninteresting noise, and this:

"Thread-9" prio=6 tid=0x0b148e40 nid=0x35c in Object.wait() [0x0c21e000..0x0c21f
b68]
at java.lang.Object.wait(Native Method)
- waiting on <0x02bdde50> (a sun.awt.windows.WToolkit)
at java.lang.Object.wait(Unknown Source)
at sun.awt.windows.WToolkit.<init>(Unknown Source)
 - locked <0x02bdde50> (a sun.awt.windows.WToolkit) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at java.lang.Class.newInstance0(Unknown Source)
at java.lang.Class.newInstance(Unknown Source)
at java.awt.Toolkit$2.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.awt.Toolkit.getDefaultToolkit(Unknown Source)
- locked <0x079ac330> (a java.lang.Class)
at com.qoppa.pdf.h.k.<clinit>(Unknown Source)

at sun.misc.Unsafe.ensureClassInitialized(Native Method)
at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(Unknown Source)
at sun.reflect.ReflectionFactory.newFieldAccessor(Unknown Source)
at java.lang.reflect.Field.acquireFieldAccessor(Unknown Source)
at java.lang.reflect.Field.getFieldAccessor(Unknown Source)
at java.lang.reflect.Field.set(Unknown Source)
at org.apache.catalina.loader.WebappClassLoader.clearReferences(WebappClassLoader.java:1644)
at org.apache.catalina.loader.WebappClassLoader.stop(WebappClassLoader.java:1524)
at org.apache.catalina.loader.WebappLoader.stop(WebappLoader.java:707)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:45
57)
- locked <0x030b88f0> (a org.apache.catalina.core.StandardContext)
at org.apache.catalina.core.ContainerBase.removeChild(ContainerBase.java
:924)
at org.apache.catalina.startup.HostConfig.undeployApps(HostConfig.java:1
191)
at ...


This is particularly interesting. Here we see Tomcat undeploying our web application, which in turn is invoking the class initializer of the PDF rendering library we use, which in turn is getting the default AWT toolkit. All of this while shutting down!?!

Off to the Tomcat source, where we find clearReferences containing a block wrapped in this:

// Null out any static or final fields from loaded classes,
// as a workaround for apparent garbage collection bugs
if (ENABLE_CLEAR_REFERENCES) {
...
}


Fortunately ENABLE_CLEAR_REFERENCES is controlled by a Java system property: org.apache.catalina.loader.WebappClassLoader.ENABLE_CLEAR_REFERENCES. Setting that property to false stops the hang. Since we are running in a dedicated Tomcat instance we are not concerned about the memory leak that happens when applications are loaded and unloaded. For others, this may be problem.

Analysis


There are two bugs at work here. The first is a bug in Windows AWT implementation. It does not either detect that the system is shutting down, nor does it handle the IllegalStateException when it tries to register its shutdown handler. This bug was tickled by a workaround for apparent bug in the garbage collector in some versions of Java, wherein static members of unloaded classes were not being deallocated.

One useful takeaway from this investigation lies in the use of java.awt.Toolkit.getDefaultToolkit during class initialization. While it is useful to cache the toolkit in a static member, doing so at class initialization time tickles this bug. If you're creating a library or an application that uses AWT and might potentially run within Tomcat, lazy initialization of the Toolkit in an accessor method might be a better way to go.


Workaround

For Code Collaborator users on Windows, who are using Java 5 (other platforms and Java 6 are immune from the issue), here's your workaround. Add the line:

-Dorg.apache.catalina.loader.WebappClassLoader.ENABLE_CLEAR_REFERENCES=false

to the ccollab-server.vmoptions file in your installation directory. Make sure the file still ends with an empty line. Restart your server; unfortunately that may mean forcibly killing ccollab-server.exe in Task Manager. In the next release, we'll make sure the vmoptions file includes the necessary line.


Close

By submitting this form, you agree to our
Terms of Use and Privacy Policy

Thanks for Subscribing

Keep an eye on your inbox for more great content.

Continue Reading

Add a little SmartBear to your life

Stay on top of your Software game with the latest developer tips, best practices and news, delivered straight to your inbox