When working with large (server side) java application, sometimes it
would be nice if you could look inside, to see what thread is taking
up so much cpu time, and why. Something similar to the Unix top
command, but then showing all threads in one (java) application,
instead of all processes in the system.

When I was looking for such a monitoring application, I came accross the 2.0 version of MC4J that
provides a "Thread Info" panel that
displays threads together with CPU usage; exactly what I
needed. Unfortunately, there is only an alpha release of this MC4J
version, that is not yet perfectly stable. Moreover, the thread info
panel doesn’t handle applications with large amounts of threads very
well. As the source code of this version of MC4J is not (yet)
publically available, this option turned out to be a dead end.

To my surprise, other applications with such functionality are hard to
find. There are probably enough profiling applications that can do the
job, but I wanted something simple, something JMX-based, that can used
also to monitor applications running in production.

There is however something called JTop, which is a plugin for
JConsole. It’s actually a demo for the new (since Java 6) JConsole
plugin API, that does show CPU usage per thread. It’s fairly basic and
only shows total CPU usage, which is not very usefull. You would
expect that (after a year), somebody would have extended the demo to
something more useful, but as I couldn’t find anything like that, I
thought I should give it a try myself.

The result is a JConsole plugin that displays the top threads, sorted
by cpu usage in the last sampling period. It also displays cpu usage
history, and an average over the last 10 sampling periods.

To avoid ending up with an unresponsive user interface when monitoring
applications with large number of threads, I took a few
precautions. First of all, the plugin has it’s own refresh rate. It’s
independent from the JConsole poll interval, which is 4 seconds by
default. For applications with large amounts of threads, this is way
too short: only retrieving all thread information can already take 4
or 5 seconds! Although you can change the JConsole poll interval with
a command line option, I thought it would be more convenient to be
able to change it from the monitoring panel. It’s default set to 10
seconds, which I think is reasonable in most cases. If you notice that
cpu usage measurement takes too much of the application under test,
just increase the sample period and see the RMI Connection thread that
processes these request, sink in the list.

Another precaution was not to list all threads in the
table. Displaying thousands of rows in a table is quite useless in any
case, and I was afraid it would seriously harm
performance. Eventually, diplaying that many rows turned out to be not
much of a problem; I guess I still suffer from an prejudice with
respect to Swing performance…

Using MX4J also showed me that in a continuously refreshing table,
it’s hard to select a thread in order to see it’s
stacktrace. Therefore, in this plugin, tracing a thread is "sticky":
when you click a row in the table, the stacktrace of that thread is
shown immediately and is refreshed with each new sample, until you
deselect it or select another thread.

Even though having threads sorted by cpu usage is the logical thing to
do, it’s not always convenient when you’re studying the results, as
rows keeping moving with each refresh. To lock the rows to there
current position, click the "fix order" button. The topmost rows
(actually all rows with a non-zero cpu usage), will stay where they
are. Rows that had a cpu usage of zero, but have a non-zero value
in the next sampling periods, will appear just below these rows, to
avoid that you oversee any thread that suddenly takes a large amount
of cpu time.

You can run the plugin by downloading the jar file and passing it to JConsole with the plugin option:
jconsole -pluginpath topthreads.jar. When JConsole connects, it should have a seventh tab named "top threads".