Berichten met label threads
New version of TopThreads JConsole plugin
Geplaatst door Peter Doornbosch in java op 26 januari 2011
Some time ago, I created the “TopThreads” plugin for JConsole, that helps you to determine why your Java application is causing such a high CPU load, by showing the most busy threads in your application and giving you the opportunity to inspect thread-stacktraces at the same time. It turned out to be quite usefull and from the responses I got, I can tell people find it still usefull today. A few days ago, I released a new version of this plugin, with one very usefull new feature: CPU usage.
Top thread?
If you’ve used the topthreads plugin, you probably seen this before: suddenly, a thread that is not supposed to be very busy, pops up at the top of the table with usage figures in the 90’s. You wonder WTF is going on, that this thread is taking so much CPU power, until you realize that this figure is only relative to the rest of the application threads. And if the application is hardly doing anything, threads that do a little more than anything might get alarming high figures (and red color). After i ran into this pitfall a few times, i decided i needed to know an absolute usage figures too.
If you enable this feature (settings -> show process cpu usage too), the top row of the table shows the CPU usage of the process as a whole. This is simply the sum of the CPU usage of all threads. The percentage shown in this row however, is the percentage this process is using the CPU, which should be approximately the same value a process viewer like top, Activity Monitor or the TaskManager would report. Although this is not always the case – more about that in a minute – it’s at least a good indication whether the process is busy or idle. And even though it may not always be as acurate as i would like it to be, it proved itself to be proficiant to help me avoid confusion.
The usual suspect: the garbage collector
In normal situations (whatever that me be… 
), the CPU usage figure is approximately the same as the figures other tools report. However, especially when the process is very busy, the CPU usage shown is far too low. After some testing, i’m rather confident that this is mainly caused by the garbage collector. As it turns out, TopThreads does not get information for all the JVM threads, which can easily be verified by comparing a thread dump with the thread listing in JConsole. For example, threads that never appear in JConsole (not in the TopThreads tab, but neither in the JConsole thread view) are the “Low Memory Detector”, compiler threads (HotSpot), “Signal Dispatcher” and “Surrogate Locker Thread (CMS)” and the garbage collector threads (the mark-and-sweep thread and the parallel gc threads). I can image that some of these threads can put a lot of load on the CPU when the application is very busy. And one thing is for sure: the cpu cycles that are taken by these threads are not counted in the totals that the TopThreads plugin computes, simply because it doesn’t know about these.
Despite these shortcomings, i find the new feature quite usefull myself. Let me know what you think.
Other improvements in this release:
- the initial poll time is not fixed to 10 seconds, but depends on the (initial) number of threads. For small apps, the updates will be much more frequent.
- there are more preferences to set and these are moved to a separate settings dialog. Settings are stored using the Java Preferences API.
- improved stacktrace panel behavivour, including automatic scroll to the top.
- better handling of security exceptions, that might occur when connecting to a remote VM.
Please let me know what you think, feedback is always welcome!
“top threads” plugin for JConsole
Geplaatst door Peter Doornbosch in Uncategorized op 24 mei 2007
When working with large (server side) java application, sometimes it would be nice if you could look inside, to see what thread is taking up so much cpu time, and why. Something similar to the Unix top command, but then showing all threads in one (java) application, instead of all processes in the system.
When I was looking for such a monitoring application, I came accross the 2.0 version of MC4J that provides a “Thread Info” panel that displays threads together with CPU usage; exactly what I needed. Unfortunately, there is only an alpha release of this MC4J version, that is not yet perfectly stable. Moreover, the thread info panel doesn’t handle applications with large amounts of threads very well. As the source code of this version of MC4J is not (yet) publically available, this option turned out to be a dead end.
To my surprise, other applications with such functionality are hard to find. There are probably enough profiling applications that can do the job, but I wanted something simple, something JMX-based, that can used also to monitor applications running in production.
There is however something called JTop, which is a plugin for JConsole. It’s actually a demo for the new (since Java 6) JConsole plugin API, that does show CPU usage per thread. It’s fairly basic and only shows total CPU usage, which is not very usefull. You would expect that (after a year), somebody would have extended the demo to something more useful, but as I couldn’t find anything like that, I thought I should give it a try myself.
The result is a JConsole plugin that displays the top threads, sorted by cpu usage in the last sampling period. It also displays cpu usage history, and an average over the last 10 sampling periods.
To avoid ending up with an unresponsive user interface when monitoring applications with large number of threads, I took a few precautions. First of all, the plugin has it’s own refresh rate. It’s independent from the JConsole poll interval, which is 4 seconds by default. For applications with large amounts of threads, this is way too short: only retrieving all thread information can already take 4 or 5 seconds! Although you can change the JConsole poll interval with a command line option, I thought it would be more convenient to be able to change it from the monitoring panel. It’s default set to 10 seconds, which I think is reasonable in most cases. If you notice that cpu usage measurement takes too much of the application under test, just increase the sample period and see the RMI Connection thread that processes these request, sink in the list.
Another precaution was not to list all threads in the table. Displaying thousands of rows in a table is quite useless in any case, and I was afraid it would seriously harm performance. Eventually, diplaying that many rows turned out to be not much of a problem; I guess I still suffer from an prejudice with
respect to Swing performance…
Using MX4J also showed me that in a continuously refreshing table, it’s hard to select a thread in order to see it’s stacktrace. Therefore, in this plugin, tracing a thread is “sticky”: when you click a row in the table, the stacktrace of that thread is shown immediately and is refreshed with each new sample, until you deselect it or select another thread.
Even though having threads sorted by cpu usage is the logical thing to do, it’s not always convenient when you’re studying the results, as rows keeping moving with each refresh. To lock the rows to there current position, click the “fix order” button. The topmost rows (actually all rows with a non-zero cpu usage), will stay where they are. Rows that had a cpu usage of zero, but have a non-zero value in the next sampling periods, will appear just below these rows, to avoid that you oversee any thread that suddenly takes a large amount of cpu time.
You can run the plugin by downloading the jar file and passing it to JConsole with the plugin option:
jconsole -pluginpath topthreads.jar. When JConsole connects, it should have a seventh tab named “top threads”.