Category Archives: technology

Monitoring and metrics – Yes, of course.

I always try to convince everybody to measure. The first reaction are very different, but after a while everyone come back and says “Wow! I didn’t realize how helpful metrics can be”.

I’ve watched video: “Using Monitoring and Metrics to Learn in Development”  (by Patrick Debois from Atlassian) with pleasure. I use Atlassian tools and talk with few guys. They really care about code of their products. Patrick not only talks about metrics but also talks about technics and ideas which helps deliver better software.

Ideas from presentation:

  • smaller and frequent changes – easier to repair (dev for ops)
  • faster and better feedback – easier to find problem (ops for dev)
  • continuous integration maturity model – see slideshare presentation.
  • reuse “workflows” across environments by using virtualization – vagrant (great tool for building dev environments), puppet/chef (configuration automation and management)
  • infrastructure code repository and application code repository have to be in sync.
  • always remember that: “a lot of different monitoring levels we have” :).
  • monitoring driven development :) – create a monitor check before implementing a feature (it is useless but you can think about similarities
  • monitoring tools grumble (around 00:19:20) – there is a projects by “Monitoring Sucks” (github), you can check but only one seems to be alive.
  • use monitoring as a service – this sound reasonable (pingdom, NewRelict, boundary, librato, and some others).
  • and  lot of other tools are mentioned – if you want to evaluate tools for metrics and monitoring you should watch video and note potential tools for evaluation.
  • always know the context of the metrics.
  • final thought: Metrics Driven Engineering (Etsy on the stage) – IMHO this is great idea to follow.

Whatever we will doing, it will happen our code do not work, our code stinks and become worser, but every time we figure out, we can do something with it. This is the reason it is so important to monitor our application continuously.

Patrick also mention about developer and operation responsibility sharing, he wrote a nice blog post: Just Enough Developed Infrastructure. I recommend you to read it too.

skipfish – fast, easy and simple

Skipfish is google code project. It is web application security scanner, high speed (they claim 2000 requests per second* – * – at local LAN :)) and due the fact it is command line tool without fancy wizards, options and so on, it is relatively easy to use, and for sure it is easy to just start scanning.

Skipfish is active scanner so it first scan application, preparing the map of web site, than recursively ran different test, the last thing is report generation. Documentation is simple and has a lot of example we can start on. So let’s see that in action.

One of such command is:

$ skipfish -m 5 -LVJ -W /dev/null -o output_dir -b ie http://www.example.com/

During the scan, Skipfish is displaying statistics:

Scan statistics
---------------

       Scan time : 0:11:07.0068
   HTTP requests : 2446 sent (3.71/s), 16228.73 kB in, 659.18 kB out (25.32 kB/s)  
     Compression : 0.00 kB in, 0.00 kB out (0.00% gain)    
 HTTP exceptions : 34 net errors, 0 proto errors, 0 retried, 0 drops
 TCP connections : 2451 total (1.09 req/conn)  
  TCP exceptions : 0 failures, 1 timeouts, 0 purged
  External links : 745 skipped
    Reqs pending : 219        

Database statistics
-------------------

          Pivots : 471 total, 94 done (19.96%)    
     In progress : 323 pending, 38 init, 12 attacks, 4 dict    
   Missing nodes : 54 spotted
      Node types : 1 serv, 269 dir, 46 file, 1 pinfo, 91 unkn, 63 par, 0 val
    Issues found : 70 info, 111 warn, 49 low, 1 medium, 13 high impact
       Dict size : 0 words (0 new), 0 extensions, 0 candidates

After few hours/minutes, it depends on the site we are scanning, we will got

[+] Copying static resources...
[+] Sorting and annotating crawl nodes: 1666
[+] Looking for duplicate entries: 1666
[+] Counting unique issues: 1158
[+] Writing scan description...
[+] Counting unique issues: 1666
[+] Generating summary views...
[+] Report saved to outputDir/index.html
[+] This was a great day for science!

The report consist of “crawl results”, “document type overview” and “issue type overview”. My last scan result has some finding, but also has a lot of false positives, it seams that a lot of work still waiting for a Skipfish team, but it looks promising.

VisualVm performance tuning tool

Few weeks ago I’ve got Performance Tuning Training made by Kirk. It was great time for me and great training. If you can manage such training go there. You won’t regret it. After this training I’ve got some deeper knowledge about performance tuning and performance tuning tools. One of such tool is VisualVm.  If you download latest Java 6 update 14 (May 28), than you have one more option to perform application tuning. Beside of jps, jstat, jmap you get jvisualvm. The tool has long story, from NetBeans plugin to standard java tool.  If you have earlier java version you can also download it separately from project page visualvm.dev.java.net .

The VisualVm developers made very good job providing plugin API. If you write any plugin for NetBeans Platform you know how to write plugin for VisualVm tool. In this post I will describe available plugins. How to install them and how to use them. The plugin menu is under Tools->Plugin. We can setup there plugins repositories. We can also manage our plugins (install, uninstall, activate, deactivate).

As I write this post currently available plugins are:.

VisualVM-Glassfish: Advanced monitoring of Glassfish server. I use Glassfish as development server in daily work and the plugin is very useful.  I will not describe this plugin. It has very nice and detailed description made by Masoud Kalali.

Java ME Priofiler Snapshot Viewer
: Java ME SDK 3.0 offline profiling session viewer.  I’ve got very little with Java ME, so I stay just on this short  description.

VisualVm-Extension : Provide support for extending VisualVM for new JDK, JVM etc. It is for extension proposes in the future.

Memory Sampler : Experimental sampling memory profiler. It’s similar to jmap -histo, an additional feature over the jmap  is possibility to display delta results. If you try to find heap memory problem or PermGen space problems this tool can help you to figure out what going there.

memory sampler

From the top on the left you can set refresh rate, on the right you can set Heap, PermGen samples and "Perform GC" and "Heap Dump" on demand. Both histograms has short information about classes, instances and overall size, and has three operation, "Pause" stops refreshing and it waits to push "Update" button, and "Deltas" switch to delta result mode. On the bottom we have filtering option to the histogram. Quite nice functionality for free.

VisualVM-JConsole: There is another command line tool called jconsole, and you can provide additional plugins for them, this plugin allows you to put the same jconsole plugin into VisualVm tool. More information.

VisualVM-MBean: MBean viewer from jconsole,  You have access to all registered MBean. I assume that you are familiar with JMX and MBeans, so I just put screen shot for you.

MBean Browser

On the left is MBean tree group by MBean’s category, and on the right are MBean’s properties, grouped by Attributes, Operations (methods), Notification (if MBean defines notifications) and Metadata (all information provided with MBean).

Visual GC: Integration of the Visual GC tool into VisualVM. It is quite nice and quick view of java heap utilization, maybe it is not so detailed but you can quickly figure out what’s going on.

VisualGC

On the left we have spaces with information how full space are, on the right we have space trends and on the bottom we’ve got heap histogram.

SysTray: Allows to minimize VisualVM into system tray. Unfortunately it is not supported on Mac OS X.

KillApplication: Very simple functionality, from menu we can kill any java attached application. Just press right mouse button on the application and choose "Kill Application".

VisualVM-JvmCapabilities: Sample plugin displaying JVM capabilities in Overview tab. It is for learning proposes.

Demo plugin

In additional tab we see more information about currently attached JVM.

VisualVm-Logfile Module: Logfile Plugin needed by TDA Plugin. Support JVM log file parsing, we can use it into our plugins.

VisualVm TDA ModuleThread Dump Analyzer Plugin. Now you can use TDA directly from VisualVm. Currently there are some problems with this plugin, but from my point of view it’s worth to manage this problems. Problems mainly are relevant for displaying properly TDA workspace. This is sample screen after manage displaying problems.

Thread Dump Analyzer

On the top you have to action button, they are responsible for Thread Dump tree operation. In the upper left corner you have standard TDA thread groups with monitors. On the right TDA displays current chose group and on the bottom TDA display current chose thread. The great thing is that in thread panel you can pressed "Thread Dump" button and after that you simply select Thread Dump and TDA is opening automatically. Great tool.

Is this all ?!?

In fact no. Of course you can look around jconsole plugin or write it by ourselves. Another option is to write by ourselves VisualVm plugin which can be very easy task due to VisualVM Sample Collection. This is a collection of NetBeans example projects which are VisualVM plugins. There are also some others sample projects, you can find them here.  Also VisualVM team provides great information how to start with VisualVM API.

Currently there is under development a plugin to Garbage Collection Histogram Tool. Another Plugin in development is BTrace plugin. We can install it by adding additional VisualVm plugin repository. All the information is provided for you here.

Btrace plugin

You can write BTrace script to find out what’s going on. User Guide is here.

With GHisto is more work to do. The team is working hard, actually you can download GCHisto. You download and run it by java -jar command.

, I’m still waiting for the VisualVM plugin.

Conclusion

I install all of this plugins, and I disabled SysTray and JavaME Profiler. I put it on the USB stick and have always with me. Unfortunately it works the best with Java 6, so if you have legacy systems it will be hard to use all the features. Also if you go into problems such as this:

Profiler Agent: JNI On Load Initializing...
Profiler Agent: JNI OnLoad Initialized succesfully
Profiler Agent: Waiting for connection on port 5140 (Protocol version: 8)
Profiler Agent: Established local connection with the tool
#
# An unexpected error has been detected by Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6ddc1475, pid=2768, tid=4136
#
# Java VM: Java HotSpot(TM) Server VM (11.3-b02 mixed mode windows-x86)
# Problematic frame:
# V  [jvm.dll+0x291475]
#
# An error report file with more information is saved as:
# c:\development\svn\unite-trunk\hs_err_pid2768.log
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

Check if both Application and VisualVM work on the same JVM.  I get this error due to run application on Java 6u13 and VisualVM running on Java 6u14.