Category Archives: technology

Monitoring and metrics – Yes, of course.

I always try to convince everybody to measure. The first reaction are very different, but after a while everyone come back and says “Wow! I didn’t realize how helpful metrics can be”.

I’ve watched video: “Using Monitoring and Metrics to Learn in Development”  (by Patrick Debois from Atlassian) with pleasure. I use Atlassian tools and talk with few guys. They really care about code of their products. Patrick not only talks about metrics but also talks about technics and ideas which helps deliver better software.

Ideas from presentation:

  • smaller and frequent changes – easier to repair (dev for ops)
  • faster and better feedback – easier to find problem (ops for dev)
  • continuous integration maturity model – see slideshare presentation.
  • reuse “workflows” across environments by using virtualization – vagrant (great tool for building dev environments), puppet/chef (configuration automation and management)
  • infrastructure code repository and application code repository have to be in sync.
  • always remember that: “a lot of different monitoring levels we have” :).
  • monitoring driven development :) – create a monitor check before implementing a feature (it is useless but you can think about similarities
  • monitoring tools grumble (around 00:19:20) – there is a projects by “Monitoring Sucks” (github), you can check but only one seems to be alive.
  • use monitoring as a service - this sound reasonable (pingdom, NewRelict, boundary, librato, and some others).
  • and  lot of other tools are mentioned - if you want to evaluate tools for metrics and monitoring you should watch video and note potential tools for evaluation.
  • always know the context of the metrics.
  • final thought: Metrics Driven Engineering (Etsy on the stage) - IMHO this is great idea to follow.

Whatever we will doing, it will happen our code do not work, our code stinks and become worser, but every time we figure out, we can do something with it. This is the reason it is so important to monitor our application continuously.

Patrick also mention about developer and operation responsibility sharing, he wrote a nice blog post: Just Enough Developed Infrastructure. I recommend you to read it too.

skipfish – fast, easy and simple

Skipfish is google code project. It is web application security scanner, high speed (they claim 2000 requests per second* – * – at local LAN :)) and due the fact it is command line tool without fancy wizards, options and so on, it is relatively easy to use, and for sure it is easy to just start scanning.

Skipfish is active scanner so it first scan application, preparing the map of web site, than recursively ran different test, the last thing is report generation. Documentation is simple and has a lot of example we can start on. So let’s see that in action.

One of such command is:

$ skipfish -m 5 -LVJ -W /dev/null -o output_dir -b ie http://www.example.com/

During the scan, Skipfish is displaying statistics:

Scan statistics
---------------

       Scan time : 0:11:07.0068
   HTTP requests : 2446 sent (3.71/s), 16228.73 kB in, 659.18 kB out (25.32 kB/s)  
     Compression : 0.00 kB in, 0.00 kB out (0.00% gain)    
 HTTP exceptions : 34 net errors, 0 proto errors, 0 retried, 0 drops
 TCP connections : 2451 total (1.09 req/conn)  
  TCP exceptions : 0 failures, 1 timeouts, 0 purged
  External links : 745 skipped
    Reqs pending : 219        

Database statistics
-------------------

          Pivots : 471 total, 94 done (19.96%)    
     In progress : 323 pending, 38 init, 12 attacks, 4 dict    
   Missing nodes : 54 spotted
      Node types : 1 serv, 269 dir, 46 file, 1 pinfo, 91 unkn, 63 par, 0 val
    Issues found : 70 info, 111 warn, 49 low, 1 medium, 13 high impact
       Dict size : 0 words (0 new), 0 extensions, 0 candidates

After few hours/minutes, it depends on the site we are scanning, we will got

[+] Copying static resources...
[+] Sorting and annotating crawl nodes: 1666
[+] Looking for duplicate entries: 1666
[+] Counting unique issues: 1158
[+] Writing scan description...
[+] Counting unique issues: 1666
[+] Generating summary views...
[+] Report saved to outputDir/index.html
[+] This was a great day for science!

The report consist of “crawl results”, “document type overview” and “issue type overview”. My last scan result has some finding, but also has a lot of false positives, it seams that a lot of work still waiting for a Skipfish team, but it looks promising.

VisualVm performance tuning tool

Few weeks ago I’ve got Performance Tuning Training made by Kirk. It was great time for me and great training. If you can manage such training go there. You won’t regret it. After this training I’ve got some deeper knowledge about performance tuning and performance tuning tools. One of such tool is VisualVm.  If you download latest Java 6 update 14 (May 28), than you have one more option to perform application tuning. Beside of jps, jstat, jmap you get jvisualvm. The tool has long story, from NetBeans plugin to standard java tool.  If you have earlier java version you can also download it separately from project page visualvm.dev.java.net .

The VisualVm developers made very good job providing plugin API. If you write any plugin for NetBeans Platform you know how to write plugin for VisualVm tool. In this post I will describe available plugins. How to install them and how to use them. The plugin menu is under Tools->Plugin. We can setup there plugins repositories. We can also manage our plugins (install, uninstall, activate, deactivate).

As I write this post currently available plugins are:.

VisualVM-Glassfish: Advanced monitoring of Glassfish server. I use Glassfish as development server in daily work and the plugin is very useful.  I will not describe this plugin. It has very nice and detailed description made by Masoud Kalali.

Java ME Priofiler Snapshot Viewer
: Java ME SDK 3.0 offline profiling session viewer.  I’ve got very little with Java ME, so I stay just on this short  description.

VisualVm-Extension : Provide support for extending VisualVM for new JDK, JVM etc. It is for extension proposes in the future.

Memory Sampler : Experimental sampling memory profiler. It’s similar to jmap -histo, an additional feature over the jmap  is possibility to display delta results. If you try to find heap memory problem or PermGen space problems this tool can help you to figure out what going there.

memory sampler

From the top on the left you can set refresh rate, on the right you can set Heap, PermGen samples and "Perform GC" and "Heap Dump" on demand. Both histograms has short information about classes, instances and overall size, and has three operation, "Pause" stops refreshing and it waits to push "Update" button, and "Deltas" switch to delta result mode. On the bottom we have filtering option to the histogram. Quite nice functionality for free.

VisualVM-JConsole: There is another command line tool called jconsole, and you can provide additional plugins for them, this plugin allows you to put the same jconsole plugin into VisualVm tool. More information.

VisualVM-MBean: MBean viewer from jconsole,  You have access to all registered MBean. I assume that you are familiar with JMX and MBeans, so I just put screen shot for you.

MBean Browser

On the left is MBean tree group by MBean’s category, and on the right are MBean’s properties, grouped by Attributes, Operations (methods), Notification (if MBean defines notifications) and Metadata (all information provided with MBean).

Visual GC: Integration of the Visual GC tool into VisualVM. It is quite nice and quick view of java heap utilization, maybe it is not so detailed but you can quickly figure out what’s going on.

VisualGC

On the left we have spaces with information how full space are, on the right we have space trends and on the bottom we’ve got heap histogram.

SysTray: Allows to minimize VisualVM into system tray. Unfortunately it is not supported on Mac OS X.

KillApplication: Very simple functionality, from menu we can kill any java attached application. Just press right mouse button on the application and choose "Kill Application".

VisualVM-JvmCapabilities: Sample plugin displaying JVM capabilities in Overview tab. It is for learning proposes.

Demo plugin

In additional tab we see more information about currently attached JVM.

VisualVm-Logfile Module: Logfile Plugin needed by TDA Plugin. Support JVM log file parsing, we can use it into our plugins.

VisualVm TDA ModuleThread Dump Analyzer Plugin. Now you can use TDA directly from VisualVm. Currently there are some problems with this plugin, but from my point of view it’s worth to manage this problems. Problems mainly are relevant for displaying properly TDA workspace. This is sample screen after manage displaying problems.

Thread Dump Analyzer

On the top you have to action button, they are responsible for Thread Dump tree operation. In the upper left corner you have standard TDA thread groups with monitors. On the right TDA displays current chose group and on the bottom TDA display current chose thread. The great thing is that in thread panel you can pressed "Thread Dump" button and after that you simply select Thread Dump and TDA is opening automatically. Great tool.

Is this all ?!?

In fact no. Of course you can look around jconsole plugin or write it by ourselves. Another option is to write by ourselves VisualVm plugin which can be very easy task due to VisualVM Sample Collection. This is a collection of NetBeans example projects which are VisualVM plugins. There are also some others sample projects, you can find them here.  Also VisualVM team provides great information how to start with VisualVM API.

Currently there is under development a plugin to Garbage Collection Histogram Tool. Another Plugin in development is BTrace plugin. We can install it by adding additional VisualVm plugin repository. All the information is provided for you here.

Btrace plugin

You can write BTrace script to find out what’s going on. User Guide is here.

With GHisto is more work to do. The team is working hard, actually you can download GCHisto. You download and run it by java -jar command.

, I’m still waiting for the VisualVM plugin.

Conclusion

I install all of this plugins, and I disabled SysTray and JavaME Profiler. I put it on the USB stick and have always with me. Unfortunately it works the best with Java 6, so if you have legacy systems it will be hard to use all the features. Also if you go into problems such as this:

Profiler Agent: JNI On Load Initializing...
Profiler Agent: JNI OnLoad Initialized succesfully
Profiler Agent: Waiting for connection on port 5140 (Protocol version: 8)
Profiler Agent: Established local connection with the tool
#
# An unexpected error has been detected by Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6ddc1475, pid=2768, tid=4136
#
# Java VM: Java HotSpot(TM) Server VM (11.3-b02 mixed mode windows-x86)
# Problematic frame:
# V  [jvm.dll+0x291475]
#
# An error report file with more information is saved as:
# c:\development\svn\unite-trunk\hs_err_pid2768.log
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

Check if both Application and VisualVM work on the same JVM.  I get this error due to run application on Java 6u13 and VisualVM running on Java 6u14.

AOSD – Finally Thoughts

We started with Keynote by John A. Stankovic from Universtity of Virginia, it was really cool keynote. He talk about defense system  The dynamics aspects where use to implement mechanism “right defense at the right time”. He describe real life problems with messages encryption, power management and sensor localization. The localization problem comes from that GPS do works only on open space. Thanks to aspects they provide sensor communication protocol. Also power management was treated as crosscutting issue and John describe protocols they develop for this propose. The final John thought is : “Flexibility offered by Dynamic AOP has great potential”.
I went to demo sessions, there was two demos, the first one is out of my interests, the second was quite interesting and open my eyes. I thought that JDK classes and aspect are not woven because of security reasons (this is what AspectJ team claims). Now I know that this issue is due to bootstrap classloader. It makes impossible to wave aspect to for example String class. The demo was about the tool which can do it. There are tools for that purpose called MAJOR and CARAJillo, one limitation of this tools is that JVM classes can only be woven statically. Great stuff.

Industry Panel was about “Challenges and Roadmap for Using AOSD in Industry”. The panel was rather boring. It was very slow and some trash talking: “it not easy to adopt new things to industry”, “risk management”, “academic solution doesn’t scale”, “no trust for aspect”,  “company policies ” and so on. Finally one of the panelist said that “Why we should use aspects if OOP currently works well”.There was few examples of industry usage from Accenture and Siemens.

Another demo session, two quite nice demos. First Lavash quite interesting probably not so aspect oriented framework of tools to automatic requirements processing. The second is in early stage and was about suggestion pointcut modification as we write new code. This may be very interesting tool in the future to keep programmers from silly mistakes. I will keep eye on it.

Industry session was more likely as panel. One good tough was that academic researcher should provide tools to make aspects fully testable. There was another aspect language proposal called “e” which mainly differs from AspectJ that AspectJ is focus on classes and “e” language on units. The session was improved by Uwe Hohenstein from Siemes. Uwe shows real project usage of aspects. Despite obvious examples such as connection pool monitoring,  performance monitoring. Siemens use aspect to address challenges of integrating 3rd party software in a maintainable manner keeping 3rd party software untouched. Most of the examples comes from Siemens Soarian project.

The last day came so quickly. AOSD is high level rather academic conference, but we should learn from the best.

AOSD Conference Day

The conference was started by Keynote. Paul Daugherty from Accenture talk about cost, and how they try to reduce it. 91 from top 100 companies are Accenture’s clients. They have consultants on every place in The Earth. He told that about 60% of cost goes to maintenance and keep live system. The solution is to use many different techniques as SOA architecture and Aspect Oriented Programming. Quite interesting and based on real example speech.
Next three papers was presented: Dependent Advice, Generic and Reflective Debugging Architecture and Expressive Scoping of Distributed AOP.  From my point of view the best of them was Generic and Reflective Debugging Architecture. Wouter has showed us problems when we must debugging system with aspect and where is the gap between program model and JVM TI. They write common aspect abstraction and it nearly ready for use. Great job.

After Lunch I went to Demo sessions. First demo was quite interesting made by Siemens. Product Lines  – features as aspects was about deliver features as crosscutting concerns implemented as aspects. Often our product is deliver to different client, and must get special features. Implementing this features as aspects can help us to keep product dependencies low and to deliver high quality software.
Next demo “Building a next generation digital news publishing” shows how introducing aspect helps to refactor and simplify code base. The problem was that news are deliver to different consumers (RSS, WWW, subscribers etc.) By using aspect they keep one source of news and many different consumers.

After break I saw demo “Druid – unexpected interaction detection”, it mangos the problem when introducing aspects can break other module tests. They make topological sort of modules and than executing junit tests, checking if something break if so, they provide special metadata. This metadata build graph about dependencies and unexpected interaction.  It may be used also to detects if every feature in module has it’s own test. The work is workable plugin for eclipse. The better integration between TDD and AOP. I talk with Andre Restivo and if you want to provide version for NetBeans or InteliJ feel free to contact with Andre. There is some kind of console magic so the plugin actually work only on Linux but should be no problem to port for OSX. There are some problems but you can download and try it now. I think that I will try to write port for InteliJ.

Last demo was funny, we have Frontend team which claims that Frontend will rule the Earth. Demo was about “AOSJ – aspect oriented programming framework for JavaScript” very interesting idea :)

The next day begin. Stay tuned.