Archive for the ‘technology’ Category

VisualVm performance tuning tool

Sunday, June 14th, 2009

Few weeks ago I’ve got Performance Tuning Training made by Kirk. It was great time for me and great training. If you can manage such training go there. You won’t regret it. After this training I’ve got some deeper knowledge about performance tuning and performance tuning tools. One of such tool is VisualVm.  If you download latest Java 6 update 14 (May 28), than you have one more option to perform application tuning. Beside of jps, jstat, jmap you get jvisualvm. The tool has long story, from NetBeans plugin to standard java tool.  If you have earlier java version you can also download it separately from project page visualvm.dev.java.net .

The VisualVm developers made very good job providing plugin API. If you write any plugin for NetBeans Platform you know how to write plugin for VisualVm tool. In this post I will describe available plugins. How to install them and how to use them. The plugin menu is under Tools->Plugin. We can setup there plugins repositories. We can also manage our plugins (install, uninstall, activate, deactivate).

As I write this post currently available plugins are:.

VisualVM-Glassfish: Advanced monitoring of Glassfish server. I use Glassfish as development server in daily work and the plugin is very useful.  I will not describe this plugin. It has very nice and detailed description made by Masoud Kalali.

Java ME Priofiler Snapshot Viewer : Java ME SDK 3.0 offline profiling session viewer.  I’ve got very little with Java ME, so I stay just on this short  description.

VisualVm-Extension : Provide support for extending VisualVM for new JDK, JVM etc. It is for extension proposes in the future.

Memory Sampler : Experimental sampling memory profiler. It’s similar to jmap -histo, an additional feature over the jmap  is possibility to display delta results. If you try to find heap memory problem or PermGen space problems this tool can help you to figure out what going there.

memory sampler

From the top on the left you can set refresh rate, on the right you can set Heap, PermGen samples and "Perform GC" and "Heap Dump" on demand. Both histograms has short information about classes, instances and overall size, and has three operation, "Pause" stops refreshing and it waits to push "Update" button, and "Deltas" switch to delta result mode. On the bottom we have filtering option to the histogram. Quite nice functionality for free.

VisualVM-JConsole: There is another command line tool called jconsole, and you can provide additional plugins for them, this plugin allows you to put the same jconsole plugin into VisualVm tool. More information.

VisualVM-MBean: MBean viewer from jconsole,  You have access to all registered MBean. I assume that you are familiar with JMX and MBeans, so I just put screen shot for you.

MBean Browser

On the left is MBean tree group by MBean’s category, and on the right are MBean’s properties, grouped by Attributes, Operations (methods), Notification (if MBean defines notifications) and Metadata (all information provided with MBean).

Visual GC: Integration of the Visual GC tool into VisualVM. It is quite nice and quick view of java heap utilization, maybe it is not so detailed but you can quickly figure out what’s going on.

VisualGC

On the left we have spaces with information how full space are, on the right we have space trends and on the bottom we’ve got heap histogram.

SysTray: Allows to minimize VisualVM into system tray. Unfortunately it is not supported on Mac OS X.

KillApplication: Very simple functionality, from menu we can kill any java attached application. Just press right mouse button on the application and choose "Kill Application".

VisualVM-JvmCapabilities: Sample plugin displaying JVM capabilities in Overview tab. It is for learning proposes.

Demo plugin

In additional tab we see more information about currently attached JVM.

VisualVm-Logfile Module: Logfile Plugin needed by TDA Plugin. Support JVM log file parsing, we can use it into our plugins.

VisualVm TDA Module:  Thread Dump Analyzer Plugin. Now you can use TDA directly from VisualVm. Currently there are some problems with this plugin, but from my point of view it’s worth to manage this problems. Problems mainly are relevant for displaying properly TDA workspace. This is sample screen after manage displaying problems.

Thread Dump Analyzer

On the top you have to action button, they are responsible for Thread Dump tree operation. In the upper left corner you have standard TDA thread groups with monitors. On the right TDA displays current chose group and on the bottom TDA display current chose thread. The great thing is that in thread panel you can pressed "Thread Dump" button and after that you simply select Thread Dump and TDA is opening automatically. Great tool.

Is this all ?!?

In fact no. Of course you can look around jconsole plugin or write it by ourselves. Another option is to write by ourselves VisualVm plugin which can be very easy task due to VisualVM Sample Collection. This is a collection of NetBeans example projects which are VisualVM plugins. There are also some others sample projects, you can find them here.  Also VisualVM team provides great information how to start with VisualVM API.

Currently there is under development a plugin to Garbage Collection Histogram Tool. Another Plugin in development is BTrace plugin. We can install it by adding additional VisualVm plugin repository. All the information is provided for you here.

Btrace plugin

You can write BTrace script to find out what’s going on. User Guide is here.

With GHisto is more work to do. The team is working hard, actually you can download GCHisto. You download and run it by java -jar command.

, I’m still waiting for the VisualVM plugin.

Conclusion

I install all of this plugins, and I disabled SysTray and JavaME Profiler. I put it on the USB stick and have always with me. Unfortunately it works the best with Java 6, so if you have legacy systems it will be hard to use all the features. Also if you go into problems such as this:

Profiler Agent: JNI On Load Initializing...
Profiler Agent: JNI OnLoad Initialized succesfully
Profiler Agent: Waiting for connection on port 5140 (Protocol version: 8)
Profiler Agent: Established local connection with the tool
#
# An unexpected error has been detected by Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6ddc1475, pid=2768, tid=4136
#
# Java VM: Java HotSpot(TM) Server VM (11.3-b02 mixed mode windows-x86)
# Problematic frame:
# V  [jvm.dll+0x291475]
#
# An error report file with more information is saved as:
# c:\development\svn\unite-trunk\hs_err_pid2768.log
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

Check if both Application and VisualVM work on the same JVM.  I get this error due to run application on Java 6u13 and VisualVM running on Java 6u14.

Posted in java, technology | 2 Comments »

AOSD – Finally Thoughts

Friday, March 6th, 2009

We started with Keynote by John A. Stankovic from Universtity of Virginia, it was really cool keynote. He talk about defense system  The dynamics aspects where use to implement mechanism “right defense at the right time”. He describe real life problems with messages encryption, power management and sensor localization. The localization problem comes from that GPS do works only on open space. Thanks to aspects they provide sensor communication protocol. Also power management was treated as crosscutting issue and John describe protocols they develop for this propose. The final John thought is : “Flexibility offered by Dynamic AOP has great potential”.
I went to demo sessions, there was two demos, the first one is out of my interests, the second was quite interesting and open my eyes. I thought that JDK classes and aspect are not woven because of security reasons (this is what AspectJ team claims). Now I know that this issue is due to bootstrap classloader. It makes impossible to wave aspect to for example String class. The demo was about the tool which can do it. There are tools for that purpose called MAJOR and CARAJillo, one limitation of this tools is that JVM classes can only be woven statically. Great stuff.

Industry Panel was about “Challenges and Roadmap for Using AOSD in Industry”. The panel was rather boring. It was very slow and some trash talking: “it not easy to adopt new things to industry”, “risk management”, “academic solution doesn’t scale”, “no trust for aspect”,  “company policies ” and so on. Finally one of the panelist said that “Why we should use aspects if OOP currently works well”.There was few examples of industry usage from Accenture and Siemens.

Another demo session, two quite nice demos. First Lavash quite interesting probably not so aspect oriented framework of tools to automatic requirements processing. The second is in early stage and was about suggestion pointcut modification as we write new code. This may be very interesting tool in the future to keep programmers from silly mistakes. I will keep eye on it.

Industry session was more likely as panel. One good tough was that academic researcher should provide tools to make aspects fully testable. There was another aspect language proposal called “e” which mainly differs from AspectJ that AspectJ is focus on classes and “e” language on units. The session was improved by Uwe Hohenstein from Siemes. Uwe shows real project usage of aspects. Despite obvious examples such as connection pool monitoring,  performance monitoring. Siemens use aspect to address challenges of integrating 3rd party software in a maintainable manner keeping 3rd party software untouched. Most of the examples comes from Siemens Soarian project.

The last day came so quickly. AOSD is high level rather academic conference, but we should learn from the best.

Tags:
Posted in java, methodology, technology | 1 Comment »

AOSD Conference Day

Thursday, March 5th, 2009

The conference was started by Keynote. Paul Daugherty from Accenture talk about cost, and how they try to reduce it. 91 from top 100 companies are Accenture’s clients. They have consultants on every place in The Earth. He told that about 60% of cost goes to maintenance and keep live system. The solution is to use many different techniques as SOA architecture and Aspect Oriented Programming. Quite interesting and based on real example speech.
Next three papers was presented: Dependent Advice, Generic and Reflective Debugging Architecture and Expressive Scoping of Distributed AOP.  From my point of view the best of them was Generic and Reflective Debugging Architecture. Wouter has showed us problems when we must debugging system with aspect and where is the gap between program model and JVM TI. They write common aspect abstraction and it nearly ready for use. Great job.

After Lunch I went to Demo sessions. First demo was quite interesting made by Siemens. Product Lines  – features as aspects was about deliver features as crosscutting concerns implemented as aspects. Often our product is deliver to different client, and must get special features. Implementing this features as aspects can help us to keep product dependencies low and to deliver high quality software.
Next demo “Building a next generation digital news publishing” shows how introducing aspect helps to refactor and simplify code base. The problem was that news are deliver to different consumers (RSS, WWW, subscribers etc.) By using aspect they keep one source of news and many different consumers.

After break I saw demo “Druid – unexpected interaction detection”, it mangos the problem when introducing aspects can break other module tests. They make topological sort of modules and than executing junit tests, checking if something break if so, they provide special metadata. This metadata build graph about dependencies and unexpected interaction.  It may be used also to detects if every feature in module has it’s own test. The work is workable plugin for eclipse. The better integration between TDD and AOP. I talk with Andre Restivo and if you want to provide version for NetBeans or InteliJ feel free to contact with Andre. There is some kind of console magic so the plugin actually work only on Linux but should be no problem to port for OSX. There are some problems but you can download and try it now. I think that I will try to write port for InteliJ.

Last demo was funny, we have Frontend team which claims that Frontend will rule the Earth. Demo was about “AOSJ – aspect oriented programming framework for JavaScript” very interesting idea :)

The next day begin. Stay tuned.

Posted in java, methodology, technology | No Comments »

AOSD – Part One

Wednesday, March 4th, 2009

This year I choose as my conference 8th International Conference on Aspect-Oriented Software Development it is rather no commercial conference, and now I know it’s for sure not commercial, probably I’m only one from industry, other fellows are from Universities and so on. It is amazing experience.

First the trip from New York to Charlottesville, was quite interesting, when I will be back home I will publish this trip. Then the first day, it was a lot of good talk, from time to time there was hard time (e.g. type systems), next we have discussion about Aspect Oriented adoption in industry. From this discussion I’ve got some great ideas to realize, we’ll see.

The first day I mainly participated in Foundations of Aspect-Oriented Languages workshop, probably the closest to my interests was Mohamed ElBendary from University of Wisconsin Milwaukee paper presentation.  The most impressive person from my point of view was Mehmet Akşit from University of Twente. It gives quite nice talk about history of programming, and he has so many papers that I will spend another year to try to read them all. We’ve finished with Open session leaded by Shmuel Katz from Technion–Israel Institute of Technology.

The second day I had hard choose between Workshop on Early Aspects and 4th Domain-Specific Aspect Languages Workshop and finally I choose Early Aspects because it was more difficult so another workshop I will have occasion to read by myself.  I was right, it was hard, probably due to fact that I rather far from modeling and aspect there is some kind of magic for me.

So BPM with Aspect, Dependencies Graph and so on was hard. On the rescue  Mehmet comes with real application example for the Netherlands government. It was about traffic system, and Mehmet tells about it in details. It was worth to here this talk as we can teach from that that we should find crosscutting concerns on early stages of software development process, otherwise we don’t use the power of aspects.
And finally my time comes, Birds of a Feather with JBoss AOP. It was very good speech made by Kabir Khan
and Flavia Rainone from JBoss. They talk about details of JBoss AOP, about new features, plans and problems with new Microkernel based on OSGi. Currently JBoss AOP is the most advanced AOP implementation for day to day use. So you find more on this blog about it soon.
The conference have just started, but the level is still high, many papers and smart people. It is good to be here.

Tags:
Posted in java, methodology, technology | No Comments »

Testing – do we have to?

Friday, May 2nd, 2008

Reliability is the core of good testing. As we know our systems are made from many objects/components. Our system is as reliable as components on which it is build. It obvious that when our components are n percent reliable and we have k components our system is (n/100)^k percent reliable. We can quickly realize that if we have components that are not one hundred percent reliable, our system goes to zero percent as the number of components grow. From the other point of view if all of our components are one hundred percent reliable and we introduce to our system one component which is 50% it makes that our system is now 50% reliable. Of course it is theoretical, in practice we should provide weights for our components, and the system is rather non linear.

Now we agreed that reliability is good thing, but to talk about it we should measure it in some way. Of course to measure our system we should measure our low level components. To do this we must ensure that low level objects are reliable through unit testing. We must run this tests as often as we can (probably every commit), so the most important is that this tests must execute quickly (probably 2-3 second is maximum). And for such task the automation is needed. The answer is continuous integration tools, such as ( Atlassian Bamboo, TeamCity, Hudson, CruiseControl, etc.. ..). If you heard this idiom for the first time, you should read Martin Flower article about Continuous Integration.  BTW if you heard but don’t read this article you should read it, anyway. Even maven site goal may help here. In my company we use Atlassian Bamboo and it’s great tool, Atlassian has great license politics so if your projects is open source you can count on your free bamboo license ;) . If your company didn’t has money to buy such software you should see Hudson.

Test Types :

Unit Tests – verify behavior of small elements (single class). Tests should be quick and run as often as we can. Every language has it’s own test libraries. Some of them : JUnit and TestNG (java), unittest (python), Test::Unit module (ruby).

Integration tests - verify behavior of portion of a system, or subsystem. Requires installed external dependencies such as databases, ldap, etc.. This test tests code via API which is rather not accessible to the clients, and this test are rather quickly to execute. There are also some tools which makes this testing simpler: DBUnit (java), fixtures (ruby), pdbseed (python).

System tests – verify behavior of complete system, so all the system part must be fully installed. This kind of test runs for the long period of time, so the good idea is schedule this tests for night activity. Some of frameworks allows to do this test easily some of them not. For example we use webwork, which action class has no dependency to HTTP objects, and we can easily can test action simulating user actions.

Functional tests – calls also as acceptance tests, provides testing from the viewpoint of client. The tools for helping us are rather language independent, they depends rather on technology which clients use, for example for web application Selenium which simulates browser behavior, but for fat client other tools are.

There are more types but they has different propose, maybe I write about it someday:

  • UI testing (GUI testing, Usability testing, Accessibility testing)
  • Speed testing (Performance testing, Load testing,   Scalability testing, Stress testing)
  • Security testing
  • Smoke testing
  • Regression testing

and many others which I can’t remember now.
Summarize

  • Categorize your tests, and run this test separately, do not trust yourself or other developer use for this task continuous integration tools or call it from cron or similar tool.
  • Remember the cycle commit-test-result must be as quick as possible, there is no benefit from result which will came after developer goes to another task.
  • Heavy tests run nightly and when developer come to work he had result from them. Remember to do this you must have this tests automated.
  • And one important thing, if you ever find a defect, write test for it. The worse thing we can met is that our defect come back, so write test for defect first than resolve the problem. One point to mention is to write this test in such way that any new behavior is also tested (for example returning null when no data found)

Question : How about your tests, the speed, the quality, the schedule ?

Pedro

Posted in methodology, technology | No Comments »

« Older Entries

about me

My name is Sebastian Pietrowski. I've finished Warsaw University as Master degree. I started my journey with Java 1.1 with Thread and JDBC programing in 1998 as I worked for merlin.pl. In 1999 I've passed Java Programer Certificate for Java 1.2, and was solution architect of merlin.pl infrastructure when we was moving from pl/sql to J2EE. It was great performance optimization with 10 times more req/sec than in requirements and 85 times faster as original solution.

Currently I work as Expert Software Development Java at F.Hoffmann-La Roche. The company was founded in 1896 and today, Roche employs over 80.000 people. After work I'm involved in activities related to Scala/Lift, Ruby/Rails/Merb, Python/Django. This is because I try to be pragmatic also I'm focused on application performance and tuning with success in my daily work.

My Yoda's motto: Do, or do not. There is no try.