Category Archives: java

Grail Quest to find where went our productivity to (part two) ?

In Part one we talked about problems with productivity instead of great support from community, adding more resources, introducing scurm or other lean processes.

We end up with resource efficiency (engineers), today we follow the next problem with throughput.  In every company there are more project, more important things to do than people.

The number of projects we are working on (WIP) is growing. Worst of all is when the project with V.I.P. priority shows up guided by C-Level (board, owner). I saw the company that had assigned more than one project to developer.  I saw the company, which switch from the “only one important” project to another “only one important” project.

As you can guess the work of developers is not easy in such environment.

It reminds me my sister current problem with “developers”. She decided to build a house and so she hired a “crew”. Quite soon, she noticed that the work was not going as fast as she had expected (as scheduled). It turned out that “Team” has learned that from time to time they have to wait for materials (eg. cement, bricks, etc.), or her (the client’s) decision. So the team worked on more than one construction site (sells time to different client) to be fully utilized and to fulfill needs from different clients. Theoretically, a brilliant idea, in practice susceptible to all kinds of “deadline missing” or unforeseen events both positive ( bricks came earlier) and negative (delay in owner decision).

This causes a lot of tension between clients and  the “team”.  It ends up in losing trust. You probably smile now: how many times IT (“team”) is taking orders (“additional work”) from different departments: marketing, sales, CTO, CIO (“clients”).

IT tries to juggle from project to project, so they appear to move forward. Usually not fulfilling its promises, and many of the solutions are makeshift. This situation negatively affects mutual relationships. On the other hand developers themselves are also unhappy because many of their solutions are not implemented,  has poor quality, with technical debt growing exponentially (this is story for another post).

The conclusion is one: it is necessary to control the amount of work performed at the same time. Switching the work kills your productivity. It makes many things “not production ready yet”, and the code not on production is WASTE.

With greater focus we can deliver things much faster and in efficient way, and after delivery move to the next task. We may say that the more things you do at the same time, the less things are done.

Here we have “Little’s Law“, the law is completely unrelated to IT, but it is useful to determine the relationship between work is progress (WIP) and the speed of execution (Lead Time). We can define it as follows:

WIP = λ * LT


  • WIP = how much task is to do (work in progress)
  • λ = how fast new task comes
  • LT = how much time the task is in queue (on average)

Assuming we have constant processing speed.  As λ can be assumed as proportional to our throughput, which gives us the relation LT = WIP / Throughput. We can conclude that the more tasks we have, the longer it takes to finish them.

Please watch this movie now. After that even my grandmother understand WIP.

This movie reminds me how often we try to accelerate the development of software. We announce deadline and watch if developers can do it (even they are saying it is impossible). The quality of this software is very similar to that what you have just watched. In the end Project Manager comes and say: “hmmm you did it, let’s go faster” 🙂

“Deciding what not to do is as important as deciding what to do.”

Steve Jobs

Let’s remember this: it is worth paying attention to the amount of simultaneous work and the pace of adding a new one.

In this part we focus on work in progress. It shows up from both direction. From the team, which try to utilize their time, and from clients, who tries to push as much work as their can. Stay tuned for final third part.

By the way, as Tech Rebels we help companies go through technical hurdles. Just hire us, and we will bring your organization to next technical level.


JIRA Tuning – part one

There was a time when I did some JVM tuning. It’s time to reactivate the part of the brain responsible for this activity.

The key principle is to prepare a baseline. What is it?

The baseline refers to:

  • the first measuring mechanisms (np. gc.log) – we should measure on different levels (cpu, I/O, database etc) if in doubt follow thus rule: “more is better”
  • a load test script that makes JIRA work (e.g. Apache JMeter script),
  • a test environment – should be identical as production (different environment create different baseline), and,
  • a desired goal (we tend to tune solutions not having any goal). Never tune just for tuning.

The next required value is time… a lot of time. The reason is simple. We make only one change at a time and when it is completed, we launch load scripts and use metrics to see if we’re going in the right direction. The whole operation is based on making several steps forward followed by a few steps back.

Our recent goal was to make JIRA stable, despite working slow (no restart during the day).

We launched the analysis of gc.loc. Due to short pauses, we used the Concurrent Mark Sweep collector. Unfortunately, there are two sides of a coin. As CMS does not compact the memory, you may not be able to place any object in free spaces despite having free memory.

This usually results in the death of JVM – when looking for some free space, you do some cleaning, which takes pretty long time. This causes a “slashdot effect”. As a result you need more and more time, because of which the main GC works and the remaining part is stuck -> reset!

OK, we update two parameters causing CMS to start working earlier – usually, it is launched when the old generation is filled up to 68% (exact value vary depending on JVM version):

  • -XX:CMSInitiatingOccupancyFraction=40
  • -XX:+UseCMSInitiatingOccupancyOnly

Because of that, the whole commotion happens on the bottom of the stack. On the top, there is a place for larger objects. Obviously, you may (and will) experience a problem with stack fragmentation.

We launch load tests. The situation was better, but….

Unfortunately, we killed JIRA. The next decision is to limit the work of GC by reducing the pipe. We lower the maxThreads parameter on tomcat. We launch tests on one of the machines. Nothing happen. We add the second machine and start acting more aggressively. Result: although slow, Jira responds in a stable manner. This is what we were hoping for!

I may still try two other options:

  • -XX:+UseCompressedStrings
  • -XX:+OptimizeStringConcat

They were added in java 6u21 and 6u20. It is an interesting option, because it optimizes String-type objects, so that – whenever possible – it used 1 byte tables instead of 2 byte char tables. Most of the texts can be squeezed in a single byte. We check the memory usage histogram (previously 98% – char[]). This time a byte[] takes precedence over a char[].

In the meantime, the jConsole operator screams with surprise: “Pedro, what have you done?!”. This draws our attention (we think we’ve f… up something). Fortunately, all this hassle was about a 50% drop in memory usage. Uff, this is yet another confirmation that our switch is working.

Time to finish. We note down our ideas for later. Migration is in just two days. Quick help of our colleague gave us a new machine. We’ve finally got a place to use the G1 collector.

Model Driven Architecture live or dead?

In my wunderlist‘s “watch this” list I found this dinosaur movie to watch: MDA: A forlorn hope. by Uncle Bob. It was posted over two years ago and it was viewed almost 55 thousand times.

MDA is easy isn’t it? The MDA Guid has only 62 pages. We need some modeling tools and another tool for model to code transformation. Sounds easy 🙂

Few years ago, we’ve been using MDA approach in our project. We used MagicDraw for UML part and AndroMDA for UML to Java code generation (of course there are many other tools). From my point of view it was great experience. I share with my opinion below:

  • (pros) Model and Factory for free.
  • (pros) Hibernate mapping for free.
  • (pros) Documentation is up to date – you have to modify it to generate changes.
  • (pros) We focused on design, before coding.
  • (pros) All you hibernate mapping/DAO/etc is similar ( standardized – we can modify template)
  • (pros/cons) We can/have to change templates to align to ours standards.
  • (cons) Lot of  code you prefer to never read :/
  • (cons) Every time you change something, you have to regenerate code.
  • (cons) You have codebase divided into “read only” and “change here”.
  • (cons) If you forget to put additional metadata into model you’ll be doomed in future.
  • (cons) Once you change template, it is harder to upgrade tools and we have to regenerate code.

Nowadays I compare that experience to modern frameworks such as Rails or Django. I have to add comment here, what I really mean is that by using MDA approach I do not think about database mapping, DAO/repository object, I started at service level, and I have similar fillings when I’m using Rails or Django framework. I’m focusing on business logic, not how to get or save data into persistence storage.

Uncle Bob is talking about analyst as software creators, and this idea fortunately for my salary is impossible ;).

I totally agree, doing software is on much more detail level than model thinking, but … I think it wasn’t so bad to think about design on high level and then generate code and go deep in business logic details. How many times Java programers do the same job: create POJO, annotate or write XML descriptor, create DAO which looks more or less the same as another one, etc.

Let cite some smart guys from Uncle Bob post:

  • Uncle Bob: “Programmers are details managers – sorry MDA” 🙂
  • Comment: “MDA is actually based on two grand ideas:
    – raising the level of abstraction above programming language.
    – satisfying everyone with universal set of *standard* abstractions. Most of MDA failures is due to the the second idea, which is why MDA (not MDE in general) may be indeed a forlorn hope.”

I don’t want to force you to use MDA, just think about it and in …meantime … find the difference :).

Old times
Blog post before 1st May
Blog post after 1st May

Yep, posterous is dead. Is MDA dead? Please share your opinion in comments.