JIRA Tuning – part one

There was a time when I did some JVM tuning. It’s time to reactivate the part of the brain responsible for this activity.

The key principle is to prepare a baseline. What is it?

The baseline refers to:

  • the first measuring mechanisms (np. gc.log) – we should measure on different levels (cpu, I/O, database etc) if in doubt follow thus rule: “more is better”
  • a load test script that makes JIRA work (e.g. Apache JMeter script),
  • a test environment – should be identical as production (different environment create different baseline), and,
  • a desired goal (we tend to tune solutions not having any goal). Never tune just for tuning.

The next required value is time… a lot of time. The reason is simple. We make only one change at a time and when it is completed, we launch load scripts and use metrics to see if we’re going in the right direction. The whole operation is based on making several steps forward followed by a few steps back.

Our recent goal was to make JIRA stable, despite working slow (no restart during the day).

We launched the analysis of gc.loc. Due to short pauses, we used the Concurrent Mark Sweep collector. Unfortunately, there are two sides of a coin. As CMS does not compact the memory, you may not be able to place any object in free spaces despite having free memory.

This usually results in the death of JVM – when looking for some free space, you do some cleaning, which takes pretty long time. This causes a “slashdot effect”. As a result you need more and more time, because of which the main GC works and the remaining part is stuck -> reset!

OK, we update two parameters causing CMS to start working earlier – usually, it is launched when the old generation is filled up to 68% (exact value vary depending on JVM version):

  • -XX:CMSInitiatingOccupancyFraction=40
  • -XX:+UseCMSInitiatingOccupancyOnly

Because of that, the whole commotion happens on the bottom of the stack. On the top, there is a place for larger objects. Obviously, you may (and will) experience a problem with stack fragmentation.

We launch load tests. The situation was better, but….

Unfortunately, we killed JIRA. The next decision is to limit the work of GC by reducing the pipe. We lower the maxThreads parameter on tomcat. We launch tests on one of the machines. Nothing happen. We add the second machine and start acting more aggressively. Result: although slow, Jira responds in a stable manner. This is what we were hoping for!

I may still try two other options:

  • -XX:+UseCompressedStrings
  • -XX:+OptimizeStringConcat

They were added in java 6u21 and 6u20. It is an interesting option, because it optimizes String-type objects, so that – whenever possible – it used 1 byte tables instead of 2 byte char tables. Most of the texts can be squeezed in a single byte. We check the memory usage histogram (previously 98% – char[]). This time a byte[] takes precedence over a char[].

In the meantime, the jConsole operator screams with surprise: “Pedro, what have you done?!”. This draws our attention (we think we’ve f… up something). Fortunately, all this hassle was about a 50% drop in memory usage. Uff, this is yet another confirmation that our switch is working.

Time to finish. We note down our ideas for later. Migration is in just two days. Quick help of our colleague gave us a new machine. We’ve finally got a place to use the G1 collector.

Model Driven Architecture live or dead?

In my wunderlist‘s “watch this” list I found this dinosaur movie to watch: MDA: A forlorn hope. by Uncle Bob. It was posted over two years ago and it was viewed almost 55 thousand times.

MDA is easy isn’t it? The MDA Guid has only 62 pages. We need some modeling tools and another tool for model to code transformation. Sounds easy :)

Few years ago, we’ve been using MDA approach in our project. We used MagicDraw for UML part and AndroMDA for UML to Java code generation (of course there are many other tools). From my point of view it was great experience. I share with my opinion below:

  • (pros) Model and Factory for free.
  • (pros) Hibernate mapping for free.
  • (pros) Documentation is up to date – you have to modify it to generate changes.
  • (pros) We focused on design, before coding.
  • (pros) All you hibernate mapping/DAO/etc is similar ( standardized – we can modify template)
  • (pros/cons) We can/have to change templates to align to ours standards.
  • (cons) Lot of  code you prefer to never read :/
  • (cons) Every time you change something, you have to regenerate code.
  • (cons) You have codebase divided into “read only” and “change here”.
  • (cons) If you forget to put additional metadata into model you’ll be doomed in future.
  • (cons) Once you change template, it is harder to upgrade tools and we have to regenerate code.

Nowadays I compare that experience to modern frameworks such as Rails or Django. I have to add comment here, what I really mean is that by using MDA approach I do not think about database mapping, DAO/repository object, I started at service level, and I have similar fillings when I’m using Rails or Django framework. I’m focusing on business logic, not how to get or save data into persistence storage.

Uncle Bob is talking about analyst as software creators, and this idea fortunately for my salary is impossible ;).

I totally agree, doing software is on much more detail level than model thinking, but … I think it wasn’t so bad to think about design on high level and then generate code and go deep in business logic details. How many times Java programers do the same job: create POJO, annotate or write XML descriptor, create DAO which looks more or less the same as another one, etc.

Let cite some smart guys from Uncle Bob post:

  • Uncle Bob: “Programmers are details managers – sorry MDA” :)
  • Comment: “MDA is actually based on two grand ideas:
    - raising the level of abstraction above programming language.
    - satisfying everyone with universal set of *standard* abstractions. Most of MDA failures is due to the the second idea, which is why MDA (not MDE in general) may be indeed a forlorn hope.”

I don’t want to force you to use MDA, just think about it and in …meantime … find the difference :).

Old times

Blog post before 1st May

nowadays

Blog post after 1st May

Yep, posterous is dead. Is MDA dead? Please share your opinion in comments.

 

It is better to harden your WordPress

During the Polish Long Weekend (actually very long), I had some speare time. I had great time with family, so after my little devils felt asleep. I though: “Let me see what I have in my draft post list” (which it is huge btw). One of interesting entry from January is “Hardening Wrodpress”.

Finish it!

The story went like that: I opened my blog at pietrowski.info, but instead of my blog I had got this (see screenshots).

chrome warning

chrome warning

safari warning

safari warning

 

firefox warning

firefox warning

What’s the hell! I check different web browsers and I had the same judgment.

Fortunately I’m Google Webmaster Tools user, and I remembered that there is diagnostic page, I ran it for pietrowski.info (click link to see actual report). Original result (4.january.2013 below).

problem-found

With my colleague (js-expert), we did backward engineering of the script. Nothing interesting,  I can put gists in comments (have to grep gTalk history :)) if you are interested in, anyway the security issue was in one of the plugin, so I made quick decision – remove it (I have no code highlighter right now ;/).

What is extremely important here is that, instead reading yet another tabloid (put your favorite name here), read security information of your platform, and perform updates if needed.

Of course I had lamer (mean standard) configuration of WordPress.

So I went through those documents:

and you should at least consider this ideas:

  • Authentication Unique Keys and Salts
  • WordPress Database Table prefix
  • Protect wp-config.php
  • Relocate Your wp-content Folder

Last but most important step was to beg google ;). You can do it through Google Webmaster Toolkit (ask for review Health->Malware). It may take up to few days to review your site (in my case 3 days). I’ve updated plugins and WordPress engine:

NewerWordpress

 

And this it is.

DevCrowd 2013

This was the 3rd time I was a presenter on the DevCrowd  conference. In fact twice on DevCrowd and once on Java4People. The variety of attendees and presenters was huge, and we saw that all the time: during questions, presentation styles, there were also presentations with meat (I  mean code) and there was a presentation about soft things too.
The venue, as always, was prepared in details, we met at Zachodniopomorski Technical University in Szczecin. The guys from Szczecin JUG did a great job (as always) and prepared for us T-shirts, great lunch, comfortably furnished rooms and last but not least an “After Party”.
Only one question remained: “Will the presenters do a great job too?“.
Let’s make some retrospectives.
  1. Prawo Javy“ (ang. The Java Law) – a quite nice presentation. From a different perspective (Rafał is a lawyer). The main thought was how different law interpretations can create new environment, and that this environment can make business die or flourish. A lot of real life examples like that Microsoft gets royalty fees for every Android phone. One extra tip is that “inspiration” :) overrides stupid patents!
  2. Six Sigma w Projektach IT”  - (ang. Six Sigma in IT projects). I think there is one interesting point: “after implementing Six Sigma you will have 3 to 4 mistakes per 1 million opportunities. Additionally, the Six Sigma organization ensures that you will make some savings.
  3. Big Data: Big problem or Big Opportunity? – this was my presentation, sooooo it is pretty hard to say if it was good or not. Anyway, I hope in the future people will think about: metrics (before implementing the feature), that we enter the information era and a lot of things are going on because of that.
  4. Jak zwiększyć wydajność pracy?“ (ang. How to increase your performance at work) and “Scalding, czyli: WordCount Hadoopem nie musi mieć 70 linii“ (ang. Scalding therefore, WordCount Hadoop doesn’t have to have 70 lines of code) – I was on the first one but my colleague was on the second one. My thoughts: great presentation, Łukasz tried to explain to us how our mind works. After his presentation I avoid beer (for a while), I will eat healthy food, and I have two more books to read. The first one: Pragmatic Thinking and Learning and the second one Think Fast and Slow (btw the second link is a mobile one :)). And one more thing, next time please give Łukasz more time :). Anyway Łukasz thanks for vodka & water and some other tips :).My colleague’s thoughts on the second presentation are: a lot of code and one thing to remember Hadoop empowered by Scala helps us to write concise code, which is easy to read.
  5. API Driven Development” – a solid presentation about REST, it is worth to learn about REST. We should follow Bezos principles and think about externalization of our API to the world from the very beginning.
  6. Księżycowo podbudowane testowanie, czyli rzecz o sklecaniu własnej podbudowy do zautomatyzowanych testów funkcjonalnych aplikacji sieciowych“ (ang. ?!?@!#%$! – If somebody wants to have self-crypted titles please talk to Filip) – Another solid presentation. Filip showed us how importatnt automatic tests are, and that a tester should not be afraid to learn and start writing those tests.
  7. Last but not least presentation was “How secure your web framework is?“- Łukasz as a lead and committer of Apache Struts2 framework told us about security holes. From Six Sigma presentation we know that bugs exist, so it is not strange to find those bugs in every web framework. Yes, even if it is written in Java. Main point: Next time instead of reading a tabloid, please check security bulletin of your framework.
As you can see, it was a huge portion of wisdom. What can I say – congratulations for both organizers and presenters.
But, wait, there is one more thing - After Party.
The after party was awesome. We went to Brama Jazz Cafe.  Great food, great beverages and great people are ingredient of awesome party. I met great people, shared a lot of different ideas and get a lot of feedback for my ideas. The locals directed us to some other place, and suggested us to drink The Flaming Poodle shot. They hadn’t to repeat this :). We went to Hormon (and. Hormone)   what can I say:  The Flaming Poodle shot was awesome and kicked ass. We finished early :) on After After After Party - the sun had risen. But definitely it was worth it and thanks to Łukasz suggestions I hadn’t no hangover at all. Once again thanks for all tips :)
To summarize this I asked my friends on our way back about their opinions:
  • Marta: It is a good place to start your presenter career here, organizers are very helpful, and people are great!
  • Irek: I’m a newbie, so I learnt a lot! OMG! I have so much to learn about right now.
  • Krzysiek: Great conference, I’m pretty sure I want to be a presenter on DevCrowd 2014, so be prepared for my speech.
My one sentence opinion is: Great job! Thanks guys. If you choose me, I will come to DevCrowd 2014 for sure.
Photos from conference are here:
  • The beginning

    The beginning

    Presenters Headquarter

    Presenters Headquarter

    Awesome Conference Organizers (part of them)

    Awesome Conference Organizers (part of them)

    Awesome Conference Organizers (part of them)

    Awesome Conference Organizers (part of them)

    Allegro & Pedro

    Allegro & Pedro

    The End

    The End

    Way back

    Way back


Monitoring and metrics – Yes, of course.

I always try to convince everybody to measure. The first reaction are very different, but after a while everyone come back and says “Wow! I didn’t realize how helpful metrics can be”.

I’ve watched video: ”Using Monitoring and Metrics to Learn in Development”  (by Patrick Debois from Atlassian) with pleasure. I use Atlassian tools and talk with few guys. They really care about code of their products. Patrick not only talks about metrics but also talks about technics and ideas which helps deliver better software.

Ideas from presentation:

  • smaller and frequent changes – easier to repair (dev for ops)
  • faster and better feedback – easier to find problem (ops for dev)
  • continuous integration maturity model – see slideshare presentation.
  • reuse “workflows” across environments by using virtualization – vagrant (great tool for building dev environments), puppet/chef (configuration automation and management)
  • infrastructure code repository and application code repository have to be in sync.
  • always remember that: “a lot of different monitoring levels we have” :).
  • monitoring driven development :) – create a monitor check before implementing a feature (it is useless but you can think about similarities
  • monitoring tools grumble (around 00:19:20) – there is a projects by “Monitoring Sucks“ (github), you can check but only one seems to be alive.
  • use monitoring as a service - this sound reasonable (pingdom, NewRelict, boundary, librato, and some others).
  • and  lot of other tools are mentioned - if you want to evaluate tools for metrics and monitoring you should watch video and note potential tools for evaluation.
  • always know the context of the metrics.
  • final thought: Metrics Driven Engineering (Etsy on the stage) - IMHO this is great idea to follow.

Whatever we will doing, it will happen our code do not work, our code stinks and become worser, but every time we figure out, we can do something with it. This is the reason it is so important to monitor our application continuously.

Patrick also mention about developer and operation responsibility sharing, he wrote a nice blog post: Just Enough Developed Infrastructure. I recommend you to read it too.