Performance tuning sucks

So, I have been doing a lot of tuning lately, and I will be the first to admit it. Performance tuning sucks, even here at YouDevise.

Let me start by telling you the reasons why performance tuning does NOT suck at YouDevise.

  • We DO have the smarts. Every developer (and most of the non-developers) can solve these performance problems once they understand the nature of the performance issue.
  • We DO have the tools. Or if we don’t today, Management (i.e. Rich/Colin/Squirrel) is sensible and can give us access to the tools tomorrow.
  • We DO have the diagnostics. Over the intervening years, YouDevise has developed some nice tools into and around IDS (like the Task Performance report, PerfManager, System Monitors, etc.) that give us great insight into problems when they appear.

Now, why does performance tuning suck at YouDevise?

  • Our application features keep changing. The easiest way to keep performance constant is to not change the code. If no one adds new code that changes the behavior of the previously tuned screen or process, then the previous optimization will not be lost. Now this isn’t fool-proof…
  • Our application’s data profile does not stay constant. A little over a year ago, when we last tuned the main Trade Idea review screen in the TIM, the heaviest user of the system was viewing 500 ideas at a time. Today, we have users regularly viewing 5000+ ideas at a time. An order of magnitude difference in data volumes warrants another round of tuning (because our customers always assume constant performance.) There are many reasons for the order of magnitude jump: simply more users, the network effects of more TIM users (i.e. the potentially exponential increase of TIM data) but…
  • Our applications are transforming. The TIM and the HIP both were initial designed to be applications that they are not today:
    • The TIM was going to be a simple trade tips system that acted as a basic central repository and transport mechanism. Now, it is morphing into a multi-purpose solution for portfolio management, feeding algorithmic systems, a compliance back-end, and a research tool. Oh, AND a trade tips system.
    • The HIP was initially conceived as a simple portfolio management tool that facilitated communication of position information between fund of funds and hedge funds. That last part has been mostly forgotten, and now the HIP is a full-fledged middle office system with performance, liquidity, and investor relation tools that is being used as a central system for fund of funds and their administrators.

I guess what I am trying to say is that performance tuning a moving target is hard, even with smart guys and good tools.

We will never win the war against performance problems.

There is no single, super-duper strategy that will make performance problems go away. The battlefield keeps changing. What we are fighting for and against keeps changing. You can win one battle, which causes you to lose two other ones.

What are we to do? Even though we will never win, we still must fight this battle.

  • Shore up your defenses. Any ground that you gain you should defend. If you tune a screen to open in 3 seconds for a certain 1000 rows, then you better create a test that confirms that. Even better automate that test case, and throw it into continuous integration. Then you will know when that one line of code is added that breaks your optimization.
  • Gather as much information you can about the enemy. Blindly attacking is worse than not attacking at all. Gather metrics and numbers. Never assume that you “know” what the problem is. Make sure you have something (slow SQL queries, profiler results, log files, etc.) to back up your performance optimization before you start on it.
  • Take it one battle at a time. Attack the enemies today, not the ones that you may have tomorrow. Wait for the performance bottleneck to appear. Do not assume that your code needs some fancy Hibernate second-level caching until you have some proof that those DB hits are there.

    Now, there needs to be some common sense applied to this rule. This does NOT mean that you should ignore performance of your code. Just keep in mind that any optimization generally adds complexity to your code and therefore decreases the maintainability of your code. There is a balance to be had. Further discussion on the topic.

Unfortunately, all of this is the nature of the business that we are in. If the TIM and the HIP stop innovating and stop providing value to our customers, then we are sunk. So, we need to accept change and realize that we are just going to need to be flexible and agile. Solve the performance problems when they appear and make sure they stay fixed.