*duh* because we want faster response times…. oh and we want to know how to scale our virtual machines…. oh and we want to tune our systems… oh and XXXXX….  there are tons of reasons. Performance testing has it’s testing rigor and we go and “hammer” the system to get at those answers.

One thing I like to do (because it’s fast and cheap) is use a calculator/spreadsheet for performance testing. I take architecture diagrams of present and future systems, infrastructure diagrams, requirements, human oracles and more and put all the numbers together. Then I check if they stack up. Like where the product tries to get 1GB of data across a 10Mbit network link in under a second. I don’t need a test to be able to tell you, that there’s a problem there.

But then it struck me today. There is something similarly simple that I am not doing (and am guessing not many performance testers do)….

Ask yourself, what is the web page that has a response time of 0.000 milliseconds and has a infinitesimally small  throughput footprint?

It’s the page that doesn’t get loaded!

Think of purchasing something online. You run through a dozen screens entering passwords, addresses, delivery types…. on and on it goes. Usually one shop worse than the next. Just as you start thinking that it is actually simpler to drive to the shop and buy the damn thing there, someone comes along and invents the 1-Click purchase. Never mind, what that did to sales of goods but think of what the advantages from the performance perspective are.

1. Fewer web pages, resources and redirects to serve up
2. Less transactions in flight at one time
3. Less database interactions
4. Less infrastructure handshaking and latency

These are just the obvious ones, there’s probably a dozen more. The example here might not even be a good one either but I think you get where I am going.

Is it not time for performance testing to look at a bit more than just response times?  This kind of analysis gets us looking beyond response times of single web pages but looks at complete flows. How much interaction does a whole flow create and can the process/business flow be optimized?

It seems to me that architects and business analysts ignore performance related issues in their designs. It can be false assumptions on what appropriate design is or just bad customer requirements that proliferate this behaviour. Too many unnecessary steps, interactive-polling overload, re-entry or confirmation of trivial data and many more that can be really annoying. Normal performance testing might highlight these issues too but the cost and effort involved might be much higher. This method is quick, easy and cheap.

So next time you front up to a performance testing gig maybe start right at the project beginning and have a look at what can be cut out of process and business flows and screen design/functionality. See if you can’t just go 1-Click. It could even make your pending performance test a lot easier by having less of and a simpler application to test. I must admit I have not yet done this but I can’t see how a simple check like this could not be worthwhile.

…and who knows, there might be a patent hidden in there too

Author: Oliver Erlewein
& thanks Aaron for the good tips for improving this post!

I’ve spent the last couple of years helping projects with their application performance in NZ (mainly Wellington). I thought it’s about time I wrote something on the experiences I’ve had during that time and the lessons learned.

NZ is comparatively a smallish place. 4.5m people live here. A large bank for example has about 0.5-0.75m customers. One of the biggest online applications running in NZ is probably TradeMe. They have 2.8m customers and about 75k-200k active customers at any point in time. On average they have less than 1m logins a day. If I contrast that to large international systems this is laughable. Ebay for instance has 83m users and  670 million page views a day (I don’t know from when these figures are though). Facebook has 750m users,…. So big international companies talk about building another datacenter, where we might start clustering.

We do things a bit smaller. That has its advantages – if we do our homework correctly. Most products used nowadays are designed to be massively scalable to the requirements of large international companies. So we should have no issues with performance….EVER!

But as you probably know from your own surfing experience this is not always the case. It gets even worse when we use web applications that are in-house. All of this should actually be a no-brainer. So what’s going wrong?

I’ll try and list the thoughts and experiences that I see are common in projects here (no particular order).

1. No performance testing/analysis
   Performance testing is still a rarity. Luckily awareness is rising and due diligence is being done more often. There are still long ways to go though.
2. Unrealistic & bad requirements
   In most projects I have encountered the requirements are about 10-100 times higher than needed. This results in tuning for the wrong sweet spot. You might have an app that works excellently under high load but performs badly under low load. Or you might agree to test something the user actually doesn’t use that much. Describing performance is not easy by any stretch.
3. Misunderstanding of terms
   Concurrent, load, stress, performance, latency, response time,… they mean different things in different projects.These need to be clarified first before discussing anything. Do not assume that a technically correct interpretation equates to what people understand what the term is.
4. Bad architecture
   My experience is that most solution architects treat performance as a minor worry, if at all. It presents one of the biggest risks to any project. I have seen re-architectures very close to go-live. Performance specialists (and I don’t mean performance testers only!) should be part of the early architecture & design phases.
5. Developers unaware of performance
   Not only architects but – in the heat of meeting the next milestone – even developers will not  think of how their code will perform. Only when confronted directly do they start tuning their code. It is less an active ignorance as a result of too narrowly focussed expectations. Non-functional requirements are often just an afterthought in development. There is the fallacy that they are minor things that are just by-products that any developer does anyway.
6. No implementation of product tuning guidelines
   Most products have tuning guidelines. They tell you things like threads per core, buffer sizes at different configurations or how to set up thread pools. Or they tell you what optimal hardware/virtualization configurations are for running the product. There might be hints on when and how to correctly cluster your solution. These are straightforward things that should be done/defined by the architects and developers before releasing their product. I have not yet seen a project do this without being told to do so by the performance tester. Oh and the excuse “there are no guidelines” doesn’t count. In that case the guideline is revealed by Google.
7. Political & bad choices for products
   IT is not as easy as the glossy brochures make it out to be. Products don’t work together, hardware won’t perform in certain set-ups and things are just plain buggy. As any good performance tester will tell you, everything will fail (that is actually always true. The question here is, does it matter). These are the things that should guide proper product selection but often products are selected by company default (we use Microsoft only/only HP servers/Cisco networking/…) or because the CIO went to dinner with XYZ product sales guy. Often products get chosen by people who have never even seen the product themselves or have any clue what it does. These people also are the same, that ignore advice from SMEs that know what they are talking about.
8. Lack of monitoring or useful monitoring
   Monitoring is one of the most important parts of running something in production. It tells you when things break. So why on earth would you not focus on it? Monitoring is also the brother of performance testing. Without it how can you see what load does to your system? But also the converse is true. Monitoring only can never replace performance testing.
9. Lack of qualified people
   There are only so many people in this country with specialist know how. We cannot keep up with the volume of new information that is being disseminated internationally. But amazingly it’s not only the complex stuff that goes wrong. It’s the basics. This to me just shows that we are not thorough enough. I think employers and customers should be asking for more and taking the lead with giving people the correct/full list of expectations and requirements.
10. Blind belief in delivery dates
    Ship and then performance test. Or “can we cut 2 weeks off the performance test?”.  Need I say more?
11. Lack of  performance testers
    My rule of thumb is the 10/20/70 rule. In performance testing 10% is the tool, 20% the training of how to performance test and 70% is just plain experience. It’s the 70% that make a performance tester. In Wellington I’d say there are probably only two-dozen performance testers that have that 70%. The market could do with a multiple of that but because this is learning from experience it’s a viscous circle. We don’t test enough because we can’t get the specialists and we can’t get specialists because there is too little performance testing.
12. Project timelines
    Ok, so when do we performance test? 99.9% of PMs will put a PT phase just before go-live. If you look at some of the issues above you might see why this can be a really bad idea. If you get architectural or product issues you might have a huge challenge on your hands and an immediate and huge delay in delivery. So performance testing should be pervasive throughout the SDLC to minimise the big risks.
13. Staff turnover
    I have worked on projects that have been running for years. Developers “get the feel” for performance once you have done a few performance testing cycles. They get better at developing performing code from the onset. As is normal for a project there is turnover. This means though that performance will drop when new team members come on board. They do not know the product well enough and will run into issues when the next PT comes. The process starts over. On projects that are highly performance critical I would suggest leaving team members in place for as long as possible and to never exchange a whole team big-bang style.
14. Testing vs. Investigation
    Last but not least it helps to understand the difference between these words. Testing means I have a clearly defined outcome that I am expecting. This is rarely the case. There are rough ideas, some data and guesses but certainly nothing that is clear and defined (see 2). So what performance testing – in all it’s diversity – is, is an investigation and tuning exercise. Think of this when creating/reading/critiquing the performance test plan or when asking for documentation. Also remember that investigation is an agile process and will form as time progresses so don’t be too rigid when prescribing what tests should look like. The investigation might lead you to unanticipated tests.
15. Complex Tools
    Performance Testing from a tool perspective is actually quite straightforward. Complex tools might make performance testing projects more intricate than they need to be. Worst case they get dumped into the “too hard” basket and don’t get done at all. Keep things simple and increase complexity when needed. In my experience the issues encountered in NZ are mostly not that intricate.
16. Following advice
    On nearly every project I have the situation, where a vendor/developer/PM/architect/… doesn’t believe that there is a problem or that the cause of the problem is something other than what I tell him/her. I have spent weeks waiting and trying to convince people to look at XYZ without success. Time and effort wasted, just to come back to XYZ and find it’s exactly what is causing it. I am not saying that I or performance testers are infallible (far from it actually). But it would help immensely if whoever could start by looking at XYZ first, as chances are that the performance tester’s experience in performance exceeds whoever’s.
17. Averages are worthless
    People love to talk about averages. They are easy to understand and from school we know they are important. Well, in performance testing they tell you very little. They actually tell you nothing without other numbers that form a context. I think it is safe to stipulate that you shouldn’t use averages in performance testing (at least when reporting). The number to go for is 90th percentile (or any other percentile between 90% and 99%). If you need details go to Wikipedia, they do a good job of explaining these. And when using these numbers always relate them back to throughput. Without this relation they reveal nothing.