I won’t go into detail about what was said and done because the website has a good Wiki where you can read-up on what went down. I hope all the scribes put up what they have. There should also be some audio tracks up there soon. This post is mainly about me posting some rough & dirty pics. AND of course to rave about how AMAZINGLY GREAT this unconference was. Lots to take home and brilliant people in attendance!

Thanks again to PJ and Jeff for helping us do our own conference! Great stuff!

Starting us all off To CITCON (speak KITKON!!!! VERY IMPORTANT!)

So…..what’s on the menu

Lively discussion

Now this is what I call OFF TOPIC

CI tools and the pro’s and con’s

2010 and WeekendTesting is gaining traction. In the process of getting involved I was confronted multiple times on how I could NOT have a twitter account. So in order to see what the fuss was about I re-activated twitter a couple of weeks back. Mainly for the whole testing scene thing.

This was during the same time the ugly FaceBook-privacy monster reared it’s head. So the motivation to look for an alternative communication channel was high.

After a few weeks of being tested-twitter-tester here’s my verdict: AWESOME!!!!

I have learned more about testing, test-tools, methodologies in about 3 weeks than in the last year (OK, I do exclude that dice-game-thing with James B. from that)! Twitter is like a huge meeting/melting of minds machine. I love it.

Unlike FaceBook it is focused. The 140 chars is annoying but extremely helpful in concentrating thought and communication. FaceBook is personal and tends to be mostly bla-bla. Cool for catching up with mom but not for debating testing or communicating information quickly. FaceBook is a wasteland when it comes to progressing your subject at hand. Twitter is the opposite.

I am encouraging the uptake of twitter as a tester. It helps the trade, it helps your know-how. FTW

Oh and my twitter account: @Oliver_NZ

The knee-jerk reaction would be to call it “automated exploratory testing” or “exploratory test automation”. Sounds good and without much ado we could put that into a test strategy/plan of some sort. But… what are we really saying? Just put yourself in the shoes of an unexperienced project manager or line manager. I could see a converstationlike the one below:

Tester: “We will be doing automated exploratory testing!”

PM: “Cool! I like that idea. …hmmmm… so you mean after some test scripting we can get rid of all those testers! Wow! Double cool!”

Tester: “No, no”

PM: “Getting confused here. Alredy wrote here ‘decrease test effort by $50k’.”

Tester: “That’s not what I said. I meant….”

PM: “So no automated exploratory testing?”

Tester: “No. What I meant was….”

PM: “On a different topic altogether,….”

Testing is really great in choosing terminology that misleads non testing people in not getting it. This being one example.

What we want to say is that exploratory testing can use automation to facilitate the process of exploring. In the wider sense automation is just one part of our “bag of goodies” that we carry around. It is a powerful and sometimes inevitable tool but it does neither exploring nor testing. All automation can really do is checking (check out Michael Bolton’s blog on checking vs. testing http://www.developsense.com/blog/) and supporting manual testing.

So I’d suggest something like “tool-supported exploratory testing”. It clearly defines action and scope. John Bach (@jbtestpilot) commented that the dash based names were not so ideal and I do tend to agree. For the moment though I’d see it as the only really viable option bar something completely new (like Bionic testing, Remote control testing – thanks John).  I personally would love to have an acronym like ETX. I have so far failed at finding the long version for ETX though.

What this discussion has sparked in me though is to think twice about the terminology I use. The pitfalls are plenty and easy to fall into. So my advice to testers is to start caring about what you call things. It might propell your testing into a favourable direction and do away with the constant need to correct mis-conceptions.

Thanks James, Michael, Ben (Simo, @QualityFrog) and John for a great twitter session!

The objective was here:
http://groups.google.co.nz/group/software-testers-new-zealand/browse_thread/thread/6cbee3a6fb040310

The raised bugs can be seen on http://bugrepository.com – project WTANZ01

The transcript is here: WTANZ Session 01 18-04-2010

As you can see the session was very lively and we had a really good attendance for a first time Weekend testing session. We pretty much kept to the time-frame of 2hrs.

The goal was to test the mortgage calculator on Sorted.org.nz. This is a popular website in NZ to learn about finances and things like retirement amongst others. From the word GO everybody really dug their heels in. One of the first insights was that there seems to be quite a big discrepancy when comparing results between various interest calculators. We really didn’t quite get to the bottom of that but it was interesting that a simple seeming thing can have so many “interpretations”.

Further testing quickly showed the obvious. This website has NO validation checking but….. it doesn’t fall over either. No matter what you threw at it it would process the results. Never mind if the answer was correct. For details see the raised defects.

The testing then focused on usability more general usability and browser testing. This might have all been  bit much for one session but nonetheless there were some good learning points to take home (check from log time 19:25 onwards).

The discussion after highlighted that in such a high pressure testing situation most used the ET path. Some ideas around planning were done and shared with some good insights. I’m sure everyone could take something home from this.

All in all great fun and great sharing. It all made my facilitation role so much easier

So after all the butterflies I’m glad it all went well and we’re all looking forward to Marlena’s session in a fortnight.

Also have a look at Ram’s blog entry that gives even more detail on what we did.

Cheers & till next time!

Oh, you probably won’t know what Weekend Testing (WT, http://weekendtesting.com) is! So let me explain… This WT stuff was “invented” in India and is spreading the globe. The aim is that Testers learn from other testers while using their skills. So the low down is, at a certain time on the weekend and undefined number of testers meet on a Skype chat and a facilitator then poses a task/mission. The testers go and do their stuff for an hour and then come back and discuss what they did and why.

The success has been quite phenomenal so far. There are three chapters in India and one in Europe. Because the time zone differences are quite extreme for us here at the other side of the globe there was a need to start our own chapter. So….

18th April is the first session
6:30PM New Zealand time and
4:30PM AUS time (east coast).

For more details have a look here: http://marlenacompton.com/?p=692

Matt Mansell’s (DIA) talk about Growing Great Test Analysts contained one comment that really struck a chord (not that the rest of the talk wasn’t excellent). He said (my own interpretation) that calling us testers is really putting down our capabilities and our profession. Instead we should make a point of calling ourselves Test Analysts. We ARE analysts and good testing is learned the hard and painful way. We are not monkeys or Business Analysts in sheep’s clothing.  I am as guilty as are most. I will in future make a real effort in using the proper term of  Test Analyst. It is a small but vital step in the right direction.

Stanz 2009 fltr: Julian Harty, Karen Johnson, Lee Copeland, James Bach

And the above really highlights what Stanz is all about. We go there to get reminded that we are in a testing profession. The speakers entice us to learn more, to read more and to get off our complacent derrierès. The 2009 Stanz definitely had the right set of speakers for this. I again enjoyed the quasi familiarity of the event. I doubt there are many places in the world where the attendees and speakers can get so up-close and personal to exchange ideas.

The effects of the recession were visible but that was to be expected. Even so I’d say the attendance was good and it was nice to see all those familiar faces again and meet some new ones.

Who knows, 2010 might be the year I switch sides from listening to talking. We’ll see…..

As most of you probably know or can guess, Rex Black’s law is quite the opposite of what I believe testing should be like. So I was quite surprised to see him talk about (A)agile testing. I couln’t see how those worlds really fit together. And I wasn’t disappointed.

[(A)gile as defined in the talk was the commonly misused cloud of anything that is not really Waterfall/V-Model. I will use this definition here and not distinguish between the project management side and the engineering practices as I usually do.]

One of the first things that he said is that he sort of doesn’t believe in (A)agile (or was that (A)agile testing?). Then he went on in his usual self to ponder about Risk Based Testing and how (A)agile is failing it’s targets because testing is being squashed at the end… He even went so far as to propose his own take on sprints (adding a second sprint after each sprint for more testing). He can do that because (A)agile is about freedom but I’m not convinced that’s what he understood why he can do that.

I think Rex misunderstands what (A)aile is all about. In my opinion it’s all about bottom up. Or to use another overused phrase “power to the people“. I asked him about that and he said that that was a dreamers perspective and would never be really implemented. I think that’s what this is really about though. As long as we have managers/sales second guessing their subject matter experts (SMEs) and coming up (dreaming up) with estimates and time-frames overruns and bad projects will continue to happen. Why not let the people in the know do what they do best?

If you are not willing to go down the path of giving power to the people who actually know what they are doing you’re still in Waterfall/V-Model world. Only if you make the mental leap to trust the people you have employed to do their job can you contemplate going (A)agile. (A)agile and all of its different methodologies have (at least) one thing in common. They rely on adhering to the proposed rules of the methodology for them to work. The nice thing is that they are pretty simple and so can be remembered easily. The problem is that they usually get “waterfallified/v-modelized”. My conviction is, that as soon as you do that, you have just migrated all the downfalls of the past to what you want to do in the future.

Rex buys fully into the “waterfallified/v-modelized” agile. That’s like saying you quit before you start. So I classify Rex’s talk as an anti-agile presentation. He would of-course vehemently oppose what I just said and went so far as to indicate that the testing job in an agile world will not change that much and he’ll earn money no matter what happens.

Wat was interesting about Rex’s talk was that there are some new things I hadn’t seen him use before. One that amazed me was the use of Test Oracles. That is something I heard from James Bach for the first time (although I’m not saying one was before the other because I plain just don’t know). Rex was using the exact same definition in his presentation and talk. For those that don’t know James and Rex are probably the most extreme positions we have in testing at the moment (each one on the opposite side). So what’s happening? Will we see a slow amalgamation of all testing lore out there? That would be like a dream come true. Or is this just a another ploy for ISTQB to make it even more encompassing (i.e. we cover that keyword too-type of encompassing)?

Anyway, the talk was good and it definitely opend my eyes a little more but I guess not quite like Rex might have wanted. I learned that the people like Rex are probably the biggest danger to (A)agile and all (A)agile methods. They say yes to agile but do not embrace the pure spirit of the movement (and reading a manifesto doesn’t really do it). Especially in people with a standing such as Rex saying it’s OK to deviate or water(fall) down the (A)agile teachings is really dangerous. They are giving the managers and non-believers out there the ammunition they need to scuttle (A)agile in the workplace.

If (A)agile does not become commonplace in the next 5-10 years it will be a huge loss to the industry. The loss of efficiency has the potential of slowing down or hampering the whole of human development. So I’d be happier if people like Rex who don’t believe that agile can work would say so and not try and fly some quirky path that leaves them somewhere in the middle just to keep in the game. I’m sure he can find enough to do in the “old” world to keep him happy until he retires. We need this revolution in the minds of leaders and managers for IT to survive as a sane engineering practice.

Thank you to SoftEd for the invite and the good breakfast.

You would be able to find offline GMail in the Google Labs tab in your GMail settings. For more details have a look here.

I’m probably not the right customer for this as I’m always online if I sit in-front of a computer but maybe this allows my wife to switch. She’s the one using the laptop. I will still give it a good go though. Who knows, the next internet-cable-cut maybe just around the corner.

I think this is a huge and important step for Google. If this works it is the corner stone that will be used to gun for Microsoft Office. This technology can then finally replace office software completely. Even companies could now join in (provided you’re not bound by legal issues of storing information off-site). I hope Google will be clever enough to provide this as one of their appliance kits as they do with their search-pizza-boxes. I know I’d love to leave Exchange behind too.

java.net.SocketException: Unconnected sockets not implemented

After a lot of searching I discovered that this seems to be an HTTPS change in the Java 1.6.0 implementation. Sockets are created in a different way than under 1.5.0. So if you run into this issue just go back to Java 1.5.0. This problem surfaced with JMeter 2.3.2 but I guess it is true for earlier versions too.

Under Ubuntu do the following:

apt-get install sun-java5-bin
update-alternatives --config java

And select Java 1.5.0 when prompted.

What is amasing though is the quick and professional responses that you get most of the time. Although these are mostly Open Source products they still have a LOT of professionalism going for them. Something I can’t really say for a lot of products that cost several arms & legs. But I’m getting off-topic….

Firefox 3.1… what can I say…. FAST, very FAST, did I mention FAST? The other things Firefox 3.1 brings are mostly of secondary interest to me. Since I use GMail, Google Calendar and other online programs I do depend on the speed of Java Script and the rendering. This version does wonders for that. I’ll keep the memory footprint in the corner of my eye though….

So if yer feel’n a li’ll adventeris taday try ye on fir size.

[Update] Looks like the Better GMail plug-in doesn’t quite work correctly. It doesn’t allow to send email.

But with Safari and Chrome using WebKit and Firefox being very HTML conforming we could end up with a somewhat clean web environment if Microsoft would switch to using WebKit. Browser testing would become a lot simpler. WebKit is also quite easy to automate provided we testers get some tools for it at some point (hello developers! This is a request for Open Source development! .

As much as I criticised Google for creating another browser I’d love Microsoft to switch to WebKit. It would make the online world a better place. But I’ll only believe it when i see it.

I have now been in the testing business since 1998. It is my profession and I do sometimes think that I am a little proficient in what I do.

I remember back in 2002/3 I was working with a colleague tester trying to come up with a glossary for testing. The company (SQS Germany) had expanded into several markets and with that different languages. In order to facilitate clear communication between projects, subsidiaries and companies we wanted an online dictionary that had definitions and the translations into various languages. Without going into too much detail the whole exercise failed miserably at the time. We didn’t even get approval as far as two tester offices further down the corridor. If there was one thing we proved it was that everybody had their own tounge. It reminded me of what had happened to the Babylonians. It seems as if we testers are doomed to an existence without standardisation.

Well…. then there’s the guys from ISTQB. They say the testing world is just so and not one bit different. I’ve been through the syllabus and it is nothing more than a learning of vocab excercise and they are trying to sell you that there’s something like standardisation in the testing world. I’ve also found a number of errors in the material. Not very convincing. I must admit though that the course I went to was way better than the ISTQB itself so I did end up learning something. To this day I refuse to sit the exam. I’d rather….well…any thing else but that!

ISTQB is corrupting the testing industry. As a so called qualification it proves nothing. I get especially cautious when I have ISTQB certified testers apply for a job with us. I question the motives and what the qualification means to them. Since they are now qualified testers I don’t have to pull punches. I can ask them anything aren’t they test professionals? Try it some day it’s fun. Most of the time it turns out they’re just poor souls trying to get some kind of recognition for their profession. Sorry dudes but ISTQB is exactely the opposite.

In 3 years time India and China will overrun us all with 60.000-gazillion ISTQB testers and they are all going to solve IT’s problems!! Sounds like a Tui ad.

OK, so by now you probably know that the ISTQB is the equivalent of a testing paper tiger. But amazingly enough I’m not the only one who thinks that. It seems that the good lady executive director of the American ISTQB is not far off the mark. Have a read here. I can’t even begin to grasp the cheek of this lady. It’s a sad day for all those that have been gobbled up by the “Big Dog” ISTQB.

So I hope long pauses in bolg entries is not hereditary and we’re going to see more good posts soon. I liked the ones so far. Keep up the good work and always use the force!

Only one little niggly thing… I am a “professional” niggler after-all…. John, your WordPress know-how still s*cks.

RAID 5

I came across [this] post. Have a read. It is really interesting. What stumped me is that the death of RAID 5 will be our ever increasing need for bigger hard drives. We usually talk about things like economies of scale. That means that when a certain thing is done many times or in a large way the price/effort/risk drops. For lots of things this is true but in this case we achieve the opposite effect. Because disks have gotten so large the infinitesimal chance of an error occurring becomes relevant again due to the terabyte sized drives. So relevant in fact that it affects something we depend on.

This bane for RAID 5 and in a few years time RAID 6 made me aware that as a tester the mind should always be kept in overdrive. I’m sure the testers of the hard drive manufacturers have not thought that far (we would have seen some kind of increase in MTBF and MBFR). As a tester with a manufacturer you’d probably test only one drive as a unit. You would not test large drives in different arrays too in-depth. Testing would take too long and be too expensive.

I will be keen to see what happens in 2009/2010 when the first real occurrences happen and valuable data is lost. I’m sure some American will sue somebody.

As for testers…. I think a calculator and a keen eye can be good tools to finding the odd defect too. You don’t need to have an application or gadget in front of you to discover defects. Take the specification and start adding, multiplying and/or dividing!

They use Apple‘s WebKit as a base rendering engine and put some -admittedly good- fluff around it. But what I don’t get is why they make another one. They would have been much more effective teaming up with Apple and Safari or the FireFox guys. It would have left the browser landscape untouched. This now adds yet again another thing you need to test when developing web applications. The cost alone worldwide to web development projects will be in the hundreds of millions. Just because Google thought it would be cool to have their own browser. I wonder if they ever thought of the follow-on-effect of their actions.

The only hope I have is that the WebKit engine they use remains unchanged. That would mean testing Chrome would test Safari and the other way round. That remains to be seen once the browser is out in the wild.

The ideas Google have and what they are trying to do is great but it’s just the wrong way. This smacks of a Microsoft stratagem and I also believe it will have a Microsoftish feel to it aka somewhat stale and big monopoly like.

As a tester I hope they will still come to their senses. We have enough browsers to have healthy competition and we have more than enough browsers to give testers a royal headache. So please not another cool browser!!!

We’ve been spoiled by speed. Every gamer today has a 1Gbit switch at home. Wireless networks are accelerating past the 54Mbit barrier so who cares if the network is doing something “funny”? Who even notices things like that? We plug the cables and it works. Little or no considderation is given to tuning/testing networks in your average project.

On this project I relied on the information I was given. I had a WAN connection of 10Mbit to my server. My estimates were that I’d probably use 2Mbit of that for the tests. A 10Mbit connection should be able to push a net load of about 7Mbit. I didn’t expect to come even close to a network bottleneck. The focus was the app. After a few tests it was apparent that there was a bottleneck somewhere. Response times were getting drastically worse the more simulated users I added. After a certain level I even got connection resets. Having dismissed the network I decided to hunt for configuration omissions first.

This took the better part of a day without the expected result. Only then did somebody ask the obvious question about the network. Now there’s an easy way to find out if there’s something wrong with your network. It’s called PING. Any network guru will tell you that ping is “No way to test” and “Won’t tell you anything”. And they are probably correct but even a simple ping shows a stressed network connection or one that isn’t clean.
So when running your performance test make it a standard task to run at least one test while running a ping alongside. On Windows use ping -t <IP-Address-of-server>. You will notice that a normal network has ping times of 1-30ms when idle. These values do tend to fluctuate somewhat. If ping times are longer there will be a good (technical) reason for it. Discuss this with the IT department or solution architect. They should know (if not then raise it as a defect .
Now you should see some degradation of ping times while running your test. If the degradation looks extreme 60-120ms (or more) or even ping time-outs then you are probably at the limit of the capacity or have a serious configuration error. At this point your only chance is a more thorough investigation by the network specialists.

Using ping is not really professional but it does do the job quickly. You can always go into more detail with further tests. And while you’re at it don’t forget tracert/traceroute.

[Update] I have now tested USB HDDs with my Airport Extreme Base Station and found out that a normal 1Gbit LAN connection to the AEBS is about twice as fast as a WiFi connection. WiFi has a read speed of about 8.2Mbit/s and a write speed of about 6.1Mbit/s. The wired connection gets 15.4Mbit/s and 10.2Mbit/s. Direct USB connection to the drive is about 22Mbit/s write and 33Mbit/s write.

I’ve been using JMeter for about 2.5 years now. It is Open Source, Java based and easy to use. I still keep a regular eye on other tools out there. To make a long story short the commercial performance testing tools are much fancier than JMeter especially in the graphing and monitoring areas. The thing is, that that is really not necessary. It’s something nice for managers to get excited about. The core function of a performance testing tool is that it generates the load in a good manner. And of course can you spend $50k-150k for a testing tool (+the training and installation).  If you’re going to do back to back performance testing of huge installations then it might pay to take a look at those. If not (and even if you are) I’d give JMeter a try.

One thing I experienced and had to deal with is discrediting the performance measuring tool. As funny as that sounds it is vital to performance analysis. The question is really what does your performance test tool measure and does it really do what you expect. Only once that is proven can you put the blame back on the application under test. When dealing with an Open Source Tool this is fairly easy. You just look at the code. I really have no idea on how you’d do that with a commercial tool.

JMeter measures results Last Byte Received (as opposed to First Byte Recieved). If you compare the “same” performance tests of different tools you are likely to get varying response time results. Always beware of that when testing with multiple tools. Please don’t try to explain this to management. It only ends in a blood bath. Companies are usually bound to reach certain service level agreements (SLAs) which are measured in ms response time. So if a tool shows 500-2000ms less per request that might give you an edge for your SLA. This is called cheating in all other disciplines and it does highlight the problem of standards around measuring response times. I just stick with JMeter and accept it’s timings.

Being Open Source and JAVA JMeter runs pretty much on any platform. It has two modes to run. GUI enabled or command line (batch mode). The batch mode is needed for unattended runs and possibly testing from multiple machines (as I do it). Speaking of running JMeter from multiple machines, JMeter supports multiple clients via RMI connections. This is an easy way to generate big amounts of Data but it also has some limitations which I’ll come to later.

When it comes to executing JMeter I do prefer a Linux environment (in my case Ubuntu or Fedora). JMeter performs much better under Linux than under Windows. Windows as a platform is very bad due to the bad quality of the networking stack and it’s default configuration. My guess is, that you can get about 4x-8x the load out of an equivalent Linux install than under Windows depending on the script.

For your normal small project or web-page it is usually enough to have one JMeter machine. My minimum requirement would be a Pentium 4 or better machine with at least 1Gb RAM. If you go smaller be careful to distinguish issues of the application under test and the JMeter PC. JAVA has some issues with threading.

It is not the best there is. Once you hit the 100-150 threads mark you can run into trouble. What I’ve noticed though is, that running multiple JMeters on one PC is totally OK (provided there’s about 512Mb per instance). You can therefore run 100 threads times the number of instances. I have tested this up to 3 instances on a Pentium 4 3Ghz with 2GB RAM.

One other thing that has a huge impact is testing encrypted protocols (e.g. HTTPS). The encryption and decryption is very intensive and limits the number of threads that can be executed. My tip is, that you should not run more than 70 threads per instance.

Oh, what I forgot is to say what a thread is. A thread is something like concurrency. Each thread executes one stream of your script. Some commercial providers call that a virtual user (definitely from a licensing perspective). A thread does usually not equate to a user. A real user could never execute that fast. So I use a factor to calculate how many real users to a thread. Usually that is 25-50 depending on application and script design. So far I think I’ve been quite accurate. The only way to check this is to get real monitoring data (which is usually only available once you have gone into production).

And then there’s scripting. Generating a script for JMeter is pretty easy. The included documentation is good and useful so I won’t go into detail here. There are some things that I might point out. When scripting don’t use the recording proxy. It records too much and is harder to change than to do it from scratch (and it doesn’t work with HTTPS). Use things like Live HTTP Headers or Tamper Data in FireFox to see what is happening. Don’t script too much! This is not functional testing. You want a few good numbers so that you can oversee and compare.

Beware of version 2.3.1 of JMeter as that has issues with cookies.

Because I run large performance tests I need to use many load generators. Currently we have six of the aforementioned Pentium 4 3Ghz machines and a seventh PC that manages them. I do not use the JMeter RMI service to spread the load but use a variation of shell scripts and Python scripts to send each PC the script and time it should execute. After it has finished all data is then transferred back to the management PC. Each PC has 3 instances of JMeter making it a total of 18 instances that can be used. That is enough to support thousands of concurrent requests even if they are encrypted.To save energy the client machines even get started and shut down for each test so they are off when not in use (I wish more people would do that with their servers).

I have also done away with the lack of graphing in JMeter by building my own parsers. I use GNUplot and Ghost Writer to produce detailed output of the results.

I think that is about all for now. JMeter is a very good tool for the performance testing job and I hope you’ll have some need for it. There can never be enough performance testing. If there are any further questions just post them in the comments. I’ll be happy to answer them.