Validate and Verify

The million dollar question which every Manager asks to testing teams is that ‘Whether they have executed all the test cases ? “. If you are a software test practitioner, you would be feeling really frustrated. How on earth, can you ever execute all the ‘Test Cases’ for a particular feature.

Well, there is no silver bullet but there is a way which you can use to get a good enough answer for the above question. Its called ‘Code Coverage’ analysis.

How does it work
There are certain software tools in the market which can be used to do ‘Code Coverage’. One of the popular software is ‘Bullseye’. The process is simple. While building binaries for your software program, configure ‘Bullseye’ with the IDE. When binaries are made, bullseye inserts some of its own signatures. This is called ‘Instrumentation’ of the code.

Read More

I have been thinking for a long time to write something on interviews. I usually take an interview every week, if not more and most of the times the candidate loses out. More so because the interview process in my company is really comprehensive, 5+ interviews, and being a small company, in terms of number of employees, we really have to ensure that we are not making a bad call. But this is not my motivation behind this post. My real motivation is something else. There are more then few times, when I personally think that the candidate is losing out because he is not really aware on whats expected out of him. He either goes too deep and hence makes more mistakes, or remain too shallow making the interviewer feel that he doesn’t know the subject. This invariably happens because the candidate doesn’t know on what to focus upon. My goal is to give you a clue on where to focus.

My first source of these ten questions are the hundreds of interviews which I have taken over last many years. This source only constitutes a minor portion, the majority of this list has been drawn from the un-ending conversations we had after we interviewed a candidate. As I mentioned above that we usually have 5+ rounds of interviews and after this all the interviewers get together in a room and discuss. During this discussion everyone shares on what he asked and what did he feel about the response. These discussions have been a great ground of learning for me as you hear from at last 5-6 people on their experience with the candidate. So without any further ado, here my top ten questions for a software testing interview.

Q1 - What is ‘Software Testing’. What do you mean by ‘Validation’ and ‘Verification’.
Q2 - Write a program in ‘C’ to reverse a array of characters without using another array as a temporary variable.
Q3 - Write a ‘Test Plan’ for testing ‘Find’ Box of MS-Word.
Q4 - Test an object, say a bottle opener.
Q5 - Write a program to find a ‘Prime Number’.
Q6 - Design a traffic control system for a junction of four roads.
Q7 - What would be the factors which you will consider to decide whether you should fix a bug or defer the bug, late in the product release cycle.
Q8 - You sent a word document to your friend.MS Word crashes when he tries to open it on his machine. Isolate the bug.
Q9 - Explain a defect/bug lifecycle and write a bug report.
Q10 - Why you think, you can be a good tester.

Read More

As Software testers, one of the big responsibility we all share is being acting like a user. While most of us love it and use this for our and user’s advantage its not a very natural thing to do. If it had been, then probably you would be that user and not really a software tester. The only way, we can most truly act like a user when we are testing either a bug-reporting application or a test-case-automation system or like wise. If we happen to be testing these applications then we are in a better situation but In real world, we end up testing banking and insurance application, graphics and document applications, storage management systems and so on. I work in a product based company and my area of testing falls under ‘Consumer Photo Applications’ so even though I click lots of photos and I spend time on them, I am not really someone who would buy a USD 100 software to manage my ever growing library of photos, correct them, share them over e-mail or cut DVDs and so on. So I am not really a true user.

Read More

This post is mostly a result of a comment from Rahul. He made the comment on a post on Validation and Verification and I think it was a great idea, so here I am with an attempt to distinguish these two closely related and often wrongly used acronyms viz QA (Quality Assurance) and QC (Quality Control).

First the definitions
Quality Assurance - Quality assurance, or QA for short, is the activity of providing evidence needed to establish quality in work, and that activities that require good quality are being performed effectively. All those planned or systematic actions necessary to provide enough confidence that a product or service will satisfy the given requirements for quality. (As per Wikipedia )

Quality Control - In engineering and manufacturing, quality control and quality engineering are involved in developing systems to ensure products or services are designed and produced to meet or exceed customer requirements. These systems are often developed in conjunction with other business and engineering disciplines using a cross-functional approach. (As per Wikipedia)

And here’s a simple version of mine.
QA - Set of processes to ensure that we do a high quality work. These processess are at all level. In s/w world, this could mean a good template to capture requirements, a check-list for testing team which needs to be ticked before we deliver the s/w, code-commenting procedures so that if someone looks at a code which is done by someone else then it makes sense and so on.

QC - Test the desired behavior. Its about testing and finding faults. QC is not about putting a right process but about exercising the process.

Simple and over. Well, mostly but let me go a little deeper and illustrate this difference with few examples
Read More

Large software organizations have the luxury to invest in labs, or test labs to help their test engineer better test a software. These labs could range from a setup having racks and racks of machines of various configs or could be just a big room with 10-20 machines. Having worked in a company (Legato Systems, now EMC) which creates Enterprise Backup Solutions (Legato Networker) I have been fortunate to see a lab which really had these large cupboard sized machines, fiber-optic powered SAN (Storage Area Network), really cool racks which support multiple machines with same keyboard, a display which would slide out and then go vertical. Probably it was done since it was needed. At the same time, out here at Adobe, we usually have a set up where a bunch of standard desktop machines are present running all kinds of operating systems, various locales and so on. Probably the reason we do not have racks is that our users do not have racks. Though I am sure that having racks might be more space efficient but thats for another day.

Some of these labs are powered by Image servers, the ones which can spit a OS image and do a raw-copy on local disk by booting the machine through network, while others may rely on OS installation through shiny disks. At some places, you have a check-in/check-out register for each of the machine, so as you get in you can work on a machine which is available and then as you go, you let go of the machine. I am sure that during your work experience you might have seen a different kind of lab as well. Also, some of you must have spent long nights in one of these labs trying to isolate a bug or just getting done with your part of test coverage. Sometimes these are also fertile grounds of new associations, more so since these are not too crowded, fall in a neutral zone (its neither harry’s office nor sally’s cubicle) and there are always many reasons to ask for help, there would always be something which is not working. Before your mind takes off and before you get nostalgic , let me come back to my intention of writing this post.

My intention here is to try to capture some of the best practices, best methods and general good housekeeping tips to harness most from a test lab. A happy test lab is key to success of test-case-execution and here are some things which you can employ to make your lab happy. These are not in any order but feel free to ask back incase it appears fragmented.

Five Things towards a Happy Test Lab

Read More

I was previously blogging at a blogspot site and over time I realized that I am getting limited in terms of tools there so I opted to do it on my own. Apart from other things which are needed to host and run a website, one of the basic initial things is to identify a name. And after being unsuccessful at getting a domain for the more common names, I was lost and dejected when suddenly I thought of validateverify, checked it and found that this is available.

Over time, I also realized that there are still a lot of people who are googling on these terms, to find their meanings, there differences and so on. While as a software testing practitioner and as a interview-holic person, ‘Validation and Verification’ are ingrained on my mind-slate, I can imagine that there would be lot of people who might just get confused. More so in places where English is not their first (and probably preferred) language.

So lets try to understand these terms and then see their relation vis-a-vis software testing.

Validation - Act of finding whether something is ‘right’ or not. The ‘Right’ is from the perspective of overall need and desire. Whether something is valid or not. 

Verification - Act of finding whether something does whats its supposed to do in defined (or may be undefined at the time of testing) condition. Verification is for built behavior, its for smaller nuances and not for the initial big goal.

Let me now take couple of examples to drive these points home.

Read More

With software getting larger and larger, the interaction between engineers who work on various parts of a software has been getting more challenging. While this is not really a big problem for development engineers, the ones who primarily write code, but its indeed a growing problem for ‘Test Engineers’, the ones who test a software and ensure that it does what its supposed to do for the end-user.

Let me take a couple of examples to explain this better. In first case, we would take the case of ‘MS Word’ which is a part of ‘MS Office’ family of products. In second case, we will take a much smaller product like ‘Adobe Reader’ (more popularly knows by its former name ‘Adobe Acrobat Reader’).

MS Word has tons of modules or components or ’section of codes’ which would have their own small teams. These may be File IO (Open, Save, SaveAs), Fonts, Print, Tables, Mail Merge, Formatting, Bookmarking, Help and so on. Some of these could be common across products, e.g. Fonts handling, Help etc. Each of these small teams will have their own test engineers and most of them test ‘their’ piece within a certain ‘Entry and Exit’ ideal boundary condition. Its always assumed that the guy who is before this node in the value chain would behave well and the guy who is next in chain would also behave well. Thats a wishful thinking.

Lets take an example

Enter Text -> Save File -> Select Text -> bold the text -> Undo bold -> Save -> Close.

In this workflow (a logical sequence of actions performed for a task), lets take the case of the team which is testing ‘Undo and Redo’ function.

Read More

Software Testing is still more in realm of ‘Arts’ than Science, especially when we are talking about testing software products,  something like MS-Office or  Mercury  WinRunner. Unless you are a part of a highly evolved and organized form of testing, in other words, a defined set of test cases on a defined set of test-bed to be executed, you can safely assume that you wont be able to find all the bugs. Being paid to find bugs and still not able to find all is an oxymoron arrangement and as Test Engineers and Test Managers, we need to live in the reality (of not being able to find all bugs) and still do business (find all bugs).

In this post, I am going to talk about a exercise which lots of organizations are started to get into their test-process as an effective tool for finding bugs, which typically act elusive to the ‘Test Engineer’ who is looking at a specific area. This exercise is called by different names like Bug Hunt, Bug Bash, Focus Day, Bug Harvest and so on. We would refer it as ‘Bug Hunt’, just a personal preference, they all work same / similar.

What is a Bug Hunt ?

A Bug Hunt is an intense bug finding mission over a small period of time, from couple of hours to a day, done by people who are not testing that area or that product as normal day-to-day work or may not be ‘testers’ at all, allows all kinds of bugs (some bugs are feature-requests) to be reported and is done to sort of give one big shake to an area or a product before its being shipped.

Read More

As software testers, we have to find faults in our own creations. Even though its lot of fun and a lot more sadistic pleasure to send a RED report every week to management on how buggy our own software is,  we can’t be holding on to  it till eternity. This is more of a problem for software managers like me then what I was few years back, a Test engineer. Somehow it never occurred to me as a Test Engineer that all my RED reports are actually not a good news to my manager and nor to company in general.

So as we like it or not, there is a time when there are very few bugs which deem a fix and we have to ship the product. Once a version is shipped, the whole game reverses. The same shortcomings which looked so much on the face are now ‘Known Issues’ or ‘Cosmetic’ and suddenly the average test engineer Mr. Joe starts to take full ownership, the same feature which he scoffed at since it was sort of developed by someone else is now something which he blessed. Any fault which gets reported later is looked at with suspicion and an explanation is tried then gladly logging them in the ‘Bug Reporting System’. Over time, the whole problem of maths-n-science and of finding logical bugs becomes a OB (Organizational Behavior) and psychology thing. What is now happening is that a tester has started to love his product too much and not able to see faults.

Read More

A good ‘Test Plan’ is often that well bound book on your glossy shelf which you never read. I must have created many test plans in my stint at Newgen Software, Legato and lately at Adobe and probably never consulted them again during the course of the testing. But its a well needed document. I was fortunate to not work in a highly process oriented company but we still need to make a plan, at least in the beginning of the project. Most of times I have seen people scouting for websites and what not to get that golden test plan template which they can just fill in and sort of get away.

I think we all do it in our youth and as I look back it all looks not-worth it :), the template hunting part. So let me take you through a decent or a good-enough test plan format w/o binding it in any template.

1. Think about the project for which you are going to create the test plan and explain it in few lines. Name this section as ‘Backround’ or ‘Context’ and have it as one of the first things. If you are testing a web-portal specializing in travel-experiences like www.ghumakkar.com (sneaking some publicity for this site) then mention something like “ghumakkar is a travel blog platform for traveler community and is primarily meant for sharing travel stories and to inspire the world……” and so on.

Read More