Saturday, November 21, 2009

Performance vs load vs stress testing

Here's a good interview question for a tester: how do you define performance/load/stress testing? Many times people use these terms interchangeably, but they have in fact quite different meanings. This post is a quick review of these concepts, based on my own experience, but also using definitions from testing literature -- in particular: "Testing computer software" by Kaner et al, "Software testing techniques" by Loveland et al, and "Testing applications on the Web" by Nguyen et al.

Performance testing

The goal of performance testing is not to find bugs, but to eliminate bottlenecks and establish a baseline for future regression testing. To conduct performance testing is to engage in a carefully controlled process of measurement and analysis. Ideally, the software under test is already stable enough so that this process can proceed smoothly.

A clearly defined set of expectations is essential for meaningful performance testing. If you don't know where you want to go in terms of the performance of the system, then it matters little which direction you take (remember Alice and the Cheshire Cat?). For example, for a Web application, you need to know at least two things:
  • expected load in terms of concurrent users or HTTP connections
  • acceptable response time
Once you know where you want to be, you can start on your way there by constantly increasing the load on the system while looking for bottlenecks. To take again the example of a Web application, these bottlenecks can exist at multiple levels, and to pinpoint them you can use a variety of tools:
  • at the application level, developers can use profilers to spot inefficiencies in their code (for example poor search algorithms)
  • at the database level, developers and DBAs can use database-specific profilers and query optimizers
  • at the operating system level, system engineers can use utilities such as top, vmstat, iostat (on Unix-type systems) and PerfMon (on Windows) to monitor hardware resources such as CPU, memory, swap, disk I/O; specialized kernel monitoring software can also be used
  • at the network level, network engineers can use packet sniffers such as tcpdump, network protocol analyzers such as ethereal, and various utilities such as netstat, MRTG, ntop, mii-tool
From a testing point of view, the activities described above all take a white-box approach, where the system is inspected and monitored "from the inside out" and from a variety of angles. Measurements are taken and analyzed, and as a result, tuning is done.

However, testers also take a black-box approach in running the load tests against the system under test. For a Web application, testers will use tools that simulate concurrent users/HTTP connections and measure response times. Some lightweight open source tools I've used in the past for this purpose are ab, siege, httperf. A more heavyweight tool I haven't used yet is OpenSTA. I also haven't used The Grinder yet, but it is high on my TODO list.

When the results of the load test indicate that performance of the system does not meet its expected goals, it is time for tuning, starting with the application and the database. You want to make sure your code runs as efficiently as possible and your database is optimized on a given OS/hardware configurations. TDD practitioners will find very useful in this context a framework such as Mike Clark's jUnitPerf, which enhances existing unit test code with load test and timed test functionality. Once a particular function or method has been profiled and tuned, developers can then wrap its unit tests in jUnitPerf and ensure that it meets performance requirements of load and timing. Mike Clark calls this "continuous performance testing". I should also mention that I've done an initial port of jUnitPerf to Python -- I called it pyUnitPerf.

If, after tuning the application and the database, the system still doesn't meet its expected goals in terms of performance, a wide array of tuning procedures is available at the all the levels discussed before. Here are some examples of things you can do to enhance the performance of a Web application outside of the application code per se:
  • Use Web cache mechanisms, such as the one provided by Squid
  • Publish highly-requested Web pages statically, so that they don't hit the database
  • Scale the Web server farm horizontally via load balancing
  • Scale the database servers horizontally and split them into read/write servers and read-only servers, then load balance the read-only servers
  • Scale the Web and database servers vertically, by adding more hardware resources (CPU, RAM, disks)
  • Increase the available network bandwidth
Performance tuning can sometimes be more art than science, due to the sheer complexity of the systems involved in a modern Web application. Care must be taken to modify one variable at a time and redo the measurements, otherwise multiple changes can have subtle interactions that are hard to qualify and repeat.

In a standard test environment such as a test lab, it will not always be possible to replicate the production server configuration. In such cases, a staging environment is used which is a subset of the production environment. The expected performance of the system needs to be scaled down accordingly.

The cycle "run load test->measure performance->tune system" is repeated until the system under test achieves the expected levels of performance. At this point, testers have a baseline for how the system behaves under normal conditions. This baseline can then be used in regression tests to gauge how well a new version of the software performs.

Another common goal of performance testing is to establish benchmark numbers for the system under test. There are many industry-standard benchmarks such as the ones published by TPC, and many hardware/software vendors will fine-tune their systems in such ways as to obtain a high ranking in the TCP top-tens. It is common knowledge that one needs to be wary of any performance claims that do not include a detailed specification of all the hardware and software configurations that were used in that particular test.

Load testing

We have already seen load testing as part of the process of performance testing and tuning. In that context, it meant constantly increasing the load on the system via automated tools. For a Web application, the load is defined in terms of concurrent users or HTTP connections.

In the testing literature, the term "load testing" is usually defined as the process of exercising the system under test by feeding it the largest tasks it can operate with. Load testing is sometimes called volume testing, or longevity/endurance testing.

Examples of volume testing:
  • testing a word processor by editing a very large document
  • testing a printer by sending it a very large job
  • testing a mail server with thousands of users mailboxes
  • a specific case of volume testing is zero-volume testing, where the system is fed empty tasks
Examples of longevity/endurance testing:
  • testing a client-server application by running the client in a loop against the server over an extended period of time
Goals of load testing:
  • expose bugs that do not surface in cursory testing, such as memory management bugs, memory leaks, buffer overflows, etc.
  • ensure that the application meets the performance baseline established during performance testing. This is done by running regression tests against the application at a specified maximum load.
Although performance testing and load testing can seem similar, their goals are different. On one hand, performance testing uses load testing techniques and tools for measurement and benchmarking purposes and uses various load levels. On the other hand, load testing operates at a predefined load level, usually the highest load that the system can accept while still functioning properly. Note that load testing does not aim to break the system by overwhelming it, but instead tries to keep the system constantly humming like a well-oiled machine.

In the context of load testing, I want to emphasize the extreme importance of having large datasets available for testing. In my experience, many important bugs simply do not surface unless you deal with very large entities such thousands of users in repositories such as LDAP/NIS/Active Directory, thousands of mail server mailboxes, multi-gigabyte tables in databases, deep file/directory hierarchies on file systems, etc. Testers obviously need automated tools to generate these large data sets, but fortunately any good scripting language worth its salt will do the job.

Stress testing

Stress testing tries to break the system under test by overwhelming its resources or by taking resources away from it (in which case it is sometimes called negative testing). The main purpose behind this madness is to make sure that the system fails and recovers gracefully -- this quality is known as recoverability.

Where performance testing demands a controlled environment and repeatable measurements, stress testing joyfully induces chaos and unpredictability. To take again the example of a Web application, here are some ways in which stress can be applied to the system:
  • double the baseline number for concurrent users/HTTP connections
  • randomly shut down and restart ports on the network switches/routers that connect the servers (via SNMP commands for example)
  • take the database offline, then restart it
  • rebuild a RAID array while the system is running
  • run processes that consume resources (CPU, memory, disk, network) on the Web and database servers
I'm sure devious testers can enhance this list with their favorite ways of breaking systems. However, stress testing does not break the system purely for the pleasure of breaking it, but instead it allows testers to observe how the system reacts to failure. Does it save its state or does it crash suddenly? Does it just hang and freeze or does it fail gracefully? On restart, is it able to recover from the last good state? Does it print out meaningful error messages to the user, or does it merely display incomprehensible hex codes? Is the security of the system compromised because of unexpected failures? And the list goes on.

Conclusion

I am aware that I only scratched the surface in terms of issues, tools and techniques that deserve to be mentioned in the context of performance, load and stress testing. I personally find the topic of performance testing and tuning particularly rich and interesting, and I intend to post more articles on this subject in the future.

Source: Grig Gheorghi

Descriptive and Prescriptive Testing

While many of us drone on about scripted testing vs. exploratory testing, the reality is on real projects we tend to execute testing with a blend of both. It often feels lop-sided - on many projects, scripted testing is the norm, and exploratory testing isn't acknowledged or supported. On others, the opposite can be true. I'll leave the debate on this topic up to others - I don't care what you do on your projects to create value. I would encourage you to try some sort of blend, particularly if you are curious about trying exploratory testing. However, I'm more interested in the styles and why some people are attracted to one side of the debate or the other.

Recently, David Hussman and I have been collaborating, and he pointed out the difference between "prescriptive" and "descriptive" team activities. A prescriptive style is a preference towards direction ("do this, do that") while a descriptive style is more reflective ("this is what we did"). Both involve desired outcomes or goals, but one attempts to plan the path to the outcome in more detail in advance, and the other relies on trying to reach the goals with the tools you have at hand, reflecting on what you did and identifying gaps, improving as you go, and moving towards that end goal.

With a descriptive style of test execution, you try to reach a goal using lightweight test guidance. You have a focus and more coarse-grained support for it that what scripted testing provides. (The guidance is there, it just isn't as explicit.) As you test, and when you report testing, you describe things like coverage, what you discovered, bugs, and your impressions and feelings. With a prescriptive style of testing, you are directed by test plans and test cases for testing guidance, and follow a more direct process of test execution.

Scripted testing is more prescriptive (in general) and exploratory testing is more descriptive (in general.) The interesting thing is that both styles work. There are merits and drawbacks of both. However, I have a strong bias towards a descriptive style. I tend to prefer an exploratory testing approach, and I can implement this with a great deal of structure, traceability utilizing different testing techniques and styles. I prefer the results the teams I work with get when they use a more descriptive style, but there are others who have credible claims that they prefer to do the opposite. I have to respect that there are different ways of solving the testing problem, and if what you're doing works for you and your team, that's great.

I've been thinking about personality styles and who might be more attracted to different test execution styles. For example, I helped a friend out with a testing project a few weeks ago. They directed me to a test plan and classic scripted test cases. Since I've spent a good deal of time on Agile teams over the past almost decade, I haven't been around a lot of scripted tests for my own test execution. Usually we use coverage outlines, feature maps, checklists, and other sources of information that are more lightweight to guide our testing. It took me back to the early days of my career and it was kind of fun to try something else for a while.
Within an hour or two of following test cases, I got worried about my mental state and energy levels. I stopped thinking and engaging actively with the application and I felt bored. I just wanted to hurry up and get through the scripted tests I'd signed on to execute and move on. I wanted to use the scripted test cases as lightweight guidance or test ideas to explore the application in far greater detail than what was described in the test cases. I got impatient and I had to work hard to keep my concentration levels up to do adequate testing. I finally wrapped up later that day, found a couple of problems, and emailed my friend my report.

The next day, mission fulfilled, I changed gears and used an exploratory testing approach. I created a coverage outline and used the test cases as a source of information to refer to if I got stuck. I also asked for the user manual and release notes. I did a small risk assessment and planned out different testing techniques that might be useful. I grabbed my favorite automated web testing tool and created some test fixtures with it so I could run through hundreds of tests using random data very quickly. That afternoon, I used my lightweight coverage to help guide my testing and found and recorded much more rich information, more bugs, and I had a lot of questions about vague requirements and inconsistencies in the application.

What was different? The test guidance I used had more sources of information, and models of coverage, and it wasn't an impediment to my thinking about testing. It put the focus on my test execution, and I used tools to help do more, better, faster test execution to get as much information as I could, in a style that helps me implement my mission as a tester. I had a regression test coverage outline to repeat what needed to be repeated, I had other outlines and maps that related to requirements, features, user goals,etc. that helped direct my inquisitive mind, and helped me be more consistent and thorough. I used tools to support my ideas and to help me extend my reach rather than try to get them to repeat what I had done. I spent more time executing tests, and many different kinds of tests using different techniques than managing the test cases, and the results reflected that.

My friend was a lot happier with my work product from day 2 (using a descriptive style) than on day 1 (using a prescriptive style). Of course, some of my prescriptive friends could rightly argue that it was my interpretation and approach that were different than theirs. But, I'm a humanist on software projects and I want to know why that happens. Why do I feel trapped and bored with much scripted testing while they feel fearful doing more exploratory testing? We tend to strike a balance somewhere in the middle on our projects, and play to the strengths and interests of the individuals anyway.

So what happened with my testing? Part of me thinks that the descriptive style is superior. However, I realize that it is better for me - it suits my personality. I had a lot of fun and used a lot of different skills to find important bugs quickly. I wasn't doing parlor trick exploratory testing and finding superficial bugs - I had a systematic, thorough traceable approach. More importantly for me, I enjoyed it thoroughly. Even more importantly, my friend, the stakeholder on the project who needed me to discover information they could use, was much happier with what I delivered on day 2 than on day 1.

I know other testers who aren't comfortable working the way I did. If I attack scripted testing, they feel personally attacked, and I think that's because the process suits their personality. Rather than debate, I prefer we work using different tools and techniques and approaches and let our results do the talking. Often, I learn something from my scripting counterpart, and they learn something from me. This fusion of ideas helps us all improve.

That realization started off my thinking in a different direction. Not in one of those "scripted testing == bad, exploratory testing == good" debates, but I wondered about testing styles and personality and what effect we might have when we encourage a style and ignore or vilify another. Some of that effect might be to drive off a certain personality type who looks at problems differently and has a different skill set.

In testing, there are often complaints about not being able to attract skilled people, or losing skilled people to other roles such as programming or marketing or technical writing. Why do we have trouble attracting and keeping skilled people in testing. Well, there are a lot of reasons, but might one be that we discourage a certain kind of personality type and related skill set by discouraging descriptive testing styles like exploratory testing? Also, on some of our zealous ET or Agile teams, are we also marginalizing worthwhile people who are more suited to a prescriptive style of working?

We also see this in testing tools. Most are geared towards one style of testing, a prescriptive model. I'm trying to help get the ball rolling on the descriptive side with the Session Tester project. There are others in this space as well, and I imagine we will see this grow.
There has to be more out there testing-style-wise other than exploratory testing and scripted testing, and manual vs. automated testing. I personally witness a lot of blends, and encourage blends of all of the above. I wonder if part of the problem with the image of testing and our problem attracting talented people is in how we insist testing must be approached. I try to look at using all the types of testing we can use on projects to discover important information and create value. Once we find the right balance, we need to monitor and change it over time to adjust to dynamics of projects. I don't understand the inflexibility we often display towards different testing ideas. How will we know if we don't try?

What's wrong with embracing different styles and creating a testing mashup on our teams? Why does it have to be one way or the other? Also, what other styles of testing other than exploratory approaches are descriptive? What other prescriptive styles other than scripted testing (test plan, test case driven) are there? I have some ideas, but email me If you'd like to see your thoughts appear in this blog.

Source:  Jonathan

Thursday, November 19, 2009

Grid-Tools Complete Data Management Suite

Datamaker Functions



 
Datamaker is an incredibly versatile tool for all testing and development phases, training, demos and more. The tool offers three distinct methods for the management of data including database subsetting, data obfuscation (data masking), and generating and using synthetic data. Each method can be deployed in isolation or combination to suit your IT environment, preference or experience.

Source : Grid-Tools

Monday, November 9, 2009

Test Director 8.0 ( TD )


Test Director 8.0

Features and Benefits, Administration

Features of TD 8.0

TestDirector simplifies and organizes test management by giving you systematic control over the testing process. It helps you create a framework and foundation for your testing workflow. TestDirector provides an intuitive and efficient method for scheduling and executing test sets, collecting test results, and analyzing the data. TestDirector also features a sophisticated system for tracking application defects, enabling you to monitor defects closely from initial detection until resolution

Features of TD 8.0

TestDirector offers integration with Mercury Interactive testing tools Winrunner, QuickTest Professional, Astra QuickTest and Load Runner. TestDirector's ability to communicate seamlessly with the testing tool of your choice provides you with a complete solution to fully automated application testing.


Test Management Process


Requirements Tab

Requirement topics are recorded in the Requirements module by creating a requirements tree. The requirements tree is a graphical representation of requirements specification, displaying the hierarchical relationship between different requirements

Print Shot of Requirement Tab:

Requirement Tab

Features of Requirements Tab
  • The Requirements menu enable you to modify requirements in the tree, generate a test from a requirement, and mail a requirement.
  • Provides a Detailed coverage analysis about test execution

Different Views - Requirements
  • Document view
  • Provides details like Author,Reviewer,Priority,Type,Product,Date,Direct Coverage Status of a Requirement.
Requirements Tab

Coverage View

Tracks the Requirement with Test case
Requirement Tab

Coverage Analysis View
Provides a Statistical coverage of Test Execution status for a requirement
Requirement Tab


Auto Conversion of Requirements to Test plan
  • TD 8.0 supports conversion of Requirements to Subjects in Test Plan.
  • Automatic conversion method is available in Test Director
  • Steps mentioned in the following slides
Requirements

Requirements

Requirements


Test Plan
  • Test plan tab allows to create tests and Map tests to Requirements.
  • Tracks version changes during the updation of Tests.
  • Checkout and Check in Facility available (Like VSS..)
  • Integration of VSS is Possible
Test Plan

Test Execution
  • TestDirector enables you to control the execution of Automated and Manual tests in a test set.
  • Set conditions, and schedule the date and time for executing the tests.
  • Set the sequence in which to execute the tests.E.g Test3 has to start only after Test2 has finished and Test1 has passed
Test Lab


Defect Tracking
  • Defects can be detected and added to TestDirector project by users during all stages of the testing process.
  • TestDirector’s Defects module report design flaws in your application, and track data derived from defect records.
Defects

Search Similar Defect
Matching defects enables you to eliminate or duplicate similar defects in your project. Each time you add a new defect, TestDirector stores lists of keywords from the Summary and Description fields. When you search for similar defects, keywords in these fields are matched against other defects.

Traceablity
  • We can Trace all changes by Clicking Trace All Changes
  • Other Features
  • Check Spelling,Spelling Options,Thesaurus features available in Test Director 8.0 (Like a word Document)
Reports and Graphs
  • Six Reports and Graphs available for each Tab (Requirements,Defects,Test Execution).
  • Graph Wizard feature available in this version
Project Customization Feature
  • We can Customize Module level access for groups
  • We can set Traceablity notification rules
Project Customization


Module Level Access

Module Level Access

Site Administrator
  • We can administer TD server in web using Site Administrator.
  • We can Check the number of connections
Site Administrator


Source : Test Director 8.0

Sunday, November 8, 2009

How to Build and Maintain a Successful Outsourced, Offshored Testing Partnership

Somebody recently asked me to review a course on “Building a Great Team” - where is the section on outsourcing? I asked. In my opinion, outsourcing should always be a consideration when trying to maximise the performance of a team. In this article I will consider this in the context of software testing. I will mainly consider software testing that is both offshored and outsourced1, although it should also be applicable to software development and to companies that just use outsourcing or offshoring. The article is based the successful application of outsourcing at ClearSpeed Technologies plc (which I’ve described elsewhere -see [1]) and in advising other companies on how to outsource their software development and testing.

Outsourcing = is the use of components or labor from external suppliers.
Offshoring = the relocation of some of a company's production, services or jobs overseas.

The objectives for offshoring are evolving. Initially companies looked for cost savings. Then companies looked externally for extra resource (for example during the Y2K crisis). Companies are now also looking for strategic benefits and better defined business impacts from their offshoring relationships. My aim in this article is to demonstrate how such benefits and impacts can be achieved with minimum risk.

To attain the strategic benefits you seek, you must first identify them. Let’s consider the benefits claimed for offshoring by managers of technology SMEs (Small/Medium Enterprises) in [2]:

To accelerate the maturation of new products: “The internal skills of rapidly developing products that meet market needs is augmented by offshored teams who have the patience, precision and perseverance to evolve young products into mature ones” (Jajiv Dholakia, Cenzic Inc.)
Agility and cost savings: “on average we are able to deliver projects at a much faster rate and save at least 30% cost to our clients” (Jing Liu, CEO, Enter Suite)
Better software through additional resources: “The additional talent and resource that we can carry allows us to develop better software than our competition can afford in the same period” (Scott Allan, Symbol technologies Inc.)
Concentrate on core competence: “Businesses should focus on clients and IP which satisfies their need. Everything else should be outsourced to those whose core competency is that business” (Bill Widmer, President and CEO, g8solutions)
Ability to undertake one-off projects: “we were entering a virtually untested market with a new product. The US team focused on the current roadmap while I was able to develop the new product at a rapid speed and at a low cost” (Gopan Madathil, Founder, TechCoire)

The strategic goals must also be agreed by a suitable forum. When I started on the offshoring route at ClearSpeed, I (as the Quality Manager) agreed the following prioritised objectives with the CFO and Engineering Manager.

  • To improve company focus.
  • To improve company resource flexibility and by doing so, reduce time to market.
  • Quality improvement.
  • Cost variablization2.
  • Cost reduction.

The above goals were shared with the offshore supplier. This gave us a common understanding which helped the day-to-day management of the projects. We also derived metrics to measure our progress against the goals. For example, we used Defect Detection Percentage (DDP) to measure the improvement in our testing (see [3] for an explanation of DDP). The time to market and the cost of testing relative to development were also tracked.

The next major decision regards on your road to offshoring is to decide what type or stage of testing to offshore. There are a number of considerations to take into account.

  • Amount of documentation: In order to test your software your offshore supplier will need to understand it. This will ultimately require good documentation.
  • Domain knowledge: How much domain knowledge will the tester need and how easy will it be to find or train testers with that knowledge?
  • Complexity of the environment: How complex is the test environment and how easy will it be for the offshore test organisation to re-create it?
  • Development process: Testing within the V model lends itself well to offshoring. Offshoring of agile development less so as it is based on nimble teams with good communication and rapid change.
  • Stability: How stable is the software that you are asking the supplier to test.
  • Level of interaction: To what extent will the offshored test team need to interact with the development team.
  • Level of independence: NASA uses an independent V&V team to improve the quality of their software (see [7]). An offshore test team allows you independence along three dimensions: managerial, financial and technical.
  • Taking a customer perspective: An offshore team can be used as a “friendly” customer, helping to identify all the problems that your real customers would otherwise find.
  • IP protection: This is always a major concern in outsourcing and the concern seems to deepen with offshoring.
  • Sign-off criteria and incoming acceptance: You will need to sign-off the work performed by your supplier through some sort of incoming Quality Assurance (yes – testing the tester!).

Now consider how the above affect your decision on what to outsource:

  • Unit testing:
    • This will probably require you to release source code which may lead to IP protection issues.
    • Unit code may not initially be stable and may not be well documented.
    • The offshore organisation can act as an independent V&V team.
    • The test environment for unit testing is not usually over-complex and the domain knowledge requirements tend to be lower.
    • Sign-off could be defined through structural coverage goals.

As an example, I successfully offshored the unit testing of a new library of memory management functions. The functions were developed using agile techniques in close collaboration with marketing and two lead customers and, once the specification was stable, we offshored the unit testing. We had an existing, trusted relationship with an offshore partner who first produced a test specification and then tests against that specification. Finally, they topped up their tests to ensure our structural coverage goals were met. Our incoming signoff consisted of signing off the test specification, reviewing a selection of actual tests against that specification and re-running the tests to check the coverage.

  • System testing:
    • In my experience by the time you reach system testing the software is more stable and the chances of some documentation are improved.
    • If you are following the V-model of development then the appropriate documentation may have been available for a while so that the offshore organisation could have already written test specifications reviewed by you.
    • The test environment and domain knowledge required can be complex.
    • The level of interaction should be reduced compared to earlier stages of testing.
    • The offshore team can perform the testing independently and can take a customer perspective in their testing.
    • IP protection issues can be avoided by releasing the binaries rather than source.
    • Testing the tester is harder as an objective measure of system test quality is more difficult - but it is not impossible. A system test specification and reviews of the test scripts for example. The test results should be also made available for audit.

Some of my most successful offshored testing has been in system testing at ClearSpeed. We had an automated test environment which gave very high unit and integration test coverage, and some system testing. We then offshored all of the system testing that was impossible or not cost-effective to automate. The offshore team was encouraged to act as a customer with no prior assumptions and not to use any work-a-rounds offered by “helpful” members of the development team!

There are numerous other types and phases of testing where the above considerations can be applied but space does not allow for here. For example: specialist testing (such as security testing) where the outsourcing arguments are very strong but expertise rather than price will be most important; compliance testing (where you are required to demonstrate conformance to a particular standard) where there is a strong argument for outsourcing to specialists in that standard; release or acceptance testing has similar arguments to system testing discussed above. And the list goes on…

There are numerous other types and phases of testing where the above considerations can be applied but space does not allow for here. For example: specialist testing (such as security testing) where the outsourcing arguments are very strong but expertise rather than price will be most important; compliance testing (where you are required to demonstrate conformance to a particular standard) where there is a strong argument for outsourcing to specialists in that standard; release or acceptance testing has similar arguments to system testing discussed above. And the list goes on…

You also need to identify your preferred outsourcing model. [5] describes three types of service deal related to the scope and complexity of the offshoring contract and your strategic objectives.

Efficiency: These deals tend to focus on cost efficiency, improved resource agility or greater organisational focus on core skills. The contract terms concentrate on service levels. So, for example, you may be outsourcing unit testing currently performed by developers to free them up for additional projects and the service level might be around test specification, structural coverage and time to completion.
Enhancement: These deals are more to do with process improvement. For example, you may outsource to a specialist test company to improve your security testing.
Transformation: These deals aim to improve your competitiveness through innovation in the relationship. So, for example your goal may be to reduce time to market or improve the quality of your software (through reduced DDP for example). Your outsource supplier would need to act in a consultancy role as well as a service supplier.
Obviously the above three imply an increasingly complex relationship with increasing levels of trust. [5] goes on to consider the appropriate commercial models for the deal and this is summarised in Table 1.

Table 1: Commercial models, service deals and risk

Commercial model Description and discussion of risk; Recipient Risk Provider Risk
Cost-plus pricing The recipient pays the providers costs plus a percentage. The provider is guaranteed cost recovery. The recipient is not guaranteed service levels. Medium Low
Appropriate for efficiency deals.
Fee-for-service pricing Price is based on the amount and/or quality of the work actually delivered. The provider’s costs are not covered unless service levels are met. The recipient costs are not entirely predictable and service levels are not guaranteed. Appropriate for efficiency and enhancement deals. Medium Medium
Fixed price Defines a specific service level and a price for achieving it. This is higher risk for the provider. The recipient has lower risk as prices are capped. Appropriate for efficiency deals. Low3 High
Shared-risk/reward pricing This involves a flat rate with additional payments for achieving specified outcomes. Appropriate for enhancement and transformation deals. High High
Business outcome achievement The provider only receives payment for achievement of specified business outcomes. The receiver has no guarantee that expected outcomes will be achieved. Appropriate for transformation deals. Medium High

You also need to consider your outsource location: on-shore, near-shore or offshore. This decision should be made in the context of your chosen service deal. I would also recommend considering multiple suppliers. In my experience this brings a number of advantages.

  • Using suppliers with different types of skills.
  • Using one supplier for an efficiency deal and another for an enhancement or transformation deal.
  • Allowing the suppliers to competitively tender for new work (rather than becoming increasingly dependent on a single supplier).
Now that you understand what you want to offshore and your preferred commercial model, the next consideration is your readiness for offshoring. A number of attempts at offshoring fail because an organisation has bad existing internal development practices: offshoring your chaos will just generate additional chaos! So, how do you measure your readiness? There are formal measurements such as CMMi (see [4]). Table 2 is taken from [8] and shows the national differences in CMMi take-up. It is also interesting to note from the table the differences in maturity from the countries more likely to be supplier to those more likely to be provider. In my experience providers tend to go for CMMi for a mixture of technical, management and marketing purposes. Table 2: Number of Appraisals and Maturity Levels Reported to the SEI by Country (where total appraisals is above 20)
Country Total App- raisals Maturity Level Reported
1 2 3 4 5
Argentina 47 31 10 2 3
Australia 29 1 8 4 2 4
Brazil 79 37 31 1 8
Canada 43 10 18 5 3
China 465 1 103 293 18 34
Egypt 27 12 11 2 2
France 112 4 67 34 1 2
Germany 51 7 27 7 1 1
India 323 1 11 127 22 151
Japan 220 16 64 88 13 15
Korea, Republic Of 107 1 31 48 11 7
Malaysia 42 15 24 3
Mexico 39 1 18 13 3 4
Spain 75 1 49 21 1 3
Taiwan 88 60 25 2
UK 71 3 36 24 1 2
US 1034 25 365 347 21 114

Most organisations will not want to gate their offshoring on such formal appraisals. In my experience, the minimum set of processes that your organisation will require to outsource software testing is:

  • Requirements management
  • Change tracking and management
  • Project planning, monitoring and control
  • Configuration management
  • Build, test and release automation
  • Validation and verification processes

You now need to select your partner. There is no magic in the selection process:

  1. Identify potential suppliers.
  2. Define your selection criteria.
  3. Create a short-list.
  4. Perform an in-depth appraisal of your short-list (including a visit if possible)
  5. Contract negotiation.
  6. Perform a pilot project with your selected supplier (or suppliers)

The major issues in the above list are identifying appropriate selection criteria and exchanging contracts. Below are some example selection criteria

  • People: Skills profile; Academic background of staff; Staff training statistics.
  • Retention: Staff attrition rates
  • Process: Process maturity; CMMi and other process maturity measures attained. You should consider how well matched the relative process maturity between your two organisations are. For example, do the outsource organisation processes require deliverables and maturity on your side that you cannot deliver on?
  • Quality: Quality processes employed; how well do they match your quality processes?
  • Project and risk management: What management processes do they follow? How well do they fit with your preferred management methodology?
  • Company profile: Financial stability; historic financial data strategic goals for the company.)
  • Technology: Domain knowledge/experience of the technologies employed by your company.
  • Cost: What are the typical pricing models? How are changes managed from a cost point-of-view? Recent cost change history.

If you are offshoring then your selection criteria will need to reflect circumstances in the providers’ country. For example, if the destination is India then staff turnover or retention rate should be a major consideration.
Regarding contractual negotiations, I have found it helpful to have two documents: a Master Services Agreement (MSA) and a Service Level Agreement (SLA). The MSA covers the legal, financial and contractual issues (such as confidentiality, ownership of deliverables, assignment of rights, indemnification, warranties, liability, changes in personnel, subcontracting, termination, penalties for not meeting agreed service and delivery targets, etc.). The Service Level Agreement (SLA) should cover the service and support goals and is strongly affected by the type of service deal that you have (efficiency, enhancement or transformation). The MSA and SLA are complex legal documents which I will not attempt to cover in detail in this article.

You are finally ready to execute your offshored software testing project and this is where it starts to becomes difficult! If you have followed the process I have described above then your chances of your success are significantly improved. However, any offshored development is inherently risky. First there are the problems of managing a distributed team (communication issues, knowledge management, change management, etc). Then, according to Ralph Kliem in [7], in offshoring there are also “the unique challenges posed by geographical, cultural, and other differences”. There is not enough space in this article to go into details of how to manage offshored projects but I will highlight the main items to consider. Firstly, you need to distinguish management and governance. Management is about responsibility for specific decisions; Governance covers decision-making processes: the who, the how and the when. It is important to define the governance rules to allow your offshore team to know when they are empowered to make decisions and how to make good decisions.

You are finally ready to execute your offshored software testing project and this is where it starts to becomes difficult! If you have followed the process I have described above then your chances of your success are significantly improved. However, any offshored development is inherently risky. First there are the problems of managing a distributed team (communication issues, knowledge management, change management, etc). Then, according to Ralph Kliem in [7], in offshoring there are also “the unique challenges posed by geographical, cultural, and other differences”. There is not enough space in this article to go into details of how to manage offshored projects but I will highlight the main items to consider. Firstly, you need to distinguish management and governance. Management is about responsibility for specific decisions; Governance covers decision-making processes: the who, the how and the when. It is important to define the governance rules to allow your offshore team to know when they are empowered to make decisions and how to make good decisions.

Good governance involves at least the following

  • Established standard decision-making procedures.
  • Well-understood processes for handling exceptions.
  • Decisions made quickly at the correct level against well-defined criteria
  • Decisions get recorded.

And good governance is helped by the following

  • Having agreed strategic objectives.
  • Formal communication methods which record project status and decisions.
  • Controls & records to ensure governance procedures are followed.
  • Regular meeting of the stakeholder group.

Good management means the usual good software project management practices plus

  • Communication
    • Who can communicate with whom? And how often?
    • Formal vs. informal communication.
    • How does the communication get recorded?
    • What happens on a miscommunication?
  • Knowledge management
    • Getting up the initial learning curve and staying there through staffing changes
    • On-going knowledge management
      • What is the process for asking a question?
      • What is the process for recording an answer?
  • Audit trails
    • What trails do you need? Decision making, Execution history, Timesheets?
    • You need to save this data quickly and cheaply but what about access time?
    • How strict does your audit need to be? Are their any legal requirements?
    • What happens on a miscommunication?
  • Well-defined incoming QA procedures

In summary, there are many benefits to offshoring your software testing, especially if you aim to build a competence based rather than transactional outsourcing relationship. However, the risks are also high and in the above I have laid out a process to improve your chances of success.

  • Agree the strategic goals in an appropriate forum and share them with your provider.
  • Decide what type or stage of testing to offshore.
  • Identify your preferred outsourcing model and location.
  • Assess your readiness for outsourcing.
  • Select your partner or partners.
By Dr.Mike Bartley

Thursday, November 5, 2009

Why Do We Need Software Testing?

For any company developing software, at some point pressure to reach the deadline in order to release the product on time will come into play. Additional pressure from project stakeholders, such as 'Marketing' will not want to delay the release date as significant effort and money may have already been spent on an expected release date.

Quite often, planned time to test the software (e.g. ascertain its quality - QA) will become reduced so as not to impact the release date. From a pure business perspective, this can be seen as a positive step as the product is reaching the intended customers on time. Careful consideration should be taken though as to the overall impact of a customer finding a 'bug' in the released product. Maybe the bug is buried deep within a very obscure functional area of the software product, and as the impact only results in a typo within a seldom-used report, the level of impact is very low. In this case, the effect on the business for this software company would probably be insignificant. But what if the bug resulted in the program crashing and losing data? Maybe this software product is used within an air traffic control system? As you can imagine, the impact of this type of bug could be incredibly high and may result in loss of life and destroying the entire company responsible. So basically, the level of risk of a bug being found (likelihood) and what is the effect of the bug (impact) prove to be critical in how much software testing is performed prior to a products release.

Due to the complexity of modern software it is impossible to ensure that software is bug-free.......it really is!

Imagine a simple form on a software application that was designed to accept one of ten specific values, in order to test this completely, you would need to create a test case for each and every permutation of the entries that could be entered by the user, for example:

10(inputs) to the 10(values)th power

10 to the 10th power

Result = 10,000,000,000 test cases

So, if you were the tester hired to perform the testing, and it only took you one second to perform each test case, it would take around 317 years to complete. Therefore, the test planning should take into consideration what is actually 'achievable.'

Software testing (synonymous with the term Quality Assurance) itself can have many different purposes (quality assurance, validation, performance etc). This is a key decision when planning the QA /software testing, as not testing enough or testing in the wrong areas will inevitably result in missed bugs. The aim should be first ascertaining 'why' we are going to test and not simply 'what' we are going to test.

Software testing and or Quality Assurance is still a kind of art, mainly due to a limited understanding of the complexities of modern software. Recent years has seen the development of software testing certification such as ISEB and ISTQB. This is good news for the software industry as a whole, as the more experienced a software tester is then the level of quality of the software they are testing can only increase.

Software testing cannot ensure software is bug-free, but it CAN increase software quality.

If we aim for perfection, we may just achieve excellence!

Article Source: EzineArticles

Sunday, November 1, 2009

What is meant by testing?

Testing is performed by any organisation to ensure the quality of the product and continual improvement on the complience.
Testing is the product certification after the product developed and before the customer release.

Testing is a process of executing a program with intent of finding an error or testing is a process used to help to identify the correctness,completeness and quality of developed computer software.

Testing is of three types
1. Black box testing,
2. White box testing,
3. Gray box testing.