Web Software Testing:

Challenges and Solutions

Abstract:

Web software-development has taken off in the past two years. Companies are not only creating dynamic, interactive web sites, but also are moving towards web-based commerce and corporate enterprises on the Internet.

Has the software-testing world kept pace with these rapid changes? Only recently have new software-testing tools been available for Java and web browsers. This presentation will provide an overview of how software-testing processes and tools need to be enhanced to handle the complexity of the web.

Brian Fromme

President - Cyborg Solutions, Inc.

CSI-98-01 Revision: 1.0
February, 1998
Cyborg Solutions, Inc.
4900 Langdale Court
Fort Collins, CO 80526-3946
Phone: (970) 223-6102
Fax: (970) 223-9219
Email: fromme@cyborgs.com
Web: www.cyborgs.com
Copyright 8 1998, Cyborg Solutions, Inc. All rights reserved.

How Can We Build High-Quality Web Software?

The Internet and corporate intranets will continue to bring many new possibilities for business communications, marketing, and electronic commerce. Dynamic web sites are now providing immediate access to new products, services, and information. Today most companies aren’t asking if they should be on the Internet, but when they should be on the Internet.

The breakneck speed at which internet programming evolves is making existing tools and methods obsolete. It is truly exciting to see the volume of new internet programming tools, but so many of these products are just entering the market. These tools often lack the maturity and sophistication of tools that have been in use for years.

In particular, software-testing technologies have not kept pace with the Internet’s rate of change. There are fewer than a dozen internet software-testing tools on the market today. When compared to the hundreds of programming tools, it becomes clear that we can easily produce more software than we can possibly test.

So how do you create and deploy high-quality web software? Web software is unique. The Internet, by its decentralized nature, presents new software-testing challenges. Our web software will require enhanced testing techniques.

This paper will show how you can make a plan for achieving high-quality web software. You’ll see how refinement of a few key software processes, tools, and metrics will indeed help you create high-quality web software.

Conventional Software versus Web Software

Conventional software development has several aspects that aid testing. Most importantly, you control the deployment platform. You make choices about the hardware and software components that will best support your applications. Market pressures may dictate that your applications run on multiple platforms, but the model remains the same, as shown in Figure 1. Tools in this model are stable and mature. In this environment, you can limit the dependencies on which you must support your applications, and therefore limit the amount of testing required.

Figure 1: Conventional development and deployment platforms.

With web software development, you control the server, but rarely control the clients (see Figure 2). In the public internet, your customers may be using one of several browsers, each with different capabilities. This model might be simpler in standardized corporate intranets, but these are the exception. You still make choices about most of the software components, but the intermediate hardware is usually unknown to you. Furthermore, new, rapidly changing languages and tools make the programming environment much less stable.

Figure 2: Web development and deployment platforms.

Unique Aspects of Web Software

Web software development is different from many conventional programming projects. Some unique aspects of web software are:

Java

Java is a new programming language developed by Sun Microsystems. Created for programming consumer electronics, it is well suited to the web. Java is a truly portable language due to its interpreter. It also provides true object-oriented capabilities that make plugging components together more reasonable. Because Java is a new language, it is going through rapid changes. New toolkits, changing APIs, and unclear strategies are just some of the problems software-development organizations have to face.

JavaScript

JavaScript is a Java-like scripting language developed by Netscape Communications, specifically for its Netscape browser. JavaScript is used to program dynamic behavior in previously static HTML web pages. Although JavaScript is available in both Netscape Navigator and Microsoft Internet Explorer, there are compatibility issues between the two browsers.

ActiveX

ActiveX is a component technology developed by Microsoft that brings Object Linking and Embedding (OLE) capabilities to the Internet Explorer. For public internet programming, the main issue with ActiveX is that it is not available in Netscape Navigator or other browsers.

Mixed code, documents, and graphics

Web software displays as a combination of program execution, formatted documents, and graphics. This multimedia experience can improve the visual communication of the program, but it also is a dramatic change from the current programming paradigm. Now, developers, writers, graphic artists, and marketing engineers all have to work together to get this job done well.

Browsers that lack debuggers

Although the Java Development Environment comes with a primitive debugger, today’s browsers do not. Therefore, debugging software for the browser is much more difficult, particularly for unit testing.

Wide variety of users

Web software is opening many computing functions to a wide range of end users. This makes application usage and training needs much more complex. Also, it is hard to anticipate all user environments, which places a burden on the web software designer.

Many configurations

Users of web software can be running on several different operating systems and many complex configurations of software. Typically these factors are not under the control of the development team.

Public network

Web software is deployed into an environment that runs over a public network. Elements such as network performance and intermediate security are often out of the developers’ hands.

To deal with the unique aspects of web software, it is important to develop a combination of processes, tools, and metrics, as shown in Figure 3. Software development processes are the manner in which teams and organizations run their projects. All software development organizations use processes although many don’t like to think about them. It is often much easier to think about how software development tools are utilized. Metrics capture and use key pieces of information about the development model.

Figure 3: Process, tools, and metrics yield high-quality software.

To best understand how to enhance processes, tools, and metrics for high-quality web software, let’s see how conventional elements are employed during the testing phase.

Conventional Software Testing: Processes, Tools, and Metrics

Processes

Conventional software projects often use a number of common software testing processes. These processes can be organized into implementation, functionality, and usability categories.

Implementation testing processes help the development organization construct better software. Code review meetings enable developers to scan and discuss source code for some or all of the project. The change request process is used to dictate how enhancement and defect requests will be propagated into applications. Perhaps the best known testing process is the test plan. Test plans are used to define how unit and system testing will take place.

Functional testing processes help the development organization determine whether the product has been implemented as designed. Functional testing is perhaps the best known testing process and is used to test a specific feature of a product. Regression tests are suites of unit tests that can be run together to determine whether changes have broken existing functionality. Stress tests are used to push a product to extremes. Many of these functional tests can be automated, but often companies will choose to have test engineers run these steps manually.

Usability testing processes focus on creating a product that is best fit for a target audience. Many companies use a Beta test procedure to allow select customers to view prerelease versions of their products. Human factors (HF) testing can also be used to pinpoint use issues that are uncommon or difficult for the product operator. Performance testing determines whether certain functions are completed in reasonable time frames.

Tools

Conventional software testing tools allow developers to achieve many tasks via their computers. Without software testing tools, most of these tasks would be manual and time consuming. Common software testing tools include:

Defect Tracking

Defect tracking tools are used to assist in the change request process. Defects are entered into a database and can be stored on a project, application, or unit basis.

Static Analysis (source code)

Static analysis tools are run over project source code to help study potential coding errors. These tools can also show areas of code that require more extensive testing.

Memory / Runtime Analysis

Memory and runtime analysis tools are built into applications and run often during regression testing. These tools can point out hard to produce bugs long before a customer would see them.

Debuggers

Debuggers are development tools that can be used during unit testing. Also, debuggers are a necessary part of fixing defects found during system testing.

Automated Software Quality (ASQ)

Automated software quality (ASQ) tools allow some testing processes to be run without human intervention. These tools have two main functions: graphical user interface (GUI) capture/replay, and load/performance testing. Automation of testing processes is almost always necessary to reduce overall testing costs.

Metrics

Conventional software testing metrics can be described in three classes: source-based, defect-based, and process-based. Source-based metrics help managers understand the state of the code, such as its size and complexity, and are used in comparison to defects. Defect-based metrics are used to tell how much additional testing is required. Process-based metrics are used to track project costs and to compare to other project schedules. Table 1 shows examples of each type of metric:

Table 1: Software Testing Metrics

Metric Class | Metric Description
Source-based | K Non-Comment Source Statements (KNCSS) Used in many other metrics. Lines of code changed per time frame Indicates rate of software change. Complexity of source statements Indicates where more testing is needed.
Defect-based | Defects found per test period Indicates rate of testing. Defects found vs. previous system total Used to track system test phase. Defects found vs. lines of code changed Indicates stability of system.
Process-based | Total testing time Used in other metrics. Cost of manual tests vs. total test cost Dictates how much automation to apply. Test cost vs. number of defects Captures test expense. Can be compared to other projects.

Life Cycle

Conventional software testing most often occurs after "code complete," when all functionality has been implemented. By capturing data about preceding projects, metrics can be used to more accurately estimate the test phase duration and to understand impacts of testing procedures and software changes.

Web Software Testing: Processes, Tools, and Metrics

Web software development should not be viewed in the same way as conventional projects. It is primarily a customer-oriented process. Web customers expect to see some functionality today and are willing to work with technology as it evolves. This implies that the traditional "waterfall" life cycle, with long, careful phases and one to two-year development cycles will not work.

A "spiral" life cycle better fits this rapid, time-to-market world. Figure 4 shows how the spiral life cycle defines short design, implement, and test phases. This model allows critical functionality to reach customers early and expects that customer requirements will drive future enhancements.

Figure 4: The spiral life cycle.

This model also implies that a change is needed for testing processes. It won’t work to wait for "code complete" versions, then test all possible problems. Instead, testing should be integrated throughout the life cycle, validating requirements, designs, and implementations. It is true that there will be a formal testing phase, but expect that phase to be significantly shorter than with conventional projects.

Web Processes

Web software testing processes do include the same set of "good habits" that were outlined for conventional development. Several processes need even more attention for web software testing. Code reviews become more important due to the object-oriented nature of web software. If you can show that a component behaves a certain way, then combining tested components should require less testing. Also, HF testing is needed to satisfy the demands of a wide variety of users. Users have come to expect certain interface features and HF testing will point out any inconsistencies.

Web Tools

Web software testing tools have a long way to go. Until recently, there were virtually no testing tools available for testing software running in a browser. Today that has changed with the introduction of ASQ tools such as Segue Software’s SILK. SILK helps you verify applications developed for multiple web browsers. Also, SunTest, the Java testing organization of Sun Microsystems, has developed several testing tools for Java applications. The SunTest Suite combines GUI testing, API testing, and source-code coverage tools for Java. Although you can use these tools to test out your Java program as an application, they are not integrated into the browser environment.

Conventional software testing tools can be reused for web testing. You would probably be wise to use the same defect tracking tool. Many other common software testing tools, though, such as static and runtime analysis, debuggers, and load/performance analysis are not available for the myriad types of programs used in web software. Except for using products like SILK for automated tests, many of the other functions will have to be hand-crafted, or not done at all!

For example, a single-page form to place an order for a product will often consist of more than half a dozen programming languages and several toolkits. The form page is generated in HTML, using JavaScript to make the page dynamic. The actual form is written in Java to better display and validate the user data. The data is passed to the server via CGI, which invokes a Perl script that runs the C++ database query via a database toolkit. There just isn’t a single development environment for all of these languages and technologies.

Web Metrics

Web software metrics have the same goal as conventional software metrics, but require different approaches. It is desirable to calculate source-based, defect-based, and process-based metrics, as these will help in making development decisions. For web software we must also add another category, customer-based metrics, to help manage the rapid changes needed during the spiral life cycle.

Source-based web metrics are more difficult, because web software has such a wide variety of programming languages and many of these languages lack source analysis tools. For example, we might be able to count Java source statements, but how do we calculate sizes of HTML, JavaScript, ActiveX, and CGI scripts?

Defect-based web metrics still provide a good model to work toward. The defect information captured during implementation and testing phases will help in understanding how long future test cycles will take. Unfortunately, it is often difficult to find comparable web software projects, due to the component nature of web applications. Also, many organizations are just launching their web development.

Process-based web metrics are useful to track development costs throughout the spiral life cycle. These measures will keep process overhead to a minimum and help make changes to the life cycle that produce high-quality applications more quickly.

Customer-based web metrics should be included as a mechanism to incorporate user requirements into the spiral life cycle. It is extremely important to focus new development on key features, as the spiral model dictates shorter cycle times. Another recommendation for customer-based web metrics is to develop an enhanced Beta-site program. Allowing more of your customers to get early versions of your web software will increase the requirements and defect feedback. This will lead to higher satisfaction levels, which ultimately means higher-quality software.

Achieving High-Quality Web Software

Web software presents a number of new challenges to development organizations. The rapid pace of technological changes is a big part of the problem. But with changes come new ideas, possibilities, and opportunities. Assuming the Internet remains the corporate and consumer information infrastructure (a good bet), it is safe to say that most companies will have to move some applications to the web.

It is possible to achieve high-quality web software. Perhaps the most important aspect of this new approach to software development is to create flexible, customer-oriented processes. Companies that take two years to produce new software are finding that they cannot keep up with customer expectations. Fundamental process changes, such as the adoption of the spiral life cycle, are required to allow innovation to reach and impact customers.

Although web software technology is changing so rapidly, new tools are becoming available to help test web applications. It is important to employ these new tools to allow development teams to maximize their time writing new features, rather than creating testing mechanisms. Furthermore, outsourcing web software-testing tasks helps companies develop more functionality with less staff.

There never seems to be enough time to accomplish all the tasks that are required during software development. Nevertheless, focus on a few key ways to measure your web software development projects. By collecting simple project statistics, such as number of hours tested, future projects will become easier to manager and estimate.

Finally, becoming customer-oriented is required throughout all aspects of web development. Business to business electronic commerce will open new markets, but will also bring the customer much closer to the application development organization. Companies doing business and writing software the "same old way" will find increased competition on the web. Companies able to make change happen quickly, yet produce high-quality software, will attract and win new customers.

Biography

Brian Fromme is the President of Cyborg Solutions, Inc., a software engineering and consulting firm. With more than fifteen years of software development experience, Brian holds a United States Patent for the Encapsulator, a CASE integration technology. A former SoftBench Product Manager with HP, Mr. Fromme has done research in CASE, Artificial Intelligence, and computer languages. Cyborg Solutions, Inc. has specialized expertise in UNIX and PC development and testing processes.