Managing Problems for Global Software Production

- Experience and Lessons

Abstract

With the increase of current software projects in size and complexity, many large companies have established global software production lines over the world to develop and delivery software products with collaborative software development process involving multiple teams located at different sites. Supporting global software production needs an effective software-engineering environment to meet the special requirements of the collaborative software development process, diverse management methods and engineering practice. The WWW technology provides a powerful mean to set up an enterprise-oriented software engineering environment for global software production due to its advantages on networking, global accesses, internationalization, and communication. Although there are many articles addressing the methods and experience on building web-based applications systems and tools, very few papers discuss the real-world problems and solutions in the development and deployment of a web-based software tool to support a collaborative software development process for global software production. This paper discusses the real world issues, and reports our experience and lessons in building and deploying a web-based problem information management system (PIMS) to support global software development process in Fujitsu. It focuses on the real issues and needs of current collaborative development process involving multiple teams, and highlights the benefits and impact of the PIMS on global software production. Moreover, it discusses our technical solutions and trade-off in the development of PIMS, and shares our experience and lessons. Furthermore, it introduces a new data-centered conceptual process model to support diverse collaborative processes for project and problem management in global software production. Finally, the paper shares our key successes and weakness, and reports our experience and lessons in the deployment of the system.

Keywords: Software engineering, software maintenance, problem management, software management tool, Web application systems.

1. Introduction

With the increase of current software projects in size and complexity, many large companies have established an enterprise-oriented global software production line to develop and delivery software application systems. Since the global production process usually involves multiple teams and organizations over the world, it is necessary to provide an effective software-engineering environment to support collaborative software processes and project managerial operations. The WWW technology provides a cost-effective mean to set up an enterprise-oriented software engineering environment for global software production due to its advantages on networking, global accesses, internationalization, and communication [7][10].

Although there are many articles reporting design methods and efforts on building web-based applications systems and tools, very few papers discuss the real-world problems and solutions in the development and deployment of a web-based software tool for global software production. Real world practitioners are seeking answers to the following questions.

• How to build software tools to support collaborative software processes for global software production? What are the trade-off in technology selection, experience and lessons in performance tuning, difficulties and challenges in software development and implementation?
• What are the suitable conceptual models for collaborative development processes? How to support the diverse work flows for global software production projects?
• How to deploy a web-based software tool to enhance a global software process? What are the encountered problems, experience and lessons during the deployment of a web-based software system?

In this paper, we made an attempt to answer these questions based on our experience and lessons in building and deploying a web-based problem information management system (PIMS) to support global software development process in Fujitsu. We use the PIMS system as a case study to expose the real issues and needs of current collaborative development process involving multiple teams, and highlights the benefits and impacts of the PIMS system to a global software development process. Moreover, we discuss our technical solutions and trade-off in the development of PIMS, and share our experience and lessons. Furthermore, we introduce a new data-centered conceptual process model to support diverse collaborative processes for project and problem management in global software production. Finally, we share our key successes and weakness, and report our experience and lessons in the deployment of PIMS.

The structure of the paper is organized as follows. Section 2 discusses the different perspectives of problem management in a collaborative software development process for global software production, including current issues, status, team structures, and requirements. Section 3 discusses technology selection, design issues and solutions, experience and lessons during the development of the PIMS system. Section 4 proposes a new data-centered conceptual process model to support diverse problem management processes in global software production. In Section 5, we report our experience and lessons of deploying our PIMS onto a global software production line. Finally, the conclusion remarks are given in Section 6.

1. Understanding Issues and Needs in Managing Problems for Global Software Production

With the increase of current software projects in size and complexity, many large companies have established an enterprise-oriented global software production line to develop and delivery software application systems with the support of multiple teams and organizations over the world. The telecommunication group in Fujitsu is an example. This group has many global software production lines. Each involves teams from three groups of organizations. The first group includes the teams from Fujitsu limited. They usually play as a global project coordinator, project planner, or project manager. They oversee and control project timeline, schedule, and deliverable. The second group includes of teams from Japanese subsidiary companies. They usually play as a supporting role to provide technical support and services for the first group, including quality control and product verification. The third group includes teams from worldwide Fujitsu subsidiary companies, such as Fujitsu Network Communication Systems Inc. In the united state, and other companies in Europe and Australia. They usually are independent business units with specific business application markets and products. They play as research and development teams on a global production line to design and implement new products.

In the past years, these teams communicate with each other to discuss the project and product problems, and share the detailed problem information in four different ways: a) email, b) fax, c) periodical teleconference meeting, and c) business face-to-face meeting. These problems can be classified into many classes, such as requirement problems, design problems, coding problems, testing problem, and customer problems. Although the problems usually are very important to a global software process, each team manages and maintains them in an ad-hoc manner. We examined several global software production processes; we found the following major issues affecting problem management:

• Inefficient problem tracking. Although every team has assigned designated persons to collect, track, and monitor different problems from global software production lines, they usually use an ad-hoc mechanism to track and monitor the problems in diverse formats and media. This causes inefficient problem tracking and problem solving.
• Limited information sharing for problems. There is no centralized problem information repository to allow all involved teams to report, access, and track all problems for a product in a global scope. This frequently causes communication issues among teams involved in concurrent development activities on a global software production line.
• Ad-hoc problem management process. Although all organizations have established its software development process and defined problem management process for projects, they may process the global problems in an ad-hoc manner. We lack an effective and systematic problem management process to support problem reporting, analysis, tracking, and management in a global software process. This affects the product productivity, cost, and quality of a collaborative development process.
• Heterogeneous and host-centered software tools and development environments. Each team has set up its own software development and management environment based on a set of host-centered tools. These tools usually have their limits and restrictions on network protocol, database repository, and platform selection. This is a major obstacle in developing a global software development environment to support problem management and analysis.

Today, with the increasing number of global software projects, there is a strong demand to establish a systematic global collaborative software development process to support all involved teams effectively. To achieve this goal, we must do the followings.

• Define and establish a conceptual model to support problem analysis and management. With this model all involved teams can define and specify their problem analysis and management process in a consistent manner.
• Develop a global network-centered problem information repository. Using this repository all involved teams for a global software project can share all types of problem information of a global software project.
• Provide a systematic method to track and monitor the states of problem processing. With this method engineers and managers can easily understand and analyze the status of problem processing for a project or a product. Based on the problem status information engineers can process and respond the problems in an efficient way.
• Provide an effective mechanism to synchronize problems and their states among collaborative development processes for a global software project. With this mechanism engineers and managers can easily process and respond problems in a very short process cycle.

In 1997, Fujitsu decided to take the advantages of the W3 technology to establish an enterprise-oriented software development environment to support global production lines worldwide [5] [6] [7]. There are several major objectives. They are a) to support the information sharing among different teams, b) to enhance the collaborative development processes, and c) to increase the productivity and reduce the project cost. Developing a web-based problem information sharing system (PIMS) to support problem management for global software production is one of the tasks.

Unlike other existing host-centered problem tracking systems, the PIMS system is a network-centered system that uses the W3 to support global user accesses. It has a lot of advantages over the host-centered systems in network communications, information sharing, internationalization, and global user access. As described in [6], the PIMS system has two sets of requirements. The first set includes the essential requirements of a problem tracking system, such as problem book keeping, status tracking, status report, statistics analysis, information search and retrieval. The other set distinguishes it from other web-based problem tracking systems [3]. This set includes the special requirements addressing the issues concerning problem management and process in global software production.

REQ 1: Support diverse workflow processes for problem report, analysis, management and monitor. A global software project usually involves many teams from different organizations. Since each team has its own pre-defined workflow process for problem report, analysis, management and monitor, it is very important to allow engineers to define and configure a workflow for their development process.

REQ 2: Provide customizable data formats and report templates. Every organization involved in global software production has its own standards on the development process and quality control system. The standards defined by different organizations usually have inconsistent definitions on problem classification and severity, and different requirements on problem data items, formats, and report templates. In addition, different projects may have different requirements and needs. Thus, to cope with diverse needs, it is essential to provide customizable problem data formats and report templates to allow users to change them according to their project needs.

REQ 3: Provide flexible user groups setting with pre-defined accessible functions. Since each team performs the specified tasks and functions in a global software project, its members may form a user group that requires a set of particular user access functions. For example, a development team member usually needs to analyze a problem, and fixes it. However, a verification team member usually needs to report a problem, and records the checking result after it is resolved by engineers. Therefore, it is necessary to allow a system administrator to set up a user group with a particular set of pre-defined accessible functions.

REQ 4: Flexible data import and export for problem repositories. With this feature a PIMS system can import a problem database from another PIMS system, or export a problem database into another PIMS system. This feature is very useful for software reuse projects and concurrent development projects in a global scope.

REQ 5: Problem notification for global communication. With this feature a user can send problem notifications through email to one or a group of users, including external users. This feature is essential to share problem statuses and synchronize problem processes in a global scope.

1. PIMS Development Experience and Lessons

In July of 1997, we started a project to develop a global web-based problem management system for global software production after we completed a couple of prototype projects using the web technology. The major motivation is to use this tool to establish a systematic problem management process for all teams involved in a reuse-driven software project in Fujitsu.

To support different user accesses and enforce system security, we divide user accesses on Internet into three scopes (see Figure 2):

• Fujitsu -Net - It is a Fujitsu private Intranet. All internal divisions and department are inside this Intranet. Different teams involving the same product line may share information and reuse common software.
• Internet - Product customers could communicate with customer support teams, marketing and sale people on Internet by accessing a customer support system and/or a business management system on Internet. Since these users could be any one, a rigorous security mechanism and security policy must be adapted.

• Extant-Net - Many global software product lines involve teams from external software workshops or warehouses. Although they have their environments and tools, they need to communicate with Fujitsu teams, and share a selective set of information crossing network firewalls. These include the selected subsets of project management information, problems, test information, and source code or documents. For example, a Fujitsu validation team may need to monitor the problems, test process and testing status of an external software testing team. Meanwhile, an external software develop team needs to receive and check the problems found by an internal validation group.

Technology selection is an important part of design trade-off in the development of a web-based software tool. Selecting technologies is always essential for a development manager. We can group the existing technologies into through groups: a) client technologies, such as HTML, JavaScript, ActiveX, Java Applets, b) client-server communication protocols, such as HTTP, SHTTP, RMI, CORBA II, OLE, and c) server and middle ware technologies, such as CORBA, EJB, COM, and Web server, Java Servlet. Each technology has its pros and cons. Because of the limitation of the paper scope, we can not list out all their features, advantages and weakness. Instead, we compare different existing approaches based on these selected technologies in the aspects of portability, security, performance, scalability, concurrence, cost, and complexity. Table 1 listed our comparison results.

In the first approach, HTTP is used as a connectionless protocol to support the communications between a web browser and a web server. At the server side, CGI is used to a communication interface to link the web server to a server program to respond a user request. Dynamic HTML pages are generated and posted to a client through a web browser. The systems generated in this approach are known as interactive HTML-based web systems. To achieve secured communications, we can adopt SHTTP protocol using a secured web server. Since HTML and CGI are simple and platform independent technologies, therefore, the major advantage of this approach is simple and cheap because all involved technologies are free here except SHTTP web server. There are three major problems in this approach. The first problem is its slow system performance. Slow execution of CGI scripts is the major cause. The second problem is the lack of support for multithreading and concurrent server processing due to the single thread limitation of CGI. The other problem has something to do with HTML. Since HTML does not support window creation, dynamic tables, and graphics display, it is very hard to provide an effective user interface to support dynamic system-user interactions to clients. With this approach, we can create simple interactive web-based systems.

Table 1. Comparison of Different Approaches and Technologies

To develop more complex interactive web-based systems, we can select some other technologies to improve system performance, client interface and security. JavaScript (or ActiveX) can be used to improve the system user interface with its primitive GUI features, such as window, dialog box, input validation, and dynamic table. Moreover, Java applets can be used to provide clients with more complicated graphic user interfaces, including multiple concurrent windows with elements (such as buttons, menus, etc), and graphic display. In addition, Java applets also enhance the security of client software because Java applets are invisible to any Internet users unlike HTML and JavaScript. In addition, CGI scripts can be replaced by Java Servlets to improve system performance. Since Java Servlets is a Java component that can be plugged into a Java-enabled web server, its performance is much better than CGI scripts because it has no overhead on program loading and interpretation.

To develop interactive web objects over the Intranet, we can use RMI as a middle ware to support the communications between Java applets and a Java-based server program. Unlike HTTP and SHTTP, RMI is not a standardized protocol without security and predefined message formats (or object formats). Compared with CGI and web server, RMI provides better system performance and supports multithreads. However, RMI can not provide secured web-based systems on the Internet because of its security weakness.

Now CORBA and COM have been used to develop interactive web-based systems with web objects over the Internet. They are very good middle ware technologies due to their support in concurrence, security, object-orientation, distributed objects and networking. In CORBA, web objects (say OrbixWeb objects) in Java applets at the client side communicate CORBA objects of the server program through CORBA IIOP protocol and ORB server. In COM, web objects (say DocObjects) communicate with server objects (such as ActiveX Doc. And DocObjects) through Network OLE and COM ORB sever. Since CORBA is platform independent technology, it has better portability than COM. However, they are very complicated to learn and use in comparison with other technologies. Although other technologies are free to use at the current point, CORBA and COM are costly to use and run.

In the development of the PIMS system, we made the following technology selections. We used HTML and Java applets to develop user friendly client software. We selected CGI and SHTTP web server as our middle ware due to the cost and availability of technologies. We created our application server based on C++ programs.

The PIMS system has a four-layered architecture.

• User Interface Layer, which refers to a user interface running on a web browser. PIMS system has its own client software, which supports distributed accesses from users. According to our experience, Java-based client software usually provides a better user interface than HTML-based client software due to Java's AWT features. Although HTML-based client software supports dynamic tables, dialog boxes and windows; it has limitations on creating dynamic graphics, and complicated windows. On the other hand, Java-based client software usually consumes much more system resources than HTML-based client software. Java applets needs a longer loading time.
• Communication Layer, which controls the communications between Java-based client software on a web browser and its corresponding server. When a web server is used, the communications between a client and its server follows the HTTP protocol or its secured version, such as HTTPS. This protocol only supports connectionless communication. Both client and server software needs packing and unpacking functions to support their communications. CGI programs at the server side supports the mapping between a client’s request and its corresponding function in the PIMS server.
• Function Layer, which is the PIMS function server. It supports problem tracking, management, analysis and reporting.
• Data Storage Layer, which consists of three parts: a) a database access library, including a collection of database access classes or programs, b) one or more database servers, and c) logic databases and physical databases. To cope with different database drivers, a standardized database definition language must be selected, related database schema conversion tool, and database migration facility is needed [6].

As shown in Figure 3, the server in PIMS includes two parts. The first part is System Administration, which supports different system administration functions. It includes the following components: a) security controller, which sets up and monitors the system security; b) access controller, which creates and maintains user accounts, user groups and user access permissions; c) workflow processor, which creates, configures, and maintains a process-oriented workflow according to a state-based workflow model; d) system configuration and customization, which configures or customizes the system persistent data, parameters, and features according to customer needs [11]; e) a database API to a system administration database; and f) a communication gateway interface for system administration functions.

The other part is problem management consisting of the several parts: a) a communication gateway interface; b) a problem database API, c) problem book keeping, which records and documents the problem information from users; d) problem tracking, which tracks and monitors problem states; e) problem analysis report, which generates a statistical analysis report based on the stored problem information; f) problem report generator, which produces an HTML-based problem report based on database query results; g) database import/export, which imports or exports a subset of the problem information from one database to another; and h) database migration, which migrates a database content to a newly created database.

In the design of the PIMS system, we include three special functions to support the information sharing and communication among different PIMS systems. An email agent is the first. It supports event notification and information exchange between two PIMS servers over Internet through an email server. The next is the FTP agent, which supports file transfer between two PIMS servers over Internet through a third party FTP server. The notification feature is the last function. This works based on the current workflow model. For example, whenever a problem fixing record is stored in a problem management system, a message should be sent out to notify the corresponding testers.

As shown in Figure 4, client software consists of several parts. The first part is a loader, which includes applet loader and data loader. Data loader is used to load data dynamically from a server to a client. These data are either client-specific or dynamic changeable. Applet loader is used to load Java applets dynamically from a server to a client machine. Although Java Virtual Machine has provided this feature to load embedded Java applets in HTML pages, we create the applet loader to load other applets during run-time. This is very useful to control the loading performance and system resource on a client machine. The next part is access controller that consists of user access control, security checking, and workflow checking. The third module is a client data store, which stores and maintains a set of client-specific data, including system persistent data/objects and dynamic data/objects. Persistent data/objects refer to the persistent information for a particular user, such as user accounts and passwords, security codes and access control information. The dynamic data (or objects) include function-specific data (or objects) loaded from a specific information repository. The fourth part is a set of GUI components support specific functions, including Text Report GUI, Analysis Report GUI, Problem Tracking, and User Data Configuration. Besides, there is a system administration GUI interface to support all administration features, and a communication interface between client software and the PIMS server. Finally, there is a utility module that includes a set of classes, such as Tool bar, Timer, Print, and Help.

System performance for a web-based application system is very critical and depends on many factors, including network performance, computer performance, selected technologies, system infrastructure and architecture, and the performance of adopted third part software, and created application server/client software. Due to the limited scope of this paper, we only highlighted some experience on software design of client and server programs.

• Use BufferedInputStream instead of DataInputStream in the implementation of Java I/O. The readLine(..) method in the DataInputStream class actually reads an input stream character by character until a "\n" or "\r\n" character is found. This is extremely slow when reading a large file. To improve this, BufferedInputSteam can be used.
• Avoid using multiple dimension Vectors in Java. Although a Java Vector is very flexible than an array, but it is very slower, particularly when a multiple dimension Vector is used. In our case, we used a 2-dimension Vector in the Java Applets of our first release, later we changed it into a single dimension vector based on string arrays for the performance reasons.
• Avoid using synchronized method in Java unless it is necessary. Synchronized methods are provided in Java to support data sharing and control among multithreading processes. Since they are slower than unsynchronized methods, it is a good idea to reduce their usage. When creating a synchronized method, be sure to minimize its source code.
• Use low-cost operators as much as you can. Like C++ and C, Java supports a set of shorthand arithmetic operations, such as ++, --, +=, -=, *=, and /=. Since they actually generate faster code, so use them whenever possible.
• Avoid dynamic object creations in Java applets. Dynamic created objects not only need extra object creation time, but also cause the extra overhead of Java garbage collector to recollect them.
• Use StringBuffer for excessive string manipulations. Since strings in Java are immutable, any change to a string creates at least one more string objects. Thus, complicated string manipulations slow down Java programs, and create temporary string objects that need to be collected by Java garbage collector later. Since StringBuffer is modifiable, no temporary string objects will be created. Therefore, it is a good idea to use StringBuffer instead of strings for excessive string manipulations.
• Cache the frequently used data in the client side. It is a good idea to create a data buffer in the client side to cache and store the session data, such as user name, pass word, and frequently used data.

(a) Performance Improvement for Data Loading

(b) Performance Improvement for Reporting

Figure 5. Performance Tuning Results for the PIMS System

• Reduce the usage of CGI scripts in the server program. Although CGI scripts provide a simple and platform independent communication interface between a web server and application programs, their execution speed is very slow due to the following two factors: a) CGI scripts must be loaded into system memory before their execution; CGI only supports single program thread. Our experience indicates that minimizing CGI code in an application server program is a very effective way to improve system performance.
• Partition the dynamic query results with fixed-size buffers. This is necessary to support user queries that generate a large volume of retrieved data from an information repository. For example, a PIMS system user may send a reporting request to generate a problem summary report based on all problems in a problem database. Since the number of problems in a database is a dynamic data, it could be 10, 50, or even 500. To achieve the reasonable performance in all cases, we must partition the result data into a number of pages, and deliver them to the client one by one.

• Reduce unnecessary database connections. In response to a user request, an application server may need to retrieve several database tables to generate the corresponding results. CGI-based implementation easily drives developers to write a script to retrieve each database table. Due to the limit of CGI, each script must connect to a database server before its database access, and disconnect from the database server after its database retrieval. This causes multiple database connections for a single user request. Since database connection time is very costly, it is important to remove the unnecessary database connection time for database accesses. During the performance tuning for the PIMS system, we improve several database access features significantly by reducing the unnecessary database connections.

• Optimize SQL queries for database accesses. The optimization of SQL queries is very useful to deal with user requests that require multiple table accesses. For example, one reporting feature in the PIMS system requires 6 SQL SELECT queries with different conditions to access 6 database tables. After optimizing the queries, we significantly improved the speed of this feature.

The above performance tuning tips did help us to improve the system performance of the PIMS system. The tables in Figure 5 shows the performance improvements for one release.

• Pay attention to software reuse. Besides reusable architecture models for client and server software, we have spent effort on finding the reusable solutions to several features, including workflow, user access control, and system customization for user-defined data. Moreover, we also developed other reusable software framework, such as database migration facility, 2D and 3D Java graphic library for statistical report generation. These reusable solutions and components have proved to be very effective to reduce the cost of software development for different systems.

• Aim at a standardized database accesses solution. For an enterprise system infrastructure, it is very important to have a standard database-access solution to cope with different database drivers and the future evolution of database technology. To achieve this goal, three things should done. Firstly, creating application database access programs using a standard database access API (such as Java JDBC) and a standard database access language (such as SQL) to separate application database access programs from specific database drivers. Secondly, developing an in-house database facility that can convert different database definitions based on an internal notation of database. Finally, implementing an in-house database migration facility that can move data from one database driver to another based on an internal database notation of a database schema.

• Focus on system customization. Customization features are the key to the success of any of systems in the infrastructure. Ideally, any system in an enterprise infrastructure must support five-customization features a) workflow customization; b) users access configuration, c) user-data customization, d) report format configuration, and e) system selection and configuration. According to our experience, these features are very import to the introduction of a system infrastructure onto a global software production line because of the diversified requirements and needs from different teams.

• Design for scalability. Scalability of a web-based software system for global software production can be viewed from two different aspects. The first is data scalability, which indicates its capability of supporting the increasing amount of data volumes in an information repository. With the increase of projects and product releases, the amount of project data for products will increase too. The next is user scalability, which indicates the capability of supporting the increasing number of users. Unlike on-line service systems in which the number of users is unpredictable, a web-based software-engineering tool usually has a set of dedicated users, for example from 10 up to several hundreds. These users usually have longer login sessions. To achieve high scalability in a web-based software-engineering tool like PIMS, we can do the followings.

• Creating one problem information repository for each global production line. This helps to partition the data volume and controls the number of potential users for a web-based software tool.
• Setting up application servers on different machines to reduce the system load of a server machine.
• Using multiple servers to share the traffic load. The basic idea is to dispatch client requests to different servers. A proxy server can be used to dispatch client requests to different web servers by reading the cookie information in the header of an HTTP request and locating the corresponding web server that has the same session object. To avoid the bottleneck problem of a proxy server, we use some hardware device, like LocalDirector Load Balancer, to dispatch client requests to multiple proxy servers.

• Design for system resource. During the development of Java-based client software, it is important for developers to consult with customers to find out the system configuration information of client machines, such as their minimum memory space. Besides, developers must understand customers’ requirements and limitations on system resource usage of client software, because the information is very important for their design and testing. For Java-based client software, its system resource usage depends on four factors. Its running environment, that is its web browser and supporting Java virtual machine (JVM), is the first factor. Usually, a web browser and its JVM have their requirements on system resources on a client machine. The next is relating to the number of Java applets and their sizes in a web browser window. The third refers to the data volume stored in the client software. The last has something to do with the complexity of GUI components and their structures, including colors, images, icons, and threads. It is clear that the design and implementation of Java applets affects the three of them. At the beginning of the project, we did not pay much attention to the system resource in the design of client software. Later, users found that they could not run PIMS Java applets with other software concurrently on their 32MB portable computers due to the shortage of system resources. This problem was resolved after developers redesigned and restructured the Java applets. Through this experience, we found several design tips, which are useful to control the system resources of Java-based client software.

• Partition a GUI interface into a number of Java applets according to high-level functions. Create a main window to hold a controlling Java applet, which can creates separate browser windows to hold function-specific Java applets. This method not only reduces the system loading time, but also cuts the size of Java applets.
• Focus on class/object reuse and avoid object creation. According to our experience, this proved to be very effective. For example, in one release of PIMS system, our developers reduced 30 to 40 percentage of system resource usage of client software after we enforced this rule in GUI design and implementation.
• Control the data volume at client side, for example, using a fixed-size data buffer, controlling the size of images and icons, reducing the number of used colors, and selecting simple and popular colors.
• Effectively use the dispose method in Java to reclaim big objects immediately when they are no longer useful.

According to our experience, testing of web-based systems is complicated and costly due to its special features such as distributed/concurrent accesses, platform independence, security, and internationalization. A web-based application usually depends on many different technologies, such as HTML, Java or J++, JavaScript or ActiveX, web browsers, web servers, third-party middle-ware. This makes it difficult to automate the test process of a web-based system. Although a few of test tools are available on today's market, they all have various limitations in different areas. Therefore, there is a strong need on practical test tools for web-based application systems.

• Platform testing is important for web-based systems. Since our PIMS system has Java-based client software, we first thought that users should have no problems to access it through any Java-enable browser on different platforms. However, this might not be true after testing it on both PC and Unix platforms using different browsers. In the best case, a Java applet work well with different browsers except that the applet's look and feel is changed in its color and size on different platforms due to the hardware differences. In the worst case, a Java applet, which works fine with one web browser release, may have problems to run and function in another browser on the same client machine. When upgrading a web browser from one version to another, we had the similar experience. Therefore, platform testing is a necessary step to make sure that a web-based system works well on a specified platform with the required web browsers.

• System resource testing is necessary for web-based systems. According to our experience, system resource testing of a web-based application system on different types of client machines is a necessary task in system testing. Its purpose is to check how much system resources (such as memory) are used by Java-based client software on a client platform. System resources include memory space, swap space, number of threads, and GUI resources (i.e. the number of colors and windows). For each platform, different administration tools are available for system testers to check the system resource usage of given software. Because of the implementation difference between web browsers, it is not surprised that on the same platform, the given Java applet needs different system resources when we change its web browser from one to another. An advice to system analysts is to find out customers' requirements on each type of client machines and its minimum system configuration at the earlier stage. An important tip for testers is to set up a benchmark test machine with the minimum system configuration for each platform based on customers' requirements. Another tip is to perform system resource testing of your client software as earlier as possible during system integration and system testing. We learned these in a hard way. During the development of PIMS system, we did not get the information from our customers at the earlier stage. Until delivering the first release, customers found that they could not run the system's Java applets on their portable machines concurrently with other software, because most of them have only 32MB memory.

• Performance testing is critical, but very expansive and time-consuming. Performance testing is very critical for a web-based application system because of its nature of distributed accesses. To prepare performance testing, testers must select a target server machine, and set up different client machines as a benchmark environment. Ideally, performance test must check the system performance according to a set of pre-defined test metrics. For PIMS system, we check its system performance using several test metrics: a) user category, such as single-user performance, and concurrent-user performance, b) platform category, such as Unix, PC, portable notebook, and c) function category, such as problem report, problem analysis, problem display, and problem book keeping. Due to the lack of web-based auto test tools, performance testing becomes very time-consuming and tedious. It is a good idea to develop a performance tracking facility inside a system to generate the performance metrics for both client and server software, so that the performance test cost can be reduced significantly.

1. Conceptual Process Model and Workflow

A global software production line may involve a number of development teams and verification teams at different locations. Each team may have its own workflow to control and manage the process of problem tracking, analysis, and resolution. Figure 6 shows two typical workflow examples. Figure 6 (a) shows a typical workflow used at Global Technology Division in a Fujitsu Network Communications Systems, Inc., and Figure 6 (b) displays a typical workflow used in Operation System Division of Fujitsu Limited. The optional steps in 6 (b) indicate the different additional steps involving in a workflow. These optional stages are either blocking or non-blocking stages. According to our customers, the PIMS must provide a simple, flexible and configurable workflow mechanism to support diversified needs for problem tracking, analysis and resolution in different teams.

The basic workflow requirements for the PIMS system are summarized below:

• Provide a simple reusable workflow mechanism. Customers require the PIMS system to provide a simple workflow mechanism to track and monitor the status of a problem in a problem repository. The workflow process should be easy to implement with a small overhead of system resources and performance. Moreover, the workflow mechanism should be reusable by other management systems in the Fujitsu global system infrastructure in [4].
• Support automatic problem tracking and monitoring. The PIMS system must be able to automatically sets up, changes, tracks, and monitors the status of each problem based on a given workflow. For each problem record, there are two problem attributes concerning the problem processing workflow:

• Problem status - It is used to track the status of a problem. It has different values, such as two pre-defined problem status values, such as OPEN and CLOSE, and other user-defined status values (see Table 2).
• Workflow status - It is used to track the progress status of a workflow process for a problem. Each status value indicates a stage of a problem process workflow.

• Allow users to configure diverse workflow. To accommodate diverse problem process workflow used by teams located worldwide, the PIMS system must allow a system administrator to configure or define a workflow based on a pre-defined default workflow. The current PIMS system can support multiple global projects. A system administrator can set up a workflow for each project to support problem analysis and processing for its software products and releases. For example, a system administrator can create and delete a workflow, or select a workflow from a pre-defined list. In addition, a user also can configure a pre-defined workflow, and convert it into a user-defined workflow by selecting state nodes and their related transition edges from a pre-defined superset. For example, a system administrator can add one or more approval nodes and its incoming and outgoing transition edges based on a pre-defined workflow (in Figure 6 (a)), and generates a new user-defined workflow as Figure 6 (b).
• Perform workflow control and checking - The PIMS system must check all users' operations based on a pre-defined workflow model to make sure that they follow the workflow to process problems.

The PIMS system uses a state-based workflow model to define a process workflow for problems in a software product. In this model, each workflow can be viewed as a finite state machine that consists of a) a set of states, and b) a set of edges between states. Each state represents a status during problem processing. For example, PR in Table 2 is a state that refers to a problem reporting stage in a problem workflow process. An edge from one state to another indicates a workflow transition between two stages in a problem workflow process. A transition usually triggers a status change of a problem. Meanwhile, it may cause some other events, such as notification event. For instance, E3 in Table 2 indicates the state transition from state PA to state PF. In some cases, a problem workflow state transition may cause different user operations. E2 is a typical example. It transfers the workflow state from PA to FR. In the course of the transition, the corresponding problem status can be changed from OPEN to HOLD, ISSUE, or ENHANCEMENT.

Edge List:

Workflow State List:

Table 2. A State-Based Workflow in PIMS for Figure 4 (a)

It is clear that this state-based workflow model is simple and easy to understand. All problems in a global production line are processed following a pre-defined workflow model - that is a state-based workflow. However, each problem has its dynamic workflow status and execution. After applying this data-centered workflow model we found that it has the following advantages:

• Simple and easy to understand and implement with low-cost and small overhead.
• Easy to define and change to fit into different software development processes.
• Good at supporting complicated user access control and checking. For example, we can easily check the user access operations following a pre-defined workflow based on some extra user access information defined the workflow state list in Table 2, including a user role (group), a component of a user graphic interface (say a screen or window), and user accessible operations.
• Scalable enough to support many different products and projects.
• Reusable to other management systems, such as test management system, customer-support system.

Based on the state-based workflow model, the PIMS system developed a low-cost workflow processor, which contains two parts. The workflow administration is the first part, which handles workflow definition, change, and configuration. It allows a system administrator to define a state-based workflow model to support a global software production line based on a set of pre-defined default workflow models. To help users, a set of pre-defined workflow models is provided in an administration database to allow a user to select, change, and configure a new workflow. In addition, a user can display the current workflow model in a graphic format.

The other part is workflow tracking that performs two functions: a) control and monitor user operations based on an existing workflow, b) dynamically change the workflow states for all problems. To reduce the communication overhead on Internet, the PIMS system implements the workflow tracking and control function at the client side only. The basic idea is explained here.

• After a user logins to PIMS and selects a software product to access its problem information, the PIMS server passes the corresponding workflow information to the client. The client software stores it at the client side.
• To perform automatic status tracking and workflow control, PIMS stores some extra information in a workflow record to represent the mappings between a destination state and its action-oriented GUI components. For example, the state list in Table 2 shows the corresponding GUI components and related action-oriented GUI elements, such as Add Button in the problem Report Window.
• The client software checks all succeeding accesses based on the workflow information. If any user access violates the workflow, then it ignores the user's request, and generates a warning message.

According to Raul Medina-Mora et al. [11], there are three types of workflow-enabled applications: workflow-initiating applications, workflow-participating applications, and workflow management applications. It is obvious that our workflow model and processor is playing as a participating role in the PIMS system. In the past, there are a number of different workflow models, which have been proposed for process management. For example, Peter H. Feiler and Watts S. Humphrey [8] uses a process transition diagram to model the structure of a process, including process entities and actions. Sergio Bandinelli et al. [2] uses a Petri-net to model a process and its related activities. In addition, Keith D. Swenson et al. [13] also use State chart diagram and Pert Chart to represent a plan in business process.

In the PIMS system, we implement a simple and low-cost workflow mechanism for problem process in a web-based software management tool. A state-based workflow model is used as a basis to model a problem tracking and resolution process for problems of a software product. Meanwhile, a data-driven method is used to automatically control, track, and monitor the problem status and its dynamic workflow. Our application experience indicates that this is a practical and simple solution with a very small overhead. Its reusable and configurable features make the adoption of the PIMS system much easy and simple.

1. PIMS System Deployment Experience and Lessons

The first beta version of the PIMS system is developed at the end of 1997. After 6 months, the first release of the PIMS system is deployed in a global software project, and used by several teams worldwide. Until now, we have delivered 4 different releases of the PIMS system to our customers, and more than 6 global software projects have used the system to support the problem management among teams in a global scope.

The deployment procedure for the PIMS system includes the following four steps.

• Play with the system as a user first. Before the deployment of the first PIMS release, both development team and verification team of the PIMS system have used the system to support problem tracking and management for their project. This enforces the developers and testers to play as users, so they can find many system problems, bugs, and possible usage issues at the earlier time.
• Provide face-to-face training for users and customers. Before installing the PIMS system into a customer site, we provide a face-to-face training to our users and customers although all user documents and on-line help are provided. We believe this is a key to the successful deployment of a web-based system.
• Provide reliable customer service and support. For the first time customers and users, we provide complete customer service, including system installation and customer support, because our system includes database system, web server, and the PIMS server. For global software projects, we even provide support and maintenance for database servers, web servers, and PIMS servers. The approach reduces a lot of efforts and problems from users at the earlier system deployment stage. Our experience suggests that this is an effective way to win user acceptance for a web-based software tool.
• Collect and review user comments and customer survey results. After installing a PIMS system for our customers, we collect the comments from the system's users. Then, we classify them into several groups: a) problems and bugs, b) stereotype comments and features, c) comments for new features, d) comments for feature improvement, and e) comments for feature enhancement. After analyzing the comments, we set up a review meeting with them to discuss their comments and feedback. This is an important step in the deployment process of a web-based software tool for global software production due to the following two reasons. Firstly, we can understand the requirements and preconceptions from our customers and users. Secondly, we can help users understand the importance of the PIMS system for a global software production, and the current technology limits.

During the development process of the PIMS system, we encountered many issues and obstacles. The major obstacles are from users and customers on global software production lines. The stereotype images and preconceptions of users is the first obstacle. Since our users and customers are engineers and managers from different organizations over worldwide, they usually have some experience on using one or more different host-centered software engineering tools. These experiences somehow create their preconceptions on a software tool, including its look and feel, label convention, working procedure, functional features. These stereotype images involuntarily become their obstacles to accept and use a new software tool. The next obstacle on the user side is their unreasonable expectations on a web-based software tool. Because of the current technology limits on the Internet, a web-based software tool usually has a slow responding speed and the limited capability to support windows and graphics in comparison with a host-centered software tool. However, most of our users expect that the PIMS system should have the same responding speed as a host-centered software tool. Some of them expect to see the MFC-based window functions and GUI style. These expectations told us that users always want to use software tools with best performance, quality and GUI style, even though they don't know that the current W3 technologies (such as Java and HTML) can not deliver web-based software tools to meet their expectations. However, these expectations did cause the difficulty during the initial deployment of the PIMS system over global software projects.

In deploying the PIMS system onto global software projects, we encountered three major issues.

• It is costly to maintain the web-based tools to keep up with the fast technology upgrades. As the advances of the Internet technology, the technology vendors upgrade their products frequently. For example, Netscape and Microsoft have updated their web browsers a number of times in the recent two years. Similarly, Sun Microsystems has updated their Java technology frequently in the past two years. For example, they updated Java from the version 1.0, 1.02, 1.1, 1.2, up to version 2.0. Although the fast technology updates are beneficial to users, they cause the high maintenance cost for web-based tools. The Java technology upgrades and the changes of the Java virtual machines in web browsers have driven us to update the Java applets of the PIMS system several times.
• It is not easy to get the commitment from managers. According to our experience, it is not difficult to get the support from senior managers in an organization to deploy a web-based software engineering tool on a global software product line. The real challenge is how to get the commitment from low-level managers. There are many factors that affect managers' views on the deployment of a web-based software tool, like the PIMS system. Senior managers usually have a better understanding of a global software production line; thus, they can consider problems in a global view. However, low-level managers usually have a very narrowed view about global software development process, hence, it is common that they only concern their localized development process without considering the needs of other teams in a global software project. In addition, a team's role and task in a global software project affects its manager's commitment and support. According to our experience, we can easily get the commitment and support from a testing manager than a development manager. Moreover, the managers from software integration and application teams show more interests in deploying a web-based software tool than the software supplier managers whose teams generate software parts and modules in a global software project.
• It is very difficult and costly to elicit the complacent user requirements for the system. Although the system functional features are not very complicated, we have encountered a lot of problems during the requirements analysis phase because of the following three reasons. First, engineers from different teams have diverse preconceptions about the system provide many inconsistent requirements. Next, people from different organizations and countries bring in diverse ideas, functional features, and approaches. Finally, managers like to stick to their original workflow and processes.

• System performance affects users' acceptance during system deployment. Although users of a web-based software tool, like PIMS, enjoy its provided network-centered global accesses to many functional features installed on a server machine, they still have high expectation on its performance. Our first beta release of the PIMS system receives very bad user comments and evaluations due to its performance problems in Applet download, report generation, and slow server responding. This does affect their acceptance of the system until we made a lot of performance improvements in the later releases. What we learned here is that users always expect a software to process their requests in 2 or 3 seconds, even though they understand that a web-based program needs some extra time to accept and process their requests through a remote server. The major difficulty is how to reduce the performance degradation when many users use the system at the same time.
• A well-defined conceptual process model is essential for collaborative development processes. Although many teams involved in a global software project have defined their development processes, most of them use an ad-hoc procedure (or workflow) to deal with the problems on a global production line. To come up a well-defined conceptual process model for problem management in a global scope, we meet with the management groups in each organization to work out a new problem management process model. This model has implemented in the PIMS system in a way to support the diverse needs of collaborative development processes in different projects. Later, this formal model has been very useful for the deployment of the system. Now whenever we deploy the PIMS system to a new organization and new production lines, we always start with the definition of a problem management process using the formal process model.
• Reliable customer service is important to engineers on global software production lines. After we installed our first system release, we assigned the designated engineers to provide reliable customer support and service to our users worldwide, including U.S.A, Japan, and other countries, such as Singapore. Our experience suggests that good customer service and support is very important to global users involved in concurrent development projects and reuse-driven projects.
• User involvement in the development of a system helps its deployment later. We found that the PIMS system is very easily introduced to teams and projects in which managers or engineers are involved in the system requirements review and design review.

The introduction of the PIMS system to several global software projects has brought out several impacts on collaborative development processes.

• Systematically digitize all problem information on global software projects. By deploying the PIMS system into a global software project, all involved teams are enforced to book keep all project problems in a systematic way even though fax, emails, and telephone calls are still used.
• Convert ad-hoc problem management processes into a systematic workflow. Using t he PIMS system, all parties involved in a global software project easily convert their ad-hoc problem management processes into a systematic workflow. This not only improves the collaborative processes, but also enhances the global software process.

• Increase the information sharing and communication among teams on a global software production line. All teams for a project can dynamic enter, view and monitor various problems into a PIMS system. Moreover, they can analyze the existing problems and generate various types of problem statistics reports to get a global picture of the problems.

According to our observations, there are two major successes of the current PIMS system. The first is its systematic workflow model, which provides an effective mean to convert ad-hoc problem processing workflow in collaborative development processes for global software production. The other is its strong customization functions that provide an essential key to support diverse user needs in system data, workflow, and report formats. This makes it easy for us to introduce the system into many teams for global development projects. After deploying the system into global projects, we realized that the current PIMS system, like other existing problem management systems, has its limitations in the following areas:

• There is no systematic technique to actively monitor and track problems for collaborative development processes in a global scope. For example, no mechanisms are provided to raise emergent alarms for urgent problems.
• There is no systematic time-out mechanism to remind engineers to process a problem before a given date and time.
• There is no support to synchronize different problem procedures in collaborative development processes for a common production line.

1. Conclusions

How is everyday software engineering practice differently when the Internet is used? How should we use Internet technology to improve problem management in a global software process? What are the needs, problems, solutions and effective strategy to establish a web-based enterprise system infrastructure for a large corporation to support software problem book keeping, problem tracking, problem analysis, and reporting for global software production?

This paper provides our answers to these questions based on our development project of the PIMS system. Besides the discussion of our reusable client and server system structure, it presents an interesting approach to design and implement the automatic problem tracking based on a configurable state-based workflow model. Moreover, the paper reports our real practical development and application experience which are valuable for people and enterprises for building their web-based software engineering environment.

Our project experience proves us that the Internet technology is a cost-effective mean to form a web-based enterprise system infrastructure to support the management of concurrent development processes for global software production. The application experience of our PIMS system shows that a web-based enterprise support environment has the distinct advantages for global software production in information sharing, problem management, project management and coordination.

2. Acknowledgement

We would like to thank all members in the R&D group at the Software Engineering Department in the Global Software Technology Division, Fujitsu Network Communications, Inc., San Jose, California. We also want to thank all members in Global Development Engineering Department at Operation System Division, in Fujitsu Limited, and all members in Software Engineering Department in Fujitsu Hokkaido Communication Systems Limited.

3. References

[1] Richard Bentley, Thilo Horstmann, Klaas Sikkel, Jonathan Trevor, "Supporting Collaborative Information Sharing with the WWW: The BSCW Shared Workspace System", Proceedings of Fourth International World Wide Web Conference, World Wide Web Journal, O’Reilly & Associates, Inc. 1995.

[2] Sergio Bandinelli, Elisabetta Di Nitto, and Alfonso Fuggetta, "Supporting Cooperation in the SPADE-1 Environment", IEEE Transactions on Software Engineering 12(1996) 841-865.

[3] John R. Callahan, Reshma R. Khatsuriya, and Randy Hefner, "Web-Based Issue Tracking For Large Software Projects", IEEE Internet Computing, 5(1998) 25-33.

[4] John C. Grundy and Mark D. Apperley, A Decentralized Architecture For Software Process Modeling And Enactment, IEEE Internet Computing, 5(1998).

[5] Jerry Z. Gao, Cris Chen, Y. Toyoshima, and David K. Leung, "Developing An Integrated Testing Environment Using The World Wide Web Technology", Proceedings of COMPSAC’97, pp. 594-601, IEEE Computer Society Press, 1997.

[6] Jerry Gao, "NMS/Foundation PIMS Function Design Specifications for NMS Foundation", Fujitsu Network Communications, Inc., Technical Document, Version 1.04, 1998.

[7] Jerry Gao, Cris Chen, Y. Toyoshima, and David K. Leung, "Engineering Internet for Global Software Production", IEEE Computer, May in 1999.

[8] Peter H. Feiler and Watts S. Humphrey, "Software Process Development and Enactment: Concepts and Definitions", Proceedings of the Second International Conference on the Software Process – Continuous Software Process Improvement, Berlin, Germany, February, 1993, pp. 28-39, IEEE Computer Society Press

[9] Eric Ly, Industry Report: Distributed Java Applets for Project Management on the Web, IEEE Internet Computing, 3(1997) 21-26.

[10] Frank Maurer and Gail Kaiser, "Software Engineering in the Internet Age", IEEE Internet Computing, 5(1998) 22-24.

[11] Raul Medina-Mora, Terry Winograd, RodrigoFlores, Fernando Flores, "The Action Workflow Approach To Workflow Management Technology", Proceedings of CSCW’92, pp. 281-288, November 1992.

[12] Walt Scacchi and John Noll, "Process-Driven Intranets: Life Cycle Support for Process Reengineering," IEEE Internet Computing, 1(1997) 39-50.

[13] Keith D. Swenson, Robin J. Maxwell, Toshikazu Matsumoto, Bahram Saghari, and Kent Irwin, "A business process environment supporting collaborative planning", Collaborative Computing, 1(1994) 15-34.