The third step was to analyze the results and to aggregate data. As it is of high interest to the regional companies, an overview of the status quo has been drawn. For reasons of space and scope it is not included in this paper but can be found in [21]. By finding interdependencies as well as aligning and judging best practices identified by the participants, we ex- tracted recommendations. Of course, particularities of the com- panies' situations were taken into account. This lead to the con- struction of a framework (see Section IV) that describes the conditions under that a recommendation applies.
IV. FRAMEWORK
Recommendations for a topic as complex and intertwined on various levels as software development require a sophisti- cated categorization. Their full value can only be accessed if it is known how to use them and which prerequisites have to be met. Besides, support on deciding which best practices are most applicable for the own business is advisable. We thus use the framework first described in [20] to classify recommenda- tions.
The level of demand shows how great the organizational change required to adopt a recommendation is. Basic recom- mendations should be adopted by any company. If recommen- dations are not only basic hints but require considerable effort to be implemented, they are considered to be advanced. Even- tually, target states are ideals that cannot be reached unlabored. In fact, they are guidelines on what level of perfection can be reached and require a process of continuous optimization. However, the benefits of an actual implementation will be im- mense in the long run.
It is important to consider the project size. Small-sized projects commonly have a single team that does development and testing. For medium-sized projects these tasks are underta- ken by at least two teams. Large projects can comprise hun- dreds of employees and include general departments that con- tribute to it. If thinking about a recommendation, not just the sole number of employees that participate in it should be taken into account. In fact, the typical size of projects as well as their character should be kept in mind.
Another important determinant is the kind of software de- veloped. Based on contracts, individual software is developed for a single customer. Usually, there is close contact to the principal. Standard or mass market software often is developed over a long period of time. This makes regression testing im- portant. Many recommendations can be applied to both kinds of software.
Similarly, the number of releases of a software product has to be taken into consideration. It is differentiated between one, several and regular releases whereas regular means that there will be releases for some month or years.
The fifth determinant distinguishes between the phases (or stages) of testing. It is divided into the phases of component test, integration test, system test and acceptance test that also can be found in the literature [35].
As represented in Figure 1, the level of demand and the phase of development are used to set up a matrix. A tick indi- cates that a recommendation is meant to be beneficial for the depicted phase and level. Ticks might be shown in brackets which indicate that benefits will be observable but might be less pronounced than for other phases and levels. The three other determinants are shown as bars. A shade of (dark) gray means that a recommendation applies under the specified con- ditions. Fading indicates that adoption of the recommendation should be considered if the depicted determinant is met. Rec- ommendations still require more detail so that companies can judge them. However, the framework can be used to get a quick overview of the main prerequisites for it.
Please consider Figure 1 for clarification:
The recommendation requires advanced effort. It is possible to be extended in order to mark a target state in which beneficial effects will be much stronger.
Implementing it especially aids integration and system testing. Positive effects are also likely to be observed for acceptance testing.
The recommendation is meant to be adopted for at least medium-sized projects and it aims at indivi- dually developed software.
It aims at individually developed software. Theo- retically, there could be a fading of the gray shade into the box for mass market software. This would mean that it would also benefit while the main fo- cus was individual software.
For full effect, there should be a greater number or regular releases of the software developed.
V. TECHNICAL RECOMMENDATIONS
The following sections present five recommendations for the optimization of technical aspects of software testing. Their order reflects the implementation complexity.
A. State-of-the-art Development Environment
The first recommendation is pretty straightforward. We en- courage using the latest development environments available, particularly integrated development environments (IDE) that are customizable and support plug-ins. They offer magnificent opportunities to increase the quality of the developed software. Using the latest IDEs is especially appealing since development software is used anyway and many of these products or at least plug-ins for them are free.
Admittedly, using an IDE is not only about testing. But the support it offers significantly helps to increase development quality. If the developer is aided in his routine work, testers do not have to struggle with defects in programs that originated in unthoughtfulness. Testers can then concentrate on finding ac- tual bugs e.g. in algorithms. Consequently, this recommenda- tion is a testing best practice even though parts of it do not di- rectly deal with testing; they have a noticeable indirect impact.
Unlike expectation, companies do not necessarily use state- of-the-art IDEs. It is common to do so for individual develop- ers in small enterprises. However, once the choice of develop- ment tools is not solely based on developers' discretion but there are general guidelines or even mandatory directives, tools that do not offer as much functionality as would be possible are used. This is especially true for situations in which developer PCs are centrally set-up by IT organization staff rather than by the developers themselves. Changing development tools could not be possible since tools for cooperative work or versioning, or software to access corporate-wide storage systems or re- source pools might not be exchangeable.
Some of the participants drew a picture of the way their de- velopment is supported by the tools used that reminded us of the 1980s. There was no kind of syntax highlighting and no built-in supportive functionality to aid the developer with cod- ing. There was no direct access to the programming languages or library documentation; developers would look it up on the Internet or use books even for the simplest questions. And, probably worst, there was no testing and debugging support. Debugging was done by putting print()-statements into the code that almost arbitrarily supplied the developers with frag- ments (or rather shreds) of information.
Seeing how much more productive developers using mod- ern IDEs are and how much these tools aid them in achieving high quality software, we strongly recommend using up-to-date development environments and the functionality that comes with them. This general recommendation is suitable for any company. It is extremely helpful for component testing (see Figure 2).