Empirical evaluation of software systems in actual use is critical in software engineering. Prototyping, beta testing, and usability testing are widely employed to refine system requirements, to detect anomalous or unexpected system and user behavior, and to evaluate software usefulness and usability. The World Wide Web enables cheap, rapid, and large-scale distribution of software for evaluation purposes. However, current techniques for collecting usage data have not kept pace with the opportunities presented by Web-based deployment. This paper presents an agent-based approach and prototype system that makes large-scale collection of usage data over the Internet a practical possibility.

Empirical evaluation of software systems in actual usage situations is critical in software engineering. Prototyping, beta testing, and usability testing are widely used to refine system requirements, detect anomalous or unexpected system and user behavior, and to evaluate software usefulness and usability. The World Wide Web enables cheap, rapid, and large-scale distribution of software for evaluation purposes. However, current techniques for collecting usage data have not kept pace with the opportunities presented by Web-based deployment. This paper presents an approach and prototype system that makes large-scale collection of usage data over the Internet a practical possibility. A general framework for comparing software monitoring systems is presented and used to compare the proposed approach to existing techniques.

Software architectures evolve as the result of numerous, interrelated design decisions. Existing approaches to analysis, however, tend to provide feedback only after numerous design decisions have been made. As a result, theydo not directly support the evolutionary nature of the architecture design process or the software architect's decision-making process. In this paper we present an approach to architectural analysis stemming from previous work in domain oriented design environments that is based on critics and criticism control mechanisms. This approach more closely supports evolution and the needs of architects by providing feedback as individual design decisions are beingconsidered. We discuss the theoretical motivations for the critic-based approach, the implementation and management of critics, support for diverse and extensible groups of critics, and the combined use of critics and existing analysis techniques.

Involving end users in the development of interactive systems increases the likelihood those systems will be useful and usable. User involvement, however, is both time and resource intensive. Intemet-and World-Wide-Web-based software release models have magnified these problems. At the same time, these practices have begun to blur the distinction between development and use and, in so doing, have provided developers with unprecedented, and currently underutilized, opportunities for increasing user involvement. We propose an approach — based on expectation-driven event monitoring and expectation agents — that leverages these opportunities to support ongoing user involvement in the software development process.