This thesis aims to contribute to a theoretical understanding of the determinants and outcomes of the use of explanations provided in knowledge-based systems. It is important to deepen our understanding of explanation use, because explanations from knowledge-based systems are thought to be valuable by researchers, systems developers, and users. Moreover, considerable effort can be expended in the design and development of explanation facilities. Nevertheless, no accepted and well-tested theory could be found to account for the use of explanations. Previous empirical studies have resulted in some contradictory findings.
The theoretical model proposed in this study is taken from cognitive learning theory, a cognitive effort perspective applied to the use of knowledge-based systems, and Toulmin's model of argumentation. The determinants of the frequency of explanation use investigated were (i) the level of expertise of the user, (ii) the goal of the user (learning or problem solving), and (iii) the nature of the problem solving situation (collaborative or non-collaborative). Collaborative situations are those in which both user and system can contribute knowledge to the problem solving process. The outcomes of explanation use investigated were (i) learning by the user, (ii) problem solving performance, and (iii) users' confidence in the system.
Two experiments were performed in laboratory settings. In the first experiment, advanced-level students in a taxation subject used an operational system for capital gains tax calculations. This experiment focussed on the effect of the user's goal, whether learning or problem solving. In the second experiment, members of the general public used a knowledge-based system designed to assist with personal financial planning. This experiment focussed on the effect of the nature of the problem solving situation, whether collaborative or non-collaborative. Two versions of the knowledge-based system were used: one with explanations and one without. In both experiments, measures of frequency of explanation use were obtained from traces of program use.
The results of the study are congruent with a cognitive effort perspective. This perspective implies that explanations will be used more when users perceive that the advantage to be gained from the explanations outweighs the cognitive effort in accessing the explanations. As predicted from this perspective, a goal of learning rather than problem solving led to higher frequency of use of explanations. Also as predicted, a requirement for collaborative problem solving led to higher frequency of use of explanations. This latter effect, however, was observed only with users classed as practiced users of explanations on the basis of their performance in familiarisation activities undertaken in preparation for the experiments.
Results offered some support for cognitive learning theory. Increased frequency of use of explanations led to increased problem solving performance under some conditions. In the first study, with non-collaborative problem solving, the use of an Answer Help explanation was significantly associated with improved performance. In the second study, the frequency of use of explanations and Answer Help in total was positively related to problem solving performance. This situation was observed in situations where explanations were available. There was no significant difference in performance between groups with and without explanations. There was some evidence that the positive relationship between explanations and improved performance was more noticeable when problems of a collaborative nature were undertaken.
The expected relationship between increased frequency of use of explanations and an increase in the amount of learning taking place was not observed. The experimental conditions were such that learning could have occurred in ways other than through use of explanations. In this light, results are not inconsistent with cognitive learning theory.
The level of expertise of the user was found to be related to higher problem solving performance. The level of expertise of the user was found also to be related to greater use of explanations in familiarisation activities. Both these relationships were observed only in the first experiment, where the measure used for level of expertise related specifically to the problem domain of the knowledge-based system.
Results did not support the propositions that were based on Touhnin's model of argumentation. Possibly, confidence in machine advice is associated with explanations that can provide the backing for arguments. In this research, however, this relationship was not observed. It appeared, rather, that the degree of confidence expressed in a system was associated with the nature of the system. Confidence ratings were significantly related to the level of expertise of individuals, but the direction of this relationship was different for the two different systems. With the system presented as fully operational, the more expert individuals gave the system a higher confidence rating. With the system presented with disclaimers because it was still under research, the more expert individuals gave the system a lower confidence rating.