5. Querying Experts

One way of determining a knowledge structure for a given domain is to query experts on prerequisite relationships (Falmagne et al., 1990; Dowling 1991, Koppen 1993, Kaluscha 1994).

In the first step, the domain of knowledge to be investigated has to be defined and the relevant items within the domain have to be identified, usually with the help of experts in the field.

In a second step, the relationships between these items have to be determined. Please note that the expertise needed for the first and the second step may be different - defining the domain is a task for experts from the field, while determining the relationships between items usually requires additional psychological and pedagogical knowledge as well as experience in teaching the items (for an example see Held, Schrepp and Fries, 1995). For instance, if the domain chosen was elementary mathematics, mathematicians could be a help for accomplishing the first step, while experienced mathematics teachers should be preferred as experts in the second step.

A computer based querying procedure has been implemented by Kaluscha (1994) based on the algorithm due to Dowling (1993). Koch and Quante have improved the procedure and provided an enriched user interface for the experts (Dowling, Koch and Quante, 1996).

As the experts usually are not able to give the complete list of prerequisite relationships, which is due to the huge number of possible relationships even in small domains, the experts have to judge standard form assertions, e.g.

Imagine a person who does not master the items . Is it then (practically) certain that this person does not master item q ?

The items $p_1,\,\ldots,\,p_k$ are called the premise (set), and the item q is called consequence. The index k is called the size of the premise. For judged assertions, the index k takes values between 1 and N-1 where N := |Q| is the total number of items considered.

The experts either accepts or rejects these assertions. By the rules of mathematical logic, inferences can be drawn from previous judgements, and thus the number of standard form assertions to be presented to the experts can be considerably reduced (Falmagne et al., 1990; Koppen, 1993; Dowling, 1993). If a suitable querying strategy is used which is optimized to present assertions for judgement which allow for a large number of inferences the necessary number of judgements is again considerably reduced (Dowling & Kaluscha, 1995). The querying procedure finishes when all standard form assertions have been judged by the expert or by inference from previous answers of the expert.

The first dynamic case, to remove an obsolete item i_obs from a knowledge structure, is a rather simple task if a knowledge space $\mbox{$\cal K$}$ is chosen as representation of the knowledge structure:

Remove from all knowledge states $K \in \mbox{$\cal K$}$ the obsolete item i_obs (and eliminate duplicate knowledge states).

This gives immediately a new knowledge space $\hat{\mbox{$\cal K$}}$ , from which the other representations can be derived (Dowling, 1993; Koppen, 1993). Please note that the algorithm mentioned above preserves the three properties P1, P2 and P3 of a knowledge space, i.e. $\hat{\mbox{$\cal K$}}$ is indeed a knowledge space on the domain $Q \setminus \{i_{obs}\}$ .

In the second dynamic case, adding a new item i_new is a more complex problem. In order to preserve property P3 (closedness under union) the introduction of the new item can result in two times as much knowledge states as before.

On the one hand, if we choose a simple approach and assume there are no relationships between the new item i_new and the other items $i_k \in Q$ , for each knowledge state $K \in \mbox{$\cal K$}$ a knowledge state $\hat{K} := K \cup \{i_{new}\}$ has to be added to obtain the new knowledge space $\hat{\mbox{$\cal K$}}$ . So we end up with twice as much knowledge states.

On the other hand, it wouldn't be feasible to start the querying procedure all over again, as the experts are rare and their time is too valuable.

So we choose another approach. The querying procedure virtually starts from scratch, but only those questions, i.e. standard form assertions, have to be presented to the expert for judgement which contain the new item i_new in the premise or consequence - the answers to the other questions are already known from the previous run of the querying procedure on the domain Q. Furthermore, not all of the standard form assertions containing the new item i_new have to be judged by the expert, as judgements can be inferred from the expert's answers to previous questions containing i_new.

Thus, already a partial run of the querying procedure yields the new knowledge space $\hat{\mbox{$\cal K$}}$ on the domain $Q \cup \{i_{new}\}$ .

If several items have become obsolete, or several items have to be newly added, the procedures above can be performed for each obsolete or new item respectively in sequence.

Another possibility for adding several new items at one time could be to first determine the knowledge structure within these new items, and then use techniques for merging two or more existing knowledge structures into a new one which are under development (Falmagne & Doignon, to appear).

Next: 6. Analysis Of Demands Up: Index Previous: 4. Analysis Of Didactics

ODL-Team
Wed Jan 12 2000