Debugging cognitive models in ACT-UP

ACT-UP comes with a range of functions designed to make debugging easy.

To boot, you can use the debugging mechanisms of your Lisp compiler. This may include very useful function to step through your model procedures and inspect variables.

ACT-UP provides functions to inspect the symbolic and subsymbolic properties of chunks and procedures (1), macros and functions to enable logging and selective logging of what happens during the execution of the model (2).

1. Inspecting a model

Declarative memory

Often, subsymbolic activation and the symbolic contents of chunks during model execution give valuable cues to the problems with a model.

To print a list of the chunks in the current model, write

(show-chunks '(:chunk-type person)
This will print all chunks of type `person'. Of course, you may use any other slot-value combinations to reduce the number of chunks output (see `retrieve-chunk' for further examples).

A single chunk may be printed with the `pc' and `pc*' functions. You may sometimes want to combine it with an expression to identify a specific chunk:

(pc* 'john-13)
(pc* (best-chunk (model-chunks)))

The first expression will print the chunk named `john-13', while the second one will identify the chunk with the highest (base-level) activation (plus noise) in the current model and print it.

Activation calculation is sometimes complicated. The following expressions will describe the activation of a chunk in terms of its base-level activation, partial matching, spreading activation and noise terms:

(explain-activation 'john-13)
(explain-activation    (best-chunk (filter-chunks (model-chunks)                       '(:location park)))     '(lawyer park))
The first expression will explain the activation of a chunk named `john-13', while the second one will identify all of the chunks in the current model with a `location' slot set to `park', select the one with the highest activation, and print its activation. The activation includes spreading activation from two sources: the chunks `lawyer', and `park'.

Procedural memory

Use the `show-utilities' function to print a list of all procedures (manually defined and compiled) and their respective utilities:
(show-utilities)

Models

To print a list of architectural parameters, use this expression:

(show-parameters)

When working with multiple models in parallel, it may be useful to give them meaningful names when creating them:

(make-model :name 'player-A)
The function `set-current-model' will permanently switch the current model to a different one.

2. Logging

Log output for all or part of your model

(debug-detail &body body) is the most important macro for logging. It executes all the forms in BODY, while printing detailed information about the goings-on inside the cognitive architecture.

You may use this macro when calling your model, such as

(debug-detail (run-experiment))
or also inside the model in particularly interesting situations: (debug-detail (retrieve-chunk '(:chunk-type person)))

Logging: warnings only

In order to to enable general logging of important messages, you can set the *debug* variable:

(setq *debug* *warning*)
Possible values for this option are *critical*, *warning*, *informational*, *all*.

Selective logging

ACT-UP allows you to keep a log without printing all of it. This can be a time-saver.

You may use `debug-grep' to extract only those messages with certain content:

(debug-grep "retrieval")
(debug-grep "person")

As you can see, you may select for any text in the output, for example in order to watch specfic chunks or procedures.

Sometimes it is useful to inspect the log output only in cases when the model fails what it is supposed to do. For example, the model may fail to execute an experimental task only in one out of 1000 cases. In such situations, you may have ACT-UP keep its log output for each model run, but not print it:

Execute your model or parts thereof with an expression like

(debug-detail* (run-experiment)
This will save the log output into an internal buffer.

Once you decide to print the log output, evaluate

(debug-log)
If you'd like to discard the previous log output (and you should do that), evaluate
(debug-clear)

Some more examples...

Of course, all of these functions may be combined with any of the ACT-UP functions to determine what is going wrong. For instance, many models require subsymbolic parameters to be in certain ranges to work.

Let's assume that our model depends on the cue-based retrieval of some chunks. Some of these chunks are created at run-time, others exist as "knowledge" before the model is run. In this case (taken from a real-world debugging situation), only the "knowledge" chunks are being retrieved. We have found out about this problem through careful analysis of the model at run time.

To figure out why this retrieval fails, we first print a list of chunks:

(show-chunks)
(CHUNK-44181 CHUNK-44173 CHUNK-44165 CHUNK-44157 CHUNK-44141 CHUNK-44125
 CHUNK-44117 CHUNK-44109 CHUNK-44093 CHUNK-44069 CHUNK-44061 CHUNK-44053
 CHUNK-44037 CHUNK-44021 CHUNK-44005 CHUNK-43973 CHUNK-43957 CHUNK-43949
 CHUNK-43941 CHUNK-43933 CHUNK-43917 CHUNK-43901 CHUNK-43893 CHUNK-43869
 CHUNK-43861 ... CHUNK-42981 CONCEPT45 CONCEPT44 CONCEPT43
 CONCEPT42 CONCEPT41 CONCEPT40 CONCEPT39 CONCEPT38 CONCEPT37 CONCEPT36
 CONCEPT35 ... CONCEPT5 CONCEPT4 CONCEPT3 CONCEPT2 CONCEPT1)
The CONCEPT... chunks represent existing knowledge (which is successfully retrieved), while the CHUNK-... chunks are generated during the model run. Let's inspect one of these:
(pc 'CHUNK-43861)
CHUNK-43861
             CHUNK-TYPE: DRAWING 
             COMPONENT1: CONCEPT22 
             COMPONENT2: CONCEPT30 
             COMPONENT3: CONCEPT41 
                CONCEPT: CONCEPT32 

The model attempts to retrieve this chunk using the three components (concepts 22,30,41) as cues. As we know that this retrieval yields the wrong result, let's print a log for it. The following command will run a retrieval for a `drawing' chunk, with just the right set of cues. It will pretty-print the resulting chunk, and most importantly, it will also print some detailed debug output during the retrieval:
(pc (debug-detail (retrieve-chunk '(:chunk-type drawing) '(CONCEPT22 CONCEPT30 CONCEPT41))))

Retrieval duration: 7.395787650346699d-4
retrieved CONCEPT4 out of 130 matching chunks.
  time: 27391.07  CONCEPT4 base-level: -.44  (8000 pres) pm:   spreading: 7.8729196
     (CONCEPT22: Sji: 5.65  CONCEPT30: Sji: 7.35  CONCEPT41: Sji: 10.62 ) -  tr.noise: -0.2207639 
  time: 27391.07  CHUNK-44181 base-level: -2.85  (1 pres) pm:   spreading: 2.302986
     (CONCEPT22: Rji: 6.91  CONCEPT30: Rji: .00  CONCEPT41: Rji: .00 ) partial match: 0.0  tr.noise: -0.08859934 
  time: 27391.07  CHUNK-44173 base-level: -3.09  (1 pres) pm:   spreading: 0.0
     (CONCEPT22: Rji: .00  CONCEPT30: Rji: .00  CONCEPT41: Rji: .00 ) partial match: 0.0  tr.noise: -0.7761554 
  ...
  time: 27391.07  CHUNK-43861 base-level: -4.46  (1 pres) pm:   spreading: 7.5304894
     (CONCEPT22: Rji: 6.91  CONCEPT30: Rji: 7.29  CONCEPT41: Rji: 8.39 ) partial match: 0.0  tr.noise: 0.47288537 
  ...
Interesting - CONCEPT4 beats out our target concept because of spreading activation. The output here is ordered by total chunk activation; our targeted chunk comes after a number of other chunks.

It seems that spreading activation from chunks established during the model isn't strong enough compared to the Sji values set initially for the "prior knowledge". Let's try to increase the `*maximum-associative-strength*' parameter, and run the retrieval again.

(setq *maximum-associative-strength* 20)
(pc (debug-detail (retrieve-chunk '(:chunk-type drawing) '(CONCEPT22 CONCEPT30 CONCEPT41))))

Retrieval duration: 1.6177249734895294d-6
retrieved CHUNK-43861 out of 130 matching chunks.
  time: 27391.07  CHUNK-43861 base-level: -4.46  (1 pres) pm:   spreading: 17.530489
     (CONCEPT22: Rji: 16.91  CONCEPT30: Rji: 17.29  CONCEPT41: Rji: 18.39 ) -  tr.noise: 0.26590148 
  time: 27391.07  CHUNK-44181 base-level: -2.85  (1 pres) pm:   spreading: 5.6363196
     (CONCEPT22: Rji: 16.91  CONCEPT30: Rji: .00  CONCEPT41: Rji: .00 ) partial match: 0.0  tr.noise: -0.17124294 
  ...
Voilá: This did the trick. Of course, we may have to consider a range of other issues, such as cognitive plausibility, when setting certain parameters. Still, this is a good start.