Hierarchical Sequence Compression
via Progressive Persistence

  1. The animation shows a hierarchical memory trace forming in response to a complex sequence (i.e., a sequence in which states can occur multiple times, e.g., natural language). Two different views of the formation process are shown side-by-side. The main point here is to illustrate the concept of progressive persistence in which codes at higher levels are mandated to have longer activation durations. This causes a code at level J to associate with a sequence of codes at level J-1, i.e., compression. This principle can be applied across as many hierarchical levels as desired, i.e., recursively nested sequence compression.
  2. The view on the left, the "processing diagram", represents codes in localist fashion: i.e., each code that becomes active is represented by a single unit (a green box). This view shows how many codes become active at each level, how long they last, and which codes synaptically associate both within level (chaining) and across level (chunking).
  3. The view on the right, the "network diagram", shows the sparse distributed codes corresponding to each green box on the left. Amongst other things, this view makes it clear that, at each level, the codes that become successively active in fact all transpire in the same representational area, e.g., macrocolumn. Thus, this view makes the simplification that only one macrocolumn is shown at each level.
  4. Note: This animation contains a lot of text bubbles. In order to read/understand these bubbles, the viewer must use the slider/buttons to manually step through it.

Processing Diagram

  • Each green box represents a single sparse distributed code active at a particular level. The nomenclature is that the Greek capital "Delta" represents a code, the leading superscript represents the hierarchical level (e.g., the model's first internal level, L2, corresponds to cortical V1, etc.)., the trailing superscript is the index of the training (or test) sequence in response to which this trace is forming (in this case, this is not really needed), and the subscript represents the level-specific ordinal code index from beginning of the sequence.
  • The animated arrows indicate the flow of signals via the bottom-up (U), top-down (D), and horizontal (H) synaptic matrices.

Network Diagram

  • Each set of co-active cells at a given layer is a single distributed code and corresponds to one box in the processing diagram at left.
  • Note that for levels, V2 and higher, the first code that becomes active in the level has two active units, not one. This is part of a novel mechanism, called overcoding-and-paring (OP) (Patent 8983884), for dealing with the problem of having to learn large numbers of sequences that start with same item or sub-sequence of items, e.g. "DOG", "DOOR", "DOORMAT", etc. OP is further explained in the animation's captions and a manuscript describing it is in revision.