Neurithmic Systems has two primary goals.
- Discover the information representation and algorithm(s), e.g., for learning, memory, recognition, inference, recall, used in the brain, in particular neocortex, but also hippocampus and other brain regions. An essential piece of this is that the brain is a hierarchy of numerous semi-autonomous coding fields, allowing explicit representation of the recursive, compositional (part-whole) structure of natural object/events.
- Develop extremely efficient, scalable models, based on these representations and algorithms, which do on-line (single/few-trial) learning of spatial and sequential patterns and probabilistic inference/reasoning over the learned patterns, e.g., similarity-based (i.e., approximate nearest-neighbor) retrieval, classification, prediction. Following on prior ONR and DARPA supported research, we are developing our model, Sparsey®, for video event recognition and understanding, as well as for other modalities and multi-modal inputs, e.g., visual + auditory + text.
Our work is founded on the idea that in the brain, especially cortex, information is represented in the form of sparse distributed representations (SDR) [a.k.a., sparse distributed codes (SDC)], a particular instantiation of Hebb's "cell assembly" and "phase sequence" concepts. Our SDR format is visible in simulation below, most clealy in each of the coding fields (hexagons) in the third panel.
Note: SDR is different than (and completely compatible with) "sparse coding" (Olshausen & Field). Sparse coding is fundamentally about the nature of the input space, whereas SDR is fundamentally about the nature of the representation space.
A hierarchical memory trace (engram)—in the form of a Hebbian phase sequence involving hundreds of cell assembly activations across two internal levels (analogs of V1 and V2)—of a visual sequence of a human bending action. Left panel shows binary pixels possibly analogous to very simplified LGN input to cortex. Next panel shows plan of array of V1 macrocolumns ("macs"), specifically, abstract versions of their L2/3 pyramidal pools, receiving input from LGN. Next panel shows array of V2 macs receiving input from the V1 macs. Last panel shows corresponding 3D view and showing some of the bottom-up (U, blue) and horizontal (H, green) connections that would be active in mediating this overall memory trace. The cyan patch on the input surface shows the union of the receptive fields (RFs) of the active V2 macs. We son't show the V1 mac RFs, but they are much smaller, e.g., 40-50 pixels. In general, the RFs of neaby macs overlap (see Figs. 2 and 3 here). Go Fullscreen to see detail. Or, see this page for more detailed discussion.
Highlights
- Invited Talk: The Coming KB Paradigm Shift: Representing Knowledge with Sparse Embeddings. Presented at “Intelligent Systems with Real-Time Learning, Knowledge Bases and Information Retrieval" Army Science Planning and Strategy Meeting (ASPSM). Hosts: Doug Summers-Stay, Brian Sadler. UT Austin. Feb 15-16, 2019.
- "A Hebbian cell assembly is formed at full strength on a single trial". (On Medium) Points out a crucial property of Hebbian cell assemblies that is not often discussed but has BIG implications.
- Submitted abstract (COSYNE 2019) "Cell assemblies encode full likelihood distributions and communicate via atemporal first-spike codes." (Rejected, here is the review)
- Blog on Sparsey at mind-builder.
- Accepted abstract to NIPS Continual Learning Wkshp 2018: "Sparsey, a memory-centric model of on-line, fixed-time, unsupervised continual learning." (also available from NIPS site)
- JAVA APP showing how Sparsey preserves similarity permitting both learning new items and retrieving the approximate most simlilar item in fixed time. A far simpler and more powerful alternative to locality sensitive hashing! Applies equally well to spatial patterns or sequences. Figure below illustrates simlarity preservation (see page).
- Accepted (as poster): First Spike Combinatorial Coding: The Key to Brain’s Computational Efficiency. (Abstract) Cognitive Computing 2018
- 8-18 Talk: Sparse distributed representation, hierarchy, critical periods, metaplasticity: the keys to lifelong fixed-time learning and best-match retrieval. at Biological Distributed Algorithms 2018 (Abstract)
- Paper: "Superposed Episodic and Semantic Memory via Sparse Distributed Representation" (arxiv)
- Rod Rinkus Extended Research Statement
- Sparsey already likely at least as fast without machine parallelism (MP) as gradient-baesd methods are with MP, can easily be sped up by 100-1,000x via simple, existing, non-GPU-based MP, e.g., SIMD, ASIC:
- Sumon Dey and Paul Franzon (2016) "Design and ASIC acceleration of cortical algorithm for text recognition" Proc. IEEE System-on-Chip Conf.
- Schabel et al. (2016) "Processor-in-memory support for artificial neural networks"
Sparse Distributed Representation (SDR) enables a revolution in probabilistic computing
SDR provides massive algorithmic speedup for both learning and approximate best-match retrieval of spatial or spatiotemporal patterns. In fact, Sparsey (formerly, TEMECOR), invented by Dr. Gerard Rinkus in the early 90's, both stores (learns) and retrieves the approximate best-matching stored sequence in fixed time for the life of the system. This was demonstrated in Dr. Rinkus's 1996 Thesis and described in his 2004 and 2006 talks at the Redwood Neuroscience Institute, amongst other places. To date, no other published information processing method achieves this level of performance! Sparsey, implements what computational scientists have long been seeking: computing directly with probability distributions, and moreover, updating from one probability distribution to the next in fixed time, i.e., time that does not increase as the number of hypotheses stored in (represented by) the distribution increases.
The magic of SDR is precisely this: any single active SDR code simultaneously functions not only as the single item (i.e., feature, concept, event) that it represents exactly, but also as the complete probability distribution over all items stored in the database (e.g., discussed here and here). With respect to the model animation shown here, each macrocolumn constitutes an independent database. Because SDR codes are fundamentally distributed entities, i.e., in our case, sets of co-active binary units chosen from some much larger pool (e.g., the pool of L2/3 pyramidals of a cortical macrocolumn), whenever one specific SDR code is active, all other stored SDR codes are also partially physically active in proportion to how many units they share with the fully active code. And, because these shared units are physically active (in neural terms, spiking), all these partially active codes also influence the next state of the computation in downstream fields as well as in the source (field via recurrent pathways). But, the next state of the computation will just be another of the stored SDR codes [or, if learning is allowed, a possibly new code that may contain portions (subsets) of previously stored codes] that becomes active, which will in general have some other pattern of overlaps with all of the stored codes, and thus embody some other probability distribution over the items.
Virtually all graphical probabilistic models to date, e.g., dynamic Bayes nets, HMMs, use localist representations. In addition, influential cortically-inspired recognition models such as Neocognitron and HMAX also use localist representations. This page shows what an SDR-based model of the cortical visual hierarchy would look like. Also see the NICE Workshop and CNS 2013 links at left.
Memory trace of 8-frame 32x32 natural event snippet playing out in a 6-level Sparsey model with 108 macs (proposed analogs of cortical macrocolumns).
This movie shows a memory trace that occurs during an event recognition test trial, when this 6-level model (with 108 macs) is presented with an identical instance to one of the the 30 training snippets. A small fraction of the U, H, and D signals underlying the trace is shown. See this page for more details. What's really happening here is that the Code Selection Algorithm (CSA) [See Rinkus (2014) for description] runs in every mac [having sufficient bottom-up (U) input to be activated] at every level and on every frame. The CSA combines the U, H, and D signals arriving at the mac, computes the overall spatiotemporal familarity of the spatiotemporal moment represented by that total input, and in so doing effectively retrieves (activates the SDR code of) the spatiotemporally closest-matching stored moment in the mac.