Higher Order Associative Memories and their Optical Implementations

An associative memory is a system which stores a mapping of N-dimensional input vectors to $N_0$-dimensional output vectors

The system should be capable of implementing all possible mappings.It’s effectiveness depends on

• –

  Capacity

• –

  Learning

• –

  Generalization

For binary outputs, capacity is given by $C\le \frac{D{\log }_{2}K}{N_0}$ ,where D is the number of independent variables and K is the separate values each can assume.

It maps an input vector to +1 or -1 by

This function dichotomizes the set of input vectors (partitions into two space, separated by a hyperplane).

All the dichotomies are possible only when the input vectors are less than N+1, so we get the capacity as : $C=N+1$

Since this capacity is low, we expand the vectors to their $r^{th}$ order to get $L=\left(\genfrac{}{}{0pt}{}{N+r}{r}\right)$ number of independent terms.

Here the number of weights used to describe the mapping is L+1, and so is the capacity of the memory.

There are $2^N$ non redundant terms in a complete polynomial expansion of a binary vector. Which is equal to the total number of possible input vectors. hence, we can say that this memory is capable of implementing all the possible mappings.

The orthogonalization property of the full expansion is interesting because it shows that higher order memories provide a complete framework that takes us from the simplest ”neuron”, the linear discriminant function to the full capacity of a Boolean truth table.

For a large N, the full expansion is too long. Hence, we only take $r^{th}$ order expansions which gives us large enough capacity to learn.

Angle between two expanded vectors is given by

where $\rho$ is $n/N$ and n is the Hamming distance between the original input vector.

The first term is identified with the desired signal and the second term with the noise.Thus we can calculate the signal to noise ratio.

Equating the signal to noise ratio of linear and $r^{th}$ order we get the $r^{th}$ order capacity as

The holographic process consists of recording and reconstruction. In the recording step, the interference between the reference plane wave and the wave originating from object ”A” is recorded on a photographic plate. This plate illuminated with thereference fied, to give a virtual projection of ”A”.

The weight of each interconnection is given by the interference pattern

The volume hologram is similar, but it records the pattern on a three dimensional medium.

To implement the quadratic memory, volume holograms are used to connect the input and output patterns. They are implementations of the weight tensor.

$N^2\to N$  schemes use the weight tensor

This allows for error driven learning where interconnections are developed by an iterative training process.

$N\to N^2$ is the inverse of the one described previously.

Input dependent weights enable quadratic memories in an $M\to N$ scheme

They do not have the extra dimension to directly implement the weight tensor, nevertheless they can be used in a similar way.

The first part of the system is a multichannel correlator, which correlates input vectors to the stored vectors. The correlation functions are then sampled at the slit and squared by the SLM. Second hologram produces the weighted of vectors, which are Fourier transformed to obtain output vectors.

One can incorporate shift invariance by lengthening the input SLM and removing slits.