Extra Encoders (Image, Video, sound, ...)

[breznak]

For some experiments, I'd like to setup encoders/extra/{vision,audio,...}/
with specialized encoders for multiple modalities.

There existed special repos as

• https://github.com/htm-community/nupic.audio
• https://github.com/numenta/nupic.vision

But basically these were

• Encoder
• normal HTM (+ problems to set up 2 repos)
• the experiment

I think it'd help our community if we provided all-around baseline.
What do you think?

EDIT:

Domains:

• NLP
  • SimHash SimHash Document Encoder in C++ with Python bindings #603
  • Universal Sentence Encoder Add new NLP encoder Google's Universal Sentence Encoder #681
  • cortical.io 's encoder
• Vision
  • dummy B/W image coding for MNIST Python MNIST examples: data not available out of the box #514
  • biological retina encoder eye Add new Vision encoder EYE  #682
  • some standard atate of the art image encoders for NNs
• Audio
  • see nupic.audio
  • MIDI/MP3 sound encoder
  • cochlea biological encoder https://github.com/mrkrd/cochlea
• Touch/haptic
  • grid cells encoder Grid Cell Encoder #353
  • coordinate encoder: Coordinate Encoder - WIP #304

[ctrl-z-9000-times]

Maybe also a Grid Cell encoder, which converts a Cartesian coordinate into a grid-cell like encoding.

[ctrl-z-9000-times]

The original nupic had a "time & date" encoder. It's not critical, but might be nice to have. I think the motivation for this encoder is that many anomalies correlate with a time of day or day of the week.

HTM school video for date/time encoder:
https://discourse.numenta.org/t/htm-school-episode-6-datetime-encoding/892

[ctrl-z-9000-times]

I wrote a vision encoder which may be added to this repo. It needs some cleanup & modification before it will be ready, and I don't know when I will get around to it. Currently it is a research prototype.

It is written in python, and uses opencv. The Opencv library has functions which do log-polar transforms, and Parvo/Magno-cellular transforms. Open-CV works well and it looks pretty well researched. I wrote an encoder which converts the processed images into sparse-distributed-representations. My encoders are less well researched.

Here are some images showing the log-polar & parvo/magno-cellular transforms.

Field of view:

Parvo-cellular:

Magno-cellular:

Here are some statistics about the SDR encoded output of eye, after viewing the dataset which contains the above image.

Parvo SDR( 250 250 1 )
    Sparsity Min/Mean/Std/Max 0.197344 / 0.200787 / 0.00118844 / 0.205056
    Activation Frequency Min/Mean/Std/Max 0 / 0.20079 / 0.29597 / 1
    Entropy 0.458866
    Overlap Min/Mean/Std/Max 0.355327 / 0.833063 / 0.0761267 / 0.956605
Magno SDR( 250 250 1 )
    Sparsity Min/Mean/Std/Max 0.19864 / 0.203853 / 0.00151922 / 0.208624
    Activation Frequency Min/Mean/Std/Max 0 / 0.203854 / 0.182725 / 0.595165
    Entropy 0.76574
    Overlap Min/Mean/Std/Max 0.00212632 / 0.848003 / 0.133877 / 0.981023

[breznak]

This is amazing!!

parvo/magno-cellular transforms.

I'll need to refresh my knowledge, this is a good starter:
https://foundationsofvision.stanford.edu/chapter-5-the-retinal-representation/#visualinformation

I cite:

When cells in the parvocellular layers of a monkey’s lateral geniculate nucleus are destroyed, performance deteriorates on a variety of tasks, such as color discrimination and pattern detection. Since the parvocellular pathway includes more than seventy percent of the retinal ganglion cells, perhaps this result is not terribly surprising. When cell bodies in the magnocellular layers are destroyed many visual performances are unaffected.

What conclusion can we draw from these lesion studies? The information carried by the neurons in the magnocellular pathway provide the best information in the low temporal and high spatial frequency components of the image.

= image classification type of tasks

Performance on motion tasks and other tasks that require this information is better when the magnocellular pathway signal is available.

= video processing in vision tasks. motion detection/tracking.

Here are some statistics [...] after viewing the dataset

So you needed to produce several SDRs from the image.

• do you use some saccadic movements (to generate a few centers of focus)
• or is this used on an "animation" (a sequence of related images)?

I am wondering how this kind of biologically plausible encoding would fare on "stupid" classification datasets, like MNIST, CIFAR10 etc.
Or we'd have to extend to real-world tasks: object recognition in video..

I'll be reading more chapters on vision, please share your retina code when you have time, even if it's not ready yet. Thank you

[ctrl-z-9000-times]

https://foundationsofvision.stanford.edu/chapter-5-the-retinal-representation/#visualinformation

That looks like a good source. My model is wrong with regards to many details, including the relative densities of magnocellular and parvocellular neurons.

do you use some saccadic movements (to generate a few centers of focus)

No, this is TODO. Currently I move the eye by a small random amount between each compute cycle. Controlling where the eye looks is an open issue, and it involves action selection and motor control.

please share your retina code

I'm keeping my latest work on the eye encoder here: https://github.com/ctrl-z-9000-times/sdr_algorithms/blob/master/eye.py I don't know when I will have time to work on it further.

[breznak]

My model is wrong with regards to many details, including

yes, according to the literature, these modifications could apply:

• magno-/parvo-cellular neurons in 70:30 proportion ( = this is "Retina")
• force abstraction by 1:10 compression (in retina to optical nerve neuron counts), SP could do that.
• then starts the (large in size) visual cortex (HTM, SP+TM)
• Q:
  • for image classification, use only parvo cells?

move the eye by a small random amount between each compute cycle. [...] involves action selection and motor control.

would the saccadic moves wrongly trigger movement/magno cells? Or would that help to encode "I moved the eye/focus, so the 'move' is caused by the move of sensor, not move of the objects in the scene"?

• I'm considering for initial, trivial use case of 28x28 MNIST images.
  • use just 1 focus point? (problem with bounding, location)
  • use "5" fixed saccades (always visit same coordinates, so this'd be consistent for different images)

eye.py I don't know when I will have time to work on it further.

cool! Would you please make just an initial PR with the eye.py (or other necessarities) when you have time? I'd like to play with it in the next week and I'll try to adapt it to the current state of htm.core. I just ask for this so you author the file, so you'd get the (c) for the lines :) I'll then continue to make modifications to it.

Btw, reviewing that repo of yours, we're pretty much synced, aren't we? ae, CP, SDR...are more or less here 👍 . eye.py is the only missing piece. Or is there something else significant?