osculari

Exploring artificial neural networks with psychophysical experiments.

Overview

The osculari package provides an easy interface for different techniques to explore and interpret the internal presentation of deep neural networks.

• Supporting the following pretrained models:
  • All classification and segmentation networks from PyTorch's official website.
  • All OpenAI CLIP language-vision models.
  • All Taskonomy networks.
• Managing convolution and transformer architectures.
• Allowing to readout the network at any given depth.
• Training a linear classifier on top of the extract features from any network/layer.
• Experimenting with 2AFC and 4AFC paradigms.

At a granular level, Kornia is a library that consists of the following components:

Module	Description
osculari	Open source library to explore and interpret pretrained deep neural networks.
osculari.datasets	A module to create datasets and dataloaders to train and test linear probes.
osculari.models	A module to readout pretrained networks and add linear layers on top of them.
osculari.paradigms	A module to implement psychophysical paradigms to experiment with deep networks.

Installation

From pip

pip install osculari

Alternative installation options

From source with symbolic links:

pip install -e .

From source using pip:

pip install git+https://github.com/ArashAkbarinia/osculari

Examples

Please check the example page of our documentation with many notebooks that can also be executed on Google Colab.

Quick start

Pretrained features

Let's create a linear classifier on top of the extracted features from a pretrained network to perform a binary classification task (i.e., 2AFC – two-alternative-force-choice). This is easily achieved by calling the paradigm_2afc_merge_concatenate from the osculari.models module.

architecture = 'resnet50'        # network's architecture
weights = 'resnet50'             # the pretrained weights
img_size = 224                   # network's input size
layer = 'block0'                 # the readout layer
readout_kwargs = {
    'architecture': architecture, 
    'weights': weights,
    'layers': layer,
    'img_size': img_size,
}
net_2afc = osculari.models.paradigm_2afc_merge_concatenate(**readout_kwargs)

Datasets

The osculari.datasets module provides datasets that are generated randomly on the fly with flexible properties that can be dynamically changed based on the experiment of interest. For instance, by passing a appearance_fun to the ShapeAppearanceDataset class, we can dynamically merge foreground masks with background images to generate stimuli of interest.

def appearance_fun(foregrounds, backgrounds):
    # implementing the required appearance (colour, texture, etc.) on foreground and merging
    # to background.
    return merged_imgs, ground_truth

num_samples = 1000               # the number of random samples generated in the dataset
num_imgs = net_2afc.input_nodes  # the number of images in each sample
background = 128                 # the background type
dataset = osculari.datasets.geometrical_shapes.ShapeAppearanceDataset(
    num_samples, num_imgs, img_size, background, appearance_fun,
    unique_bg=True, transform=net_2afc.preprocess_transform()
)

Linear probe

The osculari.paradigms module implements a set of psychophysical paradigms. The train_linear_probe function trains the network on a dataset following the paradigm passed to the function.

# experiment-dependent function to train on an epoch of data
epoch_fun = osculari.paradigms.forced_choice.epoch_loop
# calling the generic train_linear_probe function
training_log = osculari.paradigms.paradigm_utils.train_linear_probe(
    net_2afc, dataset, epoch_fun, './osculari_test/'
)

Psychophysical experiment

The osculari.paradigms module also implements a set of psychophysical experiments similar to the experiments conducted with human participants. In this example, we use the staircase function to gradually measure the network's sensitivity.

# experiment-dependent function to test an epoch of data
test_epoch_fun = osculari.paradigms.forced_choice.test_dataset
# the test dataset implementing desired stimuli.
class TestDataset(TorchDataset):
    def __getitem__(self, idx):
        return stimuli

test_log = osculari.paradigms.staircase(
    net_2afc, test_epoch_fun, TestDataset(), low_val=0, high_val=1
)

Contribution

We welcome all contributions to the project that extend or improve code and/or documentation! Please read the CONTRIBUTING page and follow the Code of Conduct.