import functions for openEphys and updated functions for intan and op…

…titrack

calc_CellMetrics/intan2buzcode.m → calc_CellMetrics/loadIntanAnalog.m

@@ -1,11 +1,11 @@
-function data_out = intan2buzcode(varargin)
+function data_out = loadIntanAnalog(varargin)
 % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % 
-% Loads analog intan files into buzcode data containers
+% Loads analog intan files into CellExplorer data containers
 % Part of CellExplorer
 %
 % Example calls:
-% intan2buzcode('session',session,'dataName','temperature','data_source_type','adc','container_type','timeseries','processing','thermistor') % Loads temperature data from a thermistor
-% wheel = intan2buzcode('session',session,'dataName','WheelPosition','data_source_type','adc','container_type','behavior','processing','wheel_position','downsample_samples',200); Loads wheel data
+% loadIntanAnalog('session',session,'dataName','temperature','data_source_type','adc','container_type','timeseries','processing','thermistor') % Loads temperature data from a thermistor
+% wheel = loadIntanAnalog('session',session,'dataName','WheelPosition','data_source_type','adc','container_type','behavior','processing','wheel_position','downsample_samples',200); Loads wheel data
 
 % By Peter Petersen
 % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % 
@@ -178,7 +178,7 @@
 end
 
 % Attaching info about the data source and how the data was processed
-data_out.processinginfo.function = 'intan2buzcode';
+data_out.processinginfo.function = 'loadIntanAnalog';
 data_out.processinginfo.version = 2;
 data_out.processinginfo.date = now;
 data_out.processinginfo.params.basepath = basepath;

calc_CellMetrics/intanDigital2buzcode.m → calc_CellMetrics/loadIntanDigital.m

@@ -1,5 +1,5 @@
-function intanDig = intanDigital2buzcode(session)
-% function gives out the digital inputs as raw datapoints.
+function intanDig = loadIntanDigital(session)
+% function gives out the digital inputs as raw datapoints. Also saves the imported data
 % Documentation for Intantech data file formats: http://intantech.com/files/Intan_RHD2000_data_file_formats.pdf
 %
 % INPUT
@@ -51,7 +51,7 @@
 end
 
 % Attaching info about how the data was processed
-intanDig.processinginfo.function = 'intanDigital2buzcode';
+intanDig.processinginfo.function = 'loadIntanDigital';
 intanDig.processinginfo.version = 1;
 intanDig.processinginfo.date = now;
 intanDig.processinginfo.params.basepath = session.general.basePath;

calc_CellMetrics/intan2buzcode_examples.m → calc_CellMetrics/loadIntan_examples.m

@@ -4,10 +4,10 @@
 session = loadSession(basepath); % Loading session info
 
 %% Importing wheel data
-wheel = intan2buzcode('session',session,'dataName','WheelPosition','data_source_type','adc','container_type','behavior','processing','wheel_position','downsample_samples',200); % Loads wheel data
+wheel = loadIntanDigital('session',session,'dataName','WheelPosition','data_source_type','adc','container_type','behavior','processing','wheel_position','downsample_samples',200); % Loads wheel data
 
 %% Importing temperature data
-temperature = intan2buzcode('session',session,'dataName','Temperature','container_type','timeseries','processing','thermocouple');
+temperature = loadIntanAnalog('session',session,'dataName','Temperature','container_type','timeseries','processing','thermocouple');
 
 %% Plotting 
 figure, 

calc_CellMetrics/loadOpenEphysDigital.m

@@ -0,0 +1,62 @@
+function openephysDig = loadOpenEphysDigital(session,TTL_paths)
+% function to load digital inputs from Open Ephys
+%
+% INPUT
+% session: session struct
+%
+% OUTPUTS
+% intanDig.on: on-state changes for each channel [in seconds]
+% intanDig.off: off-state changes for each channel [in seconds]
+%
+% By Peter Petersen
+% petersen.peter@gmail.com
+
+% Load TTL data
+% Each path must contain:
+% 1. timestamps.npy and 
+% 2: channel_states.npy
+% 3: channels.npy
+% 4: full_words.npy
+
+% TTL_paths = {'TTL_2','TTL_4'};
+% TTL_offset = double(readNPY(fullfile(TTL_path,'experiment2\recording1\continuous\Neuropix-PXI-100.0\timestamps.npy')))/session.extracellular.sr;
+
+openephysDig = {};
+openephysDig.timestamps = readNPY(fullfile(session.general.basePath, TTL_paths{1},'timestamps.npy'));
+openephysDig.channel_states = readNPY(fullfile(session.general.basePath,TTL_paths{1},'channel_states.npy'));
+openephysDig.channels = readNPY(fullfile(session.general.basePath, TTL_paths{1},'channels.npy'));
+openephysDig.full_words = readNPY(fullfile(session.general.basePath,TTL_paths{1},'full_words.npy'));
+openephysDig.on{1} = double(openephysDig.timestamps(openephysDig.channel_states == 1))/session.extracellular.sr;
+openephysDig.off{1} = double(openephysDig.timestamps(openephysDig.channel_states == -1))/session.extracellular.sr;
+
+if length(TTL_paths) > 1
+    openephysDig.nTimestampsPrFile(1) = numel(openephysDig.timestamps);
+
+    for i = 2:length(TTL_paths)
+        openephysDig.timestamps = [openephysDig.timestamps; readNPY(fullfile(session.general.basePath, TTL_paths{i},'timestamps.npy'))];
+        openephysDig.channel_states = [openephysDig.channel_states; readNPY(fullfile(session.general.basePath,TTL_paths{i},'channel_states.npy'))];
+        openephysDig.channels = [openephysDig.channels;readNPY(fullfile(session.general.basePath, TTL_paths{1},'channels.npy'))];
+        openephysDig.full_words = [openephysDig.full_words; readNPY(fullfile(session.general.basePath,TTL_paths{1},'full_words.npy'))];
+        openephysDig.on{1} = [openephysDig.on{1}; double(openephysDig.timestamps(openephysDig.channel_states == 1))/session.extracellular.sr];
+        openephysDig.off{1} = [openephysDig.off{1}; double(openephysDig.timestamps(openephysDig.channel_states == -1))/session.extracellular.sr];
+        openephysDig.nTimestampsPrFile(i) = numel(openephysDig.timestamps)-openephysDig.nTimestampsPrFile(i-1);
+    end
+end
+
+% Attaching info about how the data was processed
+openephysDig.processinginfo.function = 'loadOpenEphysDigital';
+openephysDig.processinginfo.version = 1;
+openephysDig.processinginfo.date = now;
+openephysDig.processinginfo.params.basepath = session.general.basePath;
+openephysDig.processinginfo.params.basename = session.general.name;
+openephysDig.processinginfo.params.TTL_paths = TTL_paths;
+
+try
+    openephysDig.processinginfo.username = char(java.lang.System.getProperty('user.name'));
+    openephysDig.processinginfo.hostname = char(java.net.InetAddress.getLocalHost.getHostName);
+catch
+    disp('Failed to retrieve system info.')
+end
+
+% Saving data
+saveStruct(openephysDig,'digitalseries','session',session);

calc_CellMetrics/loadOpenEphys_examples.m

@@ -0,0 +1,11 @@
+basepath = 'Z:\Homes\peterp03\IntanData\MS22\Peter_MS22_180626_110916_concat';
+cd(basepath)
+session = loadSession(basepath); % Loading session info
+
+%% Detect open ephys data
+% Coming...
+% session = detectOpenEphysData('session',session)
+
+%% Load digital pulses
+TTL_paths = {'TTL_2','TTL_4'};
+openephysDig = loadOpenEphysDigital(session,TTL_paths);

calc_CellMetrics/optitrack2buzcode.m → calc_CellMetrics/loadOptitrack.m

@@ -1,25 +1,27 @@
-function optitrack = optitrack2buzcode(varargin)
-% Loads position tracking data from Optitrack to buzcode/CellExplorer data container
+function optitrack = loadOptitrack(varargin)
+% Loads position tracking data from Optitrack (csv files) to CellExplorer behavior data container
 % https://cellexplorer.org/datastructure/data-structure-and-format/#behavior
 %
 % Example calls
-% optitrack = optitrack2buzcode('session',session)
-% optitrack = optitrack2buzcode('basepath',basepath,'basename',basename,'filenames',filenames)
+% optitrack = loadOptitrack('session',session)
+% optitrack = loadOptitrack('basepath',basepath,'basename',basename,'filenames',filenames)
+% optitrack = loadOptitrack('session',session,'dataName','theta_maze')
+% optitrack = loadOptitrack('session',session,'dataName','linear_track')
 
 p = inputParser;
 addParameter(p,'session', [], @isstruct); % A session struct
 addParameter(p,'basepath', pwd, @isstr); % Basepath of the session
 addParameter(p,'basename', [], @isstr); % Name of the session
-addParameter(p,'filenames', []); % List of tracking files 
-addParameter(p,'unit_normalization', 1, @isnumeric);
-addParameter(p,'plot_on', true, @islogical); % Creates plot with behavior
+addParameter(p,'dataName','',@isstr); % Any renaming of the behavior struct
+addParameter(p,'filenames', {''},@iscellstr); % List of tracking files 
+addParameter(p,'scaling_factor', 1, @isnumeric); % An extra scaling-factor to apply to the x,y,z data.
 addParameter(p,'saveMat', true, @islogical); % Creates behavior mat file
+addParameter(p,'plotFig', true, @islogical); % Creates plot with behavior
 addParameter(p,'saveFig', true, @islogical); % Save figure with behavior to summary folder
 parse(p,varargin{:})
 
 parameters = p.Results;
 
-
 if ~isempty(parameters.session)
     session = parameters.session;
     basename = session.general.name;
@@ -28,35 +30,36 @@
 else
     basepath = p.Results.basepath;
     basename = p.Results.basename;
-    filenames = parameters.filenames;
 end
 
 if isempty(basename)
     basename = basenameFromBasepath(basepath);
 end
 
+if ~isempty(parameters.filenames)
+    filenames = parameters.filenames;
+end
+
 % filename = [datapath recording '/' recordings(id).tracking_file];
 formatSpec = '%q%q%q%q%q%q%q%q%q%q%q%q%q%q%q%q%q%q%q%q%[^\n\r]';
 header_length = 7;
 
-if iscell(filenames)
-    fileID = fopen(fullfile(basepath,filenames{1}),'r');
-    dataArray = textscan(fileID, formatSpec, 'Delimiter', ',',  'ReturnOnError', false); 
-    fclose(fileID);
-    FramesPrFile = size(dataArray{1}(header_length:end),1);
+
+% Loading files
+fileID = fopen(fullfile(basepath,filenames{1}),'r');
+dataArray = textscan(fileID, formatSpec, 'Delimiter', ',',  'ReturnOnError', false);
+fclose(fileID);
+FramesPrFile = size(dataArray{1}(header_length:end),1);
+if length(filenames) > 1
     for i = 2:length(filenames)
         fileID = fopen(fullfile(basepath,filenames{i}),'r');
-        dataArray_temp = textscan(fileID, formatSpec, 'Delimiter', ',',  'ReturnOnError', false); 
+        dataArray_temp = textscan(fileID, formatSpec, 'Delimiter', ',',  'ReturnOnError', false);
         fclose(fileID);
         for j = 1:length(dataArray)
             dataArray{j} = [dataArray{j};dataArray_temp{j}(header_length:end)];
         end
         FramesPrFile = [FramesPrFile, size(dataArray_temp{1}(header_length:end),1)];
     end
-else
-    fileID = fopen(fullfile(basepath,filenames),'r');
-    dataArray = textscan(fileID, formatSpec, 'Delimiter', ',',  'ReturnOnError', false);
-    fclose(fileID);
 end
 
 optitrack_temp = [];
@@ -75,14 +78,12 @@
 optitrack_temp.LenghtUnit = dataArray{18}(1);
 optitrack_temp.CoorinateSpace = dataArray{20}(1);
 optitrack_temp.FrameRate = str2double(dataArray{6}{1});
-if exist('FramesPrFile')
-    optitrack_temp.FramesPrFile = FramesPrFile;
-end
+
 clear dataArray
 clearvars filename formatSpec fileID dataArray header_length;
 
-% get position out in cm, and flipping Z and Y axis
-position = 100*[-optitrack_temp.X,optitrack_temp.Z,optitrack_temp.Y]/parameters.unit_normalization;
+% getting position out in cm, and flipping Z and Y axis
+position = 100*[-optitrack_temp.X,optitrack_temp.Z,optitrack_temp.Y]/parameters.scaling_factor;
 
 % Estimating the speed of the rat
 % animal_speed = 100*Optitrack.FrameRate*(diff(Optitrack.X).^2+diff(Optitrack.Y).^2+diff(Optitrack.Z).^2).^0.5;
@@ -110,12 +111,19 @@
 optitrack.orientation.z = optitrack_temp.Zr;
 optitrack.orientation.rotationType = optitrack_temp.RotationType;
 optitrack.nSamples = numel(optitrack.timestamps);
+
+if exist('FramesPrFile')
+    optitrack.framesPrFile = ramesPrFile;
+end
+
 % Attaching info about how the data was processed
-optitrack.processinginfo.function = 'optitrack2buzcode';
+optitrack.processinginfo.function = 'loadOptitrack';
 optitrack.processinginfo.version = 1;
 optitrack.processinginfo.date = now;
 optitrack.processinginfo.params.basepath = basepath;
 optitrack.processinginfo.params.basename = basename;
+optitrack.processinginfo.params.filenames = filenames;
+
 try
     optitrack.processinginfo.username = char(java.lang.System.getProperty('user.name'));
     optitrack.processinginfo.hostname = char(java.net.InetAddress.getLocalHost.getHostName);
@@ -125,11 +133,11 @@
 
 % Saving data
 if parameters.saveMat
-    saveStruct(optitrack,'behavior','session',session);
+    saveStruct(optitrack,'dataName',dataName,'behavior','session',session);
 end
 
 % Plotting
-if parameters.plot_on
+if parameters.plotFig
     fig1 = figure;
     subplot(1,2,1)
     plot3(position(:,1),position(:,2),position(:,3)), title('Position'), xlabel('X (cm)'), ylabel('Y (cm)'), zlabel('Z (cm)'),axis tight,view(2), hold on
@@ -138,7 +146,9 @@
     xlabel('X (cm)'), ylabel('Y (cm)'),zlabel('Speed (cm/s)'), axis tight
 
     % Saving a summary figure for all cells
-    timestamp = datestr(now, '_dd-mm-yyyy_HH.MM.SS');
-    ce_savefigure(fig1,basepath,[basename, '.optitrack.behavior' timestamp])
-    disp(['optitrack2buzcode: Summary figure saved to ', fullfile(basepath, 'SummaryFigures', [basename, '.optitrack.behavior', timestamp]),'.png'])
+    if parameters.saveFig
+        timestamp = datestr(now, '_dd-mm-yyyy_HH.MM.SS');
+        ce_savefigure(fig1,basepath,[basename, '.optitrack.behavior' timestamp])
+        disp(['loadOptitrack_csv: Summary figure saved to ', fullfile(basepath, 'SummaryFigures', [basename, '.optitrack.behavior', timestamp]),'.png'])
+    end
 end

docs/datastructure/data-structure-and-format.md

@@ -182,7 +182,7 @@ A MATLAB struct `session` stored in a .mat file: `basename.session.mat`. The ses
     * `equipment` : hardware used to acquire the data
 
 ### Spikes
-A MATLAB struct `spikes` stored in a .mat file: `basename.spikes.cellinfo.mat`. It can be generated with [loadSpikes.m](https://github.com/petersenpeter/CellExplorer/blob/master/calc_CellMetrics/loadSpikes.m). The processing module `ProcessCellMetrics.m` used the script `loadSpikes.m`, to automatically load spike-data from either KiloSort, Phy or Neurosuite and saves it to a spikes struct. `basename.spikes.cellinfo.mat` is saved to the basepath. The struct has the following fields:
+A MATLAB struct `spikes` stored in a .mat file: `basename.spikes.cellinfo.mat`. It can be generated with [loadSpikes.m](https://github.com/petersenpeter/CellExplorer/blob/master/calc_CellMetrics/loadSpikes.m). The processing module `ProcessCellMetrics.m` used the script `loadSpikes.m`, to load spike-data from various pipelines/data formats, including KiloSort, Phy, and Neurosuite and saves it to a spikes struct: `basename.spikes.cellinfo.mat`, which is saved to the basepath. The struct has the following fields:
 * `ts`: a 1xN cell-struct for N units each containing a [nSpikes x 1] vector with nSpikes spike events in samples.
 * `times`: a 1xN cell-struct for N units each containing a [nSpikes x 1] vector with nSpikes spike events in seconds.
 * `cluID`: a 1xN vector with inherited IDs from the applied clustering algorithm.
@@ -335,7 +335,6 @@ This is a data container for behavioral tracking data. A MATLAB struct `behavior
 
 Any other field can be added to the struct containing behavior data. The `*.behavior.mat` files should be stored in the basepath.
 
-
 ### Trials
 A MATLAB struct `trials` stored in a .mat file: `basename.trials.behavior.mat`. The trials struct is a special behavior struct centered around behavioral trials. `trials` has the following fields:
 * `start`: trial start times in seconds.
@@ -395,15 +394,15 @@ The data is organized into data-type specific containers, a concept introduced i
 
 * `basename.session.mat`: session level metadata.
 * `basename.*.lfp.mat`: derived ephys signals including theta-band filtered lfp.
-* `basename.*.cellinfo.mat`: Spike derived data including`spikes`, `cell_metrics`, `mono_res`
+* `basename.*.cellinfo.mat`: Spike derived data including`spikes`, `cell_metrics`, `mono_res`.
 * `basename.*.firingRateMap.mat`: firing rate maps. Derived from behavior and spikes, e.g. `ratemap`.
-* `basename.*.events.mat`: events data, including `ripples`, `SWR`,
-* `basename.*.manipulation.mat`: manipulation data: 
-* `basename.*.channelinfo.mat`: channel-wise data, including impedance
-* `basename.*.timeseries.mat`: 
-* `basename.*.behavior.mat`: behavior data, including position tracking.
+* `basename.*.events.mat`: events data, including `ripples`, `SWR`.
+* `basename.*.manipulation.mat`: manipulation data.
+* `basename.*.channelinfo.mat`: channel-wise data, including impedance.
+* `basename.*.timeseries.mat`: time series data, could be temperature measures, or other data collected together with the ephys data.
+* `basename.*.behavior.mat`: behavior data, including position tracking and pupil tracking.
 * `basename.*.states.mat`: brain states derived data including SWS/REM/awake and up/down states.
 * `basename.*.intracellular.mat`: intracellular data.
 
 ## Example dataset
-There is an example dataset available to help understanding the data structure. The dataset contains: a .dat file (raw ephys data; 62GB), a .lfp file (lowpass filtered and downsampled data file; 4 GB), session.mat, spikes, events, behavior, trials, timeseries, states, firingRateMap, cell_metrics, mono_res, and spike sorted data processed with KiloSort and curated in Phy. Available from our [Webshare](https://buzsakilab.nyumc.org/datasets/PetersenP/CellExplorerExampleData/MS22/Peter_MS22_180629_110319_concat/) and our [Globus endpoint](https://app.globus.org/file-manager?origin_id=188a6110-96db-11eb-b7a9-f57b2d55370d&origin_path=%2FPetersenP%2FCellExplorerExampleData%2FMS22%2FPeter_MS22_180629_110319_concat%2F). The size of the full dataset  is 75GB, but files can be downloaded individual. 
+There is an example dataset available to help understanding the data structure. The dataset contains: a .dat file (raw ephys data; 62GB), a .lfp file (lowpass filtered and downsampled data file; 4 GB), session.mat, spikes, events, behavior, trials, timeseries, states, firingRateMap, cell_metrics, mono_res, and spike sorted data processed with KiloSort and curated in Phy. Available from our [Webshare](https://buzsakilab.nyumc.org/datasets/PetersenP/CellExplorerExampleData/MS22/Peter_MS22_180629_110319_concat/) and our [Globus endpoint](https://app.globus.org/file-manager?origin_id=188a6110-96db-11eb-b7a9-f57b2d55370d&origin_path=%2FPetersenP%2FCellExplorerExampleData%2FMS22%2FPeter_MS22_180629_110319_concat%2F). The size of the full dataset  is 75GB, but files can be downloaded individual.

docs/tutorials/behavior-tutorial.md

@@ -45,7 +45,7 @@ the optitrack output contains timestamps (from optitrack) and position data
 `optitrack.sr`:             samplingrate
 
 ```m
-optitrack = optitrack2buzcode('session',session);
+optitrack = loadOptitrack('session',session);
 
 % After this you can load the generated file:
 % optitrack = loadStruct('optitrack','behavior','session',session);
@@ -60,7 +60,7 @@ The digital timeseries contains timestamps of changes in states:
 `intanDig.off`:     cell array with off state-changes channel-wise
 
 ```m
-intanDig = intanDigital2buzcode(session);
+intanDig = loadIntanDigital(session);
 
 % After this you can load the generated file:
 % intanDig = loadStruct('intanDig','digitalseries','session',session);