diff --git a/_worksheets/synching.md b/_worksheets/synching.md index 0005375..01ab55e 100644 --- a/_worksheets/synching.md +++ b/_worksheets/synching.md @@ -9,12 +9,76 @@ title: Data Synchronization Data Synchronization ==================== -### Exercise: Synchronizing behaviour events with ephys +### **Exercise 1:** Synchronizing behaviour events with ephys -### Exercise: Synchronizing video with ephys using an LED +* Insert a `KeyDown` source. +* Insert an `Equal` transform and set its `Value` to one of the keys. The output of this operator will toggle between `True` and `False` depending on whether the key press matches the specified key. +* Insert a `DigitalOutput` sink and connect it to Arduino pin 13. +* Connect the Arduino pin 13 to OpenEphys analog input 1. +* Insert an `Rhd2000EvalBoard` source. +* Select the `Rhd2000DataFrame` > `BoardAdcData` field from the source output using the context menu. +* Insert a `SelectChannels` transform and set the `Channels` property to 0. This will select only the first analog input channel. +* Insert a `MatrixWriter` sink and configure its `Path` property with a file name ending in `.bin`. +* Run the workflow and alternately press the selected key and another key a couple of times to make the LED change state. +* Open the binary file in MATLAB/Python/R and plot the raw data. What can you conclude from it? -### Exercise (Optional): Synchronizing video with ephys using GPIO +### **Exercise 2:** Synchronizing video with ephys using an LED -### Exercise: Synchronizing a visual stimulus with ephys +* Using the workflow from the previous exercise, insert a `CameraCapture` source and point the camera such that you can see clearly both the LED and the computer keyboard. +* Insert a `VideoWriter` sink and configure the `FileName` with a path ending in `.avi`. +* Insert a `Crop` transform and set the `RegionOfInterest` property to a small area around the LED. +* Insert a `Grayscale` transform. +* Insert a `Sum (Dsp)` transform. This operator will sum the brightness values of all the pixels in the input image. +* Select the `Scalar` > `Val0` field from the right-click context menu. +* Record the output in a text file using a `CsvWriter` sink. +* Open both the text file and the binary file in MATLAB/Python/R and check that you have detected an equal number of key presses in both files. What can you conclude from these two pieces of data? +* **Optional:** Repeat the exercise, replacing the `KeyDown` source with a periodic `Timer`. Can you point out some of the limitations of synchronizing a video stream with ephys using this method? + +### **Exercise 3 (Optional):** Synchronizing video with ephys using GPIO + +Industrial grade cameras often include a GPIO connector which exposes input and output digital pins that operate similar to the pins in an Arduino or other microcontrollers. It is possible to configure these pins to report when the shutter of the camera is open or closed, i.e. when a frame is being exposed, the shutter is open and the pin goes `HIGH`, and conversely, when exposure stops, the shutter closes and the pin goes `LOW`. + +By connecting this strobe signal to the ephys system and counting the number of pulses, it is possible to reconstruct with sub-millisecond precision how many exposures were acquired by the camera, and when each of them started. One problem to consider during high-speed recordings, however, is that frames may occasionally be dropped if the system cannot handle each acquired frame fast enough. One way to work around this issue is to record the hardware frame counter which can be enabled in the drivers of all industrial grade cameras. + +* Connect one of the output GPIO camera pins to the OpenEphys analog input 1. +* Configure the camera output as *strobe*. +* Insert a `FlyCapture` source or other industrial grade camera capture operator. +* Record the embedded hardware frame counter into a text file using `CsvWriter`. +* Record the OpenEphys analog input and verify that you can recover individual camera pulses. +* Point out some of the remaining difficulties of this approach and how you would adress them. + +### **Exercise 4:** Synchronizing a visual stimulus with ephys + +Displaying visual patterns on a screen or projector can be subject to significant delays that may impact synchronization with neural signals. Unfortunately, most displays do not directly provide any kind of digital output that might be used to synchronize stimulus presentation with ephys. + +However, you can take advantage of the fact that all pixels in a frame are presented synchronously and reserve part of the display area to show a synchronization trigger. A passive photodiode can then be used to transduce this optical trigger into a digital signal that can be transmitted to the ephys auxiliary input channels. + +In this exercise you will track the display of a very simple visual stimulus: a transition between black and white. + +* Insert a `SolidColor` source and set its `Size` property to a positive value, e.g. 100,100. +* Insert a `Timer` source and set the `Period` to one second. +* Insert a `Mod` transform and set its `Value` property to 2. +* Insert a `Multiply` transform and set its `Value` property to 255. + +**Note:** The output of `Timer` is a growing count of the number of ticks. The `Mod` operator computes the remainder of the integer division of a number by another. Because every integer number in the sequence is alternately even or odd, the remainder of the division of the clock ticks by two will constantly oscillate between 0 and 1. Together with the `Multiply` operator, this is an easy way to make a periodic toggle between 0 and some value. +{: .notice--info} + +* Insert an `InputMapping` operator and connect it to the `SolidColor` source. +* Edit the `PropertyMappings` and add a mapping to the `Color` property. You will have to select four times the input to fill all the components of the `Color` scalar. +* Run the workflow and verify that the output of `SolidColor` oscillates between black and white. +* Insert an `Rhd2000EvalBoard` source. +* Select the `Rhd2000DataFrame` > `BoardAdcData` and either save or visualize its output. +* Connect the photodiode to the ephys analog input and hold it flat against the screen, on top of the visualizer window. +* Verify that you can capture the transitions between black and white in the ephys data using the photodiode. + +### **Exercise 5 (Optional):** Synchronizing video acquisition to a visual stimulus using GPIO + +The easiest way to synchronize the video acquisition with a visual stimulus is to make sure that your camera can see and track both your object of interest *and* the visual stimulus. If you can extract the stimulus events from your video, then they are synchronized with all other video events by definition, since all pixels in a video frame are acquired synchronously. + +However, sometimes this is not feasible: the stimulus may be covered by the subject, it may not be possible to recover the exact stimulus parameters from the camera view, or the stimulus may need to be filtered out entirely in order to allow for accurate tracking. In these situations, one solution for industrial grade cameras is to operate the camera in trigger mode, where the input GPIO channels can be used to align the beginning of each frame acquisition to the refresh rate of the display. + +To do this, you can use the photodiode technique described in the previous exercise, but this time the digital signal from the photodiode will be used as a trigger for the camera by connecting it to one of the GPIO inputs. + +* Assuming a DLP projector, how would you design the optical trigger for a camera system that ensures a single pulse is generated for each projected frame? *Hint:* In a DLP projector, each colour of a BGR frame is projected sequentially: first the Blue channel, then the Green, and finally the Red channel, in quick succession. +* **Optional:** Synchronize a camera with a projector using the GPIO trigger system outlined above. -### Exercise (Optional): Synchronizing video acquisition to a visual stimulus