2.1.

Animal Models

All animal-related work was done in accordance with the Tel Aviv University ethics committee for animal use and welfare, which adheres to standards established by the National Health Institute, Institutional Animal Care and Use Committee. A full description of the animal preparations procedure is detailed in Appendix A.

2.2.

System Architecture

The optical setup of the proposed system is depicted in Fig. 1 and is detailed in Appendix B. The optical path as seen in Fig. 1(a) is a standard 2PM path with the (optional) tunable acoustic gradient (TAG) lens module added before the scanning mirrors, which enables 3D imaging. For standard 2D imaging, the basic configuration of the path should be kept. The photons that fluoresce from the sample are collected by H10770PA-40SEL photomultiplier tubes (PMT), which are connected via fast preamplifiers to the TT. Photon counting permits setting the PMT gains at much higher values than in analog acquisition due to the built-in filtration step in the form of the discriminator. From a practical point of view, in our basic analog setup, we use gain values of about $4\times {10}^5$, whereas for digital acquisition we can ramp up the gain to $2\times {10}^6$, a fivefold increase, and benefit from better photon detection rates.

Fig. 1

The optical, electronic, and software setup used for online photon counting. (a) Before passing through the objective, the input beam traverses through the TAG lens (when imaging in 3D) and its relay system, consisting of two lenses $L1=45\mathrm{mm}$ and $L2=60\mathrm{mm}$. The beam bounces off the resonant-galvo scanners into the scan lens and tube lens, marked as $L3$ and $L4$, respectively. The emitted photons are collected through the objective and are routed using a dichroic mirror into the collection path, where a second dichroic mirror splits them into two spectral channels. (b) The TT wiring diagram. To build frames, rPySight requires either a line synchronization signal or a start of frame (but not both) and at least a single spectral channel to display images. Additional inputs such as the laser pulse synchronization, TAG lens phase signal, or others may be added as needed (up to 18 channels in total). (c) A screenshot showing pial blood vessels in an anesthetized mouse being imaged with rPySight. The key components are the text editor used to create or edit the configuration file, a command window and the TimeTagger server. Here, we used Visual Studio (Microsoft Inc.) both as editor and command window. In this particular case, we acquired an image of $512\times 512\text{pixels}$, at a frame rate of 30 fps. rPySight was set to display an average of 30 frames. Not shown in the image are the microscope control panels (ScanImage 2021, Vidrio Technologies, LCC). rPySight is agnostic to the software being used for microscope control as long as the required signals can be connected to the fast digitizer. RG, resonant-galvo scanner; DM, dichroic mirror; Obj., objective lens; PMT, photomultiplier tube (preamplifiers not shown); TAG, tunable acoustic gradient index of the refraction lens.

Figure 1(b) shows the electronic connections to the digitizer. The TT receives inputs into a set of channels pre-defined in the rPySight configuration file. Each input is discriminated and digitized on the fly and sent via a USB 3.0 interface to the computer. The TT has potentially up to 18 input channels, which permits experimenters to record additional data streams such as transistor-transistor logic (TTL) signals for synchronization with other experimental apparatus, such information can be recorded to disk, although it remains unused by rPySight. The data exist in binary form in the same folder of the rPySight streams. A screenshot of an acquisition session using rPySight [Fig. 1(c)] shows blood vessels at the pial level of an anesthetized mice as well as the key software tools needed to run: a text editor to create or modify the configuration file, a console to issue commands, and the TT server that runs in the background.

2.3.

Software

rPySight is an open-source application that we made available on GitHub¹⁸ licensed under the GNU Public License v3.0. Most of rPySight is written in the Rust programming language, while the parts that interact with the TT application programming interface are written in Python. rPySight is currently tested on Windows machines only but should work on UNIX-like operating systems assuming that the rest of the components of the microscope, such as the scanners and the TT, are working as well.

rPySight’s architecture is composed of three main parts. The first is a Python application that interfaces with the built-in application programming interface (API) of the TT. This module instantiates the TT with user-defined parameters and listens for events arriving to the designated channels. This polling loop, which samples at a minimum rate of 300 Hz, transmits the recorded event using inter-process communication protocols to the second rPySight module, the one responsible for assigning the detected photons to their voxel. This part of the library finds the appropriate coordinate by constructing a one-dimensional “snake,” illustrated in Figs. 2(a)–2(c). The snake is a projection of the trajectory that the scanned illumination laser beam will traverse in the sample. In the 2D case, it is a familiar raster pattern that takes into account the field fraction [i.e., the portion of the scanned path that will become the FOV displayed, i.e., pixels enclosed in the dashed-line box, Fig. 2(b)]. In 3D [Fig. 2(c)] this trajectory is not identical between volumes due to the resonant nature of the TAG lens, which means that it is unsynchronized with the rest of the scanners. The snake—built for every frame or volume on-the-fly—holds the mapping between the time of arrival to a specific pixel or voxel since the beginning of the current acquisition and the matching coordinate (2D or 3D) for photons arriving up to that moment. Then, for each detected photon, rPySight has to iterate over the snake and find the first time that the pre-computed pixel/voxel time exceeds this photon’s arrival time (computed from the beginning of the current frame or volume). The location of this occurrence (i.e., location along the snake path) is the coordinate to which the photon will be assigned. rPySight has several built-in mechanisms to render this process efficient and robust.

Fig. 2

The allocation algorithm of rPySight. (a) A one-dimensional vector (“snake”) is generated from user inputs, modeling the number of pixels or voxels that the laser beam will traverse in the next rendered frame. Each cell in this vector has two properties—its end time and its associated coordinate, or point, in the rendered image or volume. The end time is specified in time units since the start of the experiment and signals the moment after which photons detected should not be allocated to this cell. The coloring marks the time axis. (b) A unidirectional 2D image with the mapping of the snake onto it. When imaging in 2D, the snake can be mapped to the plane almost directly. The only caveat lies at the edges of the image, where the resonant mirror is rotating and image deformation occurs. rPySight—similar to all other imaging software—discards a portion of these outer pixels due to these deformations so that the visible imaged is only composed of the pixels inside the square. (c) 3D mapping of the snake to the rendered volume. Some voxels are skipped due to the mismatch in frequencies between the galvanometric mirror and the TAG lens, but they will be sampled later because the scanning elements are not synchronized. (d) The computational pipeline that processes the time tags. Once a photon-based time tag is registered and sent to rPySight, it can be converted into a coordinate or point in 3D space using the snake-like structure explained above. This point is then sent to two different paths using a hashing function—the renderer assigns a color to that point based on the previous brightness of that voxel and on the spectral channel from which the event arrived, and the serialization process simply increments the value at that voxel because each spectral channel is independent of the other ones.

The third part of rPySight is responsible for rendering and data serialization. First, it receives a list of coordinates that house a photon. This list might contain duplicates, i.e., coordinates that were assigned with more than one event; therefore, the first task of this module is to increase the brightness of the coordinates that were populated with more than one photon, and it achieves that goal with a hashmap. Following that computation, the module sends a data buffer to the graphical processing unit (GPU) for rendering and a per-spectral-channel buffer to the disk for serialization and other post-processing applications.

Appendix C describes how to use rPySight during a photon counting experiment with and without a TAG lens, which provides continuous volumetric imaging of the sample.

2.4.

Calcium Imaging Analysis and Statistical Testing

To detect and segment the active components, in the recording we used CaImAn,¹¹ a calcium imaging analysis framework written in Python. Both datasets (analog and digital) were analyzed using identical parameters, and the output of the analysis—$\mathrm{\Delta }F/F$ traces per active component—underwent peak detection using SciPy-based algorithms¹⁹ to mark all spike-like events in the recording.

The statistical tests we used are based on DABEST,²⁰ an estimation statistics library that mainly uses bootstrap-oriented methods to compare two distributions. Importantly, this process focuses on the measured effect size rather than mere $P$-values; this is to illuminate the actual differences between the two tested datasets, eliminating cases with distributions that were very similar in absolute terms but turned out to be significantly different due to a large number of samples that was used. Importantly, the bootstrap process provides a confidence interval around the effect size, which allows for determining if the effect size, regardless of its magnitude, differs from zero or not. The distribution of bootstrapped values and this confidence interval are computed based on the mean differences between groups. Here, the effect size refers to the mean value of rPySight metrics “minus” the mean value of the same metric obtained during analog acquisition.

3.1.

rPySight Enables Calcium Imaging with Expected Improvement in Related Metrics

The importance of calcium and voltage imaging in the field is undeniable, as numerous labs rely on then to provide answers to countless questions in the areas of neural coding and encoding. We have previously demonstrated that photon-counting improves both calcium and voltage performance in these imaging modalities.¹⁷ Here, we exemplify that calcium imaging is improved when rPySight is used during in vivo recordings of calcium activity in the murine neocortex. During these experiments, each FOV was imaged twice—first using analog integration and then using the new digital setup. The acquisition parameters of each imaging modality were visually tuned for a maximal SNR before acquiring the data, which is summarized in Fig. 3. We acquired here three different fields in one animal, rendering bias to a particular condition very unlikely. In Fig. 3(a) 100-frame average images in two different FOVs are contrasted, after a normalization process during which the analog image was brought to the same scale as the digital one. The digital image’s brightness values accurately represent the number of photons detected per pixel, whereas in the analog image the value is an estimation. CaImAn¹¹ detected traces for the second (bottom) FOV are displayed in Fig. 3(b).

Fig. 3

Calcium imaging using analog integration (blue) and digital photon-counting of three FOVs of a single Thy1-positive mouse. (a) Two representative FOVs from analog (left) and digital data acquisition, showing an average of 100 consecutive frames (data acquired at 30 pfs). The color bar marks the number of photons per pixel that were detected. Although for the digital image this number is accurate, the analog image was first normalized to the digital image’s scale to provide an accurate qualitative comparison. This normalization visually emphasizes the lower background levels of the photon counting image. (b) Exemplary $\mathrm{\Delta }F/F$ traces for the two modalities calculated using CaImAn.¹¹ The distinction between the relatively-quiet analog traces and the higher-amplitude digital ones, for the same FOV, is clearly visible. (c), (d) Summary statistics comparing analog and digital acquisition, showing a significant improvement in imaging conditions in favor of digital acquisition. The compared measured values were the mean number of spike-like events (c) and the mean area under the curve per spike-like event (d), and they were both significantly higher in the digital acquisition case. The effect size [right side of panes (d) and (c)] along with the corresponding confidence interval were computed around the mean difference between rPySight versus analog by bootstrap-coupled estimation using the methods introduced in Ref. 20, and the $P$-value was $<0.001$ for all comparisons, including median-based ones (not shown). The confidence intervals for the mean differences (vertical line and the entire distribution of bootstrapped estimates (green) are larger than zero, indicating the statistical significance of this comparison. Scale bar in (a) is $50\mu \mathrm{m}$.

To compare the two modalities, we pooled all three FOVs together. First, we note that the number of detected spike-like events in the digital recording is larger [0.01 to 0.026 spike-like events per second, Fig. 3(c)]. Second, the mean area under the curve (AUC) of the spike-like events in the digital recording was also larger [0.09 to 0.48 A.U., Fig. 3(d)], even though more of them were detected. Together with the fact that more active components were detected in the digital recording (145 to 119, components underwent manual filtration before being included in this count), these findings reiterate previously shown benefits of photon counting versus analog integration in calcium imaging experiments (for a detailed, cell-by-cell comparison please see Ref. 17). For the sake of completeness, we also note that classical Student’s $t$-test also found the measurements detailed above to be significantly different, with a $P$-value $<0.0001$.

3.2.

rPySight Facilitates Advanced Multiphoton Imaging Modalities

Figures 4 and 5 showcase two unique modalities that are difficult to implement in non-digital acquisition. The first (Fig. 4) is real-time demultiplexing of two or more photon streams into separate spectral channels. This is done by connecting the laser synchronization signal to one of the TT’s inputs and changing the demultiplex flag in the rPySight configuration file to true. The second modality is 3D continuous volumetric imaging, using a TAG lens, and is shown in Fig. 5. To demonstrate the demultiplexing capabilities, we built a two-arm setup before our 2PM in which one of the beams in the path is delayed by 6.25 ns [Fig. 4(a)]. The beam paths can be then alternatively blocked, and fluorescence emitted from a fluorescein bath is expected to be visible in only one of the demultiplexed data streams at the time.

Fig. 4

Real-time demultiplexing using rPySight. (a) Schematic of a simplified version that the laser path used to generate two pulse trains separated in time by half the laser inter-pulse period (top) and corresponding pulse train (bottom) for the original (green) and delayed (purple) paths. For the 80 MHz source used in this experiment, delay between each pulse train corresponds to 6.25 ns. (b) Schematics of events recorded by the TT. The original pulse train (top row), showing a typical 80 MHz signal, is duplicated and delayed internally (i.e., by the TT, middle row), which lets the TT interpret these two trains as the starting and closing signals for two alternating data streams, illustrated in green and purple in the bottom row. The arriving photons will be dynamically placed in one of the two demultiplexed event stream, which are rendered as separated channels according to their time of arrival since the last pulse. (c) Demultiplexing of photons emitted from a fluorescein bath imaged through the setup shown in (a) is demonstrated by alternatively blocking each beam path at the locations indicated by the arrows in (a). When the delayed beam path is blocked (top row), photons are detected only in the first demultiplexed stream (Demux stream 1). In turn, when the original beam path is blocked and the delayed path is open (bottom row), the opposite occurs and photons are only detected in the second demultiplexed stream (Demux stream 2), which corresponds to a 6.25 ns delay with respect to the original laser pulse emitted by the laser. Abbreviations: 2P-Mic, two-photon microscope; PBS, polarizing beam splitter; HWP, half-wave plate; × 3, three times beam expander; EOM, electro optical modulator. Scale bar is $100\mu \mathrm{m}$.

Fig. 5

Real-time rapid continuous volumetric imaging in awake murine brain using rPySight (a) Depth-color-coded maximum $z$-projection of a real-time rendering of a $480\times 480\times 300{\mu \mathrm{m}}^3$ volume continuously imaged at 30 volumes per second using a TAG lens. The temporal averaging window spans 550 frames, corresponding to 18.3 s. Following brightness normalization along the $z$ axis, the image was filtered with a 3D Gaussian window using a $1.5\times 1.5\times 0.5{\mu \mathrm{m}}^3$ kernel. Scale bar is $75\mu \mathrm{m}$. (b) Depth-color-coded maximum $z$-projection of an offline rendering of the same volume using rPySight, summed across the entire recording (22,620 frames, corresponding to 754 s). Its improved signal-to-background ratio reveals finer micro-vascular features. Following brightness normalization along the $z$ axis, the image was filtered with a 3D median filter of $1.5\times 1.5\times 2{\mu \mathrm{m}}^3$. Scale bar is $75\mu \mathrm{m}$. Both depth color codings are identical and were rendered using the $Z$-stack Depth Colorcode plugin of ImageJ.²¹

Dividing the photons in real-time requires (in the case presented here) two laser synchronization signals to be connected to the input ports of the TT, one for each data stream. Because our excitation laser only emits a single synchronization signal [Chameleon Discovery NX TPC, Coherent Inc., represented Fig. 4(a)], the TT automatically duplicates this signal and shifts it by 6.25 ns to receive two independent pulse trains. These then serve as a gating mechanism for the TT—when the first one arrives, the gate (or window) for the first channel opens while the gate for the second channel closes, and when the second pulse arrives after 6.25 ns, the gate for the first channel closes and the gate for the second opens. The TT is able to allocate the photons arriving during each of these periods into their own data stream, which is broadcasted independently to the computer. Our experiment showed that rPySight correctly assigned these demultiplexed streams into separate channels that became almost dark with the corresponding beam path was blocked [Fig. 4(c)]. The minor amount of cross-talk that remains and is visible in Figs. 4(c) and 4(d) has to do with the two beams not being exactly 6.25 ns apart in our optical setup.

The two laser pulse trains incur some post-processing penalty on rPySight due to their high rate. To reduce it as much as possible, we deploy a conditional filter with these channels and the PMT channels. The filter waits for a photon to arrive at one of the data channels, and only when an event is registered does it also register the last incoming laser pulse (which was detected but not included in a data stream to avoid processing). Recalling that a low probability of fluorophore excitation per laser pulse ($P_{\text{pulse}}<0.1$) should be retained to prevent an effective broadening of the point spread function,²² this filtration step matches the recorded event rate to the rate of photon detection, rather than to the much higher doubled laser pulse repetition rate.

As noted, rPySight also facilitates rapid 3D imaging. A TAG lens was used for continuous imaging of a $480\times 480\times 300\mu {\mathrm{m}}^3$ volume at an imaging rate of 30 volumes per second. The volumetric image shown in Fig. 5(a) is a maximum $Z$-projection of the volume rendered by our application during an in vivo recording of the vascular diameter in a Texas Red-injected mouse. The real-time rendering is sufficiently detailed to allow the experimenter to choose a volume of view centered on a penetrating artery, and an imaging depth that retains simultaneous tracking of its respective pial artery. Smaller features are buried in the background due to a short temporal averaging window (550 frames, corresponding to 18.3 s; notice that volumetric imaging is inherently reducing photon flux from each plane; hence the longer average is needed to display a volumetric image compared with a single plane imaged at 30 fps as shown in Fig. 1(c) and sub-optimal line-shift correction. These parameters can both be corrected offline. Indeed, when summing the entire 754 s long recording over time as depicted in Fig. 5(b), the improved signal-to-background ratio of the respective $z$-projection reveals finer details such as pial veins, arterioles, and capillary vessels, as well as the deeper portions of the imaged penetrating arteries.

7.1.

C.1 Preparations

The following procedures should be done on the computer that controls the 2PM with its dedicated software and hardware.

NOTE: If you have never inspected the signals emitted by your scanners and PMTs, it might be better to first connect them to an oscilloscope to check their amplitude and polarity. Signals stronger than $\pm 5\mathrm{V}$ or 50 Ω impedance will harm the TT, while a weak signal might not be detected reliably by the 50 Ω-terminated inputs.

NOTE: rPySight cannot work while the TT is connected to the web user interface. You must first shut down the TT from the settings menu before starting rPySight yet leave the server running [as shown in Fig 1(c)]

7.2.

C.2 Acquiring Data

7.3.

C.3 Data Inspection and Post-Processing

The data that were collected and displayed by rPySight may be re-used following the experiment in a few ways:

NOTE: Re-running an experiment also overwrites the .arrow_stream file, so make sure to backup the previous one if you wish to keep it.