We present an optics-free CMOS image sensor that incorporates a novel time-gated dual-photodiode pixel design to allow filter- and lens-less image acquisition of near-infrared-excited (NIR-excited) upconverting nanoparticles. Recent biomedical advances have highlighted the benefits of NIR excitation, but NIR interaction with silicon has remained a challenge, even with high-performance optical blocking filters. Using a secondary diode and a dual-photodiode design, this sensor is able to remove the 100s of mV of NIR background on pixels and bring it down to single-digit mV level, nearing its noise floor of 2.2 mV rms, not achievable with any optical filter. Non-linear effects of background cancellation using the diode pair has been mitigated using an initial one-time pixel-level curve fitting and calibration in a post-processing setting. This imager comprises a highly linear 11 fF metal-oxide-metal (MOM) capacitor and includes integrated angle-selective gratings to reject oblique light and enhance sharpness. Each pixel also includes two distinct correlated double sampling schemes, to remove low frequency flicker noise and systematic offset in the datapath. We demonstrate the performance of this imager using pulsed NIR-excited upconverting nanoparticles on standard United-States-Air-Force (USAF) resolution targets and achieve an SNR of 15 dB, while keeping NIR background below 6 mV. This 36-by-80-pixel array measures only 2.3 mm by 4.8 mm and can be thinned down to 25 µm, allowing it to become surgically compatible with intraoperative instruments and equipment, while remaining optics-free.

Purpose

Intraoperative detection and removal of microscopic residual disease (MRD) remain critical to the outcome of cancer surgeries. Today's minimally invasive surgical procedures require miniaturization and surgical integration of highly sensitive imagers to seamlessly integrate into the modern clinical workflow. However, current intraoperative imagers remain cumbersome and still heavily dependent on large lenses and rigid filters, precluding further miniaturization and integration into surgical tools.

Procedures

We have successfully engineered a chip-scale intraoperative micro-imager array-without optical filters or lenses-integrated with lanthanide-based alloyed upconverting nanoparticles (aUCNPs) to achieve tissue imaging using a single micro-chip. This imaging platform is able to leverage the unique optical properties of aUCNPs (long luminescent lifetime, high-efficiency upconversion, no photobleaching) by utilizing a time-resolved imaging method to acquire images using a 36-by-80-pixel, 2.3 mm [Formula: see text] 4.8 mm silicon-based electronic imager micro-chip, that is, less than 100-µm thin. Each pixel incorporates a novel architecture enabling automated background measurement and cancellation. We have validated the performance, spatial resolution, and the background cancellation scheme of the imaging platform, using resolution test targets and mouse prostate tumor sample intratumorally injected with aUCNPs. To demonstrate the ability to image MRD, or tumor margins, we evaluated the imaging platform in visualizing a single-cell thin section of the injected prostate tumor sample.

Results

Tested on USAF resolution targets, the imager is able to achieve a resolution of 71 µm. We have also demonstrated successful background cancellation, achieving a signal-to-background ratio of 8 when performing ex vivo imaging on aUCNP-injected prostate tumor sample, improved from originally 0.4. The performance of the imaging platform on single-cell layer sections was also evaluated and the sensor achieved a signal-to-background ratio of 4.3 in resolving cell clusters with sizes as low as 200 cells.

Conclusion

The imaging system proposed here is a scalable chip-scale ultra-thin alternative for bulky conventional intraoperative imagers. Its novel pixel architecture and background correction scheme enable visualization of microscopic-scale residual disease while remaining completely free of lenses and filters, achieving an ultra-miniaturized form factor-critical for intraoperative settings.

Scanning transmission electron microscopy is a common tool used to study the atomic structure of materials. It is an inherently multimodal tool allowing for the simultaneous acquisition of multiple information channels. Despite its versatility, however, experimental workflows currently rely heavily on experienced human operators and can only acquire data from small regions of a sample at a time. Here, we demonstrate a flexible pipeline-based system for high-throughput acquisition of atomic-resolution structural data using an all-piezo sample stage applied to large-scale imaging of nanoparticles and multimodal data acquisition. The system is available as part of the user program of the Molecular Foundry at Lawrence Berkeley National Laboratory.

In photon upconverting core-shell nanoparticles, structure strongly dictates performance. Typical imaging in scanning transmission electron microscopy has sufficient resolution to probe the atomic structure of these nanoparticles, but contrast, dose, and projection limitations make conventional methods insufficient for fully characterizing these structures. Phase retrieval techniques provide a promising alternative imaging mode, and, in particular, multislice electron ptychography can recover depth-dependent information. Here, we study beam-sensitive photon upconverting core-shell nanoparticles with a multislice ptychography approach using a low electron dose to avoid damage. Large strain fields arise in these heterostructures due to the mismatch in lattice parameter between the core and the shell. We reconstruct both a nanoparticle that appears defect-free and one that has a large break in the side and map the distribution of strain in 3D by computing distortion fields from high-resolution potential images of each slice. In the defect-free nanoparticle, we observe twisting of the shell, while in the broken nanoparticle, we measure the 3D position of the crack, the core, and dislocations. These results highlight the advantage of multislice electron ptychography to recover 3D information from a single scan, even under strict electron dose requirements from beam-sensitive samples.

Inorganic nanocrystals such as quantum dots (QDs) and upconverting nanoparticles (UCNPs) are uniquely suited for quantitative live-cell imaging and are typically functionalized with ligands to study specific receptors or cellular targets. Antibodies (Ab) are among the most useful targeting reagents owing to their high affinities and specificities, but common nanocrystal labeling methods may orient Ab incorrectly, be reversible or denaturing, or lead to Ab-NP complexes too large for some applications. Here, we show that SpyCatcher proteins, which bind and spontaneously form covalent isopeptide bonds with cognate SpyTag peptides, can conjugate engineered Ab to nanoparticle surfaces with control over stability, orientation, and stoichiometry. Compact SpyCatcher-functionalized QDs and UCNPs may be labeled with short-chain variable fragment Ab (scFv) engineered to bind urokinase-type plasminogen activator receptors (uPAR) that are overexpressed in many human cancers. Confocal imaging of anti-uPAR scFv-QD conjugates shows the antibody mediates specific binding and internalization by breast cancer cells expressing uPAR. Time-lapse imaging of photostable scFv-UCNP conjugates shows that Ab binding causes uPAR internalization with a ∼20 min half-life on the cell surface, and uPAR is internalized to endolysosomal compartments distinct from general membrane stains and without significant recycling to the cell surface. The controlled and stable conjugation of engineered Ab to NPs enables targeting of diverse receptors for live-cell study of their distribution, trafficking, and physiology.

Spontaneous emulsification, resulting from the assembly and accumulation of surfactants at liquid–liquid interfaces, is an interfacial instability where microdroplets are generated and diffusively spread from the interface until complete emulsification. Here, it is shown that an external magnetic field can modulate the assembly of paramagnetic nanoparticle surfactants (NPSs) at liquid–liquid interfaces to trigger an oversaturation in the areal density of the NPSs at the interface, as evidenced by a marked reduction in the interfacial tension, γ, and corroborated with a magnetostatic continuum theory. Despite the significant reduction in γ, the presence of the magnetic field does not cause stable interfaces to become unstable. Upon rapid removal of the field, however, an explosive ejection of a plume of microdroplets from the surface occurs, a dynamical interfacial instability which is termed explosive emulsification. This explosive event rapidly reduces the areal density of the NPSs to its pre-field level, stabilizing the interface. The ability to externally suppress or trigger the explosive emulsification and controlled generation of tens of thousands of microdroplets, uncovers an efficient energy storage and release process, that has potential applications for controlled and directed delivery of chemicals and remotely controlled soft microrobots, taking advantage of the ferromagnetic nature of the microdroplets.

Materials whose luminescence can be switched by optical stimulation drive technologies ranging from superresolution imaging^1-4, nanophotonics⁵, and optical data storage^6,7, to targeted pharmacology, optogenetics, and chemical reactivity⁸. These photoswitchable probes, including organic fluorophores and proteins, can be prone to photodegradation and often operate in the ultraviolet or visible spectral regions. Colloidal inorganic nanoparticles^6,9 can offer improved stability, but the ability to switch emission bidirectionally, particularly with near-infrared (NIR) light, has not, to our knowledge, been reported in such systems. Here, we present two-way, NIR photoswitching of avalanching nanoparticles (ANPs), showing full optical control of upconverted emission using phototriggers in the NIR-I and NIR-II spectral regions useful for subsurface imaging. Employing single-step photodarkening^10-13 and photobrightening^12,14-16, we demonstrate indefinite photoswitching of individual nanoparticles (more than 1,000 cycles over 7 h) in ambient or aqueous conditions without measurable photodegradation. Critical steps of the photoswitching mechanism are elucidated by modelling and by measuring the photon avalanche properties of single ANPs in both bright and dark states. Unlimited, reversible photoswitching of ANPs enables indefinitely rewritable two-dimensional and three-dimensional multilevel optical patterning of ANPs, as well as optical nanoscopy with sub-Å localization superresolution that allows us to distinguish individual ANPs within tightly packed clusters.

We describe the development, operation, and application of the 4D Camera-a 576 by 576 pixel active pixel sensor for scanning/transmission electron microscopy which operates at 87,000 Hz. The detector generates data at ∼480 Gbit/s which is captured by dedicated receiver computers with a parallelized software infrastructure that has been implemented to process the resulting 10-700 Gigabyte-sized raw datasets. The back illuminated detector provides the ability to detect single electron events at accelerating voltages from 30 to 300 kV. Through electron counting, the resulting sparse data sets are reduced in size by 10--300× compared to the raw data, and open-source sparsity-based processing algorithms offer rapid data analysis. The high frame rate allows for large and complex scanning diffraction experiments to be accomplished with typical scanning transmission electron microscopy scanning parameters.