Writing | Huang Longqian
Recently, from the School of Optoelectronics, Zhejiang University Hao Xiang The team of researchers is "Spectral Imaging with Deep Learning"The topic is Lighting: Science & Applications Published a review paper, sorted out the spectral imaging technology based on deep learning, explained the principle and research summary of its various technical routes, and organized the current spectral imaging dataset, outlining the possible future trends and challenges.
First, spectral images
The image we usually see in the display screen is an RGB three-color image, which has three channels of red, green and blue, that is, the color of each pixel is formed by superimposing the three colors of red, green and blue. If the three-color image is expanded to ten colors, hundred colors, and thousand colors, then our control and rendering of the image color will be extremely fine. An image with so many bands is a spectral image (term explanation >): it has more than three bands, which can be thirty or even hundreds.
Figure 1 Schematic diagram of a spectral image composed of 9 bands
The spectral information of a substance can fully reflect its internal physical structure and chemical composition. When there are enough wavelength bands in the image, each part is a spectral curve – reflecting the structural composition of the place. Therefore, the spectral image not only reflects the spatial characteristics of the imaging target, but also contains its structure and composition characteristics. Therefore, in addition to fine image rendering, the key application of spectral images is material composition analysis, which is widely used in remote sensing, medical detection, food detection and other fields.
Second, spectral imaging
Spectral imaging is a combination of imaging technology and spectral detection technology, and the main technical goal is to obtain spectral images. Traditional spectral imaging technology is scanning(Including point scanning, line scanning, band scanning)By constantly moving the spectrometer to obtain the spectrum of each point, a two-dimensional image is finally obtained. This scanning spectral imaging system is bulky and slow, which has been criticized.
Figure 2 Scanning spectral imaging (from left to right, point scan, first scan, and band scan)
With the development of computational optics, computational spectral imaging has become a new spectral imaging method. The method uses some coding to make the hidden spectral information enter the detector in a certain way, and finally uses the optimization algorithm to reconstruct the spectral image. The traditional computational spectral imaging method is iterative. Although it has effectively reduced the size of the system compared with the scanning spectral imaging, it has a great computational burden, often requires several minutes or even hours of iteration, and the reconstruction accuracy is relatively average.
The huge burden of data computing prompts people to think and look for more effective spectral reconstruction algorithms. In recent years, deep learning has developed rapidly, and the powerful pattern recognition and feature extraction capabilities of deep neural networks after good training have been favored by spectral imaging researchers. Deep learning methods have become the object of exploration, and deep learning has gradually empowered spectral imaging.
Methods of deep learning
The process of computational spectral imaging coding sampling system design and spectral image reconstruction, deep learning technology can be applied in every aspect of the process. How to effectively classify these different techniques is a key issue. The review proposes to classify according to the properties of light, that is, the computational spectral imaging system is divided into amplitude coding, phase coding and wavelength coding according to different coding methods.
(1) Amplitude-encoded spectral imaging
Figure 3: The spectral coding process in a coding aperture system
Amplitude-coded spectral imaging is achieved through a coded aperture system(CASSI)Conducted, using coding aperture(Amplitude mask)Amplitude coded spectral imaging based on deep learning replaces the iterative algorithm of compressed sensing recovery with deep neural networks, and performs efficient spectral reconstruction through codec co-optimization, iterative expansion neural networks, and untrained networks.
(2) Phase-encoded spectral imaging
Figure 4: A phase-encoded spectral imaging system
Phase-encoded spectral imaging through diffractive optics(Diffraction Optical Elements, DOE)Phase coding is achieved by designing the two-dimensional height profile of the DOE to achieve specific phase delays at different positions. Phase coding affects different spectral components through Fresnel diffraction, and then by modeling the diffraction process, the original spectral image can be reconstructed by the algorithm. Due to the complexity of diffraction calculation after phase coding, traditional iterative algorithms are difficult to achieve effective recovery of spectral images. This problem has been solved after the emergence of deep learning. At present, the spectral recovery of phase coding is mainly carried out through deep neural networks. Compared with amplitude coding, phase-encoded spectral imaging has the advantages of small light energy loss and compact system.
(3) Wavelength-encoded spectral imaging
Wavelength-encoded spectral imaging directly encodes the image in the spectral dimension and can be performed through optical filters. RGB images can be regarded as a kind of spectral coding. At present, the mainstream wavelength coding methods use existing RGB or design optical filters, and the encoded spectral reconstruction mostly relies on deep learning technology.
Direct spectral reconstruction based on RGB images is a very hot direction. With the NTIRE 2018 and NTIRE 2020 spectral reconstruction competitions, many deep learning technology teams participated, greatly expanding the existing deep learning spectral restoration techniques. The researchers analyzed the RGB spectral reconstruction and filter design spectral reconstruction involved in deep learning, and divided the reconstruction methods into point reconstruction and block reconstruction, so as to introduce each reconstruction method.
Figure 5 Codec Cooperative Design Spectral Imaging Technology [
1] [
[2] (Source: Advanced Theory and Simulations)
Wavelength coding based on custom optical filters is an emerging spectral imaging technology in recent years. By designing wide-spectrum filters, greater coding freedom can be obtained than RGB filters. Combined with effective deep learning technology, compact, fast and accurate spectral recovery can be achieved.
IV. Future Outlook
Based on current deep learning techniques and spectral imaging techniques, the researchers have made some suggestions for the future development of deep learning-based spectral imaging. These include the large-scale and normalization of datasets, and the use of large neural networks(For example Transformers)Introduce additional information such as object categories, solve the problem of layer disappearance in the process of codec collaborative design, and use neural networks architecture search technology.
Paper information
Huang, L., Luo, R., Liu, X. et al. Spectral imaging with deep learning. Light Sci Appl 11, 61 (2022).
https://doi.org/10.1038/s41377-022-00743-6
references
[1] Song H, Ma Y, Han Y, et al. Deep ? Learned Broadband Encoding Stochastic Filters for Computational Spectroscopic Instruments [J]. Advanced Theory and Simulations, 2021, 4 (3): 2000299.
[2] Zhang W, Song H, He X, et al. Deeply learned broadband encoding stochastic hyperspectral imaging [J]. Light: Science & Applications, 2021, 10 (1): 1-7.
Contributions from the research group are welcome – press release
Article reprint/business cooperation/research group submission, WeChat:447882024
Let’s read one article a day!Join >Light reading club
关于作者