Optoacoustic imaging is a new, real-time feedback and non-invasive imaging tool with increasing application in clinical and pre-clinical settings. Studies using optoacoustic imaging have already discovered its unique capabilities as a diagnostic tool for breast cancer, Crohn’s disease, atherosclerotic carotid plaques, and skin cancer. The DLBIRHOUI project tackles some of the major challenges in optoacoustic imaging to facilitate faster adoption of the technology for clinical use through developed data-driven methodologies and to increase research in the field via massive experimental datasets that are being released.
Optoacoustic (OA) imaging is a relatively new imaging technique that has complementary contrast to existing imaging technologies such as ultrasound. However, several challenges remain that are preventing OA from wider clinical adoption. Currently, customized hardware and imaging setup design are required to obtain high-quality OA images. As a tomographic imaging modality with limited tissue penetration, OA imaging suffers from limited view-associated challenges. Furthermore, various physical parameters relating to the imaged medium, such as speed of sound, are oversimplified during image reconstruction, leading to poor results and reducing the reliability of analytical techniques such as spectral unmixing.
DLBIRHOUI is a collaboration between ETH Zurich and the Swiss Data Science Center (SDSC). It has procured a first-of-its-kind collection of large, well-organized OA datasets. Furthermore, DLBIRHOUI proposed novel methodologies to overcome some of the major challenges in OA through both raw acquired signals and tomographic reconstructions.
A glimpse of challenges and solutions
Despite its vast potential, OA is a challenging field of research due to the know-how and customized hardware necessary to set up and conduct high-quality experiments. This is further exacerbated by the lack of publicly available OA data resulting in a limited number of groups that can actively research OA globally and reducing the ability to fairly compare findings across existing literature. Using our large data expertise, we procured a large OA dataset (OADAT) with an unprecedented size of 2 million instances for the community [Lafci’22]. This dataset contains samples from both volunteers and simulated data collected across multiple transducer arrays and acquisition settings in order to capture a diverse dataset to explore the many challenges in the OA imaging field, shown in Figure 1. Using these datasets, we can trivially generate further synthetic experimental images (Figure 2).
Figure 1: (a) A representation of experimental data acquisition on the forearm. (b) A semi-circle array showinga typical forearm image. The initial bright curve is the skin surface whilethe circular objects are blood vessels. (c) A multisegment array and corresponding sample OA image. (d & e) Representations of array geometries required for certain OA imaging settings and associated challenges.
Figure 2: Feel free to explore our online app and generate experimental images using our trained model for linear array (left) and semi-circle array (right).
Development of easy-to-use, open source, and stable Python packages is crucial for reproducibility and wide adoption of the imaging systems. We, therefore, developed open-source image reconstruction packages for OA (pyoat) and US (pyruct) systems in Python during the project. The “pyoat” package contains back-projection and model-based image reconstruction algorithms. The “pyruct” package includes delay and sum beamformer for reflection ultrasound computed tomography (RUCT). These packages assist users in integrating reconstruction algorithms into their deep learning pipelines using the available frameworks in Python [Lafci’22].
Hybridization of OA systems with other imaging modalities, such as US provide complementary information when assessing several biological conditions. Alternatively, the successful combination of magnetic resonance imaging (MRI) with OA can boost the anatomical, functional and molecular information that can be acquired in hybrid settings. However, the implementation of such systems requires the registration of images from these two modalities. To do so, deep learning methods were implemented, successfully registering MRI and OA images during the project [Hu’22].
Figure 3: A sample image showing (left) an optoacoustic image and its corresponding (right) ultrasound image revealing the common carotid artery and jugular vein as echo-free structures. CCA: common carotid artery; SCM: sternocleidomastoid muscle; SM: strap muscle; IJV: internal jugular vein; L: thyroid lobe. Courtesy of [Merčep’18].
Limited view associated challenges constitute a significant obstacle in OA imaging. Often, linear ultrasound arrays are coupled with lasers to acquire both optoacoustic and ultrasound images, allowing complementary contrast from both modalities (in Figure 3, for example). However, as visible in Figure 1 bottom right, the limited angular coverage of a linear array leads to distorted geometries as well as a plethora of undesired artifacts. Using paired data from our dataset, OADAT, we can optimize a data-driven method for correcting both the distorted geometries and reducing the undesired artifacts (Figure 4).
Figure 4: A sample OA image (left) with limited view related distorted geometries and artifacts and (middle) our proposed solution to correct them. Corresponding wider view experimental array image (right).
Going one step further, we also implemented a generative modeling approach that can be optimized for an image-to-image translation without the use of paired data. This is a crucial benefit that alleviates many of the feasibility problems when acquiring unpaired OA images between different kinds of transducer arrays and settings. Figure 5 provides an example where a corresponding increased angular view from an experimental array still has more artifacts than the neural network generated image. Furthermore, we can also synthesize realistic experimental images from mere simulations, as shown in Figure 6. This is particularly interesting for procuring data with controlled scenes (e.g., generating digital twins).
Figure 5: A sample OA image (left) with limited view and related distorted geometries and artifacts, and (middle) our proposed solution to correct them after an unpaired image translation optimization. A corresponding wider view experimental array image (right) that still inhibits some artifacts.
Figure 6: A sample simulated image (left) and corresponding experimental image (right)
generated using our generative model.
Direct unpaired image-to-image translation can sometimes be unpredictable due to the black-box nature of deep learning (DL) models. Accordingly, we also developed a two-step approach to overcome the limited view problem in OA [Klimovskaia ‘22] using raw OA signals. Specifically, in the first step, a DL model is optimized to find mapping into a common embedding space to encode to, and decode from for a given experimental limited view data and simulated data of matching geometry, hence limited view challenges. In a second step, a separate DL model is optimized to in-paint missing channel information for the simulated data removing the limited view challenges as shown in Figure 7. Different from other work, this approach completely removes the need to acquire any form of experimental data without limited view challenges.
Figure 7: Raw signal data (top left) is in-painted to recover missing angular coverage (top right) using our proposed model. For visualization purposes, corresponding image reconstructions are shown below the OA signals.
The methods and datasets developed throughout the DLBIRHOUI project undoubtedly brought both novel insights to the growing field of data-driven optoacoustics and a large volume of multimodal experimental and synthetic data. We tackled numerous challenges in OA through multiple methodologies using neural networks. The dataset we procured throughout the project will hopefully serve as a benchmark for the various global research groups allowing them to evaluate their methodologies and spark additional interest from other researchers with limited or no access to the customized hardware needed in the field of OA.
About the Author
Dr. Firat Ozdemir, Senior Data Scientist at the Swiss Data Science Center (SDSC).
Firat completed his undergraduate studies in Electronics Engineering at Sabanci University. He later received his MSc. in Electrical and Electronics Engineering from EPFL. He conducted his doctoral studies on medical image segmentation in Computer Vision Lab at ETH Zurich. In between, he visited INRIA (Sophia Antipolis, France) and ABB Corporate Research Center (Baden, Switzerland). His research interests revolve around computer vision and machine learning, with a focus on the medical domain. He has been with SDSC since 2019.
Berkan Lafci, ETH Zurich*
Anna Klimovskaia Susmelj, SDSC*
Xose Luis Dean-Ben, ETH Zurich*
Daniel Razansky, ETH Zurich*
Fernando Perez-Cruz, SDSC
* Affiliations throughout the DLBIRHOUI project
About the SDSC
The Swiss Data Science Center (SDSC) is a joint venture between EPFL and ETH Zurich. Its mission is to accelerate the adoption of data science and machine learning techniques within academic disciplines of the ETH domain, the Swiss academic community at large, and the industrial sector. In particular, it addresses the gap between those who create data, those who develop data analytics and systems, and those who could potentially extract value from it. The center comprises a multi-disciplinary team of data and computer scientists as well as experts in selected domains with offices in Zürich, Lausanne, and Villigen. www.datascience.ch
The author would like to thank Berkan Lafci, Anna Klimovskaia Susmelj, Xose Luis Dean-Ben, and Daniel Razansky.
Lafci, Berkan, Firat Ozdemir, Xosé Luís Deán-Ben, Daniel Razansky, and Fernando Perez-Cruz. “OADAT: Experimental and Synthetic Clinical Optoacoustic Data for Standardized Image Processing.” arXiv preprint arXiv:2206.08612 (2022).
Reference paper for pyruct:
B. Lafci, J. Robin, X. L. Deán-Ben and D. Razansky, “Expediting Image Acquisition in Reflection Ultrasound Computed Tomography,” in IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, vol. 69, no. 10, pp. 2837-2848, Oct. 2022, doi: 10.1109/TUFFC.2022.3172713.
Merčep, X. L. Deán-Ben and D. Razansky, ”Imaging of blood flow and oxygen state with a multi-segment optoacoustic ultrasound array”, Photoacoustics, 10, 48–53, 2018.
Klimovskaia, A., Lafci, B., Ozdemir, F., Davoudi, N., Dean-Ben, X. L., Perez-Cruz, F., & Razansky, D. (2021, December). Signal Domain Learning Approach for Optoacoustic Image Reconstruction from Limited View Data. In Medical Imaging with Deep Learning.
Reference paper for MRI and OA registration:
Yexing Hu, Berkan Lafci, Artur Luzgin, Hao Wang, Jan Klohs, Xose Luis Dean-Ben, Ruiqing Ni, Daniel Razansky, and Wuwei Ren, “Deep learning facilitates fully automated brain image registration of optoacoustic tomography and magnetic resonance imaging,” Biomed. Opt. Express 13, 4817-4833 (2022)
Cover Image courtesy of Elena Mercep: Transmission–reflection optoacoustic ultrasound (TROPUS) computed tomography of small animals , E Merčep, JL Herraiz, XL Deán-Ben, D Razansky, Light: Science & Applications 8 (1), 18, 2019