Theme C | Intelligent Scientific Image Analysis

Theme Description

This research theme aims to improve imaging hardware and software to enable large-scale, automated acquisition and analysis of image data in science and engineering. The grand challenge of this theme is to develop fully automated methods for capturing, processing, and analyzing visual data to gain scientific insight. Our work includes both foundational research on new methodology as well as applications to specific science and engineering problems in collaboration with domain experts.

The Intelligent Scientific Image Analysis theme encompasses the following problems:
  • Computational Imaging, i.e. the hardware-software co-design of imaging systems, including both the optical design and the computational reconstruction algorithms.

  • Inverse Problems, such as computed tomography. Topics of interest include new computational algorithms, statistical priors, or machine learning techniques for ill-posed problems like limited angle tomography and space-time reconstruction of deforming samples.

  • Steered Adaptive Image Acquisition, using on-the-fly processing of partial capture data to direct the acquisition process of further data towards regions of interest in an attempt to minimize capture time, bandwidth, and storage requirements.

  • Computer Vision, including object segmentation, object detection, and anomaly detection in 2D and 3D scientific datasets.

  • Fully Automated Analysis of Scientific Image Data, using machine learning techniques to directly extract relevant features from the captured and reconstructed data to drive scientific and engineering decisions without human intervention.

‌4D space-time tomography becomes necessary when the object deforms during the scanning process, for example, due to loss of water in organic samples, degradation of the sample due to radiation exposure (e.g. in electron microscopy), or even just due to expansion and contraction under temperature changes, for example as a CT scanner heats up over long periods of operation.

Wilting Rose
4D space-time Tomography of a wilting rose.


Example Projects

4D Space-Time Computed Tomography

A large research project in the VCC is the development of new tomographic reconstruction methods for difficult and ill-posed setting such as

  • Super-resolution reconstruction of features below the Nyquist limit of the image sensor.
  • Sparse-view reconstruction from as few as 3 projection images.
  • Limited angle tomography, where the projection images cover only a small cone of directions instead of the full 360 degrees, and
  • 4D space-time tomography of objects and phenomena undergoing deformation during the scanning process.

The same techniques can also be applied to tomographic imaging of dynamic phenomena. For example, VCC Professors Wolfgang Heidrich and Peter Wonka have collaborated with Professor Gilles Lubineau in Mechanical Engineering on dynamic loading experiments for composite materials in material science.

Space-Time Tomography
4D space-time tomography of a copper foam undergoing dynamic compression.

These methods are not limited to just X-ray computed tomography, but can also be applied for example to visible light imaging, for example in combustion research. In this setting, matters are further complicated by the fact that the number of projection images (i.e. cameras) is typically very limited. Here we show work with Professor William Roberts from the Clean Combustion Research Center on 4D space-time reconstruction of soot exhaust gases from a combustion process, using only 3 views.

Soot Somography
Sparse view, space-time tomographic imaging of a soot plume in a combustion process.

Active Learning for  Object Detection

In collaboration with Plant Scientist Prof. Salim Al-Babili, the VCC, under the leadership of Prof. Bernard Ghanem has developed an active learning framework that will automatically detect/count states of particular seeds (e.g. germinated vs. non-germinated) over time. This counting was previously the bottleneck for the plant science research since it was a tedious manual process that limited the overall number of processed specimens. Using our active learning framework, we achieve similar accuracy as human experts' orders of magnitude faster.

Seed Detection
Automatic seed detection showing seeds (left) and radicals (right)


A technologically similar project is a collaboration between the VCC (Prof. Bernard Ghanem) and Prof. Pierre Magistretti, who is working on brain energy metabolism. Prof. Magistretti is developing chemical agents that are added to neurons grown in a culture to see if they induce an increased growth of spines on the dendrites. Such spine growth has been linked to better memory capabilities, so that the corresponding agents may in the future be of help with dementia and Alzheimers. In order to check the effectiveness of the agent, it is necessary to count the number of spines before and after the agent is introduced – again a bottleneck in time. We have used the same framework as before to detect these spines for them and are getting state-of-the-art performance on standard benchmarks.
Dendritic Spines
Detection and counting of dendritic spines in brain tissue.