Abstract

Time-lapse fluorescent microscopy (TLFM) combined with predictive mathematical modelling is a powerful tool to study the inherently dynamic processes of life on the single-cell level. Such experiments are costly, complex and labour intensive. A complimentary approach and a step towards in silico experimentation, is to synthesise the imagery itself. Here, we propose Multi-StyleGAN as a descriptive approach to simulate time-lapse fluorescence microscopy imagery of living cells, based on a past experiment. This novel generative adversarial network synthesises a multi-domain sequence of consecutive timesteps. We showcase Multi-StyleGAN on imagery of multiple live yeast cells in microstructured environments and train on a dataset recorded in our laboratory. The simulation captures underlying biophysical factors and time dependencies, such as cell morphology, growth, physical interactions, as well as the intensity of a fluorescent reporter protein. An immediate application is to generate additional training and validation data for feature extraction algorithms or to aid and expedite development of advanced experimental techniques such as online monitoring or control of cells.

Method

We propose Multi-StyleGAN (Fig. 1) for high-resolution (256 x 256) multi-domain image sequence generation.

Figure 1. Architecture of Multi-StyleGAN. The style mapping network (purple) trans- forms the input noise vector into a latent vector w ∈ W, which in turn is passed to each stage of the generator (yellow) by three dual-styled-convolutional blocks (Fig. 2). The generator predicts a sequence of three consecutive images for both the BF and GFP channels. The U-Net discriminator with ADA distinguishes between real and a fake sequences by making both a scalar and a pixel-wise real/fake prediction. Residual discriminator blocks in gray and non-local blocks in blue.

Figure 2. Dual-styled-convolutional block of the Multi-StyleGAN. The incoming latent vector w is transformed into the style vector s by a linear layer. This style vector modu- lates (mod) the convolutional weights and , which are optionally demodulated (demod) before convolving the (optionally bilinearly upsampled) incoming features of the previous block. Learnable biasses ( and ) and channel-wise Gaussian noise () scaled by a learnable constant ( and ), are added to the features. The final output features are obtained by applying a leaky ReLU activation.

Results

Table 1. Evaluation metrics for Multi-StyleGAN and baselines.

Method	FID (BF)	FVD (BF)	FID (GFP)	FVD (GFP)
Multi-StyleGAN (ours)	33.3687	4.4632	207.8409	30.1650
StyleGAN 2 3D + ADA + U-Net dis.	200.5408	45.6296	224.7860	35.2169
StyleGAN 2 + ADA + U-Net dis.	76.0344	14.7509	298.7545	31.4771

Figure 3. Samples generated by Multi-StyleGAN. Brightfield channel on the top and green fluorescent protein on the bottom.

Figure 4. Latent space interpolation of Multi-StyleGAN. Brightfield channel on the top and green fluorescent protein on the bottom.

Acknowledgements

We thank Markus Baier for aid with the computational setup, Klaus-Dieter Voss for aid with the microfluidics fabrication, and Tim Kircher, Tizian Dege, and Florian Schwald for aid with the data preparation.

Citation