Project Plan

Speeding Up Examination of Heart Diseases with 3D Echocardiography and Machine Learning

View in PDF

Background

According to the World Health Organization, heart diseases are the leading cause of death and hospitalization globally. Nearly 18 million people worldwide die from heart diseases annually, accounting for 31 percent of all global deaths.

The major diagnosis tool for heart diseases is 2D echocardiography, a non-invasive, portable, yet highly time-consuming approach that results in an average waiting time of 81 weeks for an echo examination in Hong Kong public hospitals.

Thus, our project mainly focuses on speeding up the examination of heart diseases using 3D echocardiography and machine learning. It is the world’s first AI software for automatic reconstruction of the ASE-recommended cross-section views of the heart from 3D echo data. Theoretically, our project could shorten the average waiting time for an echo examination from 81 weeks to less than 10 weeks.

Objective

The essential objective of our project is speeding up the examination of heart diseases by using 3D echocardiography and machine learning. A medical research group in CUHK has finished the former part, i.e., 3D echocardiography part. They have collected 4D representations of the hearts and mark the 32 landmarks of each sample.

We have acquired their marked data and will use them to train and evaluate deep CNNs for predicting the view landmarks. We will fine-tune CNN frameworks, e.g. FCNN [1], SpatialConfiguration-Net [2], and T²DL [3]. After obtaining an accurate network, we will use the predicted landmarks to reconstruct the cross-section views, which can help the doctors to make diagnosis of heart diseases and ultimately achieve our goal of speeding up the examination of heart diseases.

Methodology

In this section, we discuss methods to recover cross-section views from 3D echocardiography data.

As each cross-section view is defined by multiple cardiac landmarks, recovering the positions of the landmarks suffices to recover the cross-section views, e.g. knowing the positions of A2C-LV apex, Anterolateral mitral annulus, and Posteromedial mitral annulus, we can easily define the 2 chamber view that passes through these three landmarks. Therefore, our primary focus would be on using machine learning techniques to predict the positions of the landmarks.

Data Preparation

The raw data in this project comprises hundreds of echocardiography in 4D that display the whole 3D heart in time series. In each echocardiography, the locations of the 32 landmarks at a specific time point are labeled by experienced medical practitioners. But there are seldom cases where some landmarks are not labeled due to their vagueness. Therefore, we extract 3D time slices when the landmarks are labeled to compose our dataset.

After obtaining the dataset, we follow conventional machine learning pre-processing methods to perform data cleansing and standardization. And we will finally form our training, validation, and test set.

Landmark Prediction

We plan to use CNN to predict the positions of the landmarks. However, the kernels of preliminary CNN frameworks tend to have restricted receptive fields, meaning that the networks focus more on local but not global features. As a result, the relative positions of the landmarks, which are usually strong indicators of their absolute positions, may be ignored.

Therefore, we plan to experiment with different improved CNN frameworks that combine local and global features. The tentative candidates include FCNN [1], SpatialConfiguration-Net [2], and T²DL [3]. These networks typically use different parts of the structures to learn different levels of detail, which would be suitable for our purposes. In practice, We will modify these network structures so that they work on our dataset. After fine-tuning the networks, we will benchmark their performance and make side-by-side comparisons on the error distances.

Cross-section Recovery

After obtaining the predicted locations of the landmarks from our networks, we will use traditional computer vision techniques to recover the cross-sections and visualize the views.

Noticeably, some cross-section views contain more than three landmarks, e.g. there are 6 landmarks on the MV short-axis view. In such cases, the predicted landmarks may not fall on the same plane. To deal with the problem, we may either use linear least square fitting to find the best plane of the cross-section or use the three landmarks with the top confidence to define the plane. We will compare the performance of different solutions and select the more suitable one.

At last, we will render 2D views from these calculated cross-section planes to visually examine the degree of satisfaction of these recovered cross-section views.

Schedule and Milestones

In this section, we will give the tentative schedule of the project, i.e. when and what will be achieved at various stages of the project.

Initial Preparation (Sep 1, 2022 - Sep 30, 2022)

We will spend the first month doing some basic preparation work, including reading relevant literature, learning about other people’s work in this area, and selecting a few advisable networks as our training baselines to optimize on top of them later. We will also divide the work so that we can move forward with the project efficiently afterward.

Milestones:

• Determine the networks we are going to train;
• Divide the work to each group member.

Data Preparation (Oct 1, 2022 - Oct 31, 2022)

We will perform data preparation in the second month. The data we obtain are 4D echocardiogram images in NRRD format, which cannot be used directly. On the one hand, some samples do not contain complete data, with partial landmarks not labeled. Hence they cannot be used for network training and should be excluded. On the other hand, we also need to complete the codes to read and pre-process these NRRD data so that they can be turned into acceptable input for the neural network.

Milestones:

• Select the samples with complete data;
• Complete the codes to read and pre-process the data.

Landmark Prediction (Nov 1, 2022 - Jan 22, 2023)

In the next two months, we will use the data pre-processed in the previous phase and feed them into different neural networks to obtain some key information, such as the number of iterations needed for each network to converge and the accuracy rate after their convergence. Additionally, we will fine-tune the networks’ parameters, evaluate their performance, and compare different networks. These research findings will be explained at the first presentation in early January.

Milestones:

• Train different networks;
• Fine-tune networks and compare them;
• Complete the slides for the first presentation (Jan 9-13, 2023);
• Complete the detailed interim report (Jan 22, 2023).

Cross-section Recovery (Jan 23, 2023 - Apr 21, 2023)

In the last phase of the project, we will use the results obtained from the previous landmark prediction phase to recover the desired cross-section planes. The details of recovery are given in the methodology section. Furthermore, we will continue to try new methods to modify and improve our network to obtain more useful information. These findings will be used to benefit speeding up the examination of heart diseases.

Milestones:

• Recover the cross-section planes and visualize them;
• Modify and improve the network continuously to obtain useful information;
• Complete the final report (Apr 18, 2023);
• Complete the slides for the final presentation (Apr 17-21, 2023).

References

[1] Julia MH Noothout, Bob D de Vos, Jelmer M Wolterink, Tim Leiner, and Ivana Išgum. Cnn-based landmark detection in cardiac cta scans. arXiv preprint arXiv:1804.04963, 2018.
[2] Christian Payer, Darko Štern, Horst Bischof, and Martin Urschler. Regressing heatmaps for multiple landmark localization using cnns. In International conference on medical image computing and computer-assisted intervention, pages 230–238. Springer, 2016.
[3] Jun Zhang, Mingxia Liu, and Dinggang Shen. Detecting anatomical landmarks from limited medical imaging data using two-stage task-oriented deep neural networks. IEEE Transactions on Image Processing, 26(10):4753–4764, 2017.

— Sep 29, 2022

Project Plan by Lu Meng is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. Permissions beyond the scope of this license may be available at About.