DETAILED ACTION
Response to Arguments
The amendments filed 12/17/2025 have been entered and made of record.
The Applicant's amendments and arguments filed 12/17/2025 have been considered but are moot in view of the new ground(s) of rejection because the Applicant has amended at least independent claim 1, and Applicant's arguments filed 12/17/2025 have been fully considered but they are not persuasive:
In the Applicant’s Remarks (on pages 5-6 of 10), Applicant also asserts that none of the cited references, Wilson, Yang and Hsiao disclose wherein the generative network comprises a feature extraction module and an upsampling module,
the feature extraction module comprises a convolution layer, cascaded convolution blocks, then passing through a convolution layer, and finally obtaining a low-resolution feature map from the low-quality CT image; each convolution block in the cascade convolution blocks comprises at least two convolution layers and a middle ReLU layer, and
the upsampling module comprises a fully connected network and a convolution layer, and each pixel position information of the input high-quality CT image is inputted into the fully connected network, and output result of the fully connected network is applied to the low- resolution feature map to obtain the high-quality high-resolution image;
However, the Examiner disagrees, because:
Wilson discloses the generative network comprises a feature extraction module and an upsampling module (see Wilson: e.g., Fig. 14, and, --[0072] Super-resolution from deep learning. Another technique that can be employed by various embodiments involves using a deep learning, generative adversarial network (GAN) to create super-resolution CT calcium score images from conventional, thick slice scans. For training, paired normal clinical resolution and high resolution volume scans can be used. These can be easily obtained with existing step and shoot multi-detector CT scan acquisition. The images can be simply reconstructed at both conventional thick slice and high resolution thin slice resolutions, giving perfectly paired data. After training, the network can create a high resolution scan volume from a low resolution clinical scan volume. Using the super-resolution volume, ICmore measures can be computed more accurately and precisely. GANs can be used to reduce noise in CT images, create super-resolution CT images, and create super-resolution MR images.
[0073] In various embodiments, different schemes can be used to create super-resolution image volumes from deep neural networks, such as the following example embodiment. This example embodiment can employ a modified version of a multi-level, densely connected super-resolution network (mDCSRN) with generative adversarial network (GAN)-guided training. By utilizing a densely connected network, the mDCSRN is extremely light-weight, and when trained with a Generative Adversarial Network (GAN), can create sharp and realistic-looking images. In various embodiments, existing network structures can be employed. This technique can employ a loss (or cost) function such as that given below in equation (2), where MSE is the mean square error between the actual and synthetic high resolution image volumes, lossCAC is the difference between the calcification mass score computed form the actual and synthetic high resolution images, lossGAN is the GAN discriminator loss, and the λ's are empirically determined constants:--, in [0072]-[0073], and, and, --[0095] Additionally or alternatively, various embodiments can employ deep learning. For example, various embodiments can extract deep features from each time point by using a pretrained convolutional neural network (e.g., such as ResNet50, Vgg16, or DenseNet-169). For each individual calcification, a bounding box can be constructed which is centered at the calcification. ICmore deep features of each calcification can be extract from the corresponding bounding box.--, in [0095]),
and, Wilson’s disclosures of the generative adversarial network (GAN) include 1) the feature extraction module, such as listed as in: “extract deep features from each time point by using a pretrained convolutional neural network (e.g., such as ResNet50, Vgg16, or DenseNet-169). For each individual calcification, a bounding box can be constructed which is centered at the calcification”;
and It is well known of VGG 16 architecture:
PNG
media_image1.png
200
400
media_image1.png
Greyscale
although Wilson does not explicitly disclose a convolution layer, cascaded convolution blocks, then passing through a convolution layer, and finally obtaining a low-resolution feature map from the low-quality CT image; each convolution block in the cascade convolution blocks comprises at least two convolution layers and a middle ReLU layer;
Hsiao discloses feature extraction module comprises a convolution layer, cascaded convolution blocks, then passing through a convolution layer, and finally obtaining a low-resolution feature map from the low-quality CT image; each convolution block in the cascade convolution blocks comprises at least two convolution layers and a middle ReLU layer (see Hsiao: e.g.,
PNG
media_image2.png
1014
686
media_image2.png
Greyscale
and see: -- CNNs have a large capacity for recalling learned features, and might supply a priori information and assumptions during inference. Based on these capabilities, CNNs should have the ability to recover high-resolution spatial detail from low-resolution images and accomplish the otherwise impossible task of single-plane super-resolution, using either a single or multiple temporal frames.--, in [0008], and
--a relatively shallow network is a SRCNN-inspired neural network with a custom hybrid loss function, which is referred to as “k-SRNet”, based on the Shallow Wide ResNet that was described by R. Yasrab (“SRNET: A Shallow Skip Connection Based Convolutional Neural Network Design for Resolving Singularities”, Journal of Computer Science and Technology 34, 924-938 (2019), which is incorporated herein by reference.) The second, a deeper, more complex network is a modified UNet, referred to as “k-UNet”. For this network, the final activation function is set to the hyperbolic tangent; a custom hybrid loss function is employed. UNet is a fully convolutional network which is known in the art, its name derived from its u-shaped architecture, as shown. (See, O. Ronneberger, et al., “U-Net: Convolutional Networks for Biomedical Image Segmentation”, (2015) arXiv: 1505.04597 [cs.CV], which incorporated herein by reference). Briefly, the UNet employs repeated application of convolutions, each followed by a rectified linear unit (ReLU) and a max pooling operation. During the contraction, the spatial information is reduced while feature information is increased. The expansive pathway combines the feature and spatial information through a sequence of up-convolutions and concatenations with high-resolution features from the contracting path. In implementations of the inventive scheme, neural networks were initially trained to perform super-resolution in image-space using synthetically-generated low-resolution data, including relatively shallow (SRNet) and relatively deep (UNet) convolutional neural networks (CNNs).
[0031] Further performance improvement was evaluated by training these neural networks to perform single-frame (k-) and multi-frame (kt-) super-resolution. Using the hypothesis that additional data from neighboring time points might improve performance, both architectures were extended to incorporate 3-dimensional convolutions, handling the temporal domain in the third dimension. Each input frame was combined with immediately flanking frames to generate input volumes. These spatiotemporal versions of k-SRNet and k-UNET are referred to as “kt-SRNet” and “kt-UNet”, respectively. The architectures of each of these networks are illustrated in FIGS. 1B-1F where layer dimensions (e.g., 128×128) are provided beside the vertical gray bars and the numbers of channels (e.g., 1, 32, 64, etc.) are shown above the gray bars. In FIGS. 1D and 1F, the 3D gray bars in kt-SRNet and kt-UNet, respectively, emphasize use of 3D convolutions. Referring to FIG. 1C, a SRNet is shown with four convolutions: (1) conv 9×9, ReLU (rectified linear activation unit); (2) conv 5×5, ReLU; (3) conv 5×5, ReLU; and (4) conv 1×1, tanh (hyperbolic tangent activation function). This network was used for single-frame super-resolution. FIG. 1D illustrates the architecture a multi-frame SRNet with four convolutions: (1) conv 3×9×9, ReLU; (2) and (3) conv 3×5×5, ReLU, and (4) conv 3×1×1, tanh. (where the “3” represents 3D convolution, which in the evaluation involved the use of three temporally adjacent cropped undersampled input images). In the final step, the central timepoint is extracted to provide a 2D output. FIG. 8 provides an alternative architecture for multi-frame resolution with three 3D convolutions as shown to generate a 2D image.--, in [0030]-[0032]);
Wilson (as modified by Yang) and Hsiao are combinable as they are in the same field of endeavor: monitoring patients using machine learning model in medical imaging and image processing and analysis. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wilson (as modified by Yang)’s method using Hsiao’s teachings by including feature extraction module comprises a convolution layer, cascaded convolution blocks, then passing through a convolution layer, and finally obtaining a low-resolution feature map from the low-quality CT image; each convolution block in the cascade convolution blocks comprises at least two convolution layers and a middle ReLU layer to Wilson (as modified by Yang)’s applications of learning model and image processing, such as VGG16 architecture in order to train neural networks and learning model to perform super-resolution in image-space using synthetically-generated low-resolution data (see Hsiao: e.g., in [0030]-[0033], and [0044]);
Wilson (as modified by Yang)’s disclosures of the upsampling module comprises a fully connected network and a convolution layer, and each pixel position information of the input high-quality CT image is inputted into the fully connected network (see Wilson: e.g., Fig. 14, and, --[0072] Super-resolution from deep learning. Another technique that can be employed by various embodiments involves using a deep learning, generative adversarial network (GAN) to create super-resolution CT calcium score images from conventional, thick slice scans. For training, paired normal clinical resolution and high resolution volume scans can be used. These can be easily obtained with existing step and shoot multi-detector CT scan acquisition. The images can be simply reconstructed at both conventional thick slice and high resolution thin slice resolutions, giving perfectly paired data. After training, the network can create a high resolution scan volume from a low resolution clinical scan volume. Using the super-resolution volume, ICmore measures can be computed more accurately and precisely. GANs can be used to reduce noise in CT images, create super-resolution CT images, and create super-resolution MR images.
[0073] In various embodiments, different schemes can be used to create super-resolution image volumes from deep neural networks, such as the following example embodiment. This example embodiment can employ a modified version of a multi-level, densely connected super-resolution network (mDCSRN) with generative adversarial network (GAN)-guided training. By utilizing a densely connected network, the mDCSRN is extremely light-weight, and when trained with a Generative Adversarial Network (GAN), can create sharp and realistic-looking images. In various embodiments, existing network structures can be employed. This technique can employ a loss (or cost) function such as that given below in equation (2), where MSE is the mean square error between the actual and synthetic high resolution image volumes, lossCAC is the difference between the calcification mass score computed form the actual and synthetic high resolution images, lossGAN is the GAN discriminator loss, and the λ's are empirically determined constants:--, in [0072]-[0073]; also see Yang: e.g., -- for CT image denoising by designing a perceptive deep CNN that relies on a perceptual loss as the objective function. During our research, it was drawn to our attention that minimizing MSE between the denoised CT image and the ground truth leads to the loss of important details, although the peak signal to noise ratio (PSNR) based evaluation numbers are excellent. That is because PSNR is equivalent to the per-pixel Euclidean intensity difference. Therefore, a model maximizing PSNR after successful training always achieves very high PSNR values.--, in page 2, also see: Section 2.2 Network Architecture, in pages 3-4, also see: --the recently popularized deep learning techniques, aim for minimizing mean-squared-error (MSE) between a denoised CT image and the ground truth--, in abstract, and, --The proposed network was implemented and trained using the Caffe toolbox [8]. At the training phrase, we randomly extracted and selected 100,000 image patches of size 32 × 32 from 2,600 CT images. We first trained a CNN with the same structure as shown in Fig. 1 but using the mean-square-error (MSE) loss, which is named CNN-MSE.--, under Section 3.1 Materials and Network Training, in page 4); and,
Hsiao discloses the upsampling module comprises a fully connected network and a convolution layer, and each pixel position information of the input high-quality image is inputted into the fully connected network, and output result of the fully connected network is applied to the low- resolution feature map to obtain the high-quality high-resolution image (see Hsiao: e.g., --[0022] FIGS. 3A and 3B are box-and-whisker plots comparing performance (SSIM) for each super-resolution method across multiple upsampling factors. FIG. 3A plots aggregate performance for each super-resolution method. FIG. 3B plots pairwise comparison of performance between each method and zero-padding (z-pad).
[0023] FIG. 4 provides an example comparing super-resolution methods across multiple upsampling factors from 2× to 64×.
[0024] FIGS. 5A-5D are examples comparing super-resolution methods at 8× upsampling. Output images and corresponding logplots of k-space are shown along with SSIM relative to ground truth.--, [0022]-[0024], and [0033]; also see:
--[0044] A unique set of UNets and SRNets were trained for multiple degrees of upsampling, from 2× to 64×. CNN-based approaches, SRNet and U-Net, and conventional methods of bicubic interpolation and z-padding were compared by calculating the Structural Similarity Index (SSIM) between each ground truth image and its corresponding super-resolved image from each method of upscaling. …in Table 1. For synthetic test data, differences between SRNet and UNet were small, but statistically significant. For all degrees of upsampling, k-UNet and kt-UNet outperformed k-SRNet and kt-SRNet, respectively. similar performance of all CNNs was observed across upsampling factors, in contrast to a widening performance gap with conventional upscaling methods with higher upsampling factors (FIG. 3A).
[0049] The performance of each of the upsampling strategies was further explored by examining representative case examples. DL-based methods consistently outperform conventional methods on both a bulk and a per-slice basis. Neural network-based methods outperformed traditional bicubic and zero-padding for nearly every slice evaluated. Zero-padding outperforms bicubic for nearly every slice evaluated.
[0050] FIG. 4 provides an example comparing super-resolution methods across different upsampling factors from 2× to 64×. At upsampling factors of 2× and 4×, the qualitative difference between each of the methods and ground truth was subtle. At 8× upsampling, bicubic interpolation showed a noticeable degradation of image quality, notably in the sharpness of the right ventricular trabeculations, and the myocardium-blood pool interface.--, in [0044]-[0050]);
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wilson (as modified by Yang)’s method using Hsiao’s teachings by including the upsampling module comprises a fully connected network and a convolution layer, and each pixel position information of the input high-quality image is inputted into the fully connected network, and output result of the fully connected network is applied to the low- resolution feature map to obtain the high-quality high-resolution image to Wilson (as modified by Yang)’s applications of learning model and image processing and creating super-resolution CT calcium score images in order to train neural networks and learning model to perform super-resolution in image-space using synthetically-generated low-resolution data (see Hsiao: e.g., in [0030]-[0033], [0044], and [0047]-[0050]);
Therefore, claims 1-8, and 10 are still not patentably distinguishable over the prior art reference(s). Further discussions are addressed in the prior art rejection section below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-8, and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Wilson (US 20200273167 A1), in view of Yang (“CT Image Denoising with Perceptive Deep Neural Networks”, Arxiv.Org Cornell University Library 201 Olin Library Cornell University Ithaca NY, (Feb. 22, 2017), 8 pgs), and further in view of Hsiao (US 20220114699 A1, claims the priority of US-Provisional-Application US 63090154 20201009).
Re Claim 1, Wilson discloses a method for enhancing quality and resolution of CT images based on deep learning (see Wilson: e.g., Fig. 14, and, --[0072] Super-resolution from deep learning. Another technique that can be employed by various embodiments involves using a deep learning, generative adversarial network (GAN) to create super-resolution CT calcium score images from conventional, thick slice scans. For training, paired normal clinical resolution and high resolution volume scans can be used. These can be easily obtained with existing step and shoot multi-detector CT scan acquisition. The images can be simply reconstructed at both conventional thick slice and high resolution thin slice resolutions, giving perfectly paired data. After training, the network can create a high resolution scan volume from a low resolution clinical scan volume. Using the super-resolution volume, ICmore measures can be computed more accurately and precisely. GANs can be used to reduce noise in CT images, create super-resolution CT images, and create super-resolution MR images.
[0073] In various embodiments, different schemes can be used to create super-resolution image volumes from deep neural networks, such as the following example embodiment. This example embodiment can employ a modified version of a multi-level, densely connected super-resolution network (mDCSRN) with generative adversarial network (GAN)-guided training. By utilizing a densely connected network, the mDCSRN is extremely light-weight, and when trained with a Generative Adversarial Network (GAN), can create sharp and realistic-looking images. In various embodiments, existing network structures can be employed. This technique can employ a loss (or cost) function such as that given below in equation (2), where MSE is the mean square error between the actual and synthetic high resolution image volumes, lossCAC is the difference between the calcification mass score computed form the actual and synthetic high resolution images, lossGAN is the GAN discriminator loss, and the λ's are empirically determined constants:--, in [0072]-[0073]),
characterized by comprising steps of: S1, pre-processing collected clinical data to obtain a data set (see Wilson: e.g., --performing pre-processing on the CT image volume obtained at 110 to improve the CT image volume. The pre-processing can comprise one or more of the techniques discussed herein, such as partial volume correction, deconvolution, or generating a super-resolution CT image volume based on the obtained CT image volume.--, in [0030], [0047], and, --wherein the operations further comprise performing pre-processing on the CT image volume to generate a pre-processed CT image volume, wherein identifying the one or more calcification candidates based on the CT image volume comprises identifying the one or more calcification candidates based on the pre-processed CT image volume.--, in [0111]);
S2, building a deep learning model comprising a generative network, a discriminator network (see Wilson: e.g., Fig.14, and, --[0072] Super-resolution from deep learning. Another technique that can be employed by various embodiments involves using a deep learning, generative adversarial network (GAN) to create super-resolution CT calcium score images from conventional, thick slice scans. For training, paired normal clinical resolution and high resolution volume scans can be used. These can be easily obtained with existing step and shoot multi-detector CT scan acquisition. The images can be simply reconstructed at both conventional thick slice and high resolution thin slice resolutions, giving perfectly paired data. After training, the network can create a high resolution scan volume from a low resolution clinical scan volume. Using the super-resolution volume, ICmore measures can be computed more accurately and precisely. GANs can be used to reduce noise in CT images, create super-resolution CT images, and create super-resolution MR images.
[0073] In various embodiments, different schemes can be used to create super-resolution image volumes from deep neural networks, such as the following example embodiment. This example embodiment can employ a modified version of a multi-level, densely connected super-resolution network (mDCSRN) with generative adversarial network (GAN)-guided training. By utilizing a densely connected network, the mDCSRN is extremely light-weight, and when trained with a Generative Adversarial Network (GAN), can create sharp and realistic-looking images. In various embodiments, existing network structures can be employed. This technique can employ a loss (or cost) function such as that given below in equation (2), where MSE is the mean square error between the actual and synthetic high resolution image volumes, lossCAC is the difference between the calcification mass score computed form the actual and synthetic high resolution images, lossGAN is the GAN discriminator loss, and the λ's are empirically determined constants:--, in [0072]-[0073], and [0110]),
Wilson however does not explicitly disclose a perceptive network;
Yang discloses including a perceptive network {in CT image processing and analysis} (see YANG: e.g., -- for CT image denoising by designing
a perceptive deep CNN that relies on a perceptual loss as the objective function.
During our research, it was drawn to our attention that minimizing MSE between the denoised CT image and the ground truth leads to the loss of important
details, although the peak signal to noise ratio (PSNR) based evaluation numbers are excellent. That is because PSNR is equivalent to the per-pixel Euclidean
intensity difference. Therefore, a model maximizing PSNR after successful training always achieves very high PSNR values.--, in page 2, also see: Section 2.2 Network Architecture, in pages 3-4);
Wilson and Yang are combinable as they are in the same field of endeavor: monitoring patients using learning model and neural networks in CT images reconstructions and enhancement in resolution. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wilson’s method using Yang’s teachings by including a perceptive network {in CT image processing and analysis} to Wilson’s deep learning model and networks in order to improve CT image denoising by designing a perceptive deep CNN that relies on a perceptual loss as the objective function (see Yang: e.g., pages 2-4);
Wilson as modified by Yang further disclose S3, building a loss function (see Wilson: e.g., Fig. 14, and, --[0072] Super-resolution from deep learning. Another technique that can be employed by various embodiments involves using a deep learning, generative adversarial network (GAN) to create super-resolution CT calcium score images from conventional, thick slice scans. For training, paired normal clinical resolution and high resolution volume scans can be used. These can be easily obtained with existing step and shoot multi-detector CT scan acquisition. The images can be simply reconstructed at both conventional thick slice and high resolution thin slice resolutions, giving perfectly paired data. After training, the network can create a high resolution scan volume from a low resolution clinical scan volume. Using the super-resolution volume, ICmore measures can be computed more accurately and precisely. GANs can be used to reduce noise in CT images, create super-resolution CT images, and create super-resolution MR images.
[0073] In various embodiments, different schemes can be used to create super-resolution image volumes from deep neural networks, such as the following example embodiment. This example embodiment can employ a modified version of a multi-level, densely connected super-resolution network (mDCSRN) with generative adversarial network (GAN)-guided training. By utilizing a densely connected network, the mDCSRN is extremely light-weight, and when trained with a Generative Adversarial Network (GAN), can create sharp and realistic-looking images. In various embodiments, existing network structures can be employed. This technique can employ a loss (or cost) function such as that given below in equation (2), where MSE is the mean square error between the actual and synthetic high resolution image volumes, lossCAC is the difference between the calcification mass score computed form the actual and synthetic high resolution images, lossGAN is the GAN discriminator loss, and the λ's are empirically determined constants:--, in [0072]-[0073]; also see Yang: e.g., -- for CT image denoising by designing a perceptive deep CNN that relies on a perceptual loss as the objective function. During our research, it was drawn to our attention that minimizing MSE between the denoised CT image and the ground truth leads to the loss of important details, although the peak signal to noise ratio (PSNR) based evaluation numbers are excellent. That is because PSNR is equivalent to the per-pixel Euclidean intensity difference. Therefore, a model maximizing PSNR after successful training always achieves very high PSNR values.--, in page 2, also see: Section 2.2 Network Architecture, in pages 3-4);
S4, using the data set and the loss function to update parameters of the iterative generative network so as to obtain a trained deep learning model (see Wilson: e.g., Fig. 14, and, --[0072] Super-resolution from deep learning. Another technique that can be employed by various embodiments involves using a deep learning, generative adversarial network (GAN) to create super-resolution CT calcium score images from conventional, thick slice scans. For training, paired normal clinical resolution and high resolution volume scans can be used. These can be easily obtained with existing step and shoot multi-detector CT scan acquisition. The images can be simply reconstructed at both conventional thick slice and high resolution thin slice resolutions, giving perfectly paired data. After training, the network can create a high resolution scan volume from a low resolution clinical scan volume. Using the super-resolution volume, ICmore measures can be computed more accurately and precisely. GANs can be used to reduce noise in CT images, create super-resolution CT images, and create super-resolution MR images.
[0073] In various embodiments, different schemes can be used to create super-resolution image volumes from deep neural networks, such as the following example embodiment. This example embodiment can employ a modified version of a multi-level, densely connected super-resolution network (mDCSRN) with generative adversarial network (GAN)-guided training. By utilizing a densely connected network, the mDCSRN is extremely light-weight, and when trained with a Generative Adversarial Network (GAN), can create sharp and realistic-looking images. In various embodiments, existing network structures can be employed. This technique can employ a loss (or cost) function such as that given below in equation (2), where MSE is the mean square error between the actual and synthetic high resolution image volumes, lossCAC is the difference between the calcification mass score computed form the actual and synthetic high resolution images, lossGAN is the GAN discriminator loss, and the λ's are empirically determined constants: loss=λ1lossMSE+λ2lossCAC+λ3lossGAN (1)
[0074] Using a properly optimized loss function, excellent results can be obtained.--, in [0072]-[0074]; also see Yang: e.g., -- for CT image denoising by designing a perceptive deep CNN that relies on a perceptual loss as the objective function. During our research, it was drawn to our attention that minimizing MSE between the denoised CT image and the ground truth leads to the loss of important details, although the peak signal to noise ratio (PSNR) based evaluation numbers are excellent. That is because PSNR is equivalent to the per-pixel Euclidean intensity difference. Therefore, a model maximizing PSNR after successful training always achieves very high PSNR values.--, in page 2, also see: Section2.1 Loss Functions; and Section 2.2 Network Architecture, in pages 3-4); and
S5, inputting a low-quality low-resolution image into the trained deep learning model to obtain a high-quality high-resolution image (see Wilson: e.g., Fig. 14, and, --[0072] Super-resolution from deep learning. Another technique that can be employed by various embodiments involves using a deep learning, generative adversarial network (GAN) to create super-resolution CT calcium score images from conventional, thick slice scans. For training, paired normal clinical resolution and high resolution volume scans can be used. These can be easily obtained with existing step and shoot multi-detector CT scan acquisition. The images can be simply reconstructed at both conventional thick slice and high resolution thin slice resolutions, giving perfectly paired data. After training, the network can create a high resolution scan volume from a low resolution clinical scan volume. Using the super-resolution volume, ICmore measures can be computed more accurately and precisely. GANs can be used to reduce noise in CT images, create super-resolution CT images, and create super-resolution MR images.
[0073] In various embodiments, different schemes can be used to create super-resolution image volumes from deep neural networks, such as the following example embodiment. This example embodiment can employ a modified version of a multi-level, densely connected super-resolution network (mDCSRN) with generative adversarial network (GAN)-guided training. By utilizing a densely connected network, the mDCSRN is extremely light-weight, and when trained with a Generative Adversarial Network (GAN), can create sharp and realistic-looking images. In various embodiments, existing network structures can be employed. This technique can employ a loss (or cost) function such as that given below in equation (2), where MSE is the mean square error between the actual and synthetic high resolution image volumes, lossCAC is the difference between the calcification mass score computed form the actual and synthetic high resolution images, lossGAN is the GAN discriminator loss, and the λ's are empirically determined constants: loss=λ1lossMSE+λ2lossCAC+λ3lossGAN (1)
[0074] Using a properly optimized loss function, excellent results can be obtained.--, in [0072]-[0074]; also see Yang: e.g., -- for CT image denoising by designing a perceptive deep CNN that relies on a perceptual loss as the objective function. During our research, it was drawn to our attention that minimizing MSE between the denoised CT image and the ground truth leads to the loss of important details, although the peak signal to noise ratio (PSNR) based evaluation numbers are excellent. That is because PSNR is equivalent to the per-pixel Euclidean intensity difference. Therefore, a model maximizing PSNR after successful training always achieves very high PSNR values.--, in page 2, also see: Section2.1 Loss Functions; and Section 2.2 Network Architecture, in pages 3-4);
Wilson as modified by Yang disclose the generative network comprises a feature extraction module and an upsampling module (see Wilson: e.g., Fig. 14, and, --[0072] Super-resolution from deep learning. Another technique that can be employed by various embodiments involves using a deep learning, generative adversarial network (GAN) to create super-resolution CT calcium score images from conventional, thick slice scans. For training, paired normal clinical resolution and high resolution volume scans can be used. These can be easily obtained with existing step and shoot multi-detector CT scan acquisition. The images can be simply reconstructed at both conventional thick slice and high resolution thin slice resolutions, giving perfectly paired data. After training, the network can create a high resolution scan volume from a low resolution clinical scan volume. Using the super-resolution volume, ICmore measures can be computed more accurately and precisely. GANs can be used to reduce noise in CT images, create super-resolution CT images, and create super-resolution MR images.
[0073] In various embodiments, different schemes can be used to create super-resolution image volumes from deep neural networks, such as the following example embodiment. This example embodiment can employ a modified version of a multi-level, densely connected super-resolution network (mDCSRN) with generative adversarial network (GAN)-guided training. By utilizing a densely connected network, the mDCSRN is extremely light-weight, and when trained with a Generative Adversarial Network (GAN), can create sharp and realistic-looking images. In various embodiments, existing network structures can be employed. This technique can employ a loss (or cost) function such as that given below in equation (2), where MSE is the mean square error between the actual and synthetic high resolution image volumes, lossCAC is the difference between the calcification mass score computed form the actual and synthetic high resolution images, lossGAN is the GAN discriminator loss, and the λ's are empirically determined constants:--, in [0072]-[0073], and, and, --[0095] Additionally or alternatively, various embodiments can employ deep learning. For example, various embodiments can extract deep features from each time point by using a pretrained convolutional neural network (e.g., such as ResNet50, Vgg16, or DenseNet-169). For each individual calcification, a bounding box can be constructed which is centered at the calcification. ICmore deep features of each calcification can be extract from the corresponding bounding box.--, in [0095]),
and, Wilson’s disclosures of the generative adversarial network (GAN) include 1) the feature extraction module, such as listed as in: “extract deep features from each time point by using a pretrained convolutional neural network (e.g., such as ResNet50, Vgg16, or DenseNet-169). For each individual calcification, a bounding box can be constructed which is centered at the calcification”;
and It is well known of VGG 16 architecture (reproduced from web: Author: Rohini G, Sep 23, 2021 “Everything you need to know about VGG16”)
PNG
media_image1.png
200
400
media_image1.png
Greyscale
although Wilson does not explicitly disclose a convolution layer, cascaded convolution blocks, then passing through a convolution layer, and finally obtaining a low-resolution feature map from the low-quality CT image; each convolution block in the cascade convolution blocks comprises at least two convolution layers and a middle ReLU layer;
Hsiao discloses feature extraction module comprises a convolution layer, cascaded convolution blocks, then passing through a convolution layer, and finally obtaining a low-resolution feature map from the low-quality CT image; each convolution block in the cascade convolution blocks comprises at least two convolution layers and a middle ReLU layer (see Hsiao: e.g.,
PNG
media_image2.png
1014
686
media_image2.png
Greyscale
and see: -- CNNs have a large capacity for recalling learned features, and might supply a priori information and assumptions during inference. Based on these capabilities, CNNs should have the ability to recover high-resolution spatial detail from low-resolution images and accomplish the otherwise impossible task of single-plane super-resolution, using either a single or multiple temporal frames.--, in [0008], and
--a relatively shallow network is a SRCNN-inspired neural network with a custom hybrid loss function, which is referred to as “k-SRNet”, based on the Shallow Wide ResNet that was described by R. Yasrab (“SRNET: A Shallow Skip Connection Based Convolutional Neural Network Design for Resolving Singularities”, Journal of Computer Science and Technology 34, 924-938 (2019), which is incorporated herein by reference.) The second, a deeper, more complex network is a modified UNet, referred to as “k-UNet”. For this network, the final activation function is set to the hyperbolic tangent; a custom hybrid loss function is employed. UNet is a fully convolutional network which is known in the art, its name derived from its u-shaped architecture, as shown. (See, O. Ronneberger, et al., “U-Net: Convolutional Networks for Biomedical Image Segmentation”, (2015) arXiv: 1505.04597 [cs.CV], which incorporated herein by reference). Briefly, the UNet employs repeated application of convolutions, each followed by a rectified linear unit (ReLU) and a max pooling operation. During the contraction, the spatial information is reduced while feature information is increased. The expansive pathway combines the feature and spatial information through a sequence of up-convolutions and concatenations with high-resolution features from the contracting path. In implementations of the inventive scheme, neural networks were initially trained to perform super-resolution in image-space using synthetically-generated low-resolution data, including relatively shallow (SRNet) and relatively deep (UNet) convolutional neural networks (CNNs).
[0031] Further performance improvement was evaluated by training these neural networks to perform single-frame (k-) and multi-frame (kt-) super-resolution. Using the hypothesis that additional data from neighboring time points might improve performance, both architectures were extended to incorporate 3-dimensional convolutions, handling the temporal domain in the third dimension. Each input frame was combined with immediately flanking frames to generate input volumes. These spatiotemporal versions of k-SRNet and k-UNET are referred to as “kt-SRNet” and “kt-UNet”, respectively. The architectures of each of these networks are illustrated in FIGS. 1B-1F where layer dimensions (e.g., 128×128) are provided beside the vertical gray bars and the numbers of channels (e.g., 1, 32, 64, etc.) are shown above the gray bars. In FIGS. 1D and 1F, the 3D gray bars in kt-SRNet and kt-UNet, respectively, emphasize use of 3D convolutions. Referring to FIG. 1C, a SRNet is shown with four convolutions: (1) conv 9×9, ReLU (rectified linear activation unit); (2) conv 5×5, ReLU; (3) conv 5×5, ReLU; and (4) conv 1×1, tanh (hyperbolic tangent activation function). This network was used for single-frame super-resolution. FIG. 1D illustrates the architecture a multi-frame SRNet with four convolutions: (1) conv 3×9×9, ReLU; (2) and (3) conv 3×5×5, ReLU, and (4) conv 3×1×1, tanh. (where the “3” represents 3D convolution, which in the evaluation involved the use of three temporally adjacent cropped undersampled input images). In the final step, the central timepoint is extracted to provide a 2D output. FIG. 8 provides an alternative architecture for multi-frame resolution with three 3D convolutions as shown to generate a 2D image.--, in [0030]-[0032]);
Wilson (as modified by Yang) and Hsiao are combinable as they are in the same field of endeavor: monitoring patients using machine learning model in medical imaging and image processing and analysis. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wilson (as modified by Yang)’s method using Hsiao’s teachings by including feature extraction module comprises a convolution layer, cascaded convolution blocks, then passing through a convolution layer, and finally obtaining a low-resolution feature map from the low-quality CT image; each convolution block in the cascade convolution blocks comprises at least two convolution layers and a middle ReLU layer to Wilson (as modified by Yang)’s applications of learning model and image processing, such as VGG16 architecture in order to train neural networks and learning model to perform super-resolution in image-space using synthetically-generated low-resolution data (see Hsiao: e.g., in [0030]-[0033], and [0044]);
Wilson (as modified by Yang)’s disclosures of the upsampling module comprises a fully connected network and a convolution layer, and each pixel position information of the input high-quality CT image is inputted into the fully connected network (see Wilson: e.g., Fig. 14, and, --[0072] Super-resolution from deep learning. Another technique that can be employed by various embodiments involves using a deep learning, generative adversarial network (GAN) to create super-resolution CT calcium score images from conventional, thick slice scans. For training, paired normal clinical resolution and high resolution volume scans can be used. These can be easily obtained with existing step and shoot multi-detector CT scan acquisition. The images can be simply reconstructed at both conventional thick slice and high resolution thin slice resolutions, giving perfectly paired data. After training, the network can create a high resolution scan volume from a low resolution clinical scan volume. Using the super-resolution volume, ICmore measures can be computed more accurately and precisely. GANs can be used to reduce noise in CT images, create super-resolution CT images, and create super-resolution MR images.
[0073] In various embodiments, different schemes can be used to create super-resolution image volumes from deep neural networks, such as the following example embodiment. This example embodiment can employ a modified version of a multi-level, densely connected super-resolution network (mDCSRN) with generative adversarial network (GAN)-guided training. By utilizing a densely connected network, the mDCSRN is extremely light-weight, and when trained with a Generative Adversarial Network (GAN), can create sharp and realistic-looking images. In various embodiments, existing network structures can be employed. This technique can employ a loss (or cost) function such as that given below in equation (2), where MSE is the mean square error between the actual and synthetic high resolution image volumes, lossCAC is the difference between the calcification mass score computed form the actual and synthetic high resolution images, lossGAN is the GAN discriminator loss, and the λ's are empirically determined constants:--, in [0072]-[0073]; also see Yang: e.g., -- for CT image denoising by designing a perceptive deep CNN that relies on a perceptual loss as the objective function. During our research, it was drawn to our attention that minimizing MSE between the denoised CT image and the ground truth leads to the loss of important details, although the peak signal to noise ratio (PSNR) based evaluation numbers are excellent. That is because PSNR is equivalent to the per-pixel Euclidean intensity difference. Therefore, a model maximizing PSNR after successful training always achieves very high PSNR values.--, in page 2, also see: Section 2.2 Network Architecture, in pages 3-4, also see: --the recently popularized deep learning techniques, aim for minimizing mean-squared-error (MSE) between a denoised CT image and the ground truth--, in abstract, and, --The proposed network was implemented and trained using the Caffe toolbox [8]. At the training phrase, we randomly extracted and selected 100,000 image patches of size 32 × 32 from 2,600 CT images. We first trained a CNN with the same structure as shown in Fig. 1 but using the mean-square-error (MSE) loss, which is named CNN-MSE.--, under Section 3.1 Materials and Network Training, in page 4); and,
Hsiao discloses the upsampling module comprises a fully connected network and a convolution layer, and each pixel position information of the input high-quality image is inputted into the fully connected network, and output result of the fully connected network is applied to the low- resolution feature map to obtain the high-quality high-resolution image (see Hsiao: e.g., --[0022] FIGS. 3A and 3B are box-and-whisker plots comparing performance (SSIM) for each super-resolution method across multiple upsampling factors. FIG. 3A plots aggregate performance for each super-resolution method. FIG. 3B plots pairwise comparison of performance between each method and zero-padding (z-pad).
[0023] FIG. 4 provides an example comparing super-resolution methods across multiple upsampling factors from 2× to 64×.
[0024] FIGS. 5A-5D are examples comparing super-resolution methods at 8× upsampling. Output images and corresponding logplots of k-space are shown along with SSIM relative to ground truth.--, [0022]-[0024], and [0033]; also see:
--[0044] A unique set of UNets and SRNets were trained for multiple degrees of upsampling, from 2× to 64×. CNN-based approaches, SRNet and U-Net, and conventional methods of bicubic interpolation and z-padding were compared by calculating the Structural Similarity Index (SSIM) between each ground truth image and its corresponding super-resolved image from each method of upscaling. …in Table 1. For synthetic test data, differences between SRNet and UNet were small, but statistically significant. For all degrees of upsampling, k-UNet and kt-UNet outperformed k-SRNet and kt-SRNet, respectively. similar performance of all CNNs was observed across upsampling factors, in contrast to a widening performance gap with conventional upscaling methods with higher upsampling factors (FIG. 3A).
[0049] The performance of each of the upsampling strategies was further explored by examining representative case examples. DL-based methods consistently outperform conventional methods on both a bulk and a per-slice basis. Neural network-based methods outperformed traditional bicubic and zero-padding for nearly every slice evaluated. Zero-padding outperforms bicubic for nearly every slice evaluated.
[0050] FIG. 4 provides an example comparing super-resolution methods across different upsampling factors from 2× to 64×. At upsampling factors of 2× and 4×, the qualitative difference between each of the methods and ground truth was subtle. At 8× upsampling, bicubic interpolation showed a noticeable degradation of image quality, notably in the sharpness of the right ventricular trabeculations, and the myocardium-blood pool interface.--, in [0044]-[0050]);
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wilson (as modified by Yang)’s method using Hsiao’s teachings by including the upsampling module comprises a fully connected network and a convolution layer, and each pixel position information of the input high-quality image is inputted into the fully connected network, and output result of the fully connected network is applied to the low- resolution feature map to obtain the high-quality high-resolution image to Wilson (as modified by Yang)’s applications of learning model and image processing and creating super-resolution CT calcium score images in order to train neural networks and learning model to perform super-resolution in image-space using synthetically-generated low-resolution data (see Hsiao: e.g., in [0030]-[0033], [0044], and [0047]-[0050]);
Re Claim 2, Wilson as modified by Yang and Hsiao further disclose characterized in that, pre-processing clinical data in step S1 comprises steps of:
S11, acquiring a low-quality CT image with low radiation dose low resolution and a high-quality CT image with normal radiation dose high resolution (see Wilson:e.g., -- For training, paired normal clinical resolution and high resolution volume scans can be used. These can be easily obtained with existing step and shoot multi-detector CT scan acquisition. The images can be simply reconstructed at both conventional thick slice and high resolution thin slice resolutions, giving perfectly paired data. After training, the network can create a high resolution scan volume from a low resolution clinical scan volume. Using the super-resolution volume, ICmore measures can be computed more accurately and precisely. GANs can be used to reduce noise in CT images, create super-resolution CT images, and create super-resolution MR images.--, in [0072]; also see Yang: e.g., -- Lowering the radiation dose tends to ignorantly increase the noise and artifacts in the reconstructed images, which can compromise diagnostic information. To reduce noise and sup-press artifacts in low-dose CT images,--, in page 1; and, -- used FBP reconstruction from 30NI dataset (high noise level) as the network input and the corresponding VEO reconstruction from 10NI dataset (low noise level) as the ground truth images.--, under Section 3.1 Materials and Network Training, in page 4);
S12, clipping the low-quality CT image according to metadata of a medical image, so that the clipped low-quality CT image corresponds to physical space information of the high- 20 quality CT image, and a data pair with same physical space information 1s obtained (see Wilson: e.g., -- For training, paired normal clinical resolution and high resolution volume scans can be used. These can be easily obtained with existing step and shoot multi-detector CT scan acquisition. The images can be simply reconstructed at both conventional thick slice and high resolution thin slice resolutions, giving perfectly paired data. After training, the network can create a high resolution scan volume from a low resolution clinical scan volume. Using the super-resolution volume, ICmore measures can be computed more accurately and precisely. GANs can be used to reduce noise in CT images, create super-resolution CT images, and create super-resolution MR images.--, in [0072]; and, --[0076] Referring to FIG. 9, illustrated is a graph showing mass scores from a QRM calcification phantom obtained via various techniques, in connection with aspects discussed herein. As can be seen in FIG. 9, mass scores can be improved with corrections towards the actual value (indicated via the horizontal line). The five mass scores shown in FIG. 9 are high resolution imaging (0.49 mm in plane, 0.67 mm thick), normal resolution imaging (2.5 mm thick) alone, normal resolution after partial volume correction (PVC), normal resolution after blind 3D deconvolution, and normal resolution after 3D deconvolution with estimated PSF (Point Spread Function). Blind deconvolution can iteratively improve an initial PSF.
[0077] FIGS. 10-12 illustrate ways in which correction techniques discussed herein can improve the accuracy and precision of mass score or other measures. Referring to FIG. 10, illustrated is a pair of example images showing a high resolution (1000) and a normal resolution (1010) cadaver image volume showing partial volume blur, in connection with various aspects discussed herein. Referring to FIG. 11, illustrated is a graph showing mass scores from scanning at 0°, −15°, and 15° at high resolution, compared to normal resolution, normal resolution after partial volume (PV) correction, and normal resolution after deconvolution, in connection with various aspects discussed herein. As can be seen in FIG. 11, after correction, the mass scores improve to match the more accurate, high resolution value. The precision was also improved with correction as COV (coefficient of variation) was 0.03, 0.11, 0.07, and 0.1, respectively. Referring to FIG. 12, illustrated is a graph showing mass scores across 4 calcifications (CA1-CA4) in a cadaver heart at high resolution, normal resolution, normal resolution with partial volume correction and normal volume with 3D deconvolution, in connection with various aspects discussed herein. After corrections on the normal resolution image volume, scores improved towards high resolution values. The average COV improved from 0.27 on normal scans to 0.1 and 0.17, following PVC and deconvolution, respectively--, in [0076]-[0077];.also see Yang: e.g., -- Lowering the radiation dose tends to ignorantly increase the noise and artifacts in the reconstructed images, which can compromise diagnostic information. To reduce noise and sup-press artifacts in low-dose CT images,--, in page 1; and, -- used FBP reconstruction from 30NI dataset (high noise level) as the network input and the corresponding VEO reconstruction from 10NI dataset (low noise level) as the ground truth images.--, under Section 3.1 Materials and Network Training, in page 4);
S13, clipping the data pair into patches of data pair, performing threshold determination, and reserving patches of data pair meeting a condition of the threshold determination (see Wilson:e.g., -- For training, paired normal clinical resolution and high resolution volume scans can be used. These can be easily obtained with existing step and shoot multi-detector CT scan acquisition. The images can be simply reconstructed at both conventional thick slice and high resolution thin slice resolutions, giving perfectly paired data. After training, the network can create a high resolution scan volume from a low resolution clinical scan volume. Using the super-resolution volume, ICmore measures can be computed more accurately and precisely. GANs can be used to reduce noise in CT images, create super-resolution CT images, and create super-resolution MR images.--, in [0072]; and, --[0076] Referring to FIG. 9, illustrated is a graph showing mass scores from a QRM calcification phantom obtained via various techniques, in connection with aspects discussed herein. As can be seen in FIG. 9, mass scores can be improved with corrections towards the actual value (indicated via the horizontal line). The five mass scores shown in FIG. 9 are high resolution imaging (0.49 mm in plane, 0.67 mm thick), normal resolution imaging (2.5 mm thick) alone, normal resolution after partial volume correction (PVC), normal resolution after blind 3D deconvolution, and normal resolution after 3D deconvolution with estimated PSF (Point Spread Function). Blind deconvolution can iteratively improve an initial PSF.
[0077] FIGS. 10-12 illustrate ways in which correction techniques discussed herein can improve the accuracy and precision of mass score or other measures. Referring to FIG. 10, illustrated is a pair of example images showing a high resolution (1000) and a normal resolution (1010) cadaver image volume showing partial volume blur, in connection with various aspects discussed herein. Referring to FIG. 11, illustrated is a graph showing mass scores from scanning at 0°, −15°, and 15° at high resolution, compared to normal resolution, normal resolution after partial volume (PV) correction, and normal resolution after deconvolution, in connection with various aspects discussed herein. As can be seen in FIG. 11, after correction, the mass scores improve to match the more accurate, high resolution value. The precision was also improved with correction as COV (coefficient of variation) was 0.03, 0.11, 0.07, and 0.1, respectively. Referring to FIG. 12, illustrated is a graph showing mass scores across 4 calcifications (CA1-CA4) in a cadaver heart at high resolution, normal resolution, normal resolution with partial volume correction and normal volume with 3D deconvolution, in connection with various aspects discussed herein. After corrections on the normal resolution image volume, scores improved towards high resolution values. The average COV improved from 0.27 on normal scans to 0.1 and 0.17, following PVC and deconvolution, respectively.--, in [0076]-[0077]; also see Yang: e.g., -- Lowering the radiation dose tends to ignorantly increase the noise and artifacts in the reconstructed images, which can compromise diagnostic information. To reduce noise and sup-press artifacts in low-dose CT images,--, in page 1; and, -- used FBP reconstruction from 30NI dataset (high noise level) as the network input and the corresponding VEO reconstruction from 10NI dataset (low noise level) as the ground truth images.--, under Section 3.1 Materials and Network Training, in page 4);
S14, performing pixel interception and normalization on the reserved patches of data pair (see Wilson: e.g., --wherein identifying the one or more territories of the at least one artery based on the registration of the atlas to the CT image volume: registering a plurality of potential atlases to a down-sampled version of the CT image volume; computing, for each potential atlas of the plurality of potential atlases, an associated normalized cross-correlation for that potential atlas; selecting, as the atlas, the potential atlas with a best associated normalized cross-correlation of the plurality of potential atlases; and registering the atlas to the CT image volume at full resolution.--, in [0115], [0124] and [0057]-[0058]; and, --[0076] Referring to FIG. 9, illustrated is a graph showing mass scores from a QRM calcification phantom obtained via various techniques, in connection with aspects discussed herein. As can be seen in FIG. 9, mass scores can be improved with corrections towards the actual value (indicated via the horizontal line). The five mass scores shown in FIG. 9 are high resolution imaging (0.49 mm in plane, 0.67 mm thick), normal resolution imaging (2.5 mm thick) alone, normal resolution after partial volume correction (PVC), normal resolution after blind 3D deconvolution, and normal resolution after 3D deconvolution with estimated PSF (Point Spread Function). Blind deconvolution can iteratively improve an initial PSF.
[0077] FIGS. 10-12 illustrate ways in which correction techniques discussed herein can improve the accuracy and precision of mass score or other measures. Referring to FIG. 10, illustrated is a pair of example images showing a high resolution (1000) and a normal resolution (1010) cadaver image volume showing partial volume blur, in connection with various aspects discussed herein. Referring to FIG. 11, illustrated is a graph showing mass scores from scanning at 0°, −15°, and 15° at high resolution, compared to normal resolution, normal resolution after partial volume (PV) correction, and normal resolution after deconvolution, in connection with various aspects discussed herein. As can be seen in FIG. 11, after correction, the mass scores improve to match the more accurate, high resolution value. The precision was also improved with correction as COV (coefficient of variation) was 0.03, 0.11, 0.07, and 0.1, respectively. Referring to FIG. 12, illustrated is a graph showing mass scores across 4 calcifications (CA1-CA4) in a cadaver heart at high resolution, normal resolution, normal resolution with partial volume correction and normal volume with 3D deconvolution, in connection with various aspects discussed herein. After corrections on the normal resolution image volume, scores improved towards high resolution values. The average COV improved from 0.27 on normal scans to 0.1 and 0.17, following PVC and deconvolution, respectively.--, in [0076]-[0077]); and
S15. expanding data of the patches of data pair processed in step S14 so as to obtain the data set for training the deep learning model (see Wilson: e.g., Fig. 14, and, --[0072] Super-resolution from deep learning. Another technique that can be employed by various embodiments involves using a deep learning, generative adversarial network (GAN) to create super-resolution CT calcium score images from conventional, thick slice scans. For training, paired normal clinical resolution and high resolution volume scans can be used. These can be easily obtained with existing step and shoot multi-detector CT scan acquisition. The images can be simply reconstructed at both conventional thick slice and high resolution thin slice resolutions, giving perfectly paired data. After training, the network can create a high resolution scan volume from a low resolution clinical scan volume. Using the super-resolution volume, ICmore measures can be computed more accurately and precisely. GANs can be used to reduce noise in CT images, create super-resolution CT images, and create super-resolution MR images.
[0073] In various embodiments, different schemes can be used to create super-resolution image volumes from deep neural networks, such as the following example embodiment. This example embodiment can employ a modified version of a multi-level, densely connected super-resolution network (mDCSRN) with generative adversarial network (GAN)-guided training. By utilizing a densely connected network, the mDCSRN is extremely light-weight, and when trained with a Generative Adversarial Network (GAN), can create sharp and realistic-looking images. In various embodiments, existing network structures can be employed. This technique can employ a loss (or cost) function such as that given below in equation (2), where MSE is the mean square error between the actual and synthetic high resolution image volumes, lossCAC is the difference between the calcification mass score computed form the actual and synthetic high resolution images, lossGAN is the GAN discriminator loss, and the λ's are empirically determined constants: loss=λ1lossMSE+λ2lossCAC+λ3lossGAN (1)
[0074] Using a properly optimized loss function, excellent results can be obtained.--, in [0072]-[0074]; and, --[0076] Referring to FIG. 9, illustrated is a graph showing mass scores from a QRM calcification phantom obtained via various techniques, in connection with aspects discussed herein. As can be seen in FIG. 9, mass scores can be improved with corrections towards the actual value (indicated via the horizontal line). The five mass scores shown in FIG. 9 are high resolution imaging (0.49 mm in plane, 0.67 mm thick), normal resolution imaging (2.5 mm thick) alone, normal resolution after partial volume correction (PVC), normal resolution after blind 3D deconvolution, and normal resolution after 3D deconvolution with estimated PSF (Point Spread Function). Blind deconvolution can iteratively improve an initial PSF.
[0077] FIGS. 10-12 illustrate ways in which correction techniques discussed herein can improve the accuracy and precision of mass score or other measures. Referring to FIG. 10, illustrated is a pair of example images showing a high resolution (1000) and a normal resolution (1010) cadaver image volume showing partial volume blur, in connection with various aspects discussed herein. Referring to FIG. 11, illustrated is a graph showing mass scores from scanning at 0°, −15°, and 15° at high resolution, compared to normal resolution, normal resolution after partial volume (PV) correction, and normal resolution after deconvolution, in connection with various aspects discussed herein. As can be seen in FIG. 11, after correction, the mass scores improve to match the more accurate, high resolution value. The precision was also improved with correction as COV (coefficient of variation) was 0.03, 0.11, 0.07, and 0.1, respectively. Referring to FIG. 12, illustrated is a graph showing mass scores across 4 calcifications (CA1-CA4) in a cadaver heart at high resolution, normal resolution, normal resolution with partial volume correction and normal volume with 3D deconvolution, in connection with various aspects discussed herein. After corrections on the normal resolution image volume, scores improved towards high resolution values. The average COV improved from 0.27 on normal scans to 0.1 and 0.17, following PVC and deconvolution, respectively.--, in [0076]-[0077]; and, --wherein identifying the one or more territories of the at least one artery based on the registration of the atlas to the CT image volume: registering a plurality of potential atlases to a down-sampled version of the CT image volume; computing, for each potential atlas of the plurality of potential atlases, an associated normalized cross-correlation for that potential atlas; selecting, as the atlas, the potential atlas with a best associated normalized cross-correlation of the plurality of potential atlases; and registering the atlas to the CT image volume at full resolution.--, in [0115], [0124] and [0057]-[0058]).
Re Claim 3, Wilson as modified by Yang and Hsiao further disclose clipping the data block into patches of data pair in step S13 comprises: clipping the high-quality CT image in the data block every fixed number of pixels/layers (see Wilson: e.g., -- For training, paired normal clinical resolution and high resolution volume scans can be used. These can be easily obtained with existing step and shoot multi-detector CT scan acquisition. The images can be simply reconstructed at both conventional thick slice and high resolution thin slice resolutions, giving perfectly paired data. After training, the network can create a high resolution scan volume from a low resolution clinical scan volume. Using the super-resolution volume, ICmore measures can be computed more accurately and precisely. GANs can be used to reduce noise in CT images, create super-resolution CT images, and create super-resolution MR images.--, in [0072]; and, --[0076] Referring to FIG. 9, illustrated is a graph showing mass scores from a QRM calcification phantom obtained via various techniques, in connection with aspects discussed herein. As can be seen in FIG. 9, mass scores can be improved with corrections towards the actual value (indicated via the horizontal line). The five mass scores shown in FIG. 9 are high resolution imaging (0.49 mm in plane, 0.67 mm thick), normal resolution imaging (2.5 mm thick) alone, normal resolution after partial volume correction (PVC), normal resolution after blind 3D deconvolution, and normal resolution after 3D deconvolution with estimated PSF (Point Spread Function). Blind deconvolution can iteratively improve an initial PSF.
[0077] FIGS. 10-12 illustrate ways in which correction techniques discussed herein can improve the accuracy and precision of mass score or other measures. Referring to FIG. 10, illustrated is a pair of example images showing a high resolution (1000) and a normal resolution (1010) cadaver image volume showing partial volume blur, in connection with various aspects discussed herein. Referring to FIG. 11, illustrated is a graph showing mass scores from scanning at 0°, −15°, and 15° at high resolution, compared to normal resolution, normal resolution after partial volume (PV) correction, and normal resolution after deconvolution, in connection with various aspects discussed herein. As can be seen in FIG. 11, after correction, the mass scores improve to match the more accurate, high resolution value. The precision was also improved with correction as COV (coefficient of variation) was 0.03, 0.11, 0.07, and 0.1, respectively. Referring to FIG. 12, illustrated is a graph showing mass scores across 4 calcifications (CA1-CA4) in a cadaver heart at high resolution, normal resolution, normal resolution with partial volume correction and normal volume with 3D deconvolution, in connection with various aspects discussed herein. After corrections on the normal resolution image volume, scores improved towards high resolution values. The average COV improved from 0.27 on normal scans to 0.1 and 0.17, following PVC and deconvolution, respectively--, in [0076]-[0077];.also see Yang: e.g., -- Lowering the radiation dose tends to ignorantly increase the noise and artifacts in the reconstructed images, which can compromise diagnostic information. To reduce noise and sup-press artifacts in low-dose CT images,--, in page 1; and, -- used FBP reconstruction from 30NI dataset (high noise level) as the network input and the corresponding VEO reconstruction from 10NI dataset (low noise level) as the ground truth images.--, under Section 3.1 Materials and Network Training, in page 4), and
scaling a number of pixels/layers of the low-quality CT image corresponding to the high-quality CT image so as to correspond to the physical space information of the high-quality CT image (see Wilson: e.g., Fig. 14, and, --[0072] Super-resolution from deep learning. Another technique that can be employed by various embodiments involves using a deep learning, generative adversarial network (GAN) to create super-resolution CT calcium score images from conventional, thick slice scans. For training, paired normal clinical resolution and high resolution volume scans can be used. These can be easily obtained with existing step and shoot multi-detector CT scan acquisition. The images can be simply reconstructed at both conventional thick slice and high resolution thin slice resolutions, giving perfectly paired data. After training, the network can create a high resolution scan volume from a low resolution clinical scan volume. Using the super-resolution volume, ICmore measures can be computed more accurately and precisely. GANs can be used to reduce noise in CT images, create super-resolution CT images, and create super-resolution MR images.
[0073] In various embodiments, different schemes can be used to create super-resolution image volumes from deep neural networks, such as the following example embodiment. This example embodiment can employ a modified version of a multi-level, densely connected super-resolution network (mDCSRN) with generative adversarial network (GAN)-guided training. By utilizing a densely connected network, the mDCSRN is extremely light-weight, and when trained with a Generative Adversarial Network (GAN), can create sharp and realistic-looking images. In various embodiments, existing network structures can be employed. This technique can employ a loss (or cost) function such as that given below in equation (2), where MSE is the mean square error between the actual and synthetic high resolution image volumes, lossCAC is the difference between the calcification mass score computed form the actual and synthetic high resolution images, lossGAN is the GAN discriminator loss, and the λ's are empirically determined constants: loss=λ1lossMSE+λ2lossCAC+λ3lossGAN (1)
[0074] Using a properly optimized loss function, excellent results can be obtained.--, in [0072]-[0074]; and, --[0076] Referring to FIG. 9, illustrated is a graph showing mass scores from a QRM calcification phantom obtained via various techniques, in connection with aspects discussed herein. As can be seen in FIG. 9, mass scores can be improved with corrections towards the actual value (indicated via the horizontal line). The five mass scores shown in FIG. 9 are high resolution imaging (0.49 mm in plane, 0.67 mm thick), normal resolution imaging (2.5 mm thick) alone, normal resolution after partial volume correction (PVC), normal resolution after blind 3D deconvolution, and normal resolution after 3D deconvolution with estimated PSF (Point Spread Function). Blind deconvolution can iteratively improve an initial PSF.
[0077] FIGS. 10-12 illustrate ways in which correction techniques discussed herein can improve the accuracy and precision of mass score or other measures. Referring to FIG. 10, illustrated is a pair of example images showing a high resolution (1000) and a normal resolution (1010) cadaver image volume showing partial volume blur, in connection with various aspects discussed herein. Referring to FIG. 11, illustrated is a graph showing mass scores from scanning at 0°, −15°, and 15° at high resolution, compared to normal resolution, normal resolution after partial volume (PV) correction, and normal resolution after deconvolution, in connection with various aspects discussed herein. As can be seen in FIG. 11, after correction, the mass scores improve to match the more accurate, high resolution value. The precision was also improved with correction as COV (coefficient of variation) was 0.03, 0.11, 0.07, and 0.1, respectively. Referring to FIG. 12, illustrated is a graph showing mass scores across 4 calcifications (CA1-CA4) in a cadaver heart at high resolution, normal resolution, normal resolution with partial volume correction and normal volume with 3D deconvolution, in connection with various aspects discussed herein. After corrections on the normal resolution image volume, scores improved towards high resolution values. The average COV improved from 0.27 on normal scans to 0.1 and 0.17, following PVC and deconvolution, respectively.--, in [0076]-[0077]; and, --wherein identifying the one or more territories of the at least one artery based on the registration of the atlas to the CT image volume: registering a plurality of potential atlases to a down-sampled version of the CT image volume; computing, for each potential atlas of the plurality of potential atlases, an associated normalized cross-correlation for that potential atlas; selecting, as the atlas, the potential atlas with a best associated normalized cross-correlation of the plurality of potential atlases; and registering the atlas to the CT image volume at full resolution.--, in [0115], [0124] and [0057]-[0058] {herein “correlation” align with “scaling”).
Re Claim 4, Wilson as modified by Yang however still do not explicitly disclose the condition of the threshold determination in step S13 is that a similarity index between the scaled low-quality CT image and the high-quality CT image in the patches of data pair is higher than a threshold,
Hsiao discloses the condition of the threshold determination in step S13 is that a similarity index between the scaled low-quality CT image and the high-quality CT image in the patches of data pair is higher than a threshold (see Hsiao: e.g., -- Specifically, the loss function is the sum of L1-loss and a modified form of the Multiscale Structural Similarity Index (MS-SSIM) loss (defined as 1-MS-SSIM). The Tensorflow implementation of MS-SSIM and its default settings for filter size=11, filter sigma=1.5, k1=0.01, and k2=0.03 were used. Due to relatively small 128×128 matrix size for training data, only the first four default MS-SSIM power factors were used and renormalized, resulting in the weights [0.0517,0.3295,0.3462,0.2726]. For multi-frame experiments, the 3D L1-loss was added to the mean of the MS-SSIM losses calculated for each of the three adjacent timeframes.
Model Analysis and Statistics
[0044] A unique set of UNets and SRNets were trained for multiple degrees of upsampling, from 2× to 64×. CNN-based approaches, SRNet and U-Net, and conventional methods of bicubic interpolation and z-padding were compared by calculating the Structural Similarity Index (SSIM) between each ground truth image and its corresponding super-resolved image from each method of upscaling. We report the mean and standard deviation of SSIM and determine statistical significance using paired Student's t-test with type I error threshold of 0.05.--, in [0043]-[0044], and also see: -- [0033] FIG. 2 illustrates a sequence for generation and use of synthetic training data to mimic the super-resolution task. The training workflow involved cropping a central 128×128 area of the short-axis (SAX) image (source image 201) to standardize image presentation and serve as ground truth 202. To simulate low-resolution MRI acquisitions, a process referred to as “Fourier downsampling” was employed to downsample ground truth image 202 to k-space to mimic a fully-sampled, low-resolution acquisition. A FFT was applied to transform the image 202 to k-space 203. Each downsampling factor was simulated by retaining central windows of k-space of varying sizes. Outer portions of k-space 203 were cropped to k-space 204 and zero-filled to a produce a matrix size of 128×128. The resulting central region of k-space 205 which was used to generate synthetic training image 206. Images transformed back to the image domain and pixel values were scaled to [0,1]. Each downsampling (and commensurate upsampling) factor was defined as the ratio of the k-space window area to the cropped, 128×128 area.--, in [0033]).
Wilson (as modified by Yang) and Hsiao are combinable as they are in the same field of endeavor: monitoring patients using machine learning model in medical imaging and image processing and analysis. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wilson (as modified by Yang)’s method using Hsiao’s teachings by including the condition of the threshold determination in step S13 is that a similarity index between the scaled low-quality CT image and the high-quality CT image in the patches of data pair is higher than a threshold to Wilson (as modified by Yang)’s applications of learning model and image processing in order to train neural networks and learning model to perform super-resolution in image-space using synthetically-generated low-resolution data (see Hsiao: e.g., in [0030]-[0033], and [0044]).
Re Claim 5, Wilson as modified by Yang further disclose expanding data in step S15 includes flipping and rotating images (see Wilson: e.g., -- [0058] Registration of serial data to identify calcification correspondence. In embodiments associated with serial CT imaging, CT volumes acquired at different time points can be co-registered, with the last volume as reference (because it likely contains the most calcifications) and earlier volumes as floating (in some embodiments, registration can be to the volume determined to have the most calcifications, etc.). Initially, gray-scale normalized-cross-correlation rigid body (translation and rotation) registration can be used to register a reference and floating volume. From this, roughly registered hearts with residual error can be obtained, such as images 400 and 410 of FIG. 4, discussed below. After rough registration, an iterative closest point (ICP) algorithm with rigid body transformation to align calcifications can be used. However, rather than just using two 3D point clouds, such as in existing techniques, various embodiments discussed herein can assign one or more morphological characteristics (e.g. center, divergence, max HU, etc.) to each calcification, and can perform ICP in a higher dimensional space so that calcifications of similar shape and size will match. This increases robustness as compared to simpler algorithms that can be trapped in local minima. ICP works with different numbers of points in reference and floating volumes, giving results robust to new and lost calcifications.--, in [0058]).
Re Claim 6, Wilson as modified by Yang further disclose the loss function is a combined loss function of a mean absolute error loss, a perceptual loss and a generation countermeasure loss (see Wilson: e.g., Fig. 14, and, --[0072] Super-resolution from deep learning. Another technique that can be employed by various embodiments involves using a deep learning, generative adversarial network (GAN) to create super-resolution CT calcium score images from conventional, thick slice scans. For training, paired normal clinical resolution and high resolution volume scans can be used. These can be easily obtained with existing step and shoot multi-detector CT scan acquisition. The images can be simply reconstructed at both conventional thick slice and high resolution thin slice resolutions, giving perfectly paired data. After training, the network can create a high resolution scan volume from a low resolution clinical scan volume. Using the super-resolution volume, ICmore measures can be computed more accurately and precisely. GANs can be used to reduce noise in CT images, create super-resolution CT images, and create super-resolution MR images.
[0073] In various embodiments, different schemes can be used to create super-resolution image volumes from deep neural networks, such as the following example embodiment. This example embodiment can employ a modified version of a multi-level, densely connected super-resolution network (mDCSRN) with generative adversarial network (GAN)-guided training. By utilizing a densely connected network, the mDCSRN is extremely light-weight, and when trained with a Generative Adversarial Network (GAN), can create sharp and realistic-looking images. In various embodiments, existing network structures can be employed. This technique can employ a loss (or cost) function such as that given below in equation (2), where MSE is the mean square error between the actual and synthetic high resolution image volumes, lossCAC is the difference between the calcification mass score computed form the actual and synthetic high resolution images, lossGAN is the GAN discriminator loss, and the λ's are empirically determined constants:--, in [0072]-[0073]; also see Yang: e.g., -- for CT image denoising by designing a perceptive deep CNN that relies on a perceptual loss as the objective function. During our research, it was drawn to our attention that minimizing MSE between the denoised CT image and the ground truth leads to the loss of important details, although the peak signal to noise ratio (PSNR) based evaluation numbers are excellent. That is because PSNR is equivalent to the per-pixel Euclidean intensity difference. Therefore, a model maximizing PSNR after successful training always achieves very high PSNR values.--, in page 2, also see: Section 2.2 Network Architecture, in pages 3-4, also see: --the recently popularized deep learning techniques, aim for minimizing mean-squared-error (MSE) between a denoised CT image and the ground truth--, in abstract, and, --The proposed network was implemented and trained using the Caffe toolbox [8]. At the training phrase, we randomly extracted and selected 100,000 image patches of size 32 × 32 from 2,600 CT images. We first trained a CNN with the same structure as shown in Fig. 1 but using the mean-square-error (MSE) loss, which is named CNN-MSE.--, under Section 3.1 Materials and Network Training, in page 4).
Re Claim 7, Wilson as modified by Yang further disclose the perceptual loss is obtained by inputting output result of the generative network and a real high-quality CT image into the perceptive network, respectively, and performing MSE loss on output result of the perceptive network (see Wilson: e.g., Fig. 14, and, --[0072] Super-resolution from deep learning. Another technique that can be employed by various embodiments involves using a deep learning, generative adversarial network (GAN) to create super-resolution CT calcium score images from conventional, thick slice scans. For training, paired normal clinical resolution and high resolution volume scans can be used. These can be easily obtained with existing step and shoot multi-detector CT scan acquisition. The images can be simply reconstructed at both conventional thick slice and high resolution thin slice resolutions, giving perfectly paired data. After training, the network can create a high resolution scan volume from a low resolution clinical scan volume. Using the super-resolution volume, ICmore measures can be computed more accurately and precisely. GANs can be used to reduce noise in CT images, create super-resolution CT images, and create super-resolution MR images.
[0073] In various embodiments, different schemes can be used to create super-resolution image volumes from deep neural networks, such as the following example embodiment. This example embodiment can employ a modified version of a multi-level, densely connected super-resolution network (mDCSRN) with generative adversarial network (GAN)-guided training. By utilizing a densely connected network, the mDCSRN is extremely light-weight, and when trained with a Generative Adversarial Network (GAN), can create sharp and realistic-looking images. In various embodiments, existing network structures can be employed. This technique can employ a loss (or cost) function such as that given below in equation (2), where MSE is the mean square error between the actual and synthetic high resolution image volumes, lossCAC is the difference between the calcification mass score computed form the actual and synthetic high resolution images, lossGAN is the GAN discriminator loss, and the λ's are empirically determined constants:--, in [0072]-[0073]; also see Yang: e.g., -- for CT image denoising by designing a perceptive deep CNN that relies on a perceptual loss as the objective function. During our research, it was drawn to our attention that minimizing MSE between the denoised CT image and the ground truth leads to the loss of important details, although the peak signal to noise ratio (PSNR) based evaluation numbers are excellent. That is because PSNR is equivalent to the per-pixel Euclidean intensity difference. Therefore, a model maximizing PSNR after successful training always achieves very high PSNR values.--, in page 2, also see: Section 2.2 Network Architecture, in pages 3-4, also see: --the recently popularized deep learning techniques, aim for minimizing mean-squared-error (MSE) between a denoised CT image and the ground truth--, in abstract, and, --The proposed network was implemented and trained using the Caffe toolbox [8]. At the training phrase, we randomly extracted and selected 100,000 image patches of size 32 × 32 from 2,600 CT images. We first trained a CNN with the same structure as shown in Fig. 1 but using the mean-square-error (MSE) loss, which is named CNN-MSE.--, under Section 3.1 Materials and Network Training, in page 4).
Re Claim 8, Wilson as modified by Yang further disclose the generation countermeasure loss is one of aGAN loss, a WGAN loss, a WGAN-GP loss or a rGAN loss (see Wilson: e.g., Fig. 14, and, --[0072] Super-resolution from deep learning. Another technique that can be employed by various embodiments involves using a deep learning, generative adversarial network (GAN) to create super-resolution CT calcium score images from conventional, thick slice scans. For training, paired normal clinical resolution and high resolution volume scans can be used. These can be easily obtained with existing step and shoot multi-detector CT scan acquisition. The images can be simply reconstructed at both conventional thick slice and high resolution thin slice resolutions, giving perfectly paired data. After training, the network can create a high resolution scan volume from a low resolution clinical scan volume. Using the super-resolution volume, ICmore measures can be computed more accurately and precisely. GANs can be used to reduce noise in CT images, create super-resolution CT images, and create super-resolution MR images.
[0073] In various embodiments, different schemes can be used to create super-resolution image volumes from deep neural networks, such as the following example embodiment. This example embodiment can employ a modified version of a multi-level, densely connected super-resolution network (mDCSRN) with generative adversarial network (GAN)-guided training. By utilizing a densely connected network, the mDCSRN is extremely light-weight, and when trained with a Generative Adversarial Network (GAN), can create sharp and realistic-looking images. In various embodiments, existing network structures can be employed. This technique can employ a loss (or cost) function such as that given below in equation (2), where MSE is the mean square error between the actual and synthetic high resolution image volumes, lossCAC is the difference between the calcification mass score computed form the actual and synthetic high resolution images, lossGAN is the GAN discriminator loss, and the λ's are empirically determined constants:--, in [0072]-[0073]; also see Yang: e.g., -- for CT image denoising by designing a perceptive deep CNN that relies on a perceptual loss as the objective function. During our research, it was drawn to our attention that minimizing MSE between the denoised CT image and the ground truth leads to the loss of important details, although the peak signal to noise ratio (PSNR) based evaluation numbers are excellent. That is because PSNR is equivalent to the per-pixel Euclidean intensity difference. Therefore, a model maximizing PSNR after successful training always achieves very high PSNR values.--, in page 2, also see: Section 2.2 Network Architecture, in pages 3-4).
Re Claim 10, Wilson as modified by Yang and Hsiao further disclose Adam optimizer is adopted to optimize the generative network and the discriminator network (see Wilson: e.g., Fig. 14, and, --[0072] Super-resolution from deep learning. Another technique that can be employed by various embodiments involves using a deep learning, generative adversarial network (GAN) to create super-resolution CT calcium score images from conventional, thick slice scans. For training, paired normal clinical resolution and high resolution volume scans can be used. These can be easily obtained with existing step and shoot multi-detector CT scan acquisition. The images can be simply reconstructed at both conventional thick slice and high resolution thin slice resolutions, giving perfectly paired data. After training, the network can create a high resolution scan volume from a low resolution clinical scan volume. Using the super-resolution volume, ICmore measures can be computed more accurately and precisely. GANs can be used to reduce noise in CT images, create super-resolution CT images, and create super-resolution MR images.
[0073] In various embodiments, different schemes can be used to create super-resolution image volumes from deep neural networks, such as the following example embodiment. This example embodiment can employ a modified version of a multi-level, densely connected super-resolution network (mDCSRN) with generative adversarial network (GAN)-guided training. By utilizing a densely connected network, the mDCSRN is extremely light-weight, and when trained with a Generative Adversarial Network (GAN), can create sharp and realistic-looking images. In various embodiments, existing network structures can be employed. This technique can employ a loss (or cost) function such as that given below in equation (2), where MSE is the mean square error between the actual and synthetic high resolution image volumes, lossCAC is the difference between the calcification mass score computed form the actual and synthetic high resolution images, lossGAN is the GAN discriminator loss, and the λ's are empirically determined constants: loss=λ1lossMSE+λ2lossCAC+λ3lossGAN (1)
[0074] Using a properly optimized loss function, excellent results can be obtained.--, in [0072]-[0074]; and, --[0095] Additionally or alternatively, various embodiments can employ deep learning. For example, various embodiments can extract deep features from each time point by using a pretrained convolutional neural network (e.g., such as ResNet50, Vgg16, or DenseNet-169). For each individual calcification, a bounding box can be constructed which is centered at the calcification. ICmore deep features of each calcification can be extract from the corresponding bounding box.--, in [0095]; also see Yang: e.g., -- for CT image denoising by designing a perceptive deep CNN that relies on a perceptual loss as the objective function. During our research, it was drawn to our attention that minimizing MSE between the denoised CT image and the ground truth leads to the loss of important details, although the peak signal to noise ratio (PSNR) based evaluation numbers are excellent. That is because PSNR is equivalent to the per-pixel Euclidean intensity difference. Therefore, a model maximizing PSNR after successful training always achieves very high PSNR values.--, in page 2, also see: Section2.1 Loss Functions; and Section 2.2 Network Architecture, in pages 3-4; also see Hsiao: e.g., --Neural Network Training
[0042] Short-axis images were randomly divided and allocated to 70% for training (37,700 1.5T images+32,380 3T images), 20% for validation (10,720 1.5T images+8,720 3T images), and 10% for testing (5,240 1.5T images+4,740 3T images). Networks on two workstations were trained running Ubuntu 16.04 equipped with either two Titan Xp graphics cards or one Titan V graphics card (NVIDIA; Mountain View, Calif.). Keras with TensorFlow-GPU backend was used for all DL experiments. For all DL networks, the Adam optimizer was used with a learning rate of 1e-4. Training with early-stopping was performed for a maximum of 25 epochs.--, in [0042]). See the similar obviousness and motivation statements for the combination of the cited references addressed above for claim 4.
Conclusion
Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEI WEN YANG whose telephone number is (571)270-5670. The examiner can normally be reached on 8:00 - 5:00 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEI WEN YANG/Primary Examiner, Art Unit 2662