DETAILED ACTION
Response to Arguments
The amendments filed 5/13/2026 have been entered and made of record.
Applicant's amendments and corresponding arguments filed 5/13/2026 have been fully considered, but are moot in view of the new ground(s) of rejection because the Applicant has substantially amended at least independent claim 1 with the newly added limitations, and made arguments and remarks based on the newly added limitations:
Re Claim 1, Applicant states (in page 6 of Applicant’s Arguments and Remarks), that the Office Action using impermissible hindsight, to combine the cited references and to piece together unrelated teachings to arrive at the claimed invention using pending claims as a blueprint.
However, the Examiner disagrees, because:
all four cited references are from IDS that Applicant consider as relevant prior art, so that they are related teachings, and particularly, FORNWALT discloses using neural network in diagnosis of congenital heart defect based on echocardiographic video; and Ko discloses The artificial intelligence model (AIM) includes various forms of normal state fetal images in the plurality of fetal image data (ID1) included in the input data, various types of images of fetuses with congenital diseases, and various forms of severe congenital diseases; and, LIU specifically teaches deep learning methods to identify motion corresponding to fetal anatomy , the ultrasonic video data can also be classified by traditional image processing methods; for example, 3D Harris corner detection operator, computational optical flow, 3D SIFT operator and other algorithms may be used to extract features of spatial and temporal dimensions, including HOG, HOF, MBH and other features, and then classification algorithms such as SVM may be adopted to classify to obtain the category of ultrasonic video data; and HAEUSSER discloses (iv) process the conditioned electrogram data and positional data in the trained machine learning model to generate the one or more predictions or results; and (v) display the one or more predictions or results on the display or monitor to the user…. (d) providing an estimate of the probability of recurrence of atrial fibrillation in the patient;
As pointed out in the Office Action, these four cited references are combinable as they are in the same field of endeavor: image processing and analysis for determine heart defects, and particularly based on echocardiographic video, fetal images, the ultrasonic video data and electrogram data, those of commonly applied medical images and measurements processing and artificial intelligences, and involving the same or similar technologies including feature extractions from these midcalf images and to make diagnosis, and diseases detection based on analysis, estimation, prediction, and classification by using neural network, or machine learning algorithms.
Applicant states that cited references do not disclose the newly added limitation: “preprocessing at least one image frame of the series of image frames to remove a portion of the image frame from such at least one image frame to generate preprocessed image data, the portion of the image frame removed from such at least one image frame comprising one or more of text or background visual information”, and,
“wherein preprocessing the at least one image frame of the series of image frames removes noise from the image data prior to applying the image data to the neural network system”;
However, the Examiner disagrees, because
First, these two newly added limitations directed to the same preprocessing step of preprocess the image frames, and remove portion of the image frame so that only meaningful portion of the image frame will be applied as input to a neural network, and the preprocessing also includes removal of noises. That is a common practice in the art of using neural network in performing image analysis, and such image preprocessing would not be considered as inventive limitations. Particularly,
FORNWALT discloses preprocessing at least one image frame of the series of image frames to remove a portion of the image frame from such at least one image frame to generate preprocessed image data, the portion of the image frame removed from such at least one image frame comprising one or more of text or background visual information (see FORNWALT: e.g., --Image Collection and Preprocessing [0072] An echocardiography study consists of several videos containing multiple views of the heart. …All continuous variables were cleaned from physiologically out of limit values, which may have been caused by input errors. In cases where no limits could be defined for a measurement, extreme outliers were removed that met two rules: 1) Value beyond the mean plus or minus three standard deviations and 2) Value below the 25th percentile minus 3 interquartile ranges or above the 75.sup.th percentile plus 3 interquartile ranges. The removed outlier values were set as missing.--, in [0076]; also see Ko: e.g., --TECH-SOLUTION: …generating respective 3D images by separating at least one pre-set fetal organ from among the baby house, the amniotic fluid, the fetus, and the fetal organ from the fetal image;--, in section under: Tech. Solution, in pages 2-3 of machine translated English version of WO 2022265345 A1, as provided with the Office Action; also see Figs. 8-11, particularly, see Fig. 10, and Fig. 11; wherein “ the amniotic fluid” and other portion/components are removed; and in Fig. 11, only “fetal organ” is left, other portion/components {as background} are removed;
also see: --Since each video from a view group could potentially have different dimensions, all videos were normalized from a view to the most common row and column dimensions. Each frame was cropped/padded with zeros to match the most common dimensions among the view group. Ultimately, Philips-generated DICOM files with raw videos and view labels were retrieved and any videos that lasted less than 1 second were excluded.--, in [0074]; {apparently, herein “cropped” is preprocessing to remove background, or extra of over dimensions from the image frame}); and
FORNWALT further discloses using filters in image preprocessing (see FORNWALT: e.g., -- reduced the fully connected layer input from the feature map size to the number of filters.--, in [0139]); and,
Furthermore, HAEUSSER discloses wherein preprocessing the at least one image frame of the series of image frames removes noise from the image data prior to applying the image data to the neural network system (see HAEUSSER: e.g., -- [0154] At step 204, a high-pass filter is applied to the acquired EP data to remove DC offsets, as well as other undesirable low-frequency noise. In one embodiment, a 5 Hz high-pass filter is applied, although other filters, including band-pass filters, are contemplated, including, but not limited to, 10 Hz high-pass filters, 5-20 Hz band-pass filters, and 5-50 Hz band-pass filters. Notch- and low-pass filtering may also be applied in step 204. Hanning, trapezoidal and other digital filtering and/or Fast Fourier Transform (FFT) filtering techniques may also be applied.
[0155] At step 206, an average far-field electrogram signal is generated by stacking and averaging all electrogram traces. In the case of atrial EP recordings, the resulting estimate of a far-field ventricular depolarization is subtracted from each trace individually, thereby removing or at least reducing the far-field component therefrom.
[0156] At step 208, the amplitudes of individual filtered electrogram signals are normalized with respect to a given standard deviation occurring over a predetermined time window (e.g., a moving window of 200 samples around a time value “x”).--, in [0154]-[0156]);
And it is a common practice to remove noises from images being further processed and analyzed, in view of easily found many relevant prior arts in the art of medical image processing with the neural networks, such as the prior art made of record and not relied upon is considered pertinent to applicant's disclosure: Roffé (US 10667776 B2), as provided in the Conclusion Section of the last Office Action, and as listed again in this Office Action.
Therefore, amended claims 1-20 are still not patentably distinguishable over the prior art reference(s). Further discussions are addressed in the prior art rejection section below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 is rejected under 35 U.S.C. 103 as being patentable over FORNWALT (US 20210150693 A1, as provided in IDS), in view of KO (WO 2022265345 A1, as provided in IDS), also in view of LIU (US 20230135046 A1, as provided in IDS), and further in view of HAEUSSER (US 20200345261 A1, as provided in IDS).
Re Claim 1, FORNWALT disclose a method for analyzing medical images (see FORNWALT: e.g., --In the method, the trained model can further include a trained submodel, and the method can further include providing the echocardiographic video to a first trained neural network included in the plurality of trained neural networks, receiving a second echocardiographic video of the heart associated with the patient, providing the second echocardiographic video to a second trained neural network included in the plurality of trained neural networks, receiving a first video risk score from the first trained neural network, and receiving a second video risk score from the second trained neural network…. diagnosis of congenital heart defect, diagnosis of dyslipidemia, and diagnosis of chronic kidney disease. The first trained neural network can be associated with a first network architecture and the second trained neural network can be associated with a second network architecture. The first network architecture can include a two-dimensional convolutional neural network, and the second network architecture can include a three-dimensional convolutional neural network--, in [0010]);
FORNWALT does not explicitly disclose above analyzing medical images corresponding to a fetus during pregnancy,
Ko discloses analyzing medical images corresponding to a fetus during pregnancy (see Ko: e.g., -- The fetal image may be characterized in that a plurality of fetal image data as metadata are learned and output based on an artificial intelligence model… outputting the 3D printer is generated by matching the state information of the fetus to the fetal image. The condition information of the fetus may include at least one of a normal state, a state with a congenital disease, and a state with a severe congenital disease.--, and, Fig. 1, and, -- the fetus image obtaining; generating respective 3D images by separating at least one pre-set fetal organ from among the baby house, the amniotic fluid, the fetus, and the fetal organ from the fetal image--, in section under: Tech. Solution, in pages 2-3 of machine translated English version of WO 2022265345 A1, as provided with the Office Action; and, -- The fetal image for each fetus may include at least one of a fetal MRI image and a fetal CT image for fetal treatment. For each fetus image, a plurality of fetal image data as metadata may be learned and output based on an artificial intelligence model. The plurality of fetal image data may include various types of images of a fetus in a normal state, images of a fetus in a state with various types of congenital diseases, and images of a fetus in a state with various types of severe congenital diseases.--, in the last paragraph in page 6 of machine translated English version of WO 2022265345 A1, and, Fig. 4, -- The artificial intelligence model (AIM) includes various forms of normal state fetal images in the plurality of fetal image data (ID1) included in the input data, various types of images of fetuses with congenital diseases, and various forms of severe congenital diseases. It can be built to learn through correlation of fetal images. The artificial intelligence model (AIM) uses a plurality of fetal image data (ID1), including images of fetuses in various forms of normal conditions, images of fetuses in various forms of congenital diseases, and images of fetuses in various forms of severe congenital diseases. A CNN algorithm or RNN algorithm can be used to build and reinforce learning as a learning data set. At this time, the fetal image data OD for each fetus may include fetal image data OD1 in a normal state, fetal image data OD2 in a state with a congenital disease, and fetal image data OD3 in a state with a severe congenital disease.--, in the first paragraph in page 7 of machine translated English version of WO 2022265345 A1);
FORNWALT and KO are combinable as they are in the same field of endeavor: image processing and analysis for determine heart defects. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify FORNWALT’s method using KO’s teachings by including analyzing medical images corresponding to a fetus during pregnancy to FORNWALT’s analyzing medical images in order to CNN algorithm or RNN algorithm can be used to build and reinforce learning of images of fetuses in various forms of normal conditions, images of fetuses in various forms of congenital diseases (see KO: e.g. in pages 2-3, and pages 6-7 of machine translated English version of WO 2022265345 A1);
FORNWALT as modified by KO further disclose the method comprising:
determining image data that is representative of a portion of the fetus’s anatomy, the image data comprising a series of image frames (see Ko: e.g., Fig. 1, and, -- the fetus image obtaining; generating respective 3D images by separating at least one pre-set fetal organ from among the baby house, the amniotic fluid, the fetus, and the fetal organ from the fetal image--, and, -- The fetal image may be characterized in that a plurality of fetal image data as metadata are learned and output based on an artificial intelligence model… outputting the 3D printer is generated by matching the state information of the fetus to the fetal image. The condition information of the fetus may include at least one of a normal state, a state with a congenital disease, and a state with a severe congenital disease.--, in section under: Tech. Solution, in pages 2-3 of machine translated English version of WO 2022265345 A1, as provided with the Office Action; and, -- The fetal image for each fetus may include at least one of a fetal MRI image and a fetal CT image for fetal treatment. For each fetus image, a plurality of fetal image data as metadata may be learned and output based on an artificial intelligence model. The plurality of fetal image data may include various types of images of a fetus in a normal state, images of a fetus in a state with various types of congenital diseases, and images of a fetus in a state with various types of severe congenital diseases.--, in the last paragraph in page 6 of machine translated English version of WO 2022265345 A1, and, Fig. 4, -- The artificial intelligence model (AIM) includes various forms of normal state fetal images in the plurality of fetal image data (ID1) included in the input data, various types of images of fetuses with congenital diseases, and various forms of severe congenital diseases. It can be built to learn through correlation of fetal images. The artificial intelligence model (AIM) uses a plurality of fetal image data (ID1), including images of fetuses in various forms of normal conditions, images of fetuses in various forms of congenital diseases, and images of fetuses in various forms of severe congenital diseases. A CNN algorithm or RNN algorithm can be used to build and reinforce learning as a learning data set. At this time, the fetal image data OD for each fetus may include fetal image data OD1 in a normal state, fetal image data OD2 in a state with a congenital disease, and fetal image data OD3 in a state with a severe congenital disease.--, in the first paragraph in page 7 of machine translated English version of WO 2022265345 A1);
preprocessing at least one image frame of the series of image frames to remove a portion of the image frame such at least one image frame to generate preprocessed image data, the portion of the image frame removed from such at least one image frame comprising one or more of text or background visual information (see FORNWALT: e.g., --Image Collection and Preprocessing [0072] An echocardiography study consists of several videos containing multiple views of the heart. …All continuous variables were cleaned from physiologically out of limit values, which may have been caused by input errors. In cases where no limits could be defined for a measurement, extreme outliers were removed that met two rules: 1) Value beyond the mean plus or minus three standard deviations and 2) Value below the 25th percentile minus 3 interquartile ranges or above the 75.sup.th percentile plus 3 interquartile ranges. The removed outlier values were set as missing.--, in [0076]; also see Ko: e.g., --TECH-SOLUTION: …generating respective 3D images by separating at least one pre-set fetal organ from among the baby house, the amniotic fluid, the fetus, and the fetal organ from the fetal image;--, in section under: Tech. Solution, in pages 2-3 of machine translated English version of WO 2022265345 A1, as provided with the Office Action; also see Figs. 8-11, particularly, see Fig. 10, and Fig. 11; wherein “ the amniotic fluid” and other portion/components are removed; and in Fig. 11, only “fetal organ” is left, other portion/components {as background} are removed;
also see: --Since each video from a view group could potentially have different dimensions, all videos were normalized from a view to the most common row and column dimensions. Each frame was cropped/padded with zeros to match the most common dimensions among the view group. Ultimately, Philips-generated DICOM files with raw videos and view labels were retrieved and any videos that lasted less than 1 second were excluded.--, in [0074]; {apparently, herein “cropped” is preprocessing to remove background, or extra of over dimensions from the image frame});
applying the preprocessed image data to a neural network system trained to identify fetal anatomy (see FORNWALT: e.g., -- receiving an echocardiographic video of a heart associated with a patient, the echocardiographic video including a plurality of video frames, analyzing one or more regions of the heart using a trained model to generate a patient analysis, and generating a mortality risk score based on the patient analysis….. the trained model can further include a trained submodel, and the method can further include providing the echocardiographic video to a first trained neural network included in the plurality of trained neural networks, receiving a second echocardiographic video of the heart associated with the patient, providing the second echocardiographic video to a second trained neural network included in the plurality of trained neural networks, receiving a first video risk score from the first trained neural network, and receiving a second video risk score from the second trained neural network. The generating the mortality risk score can include providing the first video risk score and the second video risk score to the trained submodel, and receiving the mortality risk score from the trained submodel. The trained submodel can include a trained classifier. The trained classifier can be an XGboost classifier. The generating the mortality risk score can further include providing electronic health record information associated with the patient to the trained submodel.--, in [0005]-[0010], and, -- “deep” learning (deep neural network; DNN) technologies; such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNN), Dropout Regularization, and adaptive gradient descent algorithms; in conjunction with massively parallel computational hardware (graphic processing units), have enabled state-of-the-art predictive models for image, time-series, and video-based data…. A fully 3D Convolutional Neural Network (CNN) design is utilized in this study (FIG. 1). CNNs are neural networks that exploit spatial coherence in an image to significantly reduce the number of parameters that a fully connected network would need to learn. CNNs have shown promise in image classification tasks,--, in [0051]-[0058]; and, --[0139] As opposed to the 2D CNN approach, a 3D CNN incorporated information from adjacent frames at every layer, extracting spatiotemporal dependent features which have also proven to be useful for video classification. In a 3D CNN approach, a GAP layer reduced the fully connected layer input from the feature map size to the number of filters.--, in [0138]-[0140]; also see KO: Fig. 4, -- The artificial intelligence model (AIM) includes various forms of normal state fetal images in the plurality of fetal image data (ID1) included in the input data, various types of images of fetuses with congenital diseases, and various forms of severe congenital diseases. It can be built to learn through correlation of fetal images. The artificial intelligence model (AIM) uses a plurality of fetal image data (ID1), including images of fetuses in various forms of normal conditions, images of fetuses in various forms of congenital diseases, and images of fetuses in various forms of severe congenital diseases. A CNN algorithm or RNN algorithm can be used to build and reinforce learning as a learning data set. At this time, the fetal image data OD for each fetus may include fetal image data OD1 in a normal state, fetal image data OD2 in a state with a congenital disease, and fetal image data OD3 in a state with a severe congenital disease.--, in the first paragraph in page 7 of machine translated English version of WO 2022265345 A1);
FORNWALT as modified by KO however do not explicitly disclose a neural network system including {a temporal model} trained to {process the image data} to identify motion corresponding to fetal anatomy;
LIU discloses a neural network system including {a temporal model} trained to {process the image data} to identify motion corresponding to fetal anatomy (see LIU: e.g., -- three-dimensional spatial and temporal features can be obtained by deep learning methods, that is, first using CNN, RNN, 3D full convolution network, LSTM and other deep neural networks to extract two-dimensional spatial features and temporal features from ultrasonic video data. For example, image features may be obtained from a single frame by using CNN, and then the obtained image features may be integrated in temporal dimension by using RNN network. Alternatively, 3D convolution kernels may be directly used to extract image features from frames, and to model motion sequences among adjacent frames. Alternatively, CNN is used for feature extraction of frames, and then optical flow images are used to extract temporal dimension features. Alternatively, two-dimensional CNN is used to extract the features of the scanning part frame by frame, and then LSTM network is used to model the features obtained by the two-dimensional CNN in temporal dimension.--, in [0161]-[0162]);
FORNWALT (as modified by KO) and LIU are combinable as they are in the same field of endeavor: medical images, such as ultrasound and echocardiographs processing and analysis for determine heart defects. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify FORNWALT (as modified by KO)’s method using LIU’s teachings by including {a temporal model} trained to {process the image data} to identify motion corresponding to fetal anatomy to FORNWALT (as modified by KO)’s trained neural network and artificial intelligence model in order to extract two-dimensional spatial features and temporal features from ultrasonic video data and to model motion sequences among adjacent frames (see LIU: e.g. in [0161]-[0162]);
generating a spatiotemporal output using the neural network system and based on the preprocessed image data, the spatiotemporal output corresponding to predetermined anatomy of the fetus over a time period (see FORNWALT: e.g., -- receiving an echocardiographic video of a heart associated with a patient, the echocardiographic video including a plurality of video frames, analyzing one or more regions of the heart using a trained model to generate a patient analysis, and generating a mortality risk score based on the patient analysis….. the trained model can further include a trained submodel, and the method can further include providing the echocardiographic video to a first trained neural network included in the plurality of trained neural networks, receiving a second echocardiographic video of the heart associated with the patient, providing the second echocardiographic video to a second trained neural network included in the plurality of trained neural networks, receiving a first video risk score from the first trained neural network, and receiving a second video risk score from the second trained neural network. The generating the mortality risk score can include providing the first video risk score and the second video risk score to the trained submodel, and receiving the mortality risk score from the trained submodel. The trained submodel can include a trained classifier. The trained classifier can be an XGboost classifier. The generating the mortality risk score can further include providing electronic health record information associated with the patient to the trained submodel.--, in [0005]-[0010], and, -- “deep” learning (deep neural network; DNN) technologies; such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNN), Dropout Regularization, and adaptive gradient descent algorithms; in conjunction with massively parallel computational hardware (graphic processing units), have enabled state-of-the-art predictive models for image, time-series, and video-based data…. A fully 3D Convolutional Neural Network (CNN) design is utilized in this study (FIG. 1). CNNs are neural networks that exploit spatial coherence in an image to significantly reduce the number of parameters that a fully connected network would need to learn. CNNs have shown promise in image classification tasks,--, in [0051]-[0058]; and, --[0139] As opposed to the 2D CNN approach, a 3D CNN incorporated information from adjacent frames at every layer, extracting spatiotemporal dependent features which have also proven to be useful for video classification. In a 3D CNN approach, a GAP layer reduced the fully connected layer input from the feature map size to the number of filters.--, in [0138]-[0140]; also see KO: Fig. 4, -- The artificial intelligence model (AIM) includes various forms of normal state fetal images in the plurality of fetal image data (ID1) included in the input data, various types of images of fetuses with congenital diseases, and various forms of severe congenital diseases. It can be built to learn through correlation of fetal images. The artificial intelligence model (AIM) uses a plurality of fetal image data (ID1), including images of fetuses in various forms of normal conditions, images of fetuses in various forms of congenital diseases, and images of fetuses in various forms of severe congenital diseases. A CNN algorithm or RNN algorithm can be used to build and reinforce learning as a learning data set. At this time, the fetal image data OD for each fetus may include fetal image data OD1 in a normal state, fetal image data OD2 in a state with a congenital disease, and fetal image data OD3 in a state with a severe congenital disease.--, in the first paragraph in page 7 of machine translated English version of WO 2022265345 A1; and see LIU: e.g., -- three-dimensional spatial and temporal features can be obtained by deep learning methods, that is, first using CNN, RNN, 3D full convolution network, LSTM and other deep neural networks to extract two-dimensional spatial features and temporal features from ultrasonic video data. For example, image features may be obtained from a single frame by using CNN, and then the obtained image features may be integrated in temporal dimension by using RNN network. Alternatively, 3D convolution kernels may be directly used to extract image features from frames, and to model motion sequences among adjacent frames. Alternatively, CNN is used for feature extraction of frames, and then optical flow images are used to extract temporal dimension features. Alternatively, two-dimensional CNN is used to extract the features of the scanning part frame by frame, and then LSTM network is used to model the features obtained by the two-dimensional CNN in temporal dimension.--, in [0161]-[0162]);
determining a presence of a predetermined condition of a plurality of predetermined conditions based on the spatiotemporal output (see FORNWALT: e.g., -- The generating the mortality risk score can include providing the first video risk score and the second video risk score to the trained submodel, and receiving the mortality risk score from the trained submodel. …. The diagnosis parameters can include diagnosis of acute rheumatic fever, diagnosis of chronic rheumatic heart disease, diagnosis of hypertensive diseases, diagnosis of ischemic heart diseases, diagnosis of pulmonary heart disease and diseases of pulmonary circulation, diagnosis of acute pericarditis, diagnosis of other forms of heart disease, diagnosis of acute myocarditis, diagnosis of cardiomyopathy, diagnosis of cardiac arrest, diagnosis of paroxysmal tachycardia, diagnosis of atrial fibrillation, diagnosis of heart failure, diagnosis of cerebrovascular diseases, diagnosis of diseases of arteries, arterioles and capillaries, diagnosis of diseases of veins, lymphatic vessels, and lymph nodes, diagnosis of hypotension, diagnosis of other and unspecified disorders of the circulatory system, diagnosis of diabetes mellitus, diagnosis of congenital heart defect, diagnosis of dyslipidemia, and diagnosis of chronic kidney disease. The first trained neural network can be associated with a first network architecture and the second trained neural network can be associated with a second network architecture. The first network architecture can include a two-dimensional convolutional neural network, and the second network architecture can include a three-dimensional convolutional neural network.--, in [0010]; {so that, diagnosis of congenital heart defect, read on” the spatiotemporal output indicative of a presence of a predetermined condition of a plurality of predetermined conditions}); and
causing a device to display a user interface corresponding to the spatiotemporal output (see Liu: e.g., --[0162] In addition to deep learning methods, the ultrasonic video data can also be classified by traditional image processing methods; for example, 3D Harris corner detection operator, computational optical flow, 3D SIFT operator and other algorithms may be used to extract features of spatial and temporal dimensions, including HOG, HOF, MBH and other features, and then classification algorithms such as SVM may be adopted to classify to obtain the category of ultrasonic video data.
[0163] If the ultrasonic data to be displayed is two-dimensional ultrasonic image data, the classification method thereof is similar to that of the representative frame of the ultrasonic video data. For details, please refer to the above. [0164] After the scanning part of the ultrasound data is determined, the ultrasonic data is displayed in association with the corresponding portions of the body icon according to the scanning part of the ultrasonic data, referring to FIG. 4A and FIG. 4B. The scanning part of an ultrasound image is the tissue part corresponding to the ultrasound image. Specifically, an identifier of the ultrasonic data is displayed at different portions of the body icon, the identifier of the ultrasound data can be a small map of any frame of ultrasound data; and when a selection instruction for the identifier is acquired, the ultrasonic data corresponding to the scanning part is displayed according to the scanning part corresponding to the identifier.--, in [0162]-[0164]; also see KO: e.g., Fig. 1, and, -- Referring to FIG. 5 , the processor 120 separates (segmentation) the nursery (P1), the amniotic fluid (P2), the fetus (P3), and the organ (P3-1) of the fetus (P3) from the fetal image (P). 3D images can be generated. For example, the processor 120 may perform an image of a fetus in a normal state (P11 in FIG. 1), an image of a fetus in a state with a congenital disease (P12 in FIG. 1), and an image of a fetus in a state with a severe congenital disease (P13 in FIG. 1). Each 3D image may be generated by separating at least one pre-determined fetal organ among each baby house, each amniotic fluid, each fetus, and each fetus organ. Here, the preset at least one organ of the fetus may be a major organ of the fetus. At this time, the main organ of the fetus may be any one or two or more of the heart, cerebrum, cerebellum, ventricle, stomach, liver, kidney, and bladder.
Here, “segmentation” of the fetal image P means a process of dividing a digital image into a plurality of pixel or voxel sets. The purpose of segmentation is to simplify or transform the representation of the image into something more meaningful and easier to interpret. Image segmentation is used to find objects and boundaries in images. The result of segmentation is a set of regions collectively including the entire image or a set of contours extracted from the image.--, in page 7 of machine translated English version of WO 2022265345 A1);
FORNWALT as modified by KO and Liu however do not explicitly disclose display the predetermined condition corresponding to the spatiotemporal output,
HAEUSSER (US 20200345261 A1, as provided in IDS), {which has been applied, as the prior art for the parent applications} discloses display the predetermined condition corresponding to the spatiotemporal output (see HAEUSSER: e.g., --(1) Does the patient have atrial fibrillation or not? (2) If the patient has atrial fibrillation, determining at least one of the spatial variability level, the activity level, and the flow angle stability level associated with one or more sources detected in the patient's heart; (3) If the patient has atrial fibrillation, determining the locations of one or more sources detected in the patient's heart; (4) If the patient has atrial fibrillation, whether one or more activation sources detected in the patient's heart are characterized by chaotic flow; and (5) classification of the patient as one of types A, B and C; and further wherein the computing device is configured to: (iv) process the conditioned electrogram data and positional data in the trained machine learning model to generate the one or more predictions or results; and (v) display the one or more predictions or results on the display or monitor to the user…. (d) providing an estimate of the probability of recurrence of atrial fibrillation in the patient;--, in [0282]-[0285];
also see: --The resulting sliding-window amplitude-adjusted electrogram signals were then stored for later use to generate image backgrounds in velocity vector maps, where they could be used to show low amplitude areas indicative of valve defects/artifacts, loose electrode contact, and/or areas of fibrosis in the patient's myocardium.--, in [0147]-[0148], similarly, --where electrograms processed using sliding-window techniques were used to generate the image background (including the white areas) shown in the velocity vector map of FIG. 7(j). The white areas in FIG. 7(j) thus correspond to low amplitude areas potentially indicative of valve defects or artifacts, loose electrode contact, and/or areas of fibrosis in the patient's myocardium.--, in [0170]-[0172], and see: Fig. 9, and, -- FIG. 9, there is shown another example of a vector velocity map generated from actual patient data using method 200… As shown in FIG. 9, various cardiac rhythm defects and disorders become apparent as a result of the generated vector velocity map.--, in [0176]; and, -- generating a two-dimensional spatial map, grid or representation of the electrode positions, processing the amplitude-adjusted and filtered electrogram signals to generate a plurality of three-dimensional electrogram surfaces corresponding at least partially to the 2D map, one surface being generated for each or selected discrete times, and processing the plurality of three-dimensional electrogram surfaces through time to generate a velocity vector or other type of map using one or more of optical flow,--, in abstract, [0021]-[0023], and, -- In conventional optical flow analysis, image brightness is considered at pixel (x,y) in an image plane at time t to be represented as a function I(x,y,t).--, in [0109]-[0120]; --the velocity vector or other map being generated by the computing device using at least one of an optical flow analysis technique ……(15) at least one optical flow analysis technique, if employed, being selected from the group consisting of a Horn-Schunck method, a Buxton-Buston method, a Black-Jepson method, a phase correlation method, a block-based method, a discrete optimization method, a Lucas-Kanade method, and a differential method of estimating optical flow; (16) the at least one processor and the at least one non-transitory computer readable medium being configured to determine, using a trained atrial discriminative machine learning model, predictions or results concerning atrial fibrillation in the patient's heart; (17) the trained atrial discriminative machine learning model having been trained at least partially using data obtained from a plurality of other previous patients, where intracardiac electrophysiological (EP) mapping signals for the other patients have been processed using one or more of electrographic flow (EGF), video tracking analysis, motion capture analysis, motion estimation analysis, a data association and segmentation tracking analysis, particle tracking analysis, and single-particle tracking analysis methods to detect at least one of: (I) the presence of sources of atrial fibrillation in the other patients' hearts--, in [0262], and, Fig. 18, in [0272], and [0282]-[0285]);
HAEUSSER and FORNWALT (as modified by LIU and KO) are combinable as they are in the same field of endeavor: image processing and analysis for determine heart defects. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify FORNWALT(as modified by LIU and KO)’s method using HAEUSSER’s teachings by including display the predetermined condition corresponding to the spatiotemporal output to FORNWALT (as modified by LIU and KO)’s display the predetermined condition in order to provide the determination results to users and physicians in pathological diagnosis (see HAEUSSER: e.g. in [0262], and, Fig. 18, and in [0272]);
FORNWALT as modified by KO and LIU and HAEUSSER further disclose wherein preprocessing the at least one image frame of the series of image frames removes noise from the image data prior to applying the image data to the neural network system (see HAEUSSER: e.g., -- [0154] At step 204, a high-pass filter is applied to the acquired EP data to remove DC offsets, as well as other undesirable low-frequency noise. In one embodiment, a 5 Hz high-pass filter is applied, although other filters, including band-pass filters, are contemplated, including, but not limited to, 10 Hz high-pass filters, 5-20 Hz band-pass filters, and 5-50 Hz band-pass filters. Notch- and low-pass filtering may also be applied in step 204. Hanning, trapezoidal and other digital filtering and/or Fast Fourier Transform (FFT) filtering techniques may also be applied.
[0155] At step 206, an average far-field electrogram signal is generated by stacking and averaging all electrogram traces. In the case of atrial EP recordings, the resulting estimate of a far-field ventricular depolarization is subtracted from each trace individually, thereby removing or at least reducing the far-field component therefrom.
[0156] At step 208, the amplitudes of individual filtered electrogram signals are normalized with respect to a given standard deviation occurring over a predetermined time window (e.g., a moving window of 200 samples around a time value “x”).--, in [0154]-[0156]; also see FORNWALT: e.g., -- reduced the fully connected layer input from the feature map size to the number of filters.--, in [0139]).
Re Claim 2, FORNWALT as modified by KO and LIU and HAEUSSER further disclose wherein the neural network system comprises a spatial 2. model and a temporal model and the spatial model identifies the fetal anatomy and the temporal model identifies the motion corresponding to the fetal anatomy (see FORNWALT: e.g., -- receiving an echocardiographic video of a heart associated with a patient, the echocardiographic video including a plurality of video frames, analyzing one or more regions of the heart using a trained model to generate a patient analysis, and generating a mortality risk score based on the patient analysis….. the trained model can further include a trained submodel, and the method can further include providing the echocardiographic video to a first trained neural network included in the plurality of trained neural networks, receiving a second echocardiographic video of the heart associated with the patient, providing the second echocardiographic video to a second trained neural network included in the plurality of trained neural networks, receiving a first video risk score from the first trained neural network, and receiving a second video risk score from the second trained neural network. The generating the mortality risk score can include providing the first video risk score and the second video risk score to the trained submodel, and receiving the mortality risk score from the trained submodel. The trained submodel can include a trained classifier. The trained classifier can be an XGboost classifier. The generating the mortality risk score can further include providing electronic health record information associated with the patient to the trained submodel.--, in [0005]-[0010], and, -- “deep” learning (deep neural network; DNN) technologies; such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNN), Dropout Regularization, and adaptive gradient descent algorithms; in conjunction with massively parallel computational hardware (graphic processing units), have enabled state-of-the-art predictive models for image, time-series, and video-based data…. A fully 3D Convolutional Neural Network (CNN) design is utilized in this study (FIG. 1). CNNs are neural networks that exploit spatial coherence in an image to significantly reduce the number of parameters that a fully connected network would need to learn. CNNs have shown promise in image classification tasks,--, in [0051]-[0058]; and, --[0139] As opposed to the 2D CNN approach, a 3D CNN incorporated information from adjacent frames at every layer, extracting spatiotemporal dependent features which have also proven to be useful for video classification. In a 3D CNN approach, a GAP layer reduced the fully connected layer input from the feature map size to the number of filters.--, in [0138]-[0140]; also see KO: Fig. 4, -- The artificial intelligence model (AIM) includes various forms of normal state fetal images in the plurality of fetal image data (ID1) included in the input data, various types of images of fetuses with congenital diseases, and various forms of severe congenital diseases. It can be built to learn through correlation of fetal images. The artificial intelligence model (AIM) uses a plurality of fetal image data (ID1), including images of fetuses in various forms of normal conditions, images of fetuses in various forms of congenital diseases, and images of fetuses in various forms of severe congenital diseases. A CNN algorithm or RNN algorithm can be used to build and reinforce learning as a learning data set. At this time, the fetal image data OD for each fetus may include fetal image data OD1 in a normal state, fetal image data OD2 in a state with a congenital disease, and fetal image data OD3 in a state with a severe congenital disease.--, in the first paragraph in page 7 of machine translated English version of WO 2022265345 A1; and see LIU: e.g., -- three-dimensional spatial and temporal features can be obtained by deep learning methods, that is, first using CNN, RNN, 3D full convolution network, LSTM and other deep neural networks to extract two-dimensional spatial features and temporal features from ultrasonic video data. For example, image features may be obtained from a single frame by using CNN, and then the obtained image features may be integrated in temporal dimension by using RNN network. Alternatively, 3D convolution kernels may be directly used to extract image features from frames, and to model motion sequences among adjacent frames. Alternatively, CNN is used for feature extraction of frames, and then optical flow images are used to extract temporal dimension features. Alternatively, two-dimensional CNN is used to extract the features of the scanning part frame by frame, and then LSTM network is used to model the features obtained by the two-dimensional CNN in temporal dimension.--, in [0161]-[0162]).
Re Claim 3, FORNWALT as modified by KO and LIU and HAEUSSER further disclose wherein the predetermined anatomy is a ventricle, atria, or heart valve (see KO: e.g., -- The fetal image may be characterized in that a plurality of fetal image data as metadata are learned and output based on an artificial intelligence model… outputting the 3D printer is generated by matching the state information of the fetus to the fetal image. The condition information of the fetus may include at least one of a normal state, a state with a congenital disease, and a state with a severe congenital disease.--, and, Fig. 1, and, -- the fetus image obtaining; generating respective 3D images by separating at least one pre-set fetal organ from among the baby house, the amniotic fluid, the fetus, and the fetal organ from the fetal image--, in section under: Tech. Solution, in pages 2-3 of machine translated English version of WO 2022265345 A1, as provided with the Office Action; and, -- Referring to FIG. 5 , the processor 120 separates (segmentation) the nursery (P1), the amniotic fluid (P2), the fetus (P3), and the organ (P3-1) of the fetus (P3) from the fetal image (P). 3D images can be generated. For example, the processor 120 may perform an image of a fetus in a normal state (P11 in FIG. 1), an image of a fetus in a state with a congenital disease (P12 in FIG. 1), and an image of a fetus in a state with a severe congenital disease (P13 in FIG. 1). Each 3D image may be generated by separating at least one pre-determined fetal organ among each baby house, each amniotic fluid, each fetus, and each fetus organ. Here, the preset at least one organ of the fetus may be a major organ of the fetus. At this time, the main organ of the fetus may be any one or two or more of the heart, cerebrum, cerebellum, ventricle, stomach, liver, kidney, and bladder.
Here, “segmentation” of the fetal image P means a process of dividing a digital image into a plurality of pixel or voxel sets. The purpose of segmentation is to simplify or transform the representation of the image into something more meaningful and easier to interpret. Image segmentation is used to find objects and boundaries in images. The result of segmentation is a set of regions collectively including the entire image or a set of contours extracted from the image.--, in page 7 of machine translated English version of WO 2022265345 A1).
Re Claim 4, FORNWALT as modified by KO and LIU and HAEUSSER further disclose wherein the spatiotemporal output is indicative of one of systole, diastole, contraction, or ejection (see FORNWALT: e.g., --the trained model can further include a trained submodel, and the method can further include providing the echocardiographic video to a first trained neural network included in the plurality of trained neural networks, receiving a second echocardiographic video of the heart associated with the patient, providing the second echocardiographic video to a second trained neural network included in the plurality of trained neural networks, receiving a first video risk score from the first trained neural network, and receiving a second video risk score from the second trained neural network. The generating the mortality risk score can include providing the first video risk score and the second video risk score to the trained submodel, and receiving the mortality risk score from the trained submodel. The trained submodel can include a trained classifier. The trained classifier can be an XGboost classifier. The generating the mortality risk score can further include providing electronic health record information associated with the patient to the trained submodel. The electronic health record information can include values of a number of parameters including age, tricuspid regurgitation maximum velocity, heart rate, low density lipoprotein, left ventricular ejection fraction, diastolic pressure, pulmonary artery acceleration time, systolic pressure, pulmonary artery acceleration slope, and diastolic function, the values being associated with the patient. The electronic health record information can include values of a number of parameters including demographic parameters, vitals parameters, laboratory measurement parameters, echocardiogram-based parameters, and diagnosis parameters. The demographic parameters can include age, sex, and smoking status. The vitals parameters can include height, weight, heart rate, diastolic blood pressure, and systolic blood pressure.--, in [0010]; also see KO: e.g., -- The fetal image may be characterized in that a plurality of fetal image data as metadata are learned and output based on an artificial intelligence model… outputting the 3D printer is generated by matching the state information of the fetus to the fetal image. The condition information of the fetus may include at least one of a normal state, a state with a congenital disease, and a state with a severe congenital disease.--, and, Fig. 1, and, -- the fetus image obtaining; generating respective 3D images by separating at least one pre-set fetal organ from among the baby house, the amniotic fluid, the fetus, and the fetal organ from the fetal image--, in section under: Tech. Solution, in pages 2-3 of machine translated English version of WO 2022265345 A1, as provided with the Office Action; and, -- Referring to FIG. 5 , the processor 120 separates (segmentation) the nursery (P1), the amniotic fluid (P2), the fetus (P3), and the organ (P3-1) of the fetus (P3) from the fetal image (P). 3D images can be generated. For example, the processor 120 may perform an image of a fetus in a normal state (P11 in FIG. 1), an image of a fetus in a state with a congenital disease (P12 in FIG. 1), and an image of a fetus in a state with a severe congenital disease (P13 in FIG. 1). Each 3D image may be generated by separating at least one pre-determined fetal organ among each baby house, each amniotic fluid, each fetus, and each fetus organ. Here, the preset at least one organ of the fetus may be a major organ of the fetus. At this time, the main organ of the fetus may be any one or two or more of the heart, cerebrum, cerebellum, ventricle, stomach, liver, kidney, and bladder.
Here, “segmentation” of the fetal image P means a process of dividing a digital image into a plurality of pixel or voxel sets. The purpose of segmentation is to simplify or transform the representation of the image into something more meaningful and easier to interpret. Image segmentation is used to find objects and boundaries in images. The result of segmentation is a set of regions collectively including the entire image or a set of contours extracted from the image.--, in page 7 of machine translated English version of WO 2022265345 A1).
Re Claim 5, FORNWALT as modified by KO and LIU and HAEUSSER further disclose determining a request from the device to generate a report corresponding to the spatiotemporal output (see FORNWALT: e.g., -- receiving video frames of a heart, the video frames being associated with the patient, receiving electronic health record data including a number of variables associated with the patient, providing the video frames and the electronic health record data to the trained neural network, receiving a risk score from the trained neural network, and outputting a report based on the risk score to at least one of a display or a memory.--, in abstract, and, -- The echocardiogram-based parameters can include physician-reported left ventricular ejection fraction, aortic insufficiency deceleration slope, aortic insufficiency maximum velocity, velocity-time integral of distal to aortic valve flow, maximum velocity of distal to aortic valve flow, mean velocity of distal to aortic valve flow, aortic root diameter, ascending aortic diameter, Iv end-diastolic volume: apical 2-chamber; modified ellipsoid, Iv end-diastolic volume: apical 4-chamber; modified ellipsoid, Iv end-diastolic volume: apical 2-chamber; single plane, Iv end-diastolic volume: apical 4-chamber; single plane, Iv end-systolic volume: apical 2-chamber;--, in [0010];, and, -- The chosen 3D CNN architecture (AUC range: 0.695-0.784) outperformed the 2D CNN+LSTM architecture (AUC range: 0.703-0.752) for most views. In both cases, the parasternal long-axis (“PL DEEP”) view had the best performance. This result was in line with clinical intuition, since the PL DEEP view is typically reported by cardiologists as the most informative “summary” view of overall cardiac health. This is because the PL DEEP view contains elements of the left ventricle, left atrium, right ventricle, aortic and mitral valves, and whether or not there is a pericardial or left pleural effusion all within a single view.--, in [0057]); and
causing the device to generate the report corresponding to the spatiotemporal output (see FORNWALT: e.g., -- receiving video frames of a heart, the video frames being associated with the patient, receiving electronic health record data including a number of variables associated with the patient, providing the video frames and the electronic health record data to the trained neural network, receiving a risk score from the trained neural network, and outputting a report based on the risk score to at least one of a display or a memory.--, in abstract, and, -- The echocardiogram-based parameters can include physician-reported left ventricular ejection fraction, aortic insufficiency deceleration slope, aortic insufficiency maximum velocity, velocity-time integral of distal to aortic valve flow, maximum velocity of distal to aortic valve flow, mean velocity of distal to aortic valve flow, aortic root diameter, ascending aortic diameter, Iv end-diastolic volume: apical 2-chamber; modified ellipsoid, Iv end-diastolic volume: apical 4-chamber; modified ellipsoid, Iv end-diastolic volume: apical 2-chamber; single plane, Iv end-diastolic volume: apical 4-chamber; single plane, Iv end-systolic volume: apical 2-chamber;--, in [0010];, and, -- The chosen 3D CNN architecture (AUC range: 0.695-0.784) outperformed the 2D CNN+LSTM architecture (AUC range: 0.703-0.752) for most views. In both cases, the parasternal long-axis (“PL DEEP”) view had the best performance. This result was in line with clinical intuition, since the PL DEEP view is typically reported by cardiologists as the most informative “summary” view of overall cardiac health. This is because the PL DEEP view contains elements of the left ventricle, left atrium, right ventricle, aortic and mitral valves, and whether or not there is a pericardial or left pleural effusion all within a single view.--, in [0057]).
Re Claim 6, FORNWALT as modified by KO and LIU and HAEUSSER further disclose training the spatiotemporal model using a plurality of a second image data different from the image data (see FORNWALT: e.g., --In the method, the trained model can further include a trained submodel, and the method can further include providing the echocardiographic video to a first trained neural network included in the plurality of trained neural networks, receiving a second echocardiographic video of the heart associated with the patient, providing the second echocardiographic video to a second trained neural network included in the plurality of trained neural networks, receiving a first video risk score from the first trained neural network, and receiving a second video risk score from the second trained neural network…. diagnosis of congenital heart defect, diagnosis of dyslipidemia, and diagnosis of chronic kidney disease. The first trained neural network can be associated with a first network architecture and the second trained neural network can be associated with a second network architecture. The first network architecture can include a two-dimensional convolutional neural network, and the second network architecture can include a three-dimensional convolutional neural network--, in [0010]).
Re Claim 7, FORNWALT as modified by KO and LIU and HAEUSSER further disclose wherein the image data is generated by at least one imaging system, the at least one imaging system comprising an ultrasound or echocardiogram device (see FORNWALT: e.g., --In the method, the trained model can further include a trained submodel, and the method can further include providing the echocardiographic video to a first trained neural network included in the plurality of trained neural networks, receiving a second echocardiographic video of the heart associated with the patient, providing the second echocardiographic video to a second trained neural network included in the plurality of trained neural networks, receiving a first video risk score from the first trained neural network, and receiving a second video risk score from the second trained neural network…. diagnosis of congenital heart defect, diagnosis of dyslipidemia, and diagnosis of chronic kidney disease. The first trained neural network can be associated with a first network architecture and the second trained neural network can be associated with a second network architecture. The first network architecture can include a two-dimensional convolutional neural network, and the second network architecture can include a three-dimensional convolutional neural network--, in [0010]; also see Liu: e.g., --[0162] In addition to deep learning methods, the ultrasonic video data can also be classified by traditional image processing methods; for example, 3D Harris corner detection operator, computational optical flow, 3D SIFT operator and other algorithms may be used to extract features of spatial and temporal dimensions, including HOG, HOF, MBH and other features, and then classification algorithms such as SVM may be adopted to classify to obtain the category of ultrasonic video data. [0163] If the ultrasonic data to be displayed is two-dimensional ultrasonic image data, the classification method thereof is similar to that of the representative frame of the ultrasonic video data. For details, please refer to the above. [0164] After the scanning part of the ultrasound data is determined, the ultrasonic data is displayed in association with the corresponding portions of the body icon according to the scanning part of the ultrasonic data, referring to FIG. 4A and FIG. 4B. The scanning part of an ultrasound image is the tissue part corresponding to the ultrasound image. Specifically, an identifier of the ultrasonic data is displayed at different portions of the body icon, the identifier of the ultrasound data can be a small map of any frame of ultrasound data; and when a selection instruction for the identifier is acquired, the ultrasonic data corresponding to the scanning part is displayed according to the scanning part corresponding to the identifier.--, in [0162]-[0164]).
Re Claim 8, FORNWALT as modified by KO and LIU and HAEUSSER further disclose wherein the image data comprises a first series of image frames corresponding to a first orientation of the ultrasound device or echocardiogram device and a second series of image frame corresponding to a second orientation of the ultrasound device or echocardiogram device (see FORNWALT: e.g., --In the method, the trained model can further include a trained submodel, and the method can further include providing the echocardiographic video to a first trained neural network included in the plurality of trained neural networks, receiving a second echocardiographic video of the heart associated with the patient, providing the second echocardiographic video to a second trained neural network included in the plurality of trained neural networks, receiving a first video risk score from the first trained neural network, and receiving a second video risk score from the second trained neural network…. diagnosis of congenital heart defect, diagnosis of dyslipidemia, and diagnosis of chronic kidney disease. The first trained neural network can be associated with a first network architecture and the second trained neural network can be associated with a second network architecture. The first network architecture can include a two-dimensional convolutional neural network, and the second network architecture can include a three-dimensional convolutional neural network--, in [0010]; also see Liu: e.g., --[0162] In addition to deep learning methods, the ultrasonic video data can also be classified by traditional image processing methods; for example, 3D Harris corner detection operator, computational optical flow, 3D SIFT operator and other algorithms may be used to extract features of spatial and temporal dimensions, including HOG, HOF, MBH and other features, and then classification algorithms such as SVM may be adopted to classify to obtain the category of ultrasonic video data. [0163] If the ultrasonic data to be displayed is two-dimensional ultrasonic image data, the classification method thereof is similar to that of the representative frame of the ultrasonic video data. For details, please refer to the above. [0164] After the scanning part of the ultrasound data is determined, the ultrasonic data is displayed in association with the corresponding portions of the body icon according to the scanning part of the ultrasonic data, referring to FIG. 4A and FIG. 4B. The scanning part of an ultrasound image is the tissue part corresponding to the ultrasound image. Specifically, an identifier of the ultrasonic data is displayed at different portions of the body icon, the identifier of the ultrasound data can be a small map of any frame of ultrasound data; and when a selection instruction for the identifier is acquired, the ultrasonic data corresponding to the scanning part is displayed according to the scanning part corresponding to the identifier.--, in [0162]-[0164]).
Re Claim 9, FORNWALT as modified by KO and LIU and HAEUSSER further disclose sampling the image data such that only non-adjacent image frames in the series of image frames are processed by the spatial model (see FORNWALT: e.g., --In the method, the trained model can further include a trained submodel, and the method can further include providing the echocardiographic video to a first trained neural network included in the plurality of trained neural networks, receiving a second echocardiographic video of the heart associated with the patient, providing the second echocardiographic video to a second trained neural network included in the plurality of trained neural networks, receiving a first video risk score from the first trained neural network, and receiving a second video risk score from the second trained neural network…. diagnosis of congenital heart defect, diagnosis of dyslipidemia, and diagnosis of chronic kidney disease. The first trained neural network can be associated with a first network architecture and the second trained neural network can be associated with a second network architecture. The first network architecture can include a two-dimensional convolutional neural network, and the second network architecture can include a three-dimensional convolutional neural network--, in [0010]; also see Liu: e.g., --[0162] In addition to deep learning methods, the ultrasonic video data can also be classified by traditional image processing methods; for example, 3D Harris corner detection operator, computational optical flow, 3D SIFT operator and other algorithms may be used to extract features of spatial and temporal dimensions, including HOG, HOF, MBH and other features, and then classification algorithms such as SVM may be adopted to classify to obtain the category of ultrasonic video data. [0163] If the ultrasonic data to be displayed is two-dimensional ultrasonic image data, the classification method thereof is similar to that of the representative frame of the ultrasonic video data. For details, please refer to the above. [0164] After the scanning part of the ultrasound data is determined, the ultrasonic data is displayed in association with the corresponding portions of the body icon according to the scanning part of the ultrasonic data, referring to FIG. 4A and FIG. 4B. The scanning part of an ultrasound image is the tissue part corresponding to the ultrasound image. Specifically, an identifier of the ultrasonic data is displayed at different portions of the body icon, the identifier of the ultrasound data can be a small map of any frame of ultrasound data; and when a selection instruction for the identifier is acquired, the ultrasonic data corresponding to the scanning part is displayed according to the scanning part corresponding to the identifier.--, in [0162]-[0164]).
Re Claim 10, FORNWALT as modified by KO and LIU and HAEUSSER further disclose wherein one or more of the spatiotemporal output is indicative of one or more of key-point data or contour data (see KO: e.g., -- Referring to FIG. 5 , the processor 120 separates (segmentation) the nursery (P1), the amniotic fluid (P2), the fetus (P3), and the organ (P3-1) of the fetus (P3) from the fetal image (P). 3D images can be generated. For example, the processor 120 may perform an image of a fetus in a normal state (P11 in FIG. 1), an image of a fetus in a state with a congenital disease (P12 in FIG. 1), and an image of a fetus in a state with a severe congenital disease (P13 in FIG. 1). Each 3D image may be generated by separating at least one pre-determined fetal organ among each baby house, each amniotic fluid, each fetus, and each fetus organ. Here, the preset at least one organ of the fetus may be a major organ of the fetus. At this time, the main organ of the fetus may be any one or two or more of the heart, cerebrum, cerebellum, ventricle, stomach, liver, kidney, and bladder.
Here, “segmentation” of the fetal image P means a process of dividing a digital image into a plurality of pixel or voxel sets. The purpose of segmentation is to simplify or transform the representation of the image into something more meaningful and easier to interpret. Image segmentation is used to find objects and boundaries in images. The result of segmentation is a set of regions collectively including the entire image or a set of contours extracted from the image.--, in page 7 of machine translated English version of WO 2022265345 A1).
Re Claim 11, FORNWALT as modified by KO and LIU and HAEUSSER further disclose determining one or more of key-point data or contour data based on the spatiotemporal output (see KO: e.g., -- Referring to FIG. 5 , the processor 120 separates (segmentation) the nursery (P1), the amniotic fluid (P2), the fetus (P3), and the organ (P3-1) of the fetus (P3) from the fetal image (P). 3D images can be generated. For example, the processor 120 may perform an image of a fetus in a normal state (P11 in FIG. 1), an image of a fetus in a state with a congenital disease (P12 in FIG. 1), and an image of a fetus in a state with a severe congenital disease (P13 in FIG. 1). Each 3D image may be generated by separating at least one pre-determined fetal organ among each baby house, each amniotic fluid, each fetus, and each fetus organ. Here, the preset at least one organ of the fetus may be a major organ of the fetus. At this time, the main organ of the fetus may be any one or two or more of the heart, cerebrum, cerebellum, ventricle, stomach, liver, kidney, and bladder.
Here, “segmentation” of the fetal image P means a process of dividing a digital image into a plurality of pixel or voxel sets. The purpose of segmentation is to simplify or transform the representation of the image into something more meaningful and easier to interpret. Image segmentation is used to find objects and boundaries in images. The result of segmentation is a set of regions collectively including the entire image or a set of contours extracted from the image.--, in page 7 of machine translated English version of WO 2022265345 A1 also see Liu: e.g., --- three-dimensional spatial and temporal features can be obtained by deep learning methods, that is, first using CNN, RNN, 3D full convolution network, LSTM and other deep neural networks to extract two-dimensional spatial features and temporal features from ultrasonic video data. For example, image features may be obtained from a single frame by using CNN, and then the obtained image features may be integrated in temporal dimension by using RNN network. Alternatively, 3D convolution kernels may be directly used to extract image features from frames, and to model motion sequences among adjacent frames. Alternatively, CNN is used for feature extraction of frames, and then optical flow images are used to extract temporal dimension features. Alternatively, two-dimensional CNN is used to extract the features of the scanning part frame by frame, and then LSTM network is used to model the features obtained by the two-dimensional CNN in temporal dimension.--, in [0161]-[0162], and, -[0162] In addition to deep learning methods, the ultrasonic video data can also be classified by traditional image processing methods; for example, 3D Harris corner detection operator, computational optical flow, 3D SIFT operator and other algorithms may be used to extract features of spatial and temporal dimensions, including HOG, HOF, MBH and other features, and then classification algorithms such as SVM may be adopted to classify to obtain the category of ultrasonic video data. [0163] If the ultrasonic data to be displayed is two-dimensional ultrasonic image data, the classification method thereof is similar to that of the representative frame of the ultrasonic video data. For details, please refer to the above. [0164] After the scanning part of the ultrasound data is determined, the ultrasonic data is displayed in association with the corresponding portions of the body icon according to the scanning part of the ultrasonic data, referring to FIG. 4A and FIG. 4B. The scanning part of an ultrasound image is the tissue part corresponding to the ultrasound image. Specifically, an identifier of the ultrasonic data is displayed at different portions of the body icon, the identifier of the ultrasound data can be a small map of any frame of ultrasound data; and when a selection instruction for the identifier is acquired, the ultrasonic data corresponding to the scanning part is displayed according to the scanning part corresponding to the identifier.--, in [0162]-[0164]).
Re Claim 12, FORNWALT as modified by KO and LIU and HAEUSSER further disclose causing the device to further display the one or more of key-point data or contour data (see KO: e.g., -- Referring to FIG. 5 , the processor 120 separates (segmentation) the nursery (P1), the amniotic fluid (P2), the fetus (P3), and the organ (P3-1) of the fetus (P3) from the fetal image (P). 3D images can be generated. For example, the processor 120 may perform an image of a fetus in a normal state (P11 in FIG. 1), an image of a fetus in a state with a congenital disease (P12 in FIG. 1), and an image of a fetus in a state with a severe congenital disease (P13 in FIG. 1). Each 3D image may be generated by separating at least one pre-determined fetal organ among each baby house, each amniotic fluid, each fetus, and each fetus organ. Here, the preset at least one organ of the fetus may be a major organ of the fetus. At this time, the main organ of the fetus may be any one or two or more of the heart, cerebrum, cerebellum, ventricle, stomach, liver, kidney, and bladder.
Here, “segmentation” of the fetal image P means a process of dividing a digital image into a plurality of pixel or voxel sets. The purpose of segmentation is to simplify or transform the representation of the image into something more meaningful and easier to interpret. Image segmentation is used to find objects and boundaries in images. The result of segmentation is a set of regions collectively including the entire image or a set of contours extracted from the image.--, in page 7 of machine translated English version of WO 2022265345 A1; also see Liu: e.g., --- three-dimensional spatial and temporal features can be obtained by deep learning methods, that is, first using CNN, RNN, 3D full convolution network, LSTM and other deep neural networks to extract two-dimensional spatial features and temporal features from ultrasonic video data. For example, image features may be obtained from a single frame by using CNN, and then the obtained image features may be integrated in temporal dimension by using RNN network. Alternatively, 3D convolution kernels may be directly used to extract image features from frames, and to model motion sequences among adjacent frames. Alternatively, CNN is used for feature extraction of frames, and then optical flow images are used to extract temporal dimension features. Alternatively, two-dimensional CNN is used to extract the features of the scanning part frame by frame, and then LSTM network is used to model the features obtained by the two-dimensional CNN in temporal dimension.--, in [0161]-[0162], and, -[0162] In addition to deep learning methods, the ultrasonic video data can also be classified by traditional image processing methods; for example, 3D Harris corner detection operator, computational optical flow, 3D SIFT operator and other algorithms may be used to extract features of spatial and temporal dimensions, including HOG, HOF, MBH and other features, and then classification algorithms such as SVM may be adopted to classify to obtain the category of ultrasonic video data. [0163] If the ultrasonic data to be displayed is two-dimensional ultrasonic image data, the classification method thereof is similar to that of the representative frame of the ultrasonic video data. For details, please refer to the above. [0164] After the scanning part of the ultrasound data is determined, the ultrasonic data is displayed in association with the corresponding portions of the body icon according to the scanning part of the ultrasonic data, referring to FIG. 4A and FIG. 4B. The scanning part of an ultrasound image is the tissue part corresponding to the ultrasound image. Specifically, an identifier of the ultrasonic data is displayed at different portions of the body icon, the identifier of the ultrasound data can be a small map of any frame of ultrasound data; and when a selection instruction for the identifier is acquired, the ultrasonic data corresponding to the scanning part is displayed according to the scanning part corresponding to the identifier.--, in [0162]-[0164]).
Re Claim 13, FORNWALT as modified by KO and LIU and HAEUSSER further disclose wherein the spatial output corresponds to segmentation of the fetus’s heart, stomach, and thorax and the spatiotemporal output is indicative of a presence of heterotaxy (see KO: e.g., -- Referring to FIG. 5 , the processor 120 separates (segmentation) the nursery (P1), the amniotic fluid (P2), the fetus (P3), and the organ (P3-1) of the fetus (P3) from the fetal image (P). 3D images can be generated. For example, the processor 120 may perform an image of a fetus in a normal state (P11 in FIG. 1), an image of a fetus in a state with a congenital disease (P12 in FIG. 1), and an image of a fetus in a state with a severe congenital disease (P13 in FIG. 1). Each 3D image may be generated by separating at least one pre-determined fetal organ among each baby house, each amniotic fluid, each fetus, and each fetus organ. Here, the preset at least one organ of the fetus may be a major organ of the fetus. At this time, the main organ of the fetus may be any one or two or more of the heart, cerebrum, cerebellum, ventricle, stomach, liver, kidney, and bladder.
Here, “segmentation” of the fetal image P means a process of dividing a digital image into a plurality of pixel or voxel sets. The purpose of segmentation is to simplify or transform the representation of the image into something more meaningful and easier to interpret. Image segmentation is used to find objects and boundaries in images. The result of segmentation is a set of regions collectively including the entire image or a set of contours extracted from the image.--, in page 7 of machine translated English version of WO 2022265345 A1).
Re Claim 14, FORNWALT as modified by KO and LIU and HAEUSSER further disclose wherein the spatiotemporal output corresponds to segmentation of at least one ventricle and at least one atria of the fetus, contraction of a ventricle, contraction of an atria, or a presence of an arrhythmia (see KO: e.g., -- Referring to FIG. 5 , the processor 120 separates (segmentation) the nursery (P1), the amniotic fluid (P2), the fetus (P3), and the organ (P3-1) of the fetus (P3) from the fetal image (P). 3D images can be generated. For example, the processor 120 may perform an image of a fetus in a normal state (P11 in FIG. 1), an image of a fetus in a state with a congenital disease (P12 in FIG. 1), and an image of a fetus in a state with a severe congenital disease (P13 in FIG. 1). Each 3D image may be generated by separating at least one pre-determined fetal organ among each baby house, each amniotic fluid, each fetus, and each fetus organ. Here, the preset at least one organ of the fetus may be a major organ of the fetus. At this time, the main organ of the fetus may be any one or two or more of the heart, cerebrum, cerebellum, ventricle, stomach, liver, kidney, and bladder.
Here, “segmentation” of the fetal image P means a process of dividing a digital image into a plurality of pixel or voxel sets. The purpose of segmentation is to simplify or transform the representation of the image into something more meaningful and easier to interpret. Image segmentation is used to find objects and boundaries in images. The result of segmentation is a set of regions collectively including the entire image or a set of contours extracted from the image.--, in page 7 of machine translated English version of WO 2022265345 A1; also see Liu: e.g., --- three-dimensional spatial and temporal features can be obtained by deep learning methods, that is, first using CNN, RNN, 3D full convolution network, LSTM and other deep neural networks to extract two-dimensional spatial features and temporal features from ultrasonic video data. For example, image features may be obtained from a single frame by using CNN, and then the obtained image features may be integrated in temporal dimension by using RNN network. Alternatively, 3D convolution kernels may be directly used to extract image features from frames, and to model motion sequences among adjacent frames. Alternatively, CNN is used for feature extraction of frames, and then optical flow images are used to extract temporal dimension features. Alternatively, two-dimensional CNN is used to extract the features of the scanning part frame by frame, and then LSTM network is used to model the features obtained by the two-dimensional CNN in temporal dimension.--, in [0161]-[0162], and, -[0162] In addition to deep learning methods, the ultrasonic video data can also be classified by traditional image processing methods; for example, 3D Harris corner detection operator, computational optical flow, 3D SIFT operator and other algorithms may be used to extract features of spatial and temporal dimensions, including HOG, HOF, MBH and other features, and then classification algorithms such as SVM may be adopted to classify to obtain the category of ultrasonic video data. [0163] If the ultrasonic data to be displayed is two-dimensional ultrasonic image data, the classification method thereof is similar to that of the representative frame of the ultrasonic video data. For details, please refer to the above. [0164] After the scanning part of the ultrasound data is determined, the ultrasonic data is displayed in association with the corresponding portions of the body icon according to the scanning part of the ultrasonic data, referring to FIG. 4A and FIG. 4B. The scanning part of an ultrasound image is the tissue part corresponding to the ultrasound image. Specifically, an identifier of the ultrasonic data is displayed at different portions of the body icon, the identifier of the ultrasound data can be a small map of any frame of ultrasound data; and when a selection instruction for the identifier is acquired, the ultrasonic data corresponding to the scanning part is displayed according to the scanning part corresponding to the identifier.--, in [0162]-[0164]; also see FORNWALT: e.g., -- receiving video frames of a heart, the video frames being associated with the patient, receiving electronic health record data including a number of variables associated with the patient, providing the video frames and the electronic health record data to the trained neural network, receiving a risk score from the trained neural network, and outputting a report based on the risk score to at least one of a display or a memory.--, in abstract, and, -- The echocardiogram-based parameters can include physician-reported left ventricular ejection fraction, aortic insufficiency deceleration slope, aortic insufficiency maximum velocity, velocity-time integral of distal to aortic valve flow, maximum velocity of distal to aortic valve flow, mean velocity of distal to aortic valve flow, aortic root diameter, ascending aortic diameter, Iv end-diastolic volume: apical 2-chamber; modified ellipsoid, Iv end-diastolic volume: apical 4-chamber; modified ellipsoid, Iv end-diastolic volume: apical 2-chamber; single plane, Iv end-diastolic volume: apical 4-chamber; single plane, Iv end-systolic volume: apical 2-chamber;--, in [0010];, and, -- The chosen 3D CNN architecture (AUC range: 0.695-0.784) outperformed the 2D CNN+LSTM architecture (AUC range: 0.703-0.752) for most views. In both cases, the parasternal long-axis (“PL DEEP”) view had the best performance. This result was in line with clinical intuition, since the PL DEEP view is typically reported by cardiologists as the most informative “summary” view of overall cardiac health. This is because the PL DEEP view contains elements of the left ventricle, left atrium, right ventricle, aortic and mitral valves, and whether or not there is a pericardial or left pleural effusion all within a single view.--, in [0057]).
Re Claim 15, FORNWALT as modified by KO and LIU and HAEUSSER further disclose wherein the spatiotemporal output corresponds to segmentation of ventricles of the fetus and the spatiotemporal output is indicative of a presence of ventricular akinesia (see KO: e.g., -- Referring to FIG. 5 , the processor 120 separates (segmentation) the nursery (P1), the amniotic fluid (P2), the fetus (P3), and the organ (P3-1) of the fetus (P3) from the fetal image (P). 3D images can be generated. For example, the processor 120 may perform an image of a fetus in a normal state (P11 in FIG. 1), an image of a fetus in a state with a congenital disease (P12 in FIG. 1), and an image of a fetus in a state with a severe congenital disease (P13 in FIG. 1). Each 3D image may be generated by separating at least one pre-determined fetal organ among each baby house, each amniotic fluid, each fetus, and each fetus organ. Here, the preset at least one organ of the fetus may be a major organ of the fetus. At this time, the main organ of the fetus may be any one or two or more of the heart, cerebrum, cerebellum, ventricle, stomach, liver, kidney, and bladder.
Here, “segmentation” of the fetal image P means a process of dividing a digital image into a plurality of pixel or voxel sets. The purpose of segmentation is to simplify or transform the representation of the image into something more meaningful and easier to interpret. Image segmentation is used to find objects and boundaries in images. The result of segmentation is a set of regions collectively including the entire image or a set of contours extracted from the image.--, in page 7 of machine translated English version of WO 2022265345 A1).
Re Claim 16, FORNWALT as modified by KO and LIU and HAEUSSER further disclose wherein the temporal output corresponds to a presence of a valve at a given time and the spatiotemporal output is corresponds to whether the valve is open, the spatiotemporal output indicative of a presence of valve atresia (see KO: e.g., -- Referring to FIG. 5 , the processor 120 separates (segmentation) the nursery (P1), the amniotic fluid (P2), the fetus (P3), and the organ (P3-1) of the fetus (P3) from the fetal image (P). 3D images can be generated. For example, the processor 120 may perform an image of a fetus in a normal state (P11 in FIG. 1), an image of a fetus in a state with a congenital disease (P12 in FIG. 1), and an image of a fetus in a state with a severe congenital disease (P13 in FIG. 1). Each 3D image may be generated by separating at least one pre-determined fetal organ among each baby house, each amniotic fluid, each fetus, and each fetus organ. Here, the preset at least one organ of the fetus may be a major organ of the fetus. At this time, the main organ of the fetus may be any one or two or more of the heart, cerebrum, cerebellum, ventricle, stomach, liver, kidney, and bladder.
Here, “segmentation” of the fetal image P means a process of dividing a digital image into a plurality of pixel or voxel sets. The purpose of segmentation is to simplify or transform the representation of the image into something more meaningful and easier to interpret. Image segmentation is used to find objects and boundaries in images. The result of segmentation is a set of regions collectively including the entire image or a set of contours extracted from the image.--, in page 7 of machine translated English version of WO 2022265345 A1; also see Liu: e.g., --- three-dimensional spatial and temporal features can be obtained by deep learning methods, that is, first using CNN, RNN, 3D full convolution network, LSTM and other deep neural networks to extract two-dimensional spatial features and temporal features from ultrasonic video data. For example, image features may be obtained from a single frame by using CNN, and then the obtained image features may be integrated in temporal dimension by using RNN network. Alternatively, 3D convolution kernels may be directly used to extract image features from frames, and to model motion sequences among adjacent frames. Alternatively, CNN is used for feature extraction of frames, and then optical flow images are used to extract temporal dimension features. Alternatively, two-dimensional CNN is used to extract the features of the scanning part frame by frame, and then LSTM network is used to model the features obtained by the two-dimensional CNN in temporal dimension.--, in [0161]-[0162], and, -[0162] In addition to deep learning methods, the ultrasonic video data can also be classified by traditional image processing methods; for example, 3D Harris corner detection operator, computational optical flow, 3D SIFT operator and other algorithms may be used to extract features of spatial and temporal dimensions, including HOG, HOF, MBH and other features, and then classification algorithms such as SVM may be adopted to classify to obtain the category of ultrasonic video data. [0163] If the ultrasonic data to be displayed is two-dimensional ultrasonic image data, the classification method thereof is similar to that of the representative frame of the ultrasonic video data. For details, please refer to the above. [0164] After the scanning part of the ultrasound data is determined, the ultrasonic data is displayed in association with the corresponding portions of the body icon according to the scanning part of the ultrasonic data, referring to FIG. 4A and FIG. 4B. The scanning part of an ultrasound image is the tissue part corresponding to the ultrasound image. Specifically, an identifier of the ultrasonic data is displayed at different portions of the body icon, the identifier of the ultrasound data can be a small map of any frame of ultrasound data; and when a selection instruction for the identifier is acquired, the ultrasonic data corresponding to the scanning part is displayed according to the scanning part corresponding to the identifier.--, in [0162]-[0164]; also see FORNWALT: e.g., -- receiving video frames of a heart, the video frames being associated with the patient, receiving electronic health record data including a number of variables associated with the patient, providing the video frames and the electronic health record data to the trained neural network, receiving a risk score from the trained neural network, and outputting a report based on the risk score to at least one of a display or a memory.--, in abstract, and, -- The echocardiogram-based parameters can include physician-reported left ventricular ejection fraction, aortic insufficiency deceleration slope, aortic insufficiency maximum velocity, velocity-time integral of distal to aortic valve flow, maximum velocity of distal to aortic valve flow, mean velocity of distal to aortic valve flow, aortic root diameter, ascending aortic diameter, Iv end-diastolic volume: apical 2-chamber; modified ellipsoid, Iv end-diastolic volume: apical 4-chamber; modified ellipsoid, Iv end-diastolic volume: apical 2-chamber; single plane, Iv end-diastolic volume: apical 4-chamber; single plane, Iv end-systolic volume: apical 2-chamber;--, in [0010];, and, -- The chosen 3D CNN architecture (AUC range: 0.695-0.784) outperformed the 2D CNN+LSTM architecture (AUC range: 0.703-0.752) for most views. In both cases, the parasternal long-axis (“PL DEEP”) view had the best performance. This result was in line with clinical intuition, since the PL DEEP view is typically reported by cardiologists as the most informative “summary” view of overall cardiac health. This is because the PL DEEP view contains elements of the left ventricle, left atrium, right ventricle, aortic and mitral valves, and whether or not there is a pericardial or left pleural effusion all within a single view.--, in [0057]).
Re Claim 17, FORNWALT as modified by KO and LIU and HAEUSSER further disclose wherein the spatiotemporal output corresponds to segmentation of a left ventricular outflow tract and an aorta of the fetus, a presence of blood flow between the right ventricle, or the aorta at a certain time in the time period, and the spatiotemporal output is indicative of a presence of an overriding aorta (see KO: e.g., -- Referring to FIG. 5 , the processor 120 separates (segmentation) the nursery (P1), the amniotic fluid (P2), the fetus (P3), and the organ (P3-1) of the fetus (P3) from the fetal image (P). 3D images can be generated. For example, the processor 120 may perform an image of a fetus in a normal state (P11 in FIG. 1), an image of a fetus in a state with a congenital disease (P12 in FIG. 1), and an image of a fetus in a state with a severe congenital disease (P13 in FIG. 1). Each 3D image may be generated by separating at least one pre-determined fetal organ among each baby house, each amniotic fluid, each fetus, and each fetus organ. Here, the preset at least one organ of the fetus may be a major organ of the fetus. At this time, the main organ of the fetus may be any one or two or more of the heart, cerebrum, cerebellum, ventricle, stomach, liver, kidney, and bladder.
Here, “segmentation” of the fetal image P means a process of dividing a digital image into a plurality of pixel or voxel sets. The purpose of segmentation is to simplify or transform the representation of the image into something more meaningful and easier to interpret. Image segmentation is used to find objects and boundaries in images. The result of segmentation is a set of regions collectively including the entire image or a set of contours extracted from the image.--, in page 7 of machine translated English version of WO 2022265345 A1; also see Liu: e.g., --- three-dimensional spatial and temporal features can be obtained by deep learning methods, that is, first using CNN, RNN, 3D full convolution network, LSTM and other deep neural networks to extract two-dimensional spatial features and temporal features from ultrasonic video data. For example, image features may be obtained from a single frame by using CNN, and then the obtained image features may be integrated in temporal dimension by using RNN network. Alternatively, 3D convolution kernels may be directly used to extract image features from frames, and to model motion sequences among adjacent frames. Alternatively, CNN is used for feature extraction of frames, and then optical flow images are used to extract temporal dimension features. Alternatively, two-dimensional CNN is used to extract the features of the scanning part frame by frame, and then LSTM network is used to model the features obtained by the two-dimensional CNN in temporal dimension.--, in [0161]-[0162], and, -[0162] In addition to deep learning methods, the ultrasonic video data can also be classified by traditional image processing methods; for example, 3D Harris corner detection operator, computational optical flow, 3D SIFT operator and other algorithms may be used to extract features of spatial and temporal dimensions, including HOG, HOF, MBH and other features, and then classification algorithms such as SVM may be adopted to classify to obtain the category of ultrasonic video data. [0163] If the ultrasonic data to be displayed is two-dimensional ultrasonic image data, the classification method thereof is similar to that of the representative frame of the ultrasonic video data. For details, please refer to the above. [0164] After the scanning part of the ultrasound data is determined, the ultrasonic data is displayed in association with the corresponding portions of the body icon according to the scanning part of the ultrasonic data, referring to FIG. 4A and FIG. 4B. The scanning part of an ultrasound image is the tissue part corresponding to the ultrasound image. Specifically, an identifier of the ultrasonic data is displayed at different portions of the body icon, the identifier of the ultrasound data can be a small map of any frame of ultrasound data; and when a selection instruction for the identifier is acquired, the ultrasonic data corresponding to the scanning part is displayed according to the scanning part corresponding to the identifier.--, in [0162]-[0164]; also see FORNWALT: e.g., -- receiving video frames of a heart, the video frames being associated with the patient, receiving electronic health record data including a number of variables associated with the patient, providing the video frames and the electronic health record data to the trained neural network, receiving a risk score from the trained neural network, and outputting a report based on the risk score to at least one of a display or a memory.--, in abstract, and, -- The echocardiogram-based parameters can include physician-reported left ventricular ejection fraction, aortic insufficiency deceleration slope, aortic insufficiency maximum velocity, velocity-time integral of distal to aortic valve flow, maximum velocity of distal to aortic valve flow, mean velocity of distal to aortic valve flow, aortic root diameter, ascending aortic diameter, Iv end-diastolic volume: apical 2-chamber; modified ellipsoid, Iv end-diastolic volume: apical 4-chamber; modified ellipsoid, Iv end-diastolic volume: apical 2-chamber; single plane, Iv end-diastolic volume: apical 4-chamber; single plane, Iv end-systolic volume: apical 2-chamber;--, in [0010];, and, -- The chosen 3D CNN architecture (AUC range: 0.695-0.784) outperformed the 2D CNN+LSTM architecture (AUC range: 0.703-0.752) for most views. In both cases, the parasternal long-axis (“PL DEEP”) view had the best performance. This result was in line with clinical intuition, since the PL DEEP view is typically reported by cardiologists as the most informative “summary” view of overall cardiac health. This is because the PL DEEP view contains elements of the left ventricle, left atrium, right ventricle, aortic and mitral valves, and whether or not there is a pericardial or left pleural effusion all within a single view.--, in [0057]).
Re Claim 18, FORNWALT as modified by KO and LIU and HAEUSSER further disclose wherein the spatiotemporal output corresponds to segmentation of ventricles, an aorta, and a pulmonary artery of the fetus and the spatiotemporal output is indicative of whether a connection between arteries and the ventricles of the fetus is normal (see KO: e.g., -- Referring to FIG. 5 , the processor 120 separates (segmentation) the nursery (P1), the amniotic fluid (P2), the fetus (P3), and the organ (P3-1) of the fetus (P3) from the fetal image (P). 3D images can be generated. For example, the processor 120 may perform an image of a fetus in a normal state (P11 in FIG. 1), an image of a fetus in a state with a congenital disease (P12 in FIG. 1), and an image of a fetus in a state with a severe congenital disease (P13 in FIG. 1). Each 3D image may be generated by separating at least one pre-determined fetal organ among each baby house, each amniotic fluid, each fetus, and each fetus organ. Here, the preset at least one organ of the fetus may be a major organ of the fetus. At this time, the main organ of the fetus may be any one or two or more of the heart, cerebrum, cerebellum, ventricle, stomach, liver, kidney, and bladder.
Here, “segmentation” of the fetal image P means a process of dividing a digital image into a plurality of pixel or voxel sets. The purpose of segmentation is to simplify or transform the representation of the image into something more meaningful and easier to interpret. Image segmentation is used to find objects and boundaries in images. The result of segmentation is a set of regions collectively including the entire image or a set of contours extracted from the image.--, in page 7 of machine translated English version of WO 2022265345 A1; also see Liu: e.g., --- three-dimensional spatial and temporal features can be obtained by deep learning methods, that is, first using CNN, RNN, 3D full convolution network, LSTM and other deep neural networks to extract two-dimensional spatial features and temporal features from ultrasonic video data. For example, image features may be obtained from a single frame by using CNN, and then the obtained image features may be integrated in temporal dimension by using RNN network. Alternatively, 3D convolution kernels may be directly used to extract image features from frames, and to model motion sequences among adjacent frames. Alternatively, CNN is used for feature extraction of frames, and then optical flow images are used to extract temporal dimension features. Alternatively, two-dimensional CNN is used to extract the features of the scanning part frame by frame, and then LSTM network is used to model the features obtained by the two-dimensional CNN in temporal dimension.--, in [0161]-[0162], and, -[0162] In addition to deep learning methods, the ultrasonic video data can also be classified by traditional image processing methods; for example, 3D Harris corner detection operator, computational optical flow, 3D SIFT operator and other algorithms may be used to extract features of spatial and temporal dimensions, including HOG, HOF, MBH and other features, and then classification algorithms such as SVM may be adopted to classify to obtain the category of ultrasonic video data. [0163] If the ultrasonic data to be displayed is two-dimensional ultrasonic image data, the classification method thereof is similar to that of the representative frame of the ultrasonic video data. For details, please refer to the above. [0164] After the scanning part of the ultrasound data is determined, the ultrasonic data is displayed in association with the corresponding portions of the body icon according to the scanning part of the ultrasonic data, referring to FIG. 4A and FIG. 4B. The scanning part of an ultrasound image is the tissue part corresponding to the ultrasound image. Specifically, an identifier of the ultrasonic data is displayed at different portions of the body icon, the identifier of the ultrasound data can be a small map of any frame of ultrasound data; and when a selection instruction for the identifier is acquired, the ultrasonic data corresponding to the scanning part is displayed according to the scanning part corresponding to the identifier.--, in [0162]-[0164]; also see FORNWALT: e.g., -- receiving video frames of a heart, the video frames being associated with the patient, receiving electronic health record data including a number of variables associated with the patient, providing the video frames and the electronic health record data to the trained neural network, receiving a risk score from the trained neural network, and outputting a report based on the risk score to at least one of a display or a memory.--, in abstract, and, -- The echocardiogram-based parameters can include physician-reported left ventricular ejection fraction, aortic insufficiency deceleration slope, aortic insufficiency maximum velocity, velocity-time integral of distal to aortic valve flow, maximum velocity of distal to aortic valve flow, mean velocity of distal to aortic valve flow, aortic root diameter, ascending aortic diameter, Iv end-diastolic volume: apical 2-chamber; modified ellipsoid, Iv end-diastolic volume: apical 4-chamber; modified ellipsoid, Iv end-diastolic volume: apical 2-chamber; single plane, Iv end-diastolic volume: apical 4-chamber; single plane, Iv end-systolic volume: apical 2-chamber;--, in [0010];, and, -- The chosen 3D CNN architecture (AUC range: 0.695-0.784) outperformed the 2D CNN+LSTM architecture (AUC range: 0.703-0.752) for most views. In both cases, the parasternal long-axis (“PL DEEP”) view had the best performance. This result was in line with clinical intuition, since the PL DEEP view is typically reported by cardiologists as the most informative “summary” view of overall cardiac health. This is because the PL DEEP view contains elements of the left ventricle, left atrium, right ventricle, aortic and mitral valves, and whether or not there is a pericardial or left pleural effusion all within a single view.--, in [0057]).
Re Claim 19, FORNWALT as modified by KO and LIU and HAEUSSER further disclose wherein the spatiotemporal output corresponds to contours of ventricles of the fetus, an end of diastole for a heart of the fetus, or at least one measurement of at least one ventricle at the end of diastole (see KO: e.g., -- Referring to FIG. 5 , the processor 120 separates (segmentation) the nursery (P1), the amniotic fluid (P2), the fetus (P3), and the organ (P3-1) of the fetus (P3) from the fetal image (P). 3D images can be generated. For example, the processor 120 may perform an image of a fetus in a normal state (P11 in FIG. 1), an image of a fetus in a state with a congenital disease (P12 in FIG. 1), and an image of a fetus in a state with a severe congenital disease (P13 in FIG. 1). Each 3D image may be generated by separating at least one pre-determined fetal organ among each baby house, each amniotic fluid, each fetus, and each fetus organ. Here, the preset at least one organ of the fetus may be a major organ of the fetus. At this time, the main organ of the fetus may be any one or two or more of the heart, cerebrum, cerebellum, ventricle, stomach, liver, kidney, and bladder.
Here, “segmentation” of the fetal image P means a process of dividing a digital image into a plurality of pixel or voxel sets. The purpose of segmentation is to simplify or transform the representation of the image into something more meaningful and easier to interpret. Image segmentation is used to find objects and boundaries in images. The result of segmentation is a set of regions collectively including the entire image or a set of contours extracted from the image.--, in page 7 of machine translated English version of WO 2022265345 A1; also see Liu: e.g., --- three-dimensional spatial and temporal features can be obtained by deep learning methods, that is, first using CNN, RNN, 3D full convolution network, LSTM and other deep neural networks to extract two-dimensional spatial features and temporal features from ultrasonic video data. For example, image features may be obtained from a single frame by using CNN, and then the obtained image features may be integrated in temporal dimension by using RNN network. Alternatively, 3D convolution kernels may be directly used to extract image features from frames, and to model motion sequences among adjacent frames. Alternatively, CNN is used for feature extraction of frames, and then optical flow images are used to extract temporal dimension features. Alternatively, two-dimensional CNN is used to extract the features of the scanning part frame by frame, and then LSTM network is used to model the features obtained by the two-dimensional CNN in temporal dimension.--, in [0161]-[0162], and, -[0162] In addition to deep learning methods, the ultrasonic video data can also be classified by traditional image processing methods; for example, 3D Harris corner detection operator, computational optical flow, 3D SIFT operator and other algorithms may be used to extract features of spatial and temporal dimensions, including HOG, HOF, MBH and other features, and then classification algorithms such as SVM may be adopted to classify to obtain the category of ultrasonic video data. [0163] If the ultrasonic data to be displayed is two-dimensional ultrasonic image data, the classification method thereof is similar to that of the representative frame of the ultrasonic video data. For details, please refer to the above. [0164] After the scanning part of the ultrasound data is determined, the ultrasonic data is displayed in association with the corresponding portions of the body icon according to the scanning part of the ultrasonic data, referring to FIG. 4A and FIG. 4B. The scanning part of an ultrasound image is the tissue part corresponding to the ultrasound image. Specifically, an identifier of the ultrasonic data is displayed at different portions of the body icon, the identifier of the ultrasound data can be a small map of any frame of ultrasound data; and when a selection instruction for the identifier is acquired, the ultrasonic data corresponding to the scanning part is displayed according to the scanning part corresponding to the identifier.--, in [0162]-[0164]; also see FORNWALT: e.g., -- receiving video frames of a heart, the video frames being associated with the patient, receiving electronic health record data including a number of variables associated with the patient, providing the video frames and the electronic health record data to the trained neural network, receiving a risk score from the trained neural network, and outputting a report based on the risk score to at least one of a display or a memory.--, in abstract, and, -- The echocardiogram-based parameters can include physician-reported left ventricular ejection fraction, aortic insufficiency deceleration slope, aortic insufficiency maximum velocity, velocity-time integral of distal to aortic valve flow, maximum velocity of distal to aortic valve flow, mean velocity of distal to aortic valve flow, aortic root diameter, ascending aortic diameter, Iv end-diastolic volume: apical 2-chamber; modified ellipsoid, Iv end-diastolic volume: apical 4-chamber; modified ellipsoid, Iv end-diastolic volume: apical 2-chamber; single plane, Iv end-diastolic volume: apical 4-chamber; single plane, Iv end-systolic volume: apical 2-chamber;--, in [0010];, and, -- The chosen 3D CNN architecture (AUC range: 0.695-0.784) outperformed the 2D CNN+LSTM architecture (AUC range: 0.703-0.752) for most views. In both cases, the parasternal long-axis (“PL DEEP”) view had the best performance. This result was in line with clinical intuition, since the PL DEEP view is typically reported by cardiologists as the most informative “summary” view of overall cardiac health. This is because the PL DEEP view contains elements of the left ventricle, left atrium, right ventricle, aortic and mitral valves, and whether or not there is a pericardial or left pleural effusion all within a single view.--, in [0057]).
Re Claim 20, claim 20 is the corresponding system claim to claim 1 respectively. Thus, claim 20 is rejected for the similar reasons as for claim 1. Furthermore, FORNWALT as modified by KO and LIU and HAEUSSER further disclose a system for determining a presence of one or more defects or conditions in a fetus during pregnancy, the system comprising. memory configured to store computer-executable instructions; and at least one computer processor configured to access memory and execute the computer- executable instructions to performing the method (see FORNWALT: e.g., -- [0103] Turning now to FIG. 12 as well as FIG. 1, an exemplary process 100 for predicting a relevant clinical endpoint such as all-cause mortality in a patient for a predetermined time period (i.e., one year) based on a video of the heart (in this case echocardiography data) as well as any additional available EHR data is shown. The process 100 predicts a risk score for the patient based on a neural network, which can be a deep neural network such as a convolutional neural network, trained using videos of the heart such as echocardiogram videos and EHR variables as described above. The process 100 can be employed in a health analytics module that is used by a care team including the physician in order to treat the patient or for population level management of patients, for example a physician deploying resources to an entire population of ten thousand patients with heart failure. In some embodiments, the process 100 can be implemented as instructions (e.g., computer readable instructions) on at least one memory, and executed by one or more processors coupled to the at least one memory.--, in [0103]; and, -- receiving an echocardiographic video of a heart associated with a patient, the echocardiographic video including a plurality of video frames, analyzing one or more regions of the heart using a trained model to generate a patient analysis, and generating a mortality risk score based on the patient analysis….. the trained model can further include a trained submodel, and the method can further include providing the echocardiographic video to a first trained neural network included in the plurality of trained neural networks, receiving a second echocardiographic video of the heart associated with the patient, providing the second echocardiographic video to a second trained neural network included in the plurality of trained neural networks, receiving a first video risk score from the first trained neural network, and receiving a second video risk score from the second trained neural network. The generating the mortality risk score can include providing the first video risk score and the second video risk score to the trained submodel, and receiving the mortality risk score from the trained submodel. The trained submodel can include a trained classifier. The trained classifier can be an XGboost classifier. The generating the mortality risk score can further include providing electronic health record information associated with the patient to the trained submodel.--, in [0005]-[0010], and, -- “deep” learning (deep neural network; DNN) technologies; such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNN), Dropout Regularization, and adaptive gradient descent algorithms; in conjunction with massively parallel computational hardware (graphic processing units), have enabled state-of-the-art predictive models for image, time-series, and video-based data…. A fully 3D Convolutional Neural Network (CNN) design is utilized in this study (FIG. 1). CNNs are neural networks that exploit spatial coherence in an image to significantly reduce the number of parameters that a fully connected network would need to learn. CNNs have shown promise in image classification tasks,--, in [0051]-[0058]; and, --[0139] As opposed to the 2D CNN approach, a 3D CNN incorporated information from adjacent frames at every layer, extracting spatiotemporal dependent features which have also proven to be useful for video classification. In a 3D CNN approach, a GAP layer reduced the fully connected layer input from the feature map size to the number of filters.--, in [0138]-[0140]; also see KO: Fig. 4, -- The artificial intelligence model (AIM) includes various forms of normal state fetal images in the plurality of fetal image data (ID1) included in the input data, various types of images of fetuses with congenital diseases, and various forms of severe congenital diseases. It can be built to learn through correlation of fetal images. The artificial intelligence model (AIM) uses a plurality of fetal image data (ID1), including images of fetuses in various forms of normal conditions, images of fetuses in various forms of congenital diseases, and images of fetuses in various forms of severe congenital diseases. A CNN algorithm or RNN algorithm can be used to build and reinforce learning as a learning data set. At this time, the fetal image data OD for each fetus may include fetal image data OD1 in a normal state, fetal image data OD2 in a state with a congenital disease, and fetal image data OD3 in a state with a severe congenital disease.--, in the first paragraph in page 7 of machine translated English version of WO 2022265345 A1).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: Roffé (US 10667776 B2) discloses (19) a denoising filter is applied to each image frame. An adaptive Gaussian thresholder is used to emphasize the dark areas in the image and turn the image from grayscale to a binary image, and (20) FIG. 3 depicts detection of contrast in image frames. FIG. 3A depicts a raw image acquired in act A110. The LCA 110 is visible, but there is a lot of noise in the image. FIG. 3B depicts a denoised image. The LCA 110 is visible in the image and the noise that dominated portions of FIG. 3A has been removed.
Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEI WEN YANG whose telephone number is (571)270-5670. The examiner can normally be reached on 8:00 - 5:00 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEI WEN YANG/Primary Examiner, Art Unit 2662
Prosecution Insights