DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. 202310733977.3, filed on 06/20/2023.
Specification
The use of the terms NVIDIA, GeForce, RTX, PyTorch, Python, OpenCV and Portable Network Graphics (PNG) format, which is a trade name or a mark used in commerce, has been noted in this application. The term should be accompanied by the generic terminology; furthermore the term should be capitalized wherever it appears or, where appropriate, include a proper symbol indicating use in commerce such as ™, SM , or ® following the term.
Although the use of trade names and marks used in commerce (i.e., trademarks, service marks, certification marks, and collective marks) are permissible in patent applications, the proprietary nature of the marks should be respected and every effort made to prevent their use in any manner which might adversely affect their validity as commercial marks.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3-7 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 3 contains the trademarks/trade names Python, OpenCV and Portable Network Graphics (PNG) format. Where a trademark or trade name is used in a claim as a limitation to identify or describe a particular material or product, the claim does not comply with the requirements of 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph. See Ex parte Simpson, 218 USPQ 1020 (Bd. App. 1982). The claim scope is uncertain since the trademark or trade name cannot be used properly to identify any particular material or product. A trademark or trade name is used to identify a source of goods, and not the goods themselves. Thus, a trademark or trade name does not identify or describe the goods associated with the trademark or trade name. In the present case, the trademarks/trade names Python and OpenCV to identify/describe “batch-processing video/images, and PNG is used to identify/describe video/frames saving formats accordingly, the identification/description is indefinite.
Dependent claims 4-7 are rejected for fully incorporating the dependencies of their bases.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-7 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claims 1 and 6 are directed to a method, and system, which is directed to a process, one of the statutory categories.
Step 2A prong 1: Does the claim recite a judicial exception? Yes, claim 1 recites similar limitations of: “annotating the preprocessed videos and images of adult echocardiograms, dividing the adult echocardiogram dataset into a training set, a validation set, and a test set, selecting an optimal classification model, evaluating performance of the optimal adult echocardiogram view classification model”. The broadest reasonable interpretation “falls under the mental process groupings of abstract ideas because they cover concepts performed in the human mind, including observation, evaluation, judgment, and opinion.
Step 2A prong 2: Does the claim recite additional elements? Do those additional elements, individually and in combination, integrate the judicial exception into a practical application?
The additional element of “collecting videos and images of multi-modal multi-view adult echocardiograms, a deep learning algorithm, generate an adult echocardiogram dataset, constructing an adult echocardiogram view classification model based on a ResNet network, training the model using the training set and inputting a to-be-tested image or video into the adult echocardiogram view classification model to obtain a classification result”
The recites “collecting videos and images of multi-modal multi-view adult echocardiograms” using a “deep learning algorithm,” to “construct an adult echocardiogram view classification model based on a ResNet network, training the model using the training set” for “inputting to-be-tested images or videos into a model for obtaining classification results”, thus falling under the “apply it” consideration (MPEP 2106.05 (f)). The use of a “deep learning algorithm” to analyzed images or videos is similar to an “off the shelf” component.
The additional elements, alone and in combination, fail to integrate the abstract idea into a practical application. Thus, the claims is directed to an abstract idea.
Step 2B: Do the additional elements, considered individually and in combination, amount to significantly more than the judicial exception? No, As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of “collecting videos and images of multi-modal multi-view adult echocardiograms” “a deep learning algorithm, generate an adult echocardiogram dataset, constructing an adult echocardiogram view classification model based on a ResNet network, training the model using the training set and inputting a to-be-tested image or video into the adult echocardiogram view classification model to obtain a classification result” only amount to “apply it” consideration (MPEP 2106.5 (f)
As discussed in Step 2A, Prong 2 above, the recitations of “collecting videos and images of multi-modal multi-view adult echocardiograms” using a “deep learning algorithm,” to “construct an adult echocardiogram view classification model based on a ResNet network, training the model using the training” for “inputting to-be-tested images or videos into a model for obtaining classification results”, are recited at a high level of generality and is an improvement to the abstract idea itself. Thus, only amounts to insignificant extra-solution activity and WURC activities to condition the data for input into the classification model, similar to “receiving or transmitting data over the network, e.g., using the Internet to gather data, Symantec, 832 F.3d at 1362 (utilizing intermediary computer to forward information)” See MPEP 2106.05 (d). These limitations, taken alone or in combination, fail to provide an inventive concept. Thus, the claim is not patent eligible.
The dependent claims the additional limitations (in claims 2-7) also constitute concepts to “apply it” which fall within the “Mental Processes” and “Mathematical Concepts” groupings of abstract ideas.
This judicial exception is not integrated into a practical application and amount to no more than adding insignificant extra-solution activity/specifications related to data gathering, data input, or data transmittal. These additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The dependent claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above.
Examiner Notes
Multi-modal refers to data from different types of sources and/or views. Multi-view, different perspectives of the object.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-4 are rejected under 35 U.S.C. 103 as being unpatentable over Fornwalt (US 20210150693 A1, Filed Date: Nov. 16, 2020) in view of Ouyang (US 20210304410 A1, Filed Date: Mar. 30, 2021).
Regarding independent claim 1, Fornwalt teaches: A multi-modal multi-view classification method for echocardiograms based on a deep learning algorithm, comprising the following steps:
step 1: collecting videos and images of multi-modal multi-view adult echocardiograms; (Fornwalt − [0058] During the acquisition of an echocardiogram (or any other medical video acquisition of the heart including but not limited to videos generated using cardiac MRI or CT), images of the heart and large blood vessels are acquired in different two-dimensional planes, or “views”; [0067] Table 2 below shows that all videos combined with the multi-modal DNN approach; [0104] At 102, the process 100 can receive an echocardiographic video of a heart associated with a patient. The echocardiographic video can include echocardiography video frames. The video frames can include video frames taken from one or more views of the heart of the patient. For example, the video frames can include video frames taken at twenty-one different views of the heart.)
step 2: preprocessing the collected videos and images of adult echocardiograms; (Fornwalt − [Image Collection and Preprocessing] [0073] [0115] An echocardiography study consists of several videos containing multiple views of the heart. The retrieved DICOM files contained an annotated video and a raw video when the equipment was configured to store it.)
step 3: annotating the preprocessed videos and images of adult echocardiograms to generate an adult echocardiogram dataset; (Fornwalt − [Image Collection and Preprocessing] [0115-0116] An echocardiography study consists of several videos containing multiple views of the heart. The retrieved DICOM files contained an annotated video and a raw video when the equipment was configured to store it. Examiner Note: the stored data is the data set;)
step 5: constructing an adult echocardiogram view classification model based on a ResNet network, (Fornwalt − [0010] the trained model can further include a trained submodel, The trained submodel can include a trained classifier. [0137] the models 300, 304, 400, 404 are low parameter designs [0140] Deep learning models typically consist of millions of parameters; and ResNet more than forty million parameters) training the model using the training set, (Fornwalt − [0143] Using the cross-validation set, For each training set) and selecting an optimal classification model using the validation set; (Fornwalt − [0092] [0092] All four candidate architectures were applied to all the identified echocardiography views with a 1-year mortality label, and the 3D CNN consistently showed the best performance (FIG. 3))
step 6: evaluating performance of the optimal adult echocardiogram view classification model based on the test set; (Fornwalt − [0092] [0092] All four candidate architectures were applied to all the identified echocardiography views with a 1-year mortality label, and the 3D CNN consistently showed the best performance (FIG. 3))
and step 7: inputting a to-be-tested image or video into the adult echocardiogram view classification model to obtain a classification result. (Fornwalt − [0106] At 106, the process 100 can provide the video frames to the trained neural network [0109] At 110, the process 100 can output the raw risk score to at least one of a memory or a display for viewing by a medical practitioner or healthcare administrator. In some embodiments, the process 100 can generate and output a report based on the risk score. The report can include the raw risk score. The report can include any appropriate graphs and/or charts generated based on the risk score)
Fornwalt does not explicitly teach: dividing the adult echocardiogram dataset
However, Ouyang teaches: step 4: dividing the adult echocardiogram dataset into a training set, a validation set, and a test set; (Ouyang − [0065] A standard full resting echocardiogram study consists of a series of 50-100 videos and still images visualizing the heart from different angles, locations and image acquisition techniques (two-dimensional images, tissue Doppler images, color Doppler images and others). Videos were randomly split into 7,465, 1,277 and 1,288 patients, respectively, for the training, validation and test sets.)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teaching of Fornwalt and Ouyang as each invention same field of image analysis of medical images. One of ordinary skill in the art would have been motivated to make these modification in order to improves performance of machine learning models for diagnosis risks/diseases of a patient organs.
Regarding dependent claim 2, depends on claim 1, Fornwalt teaches: wherein in step 1, said collecting the videos and images of multi-modal multi-view adult echocardiograms specifically comprises: collecting the videos and images of multi-modal multi-view adult echocardiograms, comprising three-dimensional grayscale echocardiograms, two-dimensional grayscale parasternal left ventricular long-axis views, two-dimensional grayscale parasternal left ventricular short-axis views, two-dimensional grayscale parasternal aorta short-axis views, two-dimensional grayscale subxiphoid views, two-dimensional grayscale apical two-chamber views, two-dimensional grayscale apical three-chamber views, two-dimensional grayscale apical four-chamber views, two-dimensional apical two-chamber views based on left ventricular opacification, two-dimensional apical three-chamber views based on left ventricular opacification, two-dimensional apical four-chamber views based on left ventricular opacification, and two-dimensional parasternal left ventricular short-axis views based on left ventricular opacification. (Fornwalt – [0003] videos of the heart, for example those acquired during an echocardiogram, [0004] the present disclosure provides a neural network capable of receiving echocardiography videos; [0009-0010]] In the method, the plurality of echocardiographic views can include at least two of an apical two-chamber view, an apical three-chamber view, an apical four-chamber view, an apical four-chamber focused to right ventricle view, an apical five chamber view, a parasternal long axis view, a parasternal long descending aorta view, a parasternal long mitral valve view, a parasternal long pulmonic valve view, a parasternal long right ventricle inflow view, a parasternal long zoom aortic valve view, a parasternal short aortic valve view, a parasternal short pulmonic valve and pulmonary artery view, a parasternal short tricuspid valve view, a short axis apex view, a short axis base view, a short axis mid papillary view, a subcostal four-chamber view, a subcostal hepatic vein view, a subcostal inter-atrial septum view, a subcostal inferior vena cava view, a subcostal right ventricle view, a suprasternal notch view, a short axis mid papillary view, a short axis apex view, an apical three-chamber zoom view, an apical two-chamber zoom view, or a short axis base view.)
Fornwalt does not explicitly teach: color Doppler echocardiograms
However, Ouyang teaches: color Doppler echocardiograms,(Ouyang − [0074] videos with injection of ultrasonic contrast agents and videos with color Doppler)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teaching of Fornwalt and Ouyang as each invention same field of image analysis of medical images. One of ordinary skill in the art would have been motivated to make these modification in order to improves performance of machine learning models for diagnosis risks/diseases of a patient organs.
Regarding dependent claim 3, depends on claim 2, Fornwalt teaches: wherein in step 2, said preprocessing the collected videos and images of adult echocardiograms specifically comprises: batch-processing the videos and images of adult echocardiograms using a Python third-party library OpenCV, extracting sector regions of interest based on pixel changes in consecutive frames by using image preprocessing operations, (Fornwalt – [0099] algorithm as implemented in the OpenCV (version 2.4.13.7) software library [0141] In some embodiments, the models can be implemented using Python; [0147 ] The analysis was conducted using the lifelines python package) and saving the videos as frames in Portable Network Graphics (PNG) format ([0073] storing video frame; Official Notice: saving videos as frames in PNG is well known and obvious that Fornwalt computer saves PNG format frames.)
Regarding dependent claim 4, depends on claim 3, Fornwalt does not explicitly teach: wherein in step 4, said dividing the adult echocardiogram dataset into a training set, a validation set, and a test set specifically comprises: dividing the adult echocardiogram dataset into the training set, the validation set, and the test set in a ratio of 8:1:1,
However, Ouyang teaches: wherein in step 4, said dividing the adult echocardiogram dataset into a training set, a validation set, and a test set specifically comprises: dividing the adult echocardiogram dataset into the training set, the validation set, and the test set in a ratio of 8:1:1, wherein with view data having a smallest sample volume as a standard, datasets of other view categories are sampled at equal proportions to create a dataset with balanced sample volumes for each view category; the training set is used to train the adult echocardiogram view classification model, the validation set is used to adjust model hyperparameters and select the optimal classification model, and the test set is used to evaluate classification performance of the model. (Ouyang − [0065] A standard full resting echocardiogram study consists of a series of 50-100 videos and still images visualizing the heart from different angles, locations and image acquisition techniques (two-dimensional images, tissue Doppler images, color Doppler images and others). Videos were randomly split into 7,465, 1,277 and 1,288 patients, respectively, for the training, validation and test sets. 98%, 1%, 1% ratio equal to 8:1:1 ratio)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teaching of Fornwalt and Ouyang as each invention same field of image analysis of medical images. One of ordinary skill in the art would have been motivated to make these modification in order to improves performance of machine learning models for diagnosis risks/diseases of a patient organs.
Claim(s) 5 is rejected under 35 U.S.C. 103 as being unpatentable over Fornwalt and Ouyang as applied to claim 4 above, and further in view of Liu (US 20240282090 A1, Filed Date: Feb. 16, 2023).
Regarding dependent claim 5, depends on claim 4, Fornwalt teaches: wherein in step 5, said constructing the adult echocardiogram view classification model based on the ResNet network, training the model using the training set, and selecting the optimal classification model using the validation set specifically comprises: constructing the adult echocardiogram view classification model based on a 101-layer ResNet network; (Fornwalt − [0010] the trained model can further include a trained submodel, The trained submodel can include a trained classifier. [0137] the models 300, 304, 400, 404 are low parameter designs [0140] Deep learning models typically consist of millions of parameters; and ResNet more than forty million parameters)
Fornwalt does not explicitly teach: a softmax function and an Adam optimizer
However, Liu teaches: training the model using the training set, wherein a transfer learning strategy is employed during a training phase, training results on an ImageNet dataset are used as pre-trained weights, an output layer classifier uses a Softmax function, with 13 categories for classification, (Liu− [0004] [0024] calculating a softmax) an Adam optimizer with an initial learning rate of 0.0001 is used, (Liu− [0104] Adam optimizer is used model training and model output) model fine-tuning is performed with a batch size of 128 for 100 iterations, and a convolutional neural network with residual structures is used to extract ultrasound image features; after training, evaluating model performance on the validation set by minimizing a cross-entropy loss between real labels and predicted results; and selecting a model weight with highest classification accuracy as the optimal [adult echocardiogram view] classification model. (Liu− [0010] and taking a model parameter value with a highest classification accuracy rate on the test set as a final AmmH model parameter value for classifying)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teaching of Fornwalt, Ouyang and Liu as each invention same field of image analysis of medical images. One of ordinary skill in the art would have been motivated to make these modification in order to improves performance of machine learning models for diagnosis risks/diseases of a patient organs.
Claim(s) 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over Fornwalt, Ouyang, Liu as applied to claim 5 above, and further in view of (Daughton US 11901076 B1, Filed Date: Jun. 11, 2021).
Regarding dependent claim 6, depends on claim 5, Fornwalt teaches: wherein in step 6, said evaluating the performance of the optimal adult echocardiogram view classification model based on the test set specifically comprises: confusion matrix, (Fornwalt − [0092] [0092] All four candidate architectures were applied to all the identified echocardiography views with a 1-year mortality label, and the 3D CNN consistently showed the best performance (FIG. 3))
Fornwalt does not explicitly teach: precision, recall, specificity, and F1 score
However, Daughton teaches: evaluating the performance of the optimal adult echocardiogram view classification model on the validation set and the test set based on confusion matrix, accuracy, precision, recall, specificity, and F1 score. (Daughton − [Col. 18 ll. 53-60] Note that the predefined target performance may include an AUC. However, in other embodiments, a wide variety of target performance metrics may be used, including: specificity, sensitivity or recall, precision, F1-score, p-value, accuracy, a confusion matrix, mean square error, mean absolute error, a receiver operator characteristic curve (ROC) and/or another performance metric (which may be determined from the ROC and/or the confusion matrix).)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teaching of Fornwalt, Ouyang, Liu and Daughton as each invention same field of image analysis of medical images. One of ordinary skill in the art would have been motivated to make these modification in order to improves performance of machine learning models for diagnosis risks/diseases of a patient organs.
Regarding dependent claim 7, depends on claim 6, Fornwalt teaches: wherein in step 7, said inputting the to-be-tested image or video into the adult echocardiogram view classification model to obtain the classification result specifically comprises: inputting the to-be-tested image into the adult echocardiogram view classification model, wherein for adult echocardiogram images, the adult echocardiogram view classification model predicts a classification result for each image, while for adult echocardiogram videos, the adult echocardiogram view classification model samples 10 frames from each video at regular intervals for prediction, takes an average value of predictions results, and uses a view class corresponding to a maximum prediction probability as a classification result for the video; (Fornwalt − [0106] At 106, the process 100 can provide the video frames to the trained neural network [0109] At 110, the process 100 can output the raw risk score to at least one of a memory or a display for viewing by a medical practitioner or healthcare administrator. In some embodiments, the process 100 can generate and output a report based on the risk score. The report can include the raw risk score. The report can include any appropriate graphs and/or charts generated based on the risk score)
based on a gradient-weighted class activation map visualization analysis method, generating a heatmap by using final-layer feature weights of the adult echocardiogram view classification model, to visualize focus areas of the view classification model, and performing an interpretable analysis on the classification result. (Fornwalt − [0087] he final fully connected layer after the GAP would provide a weighted average of the CNN features, which could indicate what sections of the video weighted more in the final decision.)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teaching of Fornwalt, Ouyang, Liu and Daughton as each invention same field of image analysis of medical images. One of ordinary skill in the art would have been motivated to make these modification in order to improves performance of machine learning models for diagnosis risks/diseases of a patient organs.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Ye Zhu, Automatic view classification of contrast and non-contrast echocardiography.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CARL E BARNES JR whose telephone number is (571)270-3395. The examiner can normally be reached Monday-Friday 9am-6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen Hong can be reached at (571) 272-4124. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CARL E BARNES JR/Examiner, Art Unit 2178
/STEPHEN S HONG/Supervisory Patent Examiner, Art Unit 2178