Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities:
On pg. 4, section heading “PDETAILED DESCRIPTION” should read “DETAILED DESCRIPTION”.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Madabhushi et al. (US 20190357870 A1), (hereinafter Madabhushi) in view of Lee (“Performing Image Augmentation For Machine Learning”, Datature Blog, Oct. 2022).
Regarding claim 1, Madabhushi teaches a computer device, comprising: a storage device, configured to store an image pre-processing application and an artificial intelligence model; and a processor, configured to execute the image pre-processing application and the artificial intelligence model to perform the following operations (Madabhushi, “Embodiments extract radiomic features from the inside (intratumoral) and outside (peri-tumoral) tumoral regions to differentiate MIA and AIS from invasive (INV) adenocarcinoma regions represented on medical imagery, including CT scans.”, pgs. 1, paragraph 0019, “Embodiments may validate the ability of a radiomic feature set that includes intratumoral and peritumoral features to distinguish MIA and AIS from invasive adenocarcinoma using different types of machine learning classifiers.”, pg. 3, paragraph 0027, lines 1-5, “In one example, a method may be implemented as computer executable instructions. Thus, in one example, a computer-readable storage device may store computer executable instructions that if executed by a machine (e.g., computer, processor) cause the machine to perform methods or operations described or claimed herein including methods or operations 400, 500, or 900.”, pg. 6, paragraph 0048, lines 1-7, see Figs. 7 and 8):
obtaining a first image set, wherein the first image set comprises at least two images captured with different parameters (Madabhushi, “In one embodiment, a set of 146 CT scans from four different institutions was accessed. Only Tla INV cancer cases were chosen (predominantly GGO < = 2 cm diameter) from the entire cohort, matching the diameter of the MIA and AIS subset. The first data set (N = 39) which contained 7 AIS and 2 MIA and 30 INV cancer cases was used for training a machine learning classifier”, pg. 3, paragraph 0020, “In one embodiment , apparatus 800 also includes training and testing circuit 851. Training and testing circuit 851 is configured to train nodule classification circuit 757 according to techniques described herein. Training and testing circuit 851 is configured to train the nodule classification circuit 757 to compute the probability that the GGO nodule is invasive adenocarcinoma using a set of training images, where a member of the set of training images is acquired using different imaging parameters than the diagnostic image.”, pg. 7, paragraph 0061, lines 1-10, A training dataset of CT images is obtained for training a machine learning classifier. This dataset includes images acquired using different imaging parameters.);
performing image pre-processing on each image of the first image set to obtain a second image set; adding the second image set to a training image data set; and training the artificial intelligence model using the training image data set (Madabhushi, “In one embodiment, training and testing circuit 851 is configured to access a training dataset of digitized images of a region of interest demonstrating lung nodules. The training dataset includes images of tissue that were classified as AIS or MIA, and images of tissue that were classified as invasive adenocarcinoma... In this embodiment, the machine learning classifier is trained using the training dataset of images and tested using the testing dataset of images. Training the machine learning classifier may include training the machine learning classifier until a threshold level of accuracy is achieved, until a threshold time has been spent training the machine learning classifier, until a threshold amount of computational resources have been expended training the machine learning classifier, or until a user terminates training.”, pgs. 7 and 8, paragraph 0061, lines 13-33, Once obtained the training images are pre-processed, such as the segmentation and/or feature extraction of steps 420-440 in Fig. 5, to generate a final training dataset used to train the machine learning model.).
Madabhushi does not teach performing image augmentation on the second image set to obtain a third image set; and adding the third image set to a training image data set.
However, Lee teaches performing image augmentation on the second image set to obtain a third image set; and adding the third image set to a training image data set (Lee, “Image augmentation
is a technique that creates new images from existing ones. To create new images from the existing ones, you make some small changes to them, such as adjusting the brightness of the image, or rotating the image, or shifting the subject in the image horizontally or vertically.”, pg. 2, lines 9-13, “Instead of performing the image augmentation by hand (well, code actually), wouldn’t it be cool if there is a tool that can help us automate this entire process? This is where the Datature platform comes in. Datature is a platform for training and deploying machine learning models, all without needing to code. The platform allows users to manage multiple machine learning projects, across multiple data sources. With Datature, you can simply onboard your image dataset as well as existing annotations to the platform.”, pg. 6, 18-27, “You can see that the Augmentations node offers a series of augmentation techniques broadly classified into: Random Position Augmentations and Color Space Augmentations… You should now see the various augmented images created by the tool.”, pg. 9, lines 1-22, “With the images annotated, it is time to train a model to perform the detection of workers in an image”, pg. 9, lines 24-25, Image augmentation is performed with respect to an annotated dataset for training a machine learning model.).
Madabhushi teaches obtaining a training dataset of CT images acquired using different imaging parameters (Madabhushi, see paragraph 0037 and 0061), performing image pre-processing including segmentation and feature extraction (see Fig. 5, steps 420-440), and using the resulting dataset to train a machine learning model (see Fig. 5, step 542). Madabhushi does not teach performing image augmentation on the pre-processed training images. Lee teaches performing image augmentation on annotated training images prior to training machine learning models (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Madabhushi to include image augmentation applied to pre-processed training images as taught by Lee (Lee, pg. 6, 18-27, pg. 9, lines 24-25). The motivation for doing so would have been to diversify the training dataset, thereby reducing overfitting of the machine learning model (as suggested by Lee, “Image augmentation techniques allow you to artificially diversify your dataset. This can be significant in a variety of situations. Image augmentation can assist in artificially creating different images to bolster underrepresented classes or challenge your model to learn more critical features despite visual variance.”, pg. 2, lines 15-19). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Madabhushi with Lee to obtain the invention as specified in claim 1.
Claim 6 corresponds to claim 1, with the addition of a deep learning method of an artificial intelligence model for medical image recognition to perform the functions according to claim 1. Madabhushi in view of Lee teaches the addition of a deep learning method of an artificial intelligence model for medical image recognition (Madabhushi, “Thus, in one example, a computer-readable storage device may store computer executable instructions that if executed by a machine (e.g., computer, processor) cause the machine to perform methods or operations described or claimed herein including methods or operations 400, 500, or 900.”, pg. 6, paragraph 0048, lines 2-7, see Figs. 4 and 5) to perform the functions according to claim 1. As indicated in the analysis of claim 1, Madabhushi in view of Lee teaches all the limitations according to claim 1. Therefore, claim 6 is rejected for the same reasons of obviousness as claim 1.
Claims 2 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Madabhushi et al. (US 20190357870 A1) in view of Lee (“Performing Image Augmentation For Machine Learning”, Datature Blog, Oct. 2022) and further in view of Das et al. (US 20230018833 A1), (hereinafter Das).
Regarding claim 2, Madabhushi in view of Lee teaches the computer device according to claim 1. Madabhushi in view of Lee does not teach wherein the at least two images are at least two medical images obtained by photographing the same body part of the same patient during substantially the same time interval using different parameters, and comprise corresponding labeled organ regions.
However, Das teaches wherein the at least two images are at least two medical images obtained by photographing the same body part of the same patient during substantially the same time interval using different parameters, and comprise corresponding labeled organ regions (Das, “As described above with reference to FIG. 2, a variety of different clinical and non-clinical data may be included in the multimodal clinical data 104 for each patient, and the amount of data available can vary for different patients. For example, some patients may be associated with multiple different imaging studies captured in different modalities. Each imaging study can also include several different images (scan slices in CT an MR for example) with different perspectives of an anatomical region of interest and captured with different capture parameters/protocols (e.g., contrast vs. non-contrast, different reconstruction kernel sizes, different MRI frequencies, different scanner devices, etc.). This imaging data needs to be analyzed and filtered to select the best images for training the clinical inferencing
model 132.… In this regard, initial selection criteria can define at least some high-level filtering parameters for selecting the appropriate patient/subjects and the appropriate subset of data for the patient/subjects from the multimodal clinical data 104. For example, in one implementation in which the clinical inferencing model 132/132' is adapted to classify a lung condition based at least in part on medical image data captured of the lungs, the initial inclusion criteria may include the following: 1. both CT and XR images captured for the same patient, 2. the CT and XR images to be captured within a 48 hour time window, 3. the images to include non-contrast images, and 4. the images to have a slice thickness of 3 .0 millimeters or less.”, pg. 8, paragraphs 0061-0062, Training datasets are generated for each patient including multiple images of the same patient captured using different imaging parameters during a set time interval.).
Madabhushi in view of Lee teaches obtaining training images captured using different imaging parameters for multiple patients and labeling organ regions for the training images (Madabhushi, pg. 4, paragraph 0037, see Fig. 5, step 420). Madabhushi in view of Lee does not teach obtaining multiple training images of the same patient captured during substantially the same time interval. Das teaches generating training datasets which include multiple images of a same patient, captured using different imaging parameters during a defined time window (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the training images of Madabhushi in view of Lee to include the training datasets as taught by Das (Das, pg. 8, paragraphs 0061-0062). The motivation for doing so would have been select training samples which reflect consistent anatomical features across imaging parameters, thereby improving training data quality and model accuracy (as suggested by Das, “The mutual information analysis component 310 can further evaluate mutual information included in and/or associated with each of the initial datasets 106 to sub-select only those datasets whose mutual information reflects a consistent anatomy, pathology or diagnosis.”, pg. 12, paragraph 0083, lines 1-5). The combination of Madabhushi in view of Lee and further in view of Das would result in training datasets for individual patients that include multiple images of the same organ obtained using different imaging parameters, with corresponding labels assigned to the same organ across images. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Madabhushi in view of Lee with Das to obtain the invention as specified in claim 2.
Claim 7 corresponds to claim 2, with the addition of a deep learning method of an artificial intelligence model for medical image recognition to perform the functions according to claim 2. Madabhushi in view of Lee and further in view of Das teaches the addition of a deep learning method of an artificial intelligence model for medical image recognition (see analysis of claim 6) to perform the functions according to claim 2. As indicated in the analysis of claim 2, Madabhushi in view of Lee and further in view of Das teaches all the limitations according to claim 2. Therefore, claim 7 is rejected for the same reasons of obviousness as claim 2.
Claims 3 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Madabhushi et al. (US 20190357870 A1) in view of Lee (“Performing Image Augmentation For Machine Learning”, Datature Blog, Oct. 2022) and further in view of Knapp et al. (US 20130223711 A1), (hereinafter Knapp).
Regarding claim 3, Madabhushi in view of Lee teaches the computer device according to claim 1. Madabhushi in view of Lee does not teach wherein the image pre-processing comprises coordinate system conversion processing, locus-to-mask processing, image padding processing, image normalization processing, or a combination thereof.
However Knapp teaches wherein the image pre-processing comprises coordinate system conversion processing, locus-to-mask processing, image padding processing, image normalization processing, or a combination thereof (Knapp, “As illustrated in FIG. 2, which shows an overall process of building a predictor for equalized images, according to various embodiments of the invention (and it is noted that this need not be limited to pectoral equalized images, but rather, it may be generalized to other types of equalized images), the process may involve image normalization 22 of an existing image (which may be obtained, for example, from an image database 21; however, the invention is not limited to this, and images may be obtained from elsewhere). The normalization 22 may involve an algorithm used for chest X-rays… The process may then proceed by applying breast segmentation 23 (or other segmentation, as may be appropriate to the type of image) to the normalized image.”, pgs. 1 and 2, paragraphs 0018-0019, “The above may be repeated for a set of images in order to obtain sets of image data that may be used for bias estimation 24 and to train predictive models 26, as will be explained further below.”, pg. 2, paragraph 0021, lines 1-4, Chest x-ray training image datasets are normalized prior to performing segmentation and model training.).
Madabhushi in view of Lee teaches performing image pre-processing for training images including segmentation and feature extraction (see Fig. 5, steps 420-440) prior to model training. Madabhushi in view of Lee does not teach pre-processing including coordinate system conversion processing, locus-to-mask processing, image padding processing or image normalization processing. Knapp teaches performing image normalization for training images prior to segmentation and model training (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the training image pre-processing of Madabhushi in view of Lee to include image normalization as taught by Knapp (Knapp, pgs. 1 and 2, paragraphs 0018-0021). The motivation for doing so would have been to standardize pixel values across training image, thereby improving numerical stability and convergence during model training. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Madabhushi in view of Lee with Knapp to obtain the invention as specified in claim 3.
Claim 8 corresponds to claim 3, with the addition of a deep learning method of an artificial intelligence model for medical image recognition to perform the functions according to claim 3. Madabhushi in view of Lee and further in view of Knapp teaches the addition of a deep learning method of an artificial intelligence model for medical image recognition (see analysis of claim 6) to perform the functions according to claim 3. As indicated in the analysis of claim 3, Madabhushi in view of Lee and further in view of Knapp teaches all the limitations according to claim 3. Therefore, claim 8 is rejected for the same reasons of obviousness as claim 3.
Claims 4 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Madabhushi et al. (US 20190357870 A1) in view of Lee (“Performing Image Augmentation For Machine Learning”, Datature Blog, Oct. 2022) and further in view of Hillen (US 11553874 B2).
Regarding claim 4, Madabhushi in view of Lee teaches the computer device according to claim 1, wherein the processor is further configured to perform the following operations:
receiving an input image; performing image pre-processing on the input image to obtain a first image; and (Madabhushi, “The set of operations 400 includes, at 410, accessing an image of a region of tissue. The region of tissue includes lung tissue… The set of operations 400 also includes, at 420 defining a tumoral region by segmenting the GGO nodule, where defining the tumoral region includes defining a tumoral boundary.”, pg. 4, paragraphs 0037-0038, )
utilizing the artificial intelligence model to recognize the first image (Madabhushi, “The set of operations 400 also includes, at 440, extracting a set of radiomic features from the peri-tumoral region and the tumoral region… The set of operations 400 also includes, at 450, providing the set of radiomic features to a machine learning classifier trained to distinguish minimally invasive adenocarcinoma (MIA) and adenocarcinoma in situ (AIS) from invasive adenocarcinoma.”, pg. 5, paragraphs 0040-0041).
Madabhushi in view of Lee does not teach utilizing the artificial intelligence model to recognize the first image to obtain a second image; and performing image post-processing on the input image according to the second image to obtain an output image.
However, Hillen teaches utilizing the artificial intelligence model to recognize the first image to obtain a second image; and performing image post-processing on the input image according to the second image to obtain an output image (Hillen, “In this example, the output matrix (e.g., with 1440 by 1920 elements) from the machine learning inference 412 is stored in an output data database 410. A renderer 414 determines whether a detected image feature (e.g. carious lesion) are present based on the value of the confidence score and, for any lesion present, generates the coordinates of the lesion bounding box. The results determined by the renderer 414 ( e.g., a list of pixel-coordinates of the detected feature and its rendered bounding box) can be stored on the storage device 316 (e.g., in an output data database 410) for later retrieval and use.”, column 9, lines 29-39, “In production phase, the Input 516 is typically an image (or a set of images) without any annotation. This image is usually processed with the same Image Preprocessing Module 504 that is used in Machine Learning Trainer 408. Then, without any further processing, the image is fed to the Trained Model 514 and the model predict the target output (e.g., a bounding box, a heatmap, or a binary mask) for any present detected feature. These intermediate outputs are put together and superimposed on the original input image in Postprocessing 518 and results in the Output 520 that can be rendered on the users' workstation.”, column 11, lines 3-13, Confidence scores are computed for regions of input images using a machine learning model. These confidence scores are post-processed to generate bounding box for detected regions. The model outputs are then superimposed on the original input image for visualization.).
Madabhushi in view of Lee teaches classifying pre-processed input images using an artificial intelligence model to determine disease classification (Madabhushi, pg. 5, paragraph 0041, lines 1-5, see Fig. 4, steps 410-470). Madabhushi in view of Lee does not teach post-processing of the input image according to model outputs. Hillen teaches post-processing outputs of a machine learning model, including transforming model outputs to image representations for superimposing onto original input images for user visualization (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Madabhushi in view of Lee to include the post-processing as taught by Hillen (Hillen, column 11, lines 3-13), thereby superimposing model classification results on input images. The motivation for doing so would have been to allow users to simultaneously visualize both disease classification and position relative to the input image. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Madabhushi in view of Lee with Hillen to obtain the invention as specified in claim 4.
Claim 9 corresponds to claim 4, with the addition of a deep learning method of an artificial intelligence model for medical image recognition to perform the functions according to claim 4. Madabhushi in view of Lee and further in view of Hillen teaches the addition of a deep learning method of an artificial intelligence model for medical image recognition (see analysis of claim 6) to perform the functions according to claim 4. As indicated in the analysis of claim 4, Madabhushi in view of Lee and further in view of Hillen teaches all the limitations according to claim 4. Therefore, claim 9 is rejected for the same reasons of obviousness as claim 4.
Allowable Subject Matter
Claims 5 and 10 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CONNOR LEVI HANSEN whose telephone number is (703)756-5533. The examiner can normally be reached Monday-Friday 9:00-5:00 (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CONNOR L HANSEN/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672