Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-21 are rejected under 35 U.S.C. 101 because the
claimed invention is directed to non-statutory subject matter as follows. Regarding
claims 1, 8, and 15, the claims are directed to an abstract idea, namely mathematical operations and information processing. The claims are not integrated into a practical application and the claims lack an inventive concept. Furthermore, claims 2-7, 9-14, 16-21 are also directed to an abstract idea, specifically, describing datasets and methods to train an AI model to output specific types of data.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2, 4-9, 11-16, 18-21 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Rajagopal et al. (US 20230410301 A1), hereinafter Rajagopal.
Regarding claim 1, Rajagopal teaches A system comprising: a memory to store instructions; a processor to execute the instructions stored in the memory; wherein the system is specially configured to execute instructions for implementing (Para. 16 see "a system is provided that includes one or more data processors and a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein." Para. 96 see "the processing units can be implemented within one or more algorithm specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, other electronic units designed to perform the functions described above." Para. 99 see "As used herein the term “memory” refers to any type of long term, short term, volatile, nonvolatile, or other storage medium and is not to be limited to any particular type of memory or number of memories, or type of media upon which memory is stored."). a unified AI model pre-trained for use with medical image classification, (Para. 35 see "the present disclosure introduces techniques for incorporating multiple sources of pathology data under a single umbrella to improve a machine-learning based cancer classification system over what was possible using any individual pathology source." Para. 38 see "As shown in FIG. 1, the tumor identification, classification, and grading performed by the computing environment 100 in this example includes several stages: an image acquisition stage 105, a model training stage 110, an identification, classification, and grading stage 115, and an analysis stage 120." Para. 51 see "The model 130 is constructed to take as input the subset of images 135a (e.g., a registered three channel 3D MRI input using T2w, DWI, ADC) and for each pixel or voxel the model 130 predicts a cancer grade (e.g., 0-5) to produce voxel-wise maps of lesion occupancy (semantic segmentation) and cancer grade as two output channels (i.e., a 3D cancer grade map) with a same resolution as the subset of images 135a. Thus, the encoded feature representation generated from the model 130 is used for related but distinct tasks of segmenting lesions and predicting cancer grade throughout the localized region or object of interest within the images 135a, whether binary or multi-level."). medical image localization, and medical image segmentation, in the context of medical image analysis, by performing the following operations: (Para. 46 see "the labeling and training uses lesion locations provided as volumetric masks and multi-level lesion Gleason grade values... the labeling and training uses region locations in the form of volumetric masks and multi-level region Gleason grade values." Para. 51 see "The model 130 is constructed to take as input the subset of images 135a (e.g., a registered three channel 3D MRI input using T2w, DWI, ADC) and for each pixel or voxel the model 130 predicts a cancer grade (e.g., 0-5) to produce voxel-wise maps of lesion occupancy (semantic segmentation) and cancer grade as two output channels." Abstract see "aspects are directed to a computer implemented method that includes obtaining medical images of a subject, inputting the medical images into a three-dimensional neural network model constructed to produce a voxelwise cancer risk map of lesion occupancy and cancer grade." (Examiner note: localization is inherently down with image segmentation, volumetric masks are used for localization)). receiving medical image data at the system from a plurality of datasets provided via publicly or privately available sources; (Para. 41 see "The model training stage 110 builds and trains one or more models 130a-130n." Para. 42 see "To train a model 130 in this example, samples 135 are generated by acquiring digital images, splitting the images into a subset of images 135a for training (e.g., 90%) and a subset of images 135b for validation." Para. 68 see "The cohort used in this example is comprised of 973 multiparametric prostate MRI exams from 921 subjects that subsequently underwent MRI/US fusion prostate biopsy of 1440 radiologist-identified MRI targets as well as biopsy of systematic biopsy sites."). training the AI model on the datasets to learn image classification and output (i) an image-level classification function, (ii) an object-level classification function, and (iii) a plurality of image classification weights; (Para. 41 see "The model training stage 110 builds and trains one or more models 130a-130n… The model 130 can also be any other suitable ML model trained in object detection and/or segmentation from images… The computing environment 100 may employ the same type of model or different types of models for segmenting instances of an object of interest." Para. 49 see "The training process includes selecting hyperparameters for the model 130 and performing iterative operations of inputting images from the subset of images 135a into the model 130 to find a set of model parameters (e.g., weights and/or biases) that minimizes the objective function for the model 130." Para. 50 "Each iteration of training can involve finding a set of model parameters for the model 130... so that the value of the objective function using the set of model parameters is smaller than the value of the objective function using a different set of model parameters in a previous iteration." Para. 81 see "the final classification layers can be very shallow (e.g., 2 layers, and a total of 93 parameters for the classification layers)" (This covers the classification weights.) Para. 82 see "region-wise or exam-wise Gleason grade classification loss." (This covers image-level and object-level classification)). training the AI model on the datasets to learn image localization and output an object localization function and a plurality of image localization weights; training the AI model on the datasets to learn image segmentation and output an object segmentation function and a plurality of image segmentation weights; (Para. 46 see "the labeling and training uses lesion locations provided as volumetric masks and multi-level lesion Gleason grade values... the labeling and training uses region locations in the form of volumetric masks and multi-level region Gleason grade values." Para. 51 see "The model 130 is constructed to take as input the subset of images 135a (e.g., a registered three channel 3D MRI input using T2w, DWI, ADC) and for each pixel or voxel the model 130 predicts a cancer grade (e.g., 0-5) to produce voxel-wise maps of lesion occupancy (semantic segmentation) and cancer grade as two output channels." Para. 55 see "The model training stage 110 outputs trained models including one or more trained segmentation and map models 145. In some instance, images 125 are obtained by a segmentation and map controller 150 within the identification, classification, and grading stage 115... The segmenting includes: (i) generating, using the segmentation model 145, an estimated segmentation boundary around a region or object of interest; and (ii) outputting, using the segmentation model 145, images 125 with the estimated segmentation boundary around the region or object of interest. The images with the estimated segmentation boundary around the region or object of interest may then be cropped to localize the region or object of interest."). integrating each of the image classification weights, the image localization weights, and the image segmentation weights into a single pre-trained AI model; integrating each of the image-level classification function, the object-level classification function, the object localization function and the object segmentation function into the single pre-trained AI model; (Para. 38 see "As shown in FIG. 1, the tumor identification, classification, and grading performed by the computing environment 100 in this example includes several stages: an image acquisition stage 105, a model training stage 110, an identification, classification, and grading stage 115, and an analysis stage 120." Para. 41 see "The model training stage 110 builds and trains one or more models 130a-130n." Para. 46 see "the labeling and training uses lesion locations provided as volumetric masks and multi-level lesion Gleason grade values... the labeling and training uses region locations in the form of volumetric masks and multi-level region Gleason grade values." Para. 51 see "The model 130 is constructed to take as input the subset of images 135a (e.g., a registered three channel 3D MRI input using T2w, DWI, ADC) and for each pixel or voxel the model 130 predicts a cancer grade (e.g., 0-5) to produce voxel-wise maps of lesion occupancy (semantic segmentation) and cancer grade as two output channels." Abstract see "aspects are directed to a computer implemented method that includes obtaining medical images of a subject, inputting the medical images into a three-dimensional neural network model constructed to produce a voxelwise cancer risk map of lesion occupancy and cancer grade." (Examiner note: The trained model integrates the weights and the functions because the model IS the function and the weights are contained within the model.)). and outputting the pre-trained AI model for use with medical image analysis. (Abstract see "The present disclosure relates to techniques for non-invasive tumor identification, classification, and grading using mixed exam-, region-, and voxel-wise supervision." Para. 55 see "The model training stage 110 outputs trained models including one or more trained segmentation and map models 145. In some instance, images 125 are obtained by a segmentation and map controller 150 within the identification, classification, and grading stage 115. The images 125 depict an object of interest. In certain instances, the images are MRI images (e.g., ADC and DWI registered to T2WI) of a sub-volume of a full volume scan of a subject. Optionally, the segmentation and map controller 150 includes processes for preprocess segmenting, using a segmentation model 145, the object of interest (e.g., the prostate gland) within the image(s) 125. The segmenting includes: (i) generating, using the segmentation model 145, an estimated segmentation boundary around a region or object of interest; and (ii) outputting, using the segmentation model 145, images 125 with the estimated segmentation boundary around the region or object of interest. The images with the estimated segmentation boundary around the region or object of interest may then be cropped to localize the region or object of interest.").
Regarding claim 2, Rajagopal teaches The system of claim 1. wherein receiving the medical image data at the system further comprises receiving a plurality of private datasets provided via non-public sources. (Para. 68 see "The cohort used in this example is comprised of 973 multiparametric prostate MRI exams from 921 subjects that subsequently underwent MRI/US fusion prostate biopsy of 1440 radiologist-identified MRI targets as well as biopsy of systematic biopsy sites." (Examiner note: Medical records are inherently private.)).
Regarding claim 4, Rajagopal teaches The system of claim 1. wherein training the AI model on the datasets comprises executing supervised learning operations on the datasets via the AI model. (Para. 50 see "Each iteration of training can involve finding a set of model parameters for the model 130... the objective function (Equation 4 described in detail with respect to Section IV) is constructed combining the strongly supervised loss for regression in lesions that compares cancer detections at each voxel (Equation 1 described in detail with respect to Section IV), with weakly supervised loss for regression in regions that approximates and penalizes the max peak of a region (Equation 3 described in detail with respect to Section IV), averaging these losses over each region and observed grade group, and adding an additional semantic segmentation (DICE) loss. In certain instances, additional objective functions may be added even if they do not provide direct strong or weak supervision to the lesion occupancy detection or cancer grading in voxels or regions, but instead provide indirect supervision to other desirable properties of these 3D maps (e.g. smooth spatial distribution, or sharp histogram distribution) that may improve the overall interpretation of the maps or derivative classifications.").
Regarding claim 5, Rajagopal teaches The system of claim 1. wherein training the AI model on the datasets comprises executing deep learning operations on the datasets via the AI model. (Para. 38 see "FIG. 1 illustrates an example computing environment 100 (i.e., a data processing system) for tumor identification, classification, and grading using deep neural networks according to various embodiments. As shown in FIG. 1, the tumor identification, classification, and grading performed by the computing environment 100 in this example includes several stages: an image acquisition stage 105, a model training stage 110, an identification, classification, and grading stage 115, and an analysis stage 120." Para. 41 see "The model training stage 110 builds and trains one or more models 130a-130n (‘n’ represents any natural number)(which may be referred to herein individually as a model 130 or collectively as the models 130) to be used by the other stages. The model 130 can be a machine-learning (“ML”) model, such as a convolutional neural network (“CNN”), e.g. an inception neural network, a residual neural network (“Resnet”), a U-Net, a V-Net, a single shot multibox detector (“SSD”) network, or a recurrent neural network (“RNN”), e.g., long short-term memory (“LSTM”) models or gated recurrent units (“GRUs”) models, or any combination thereof." Para. 75 see "Note here that penalization of the maximum of a group of voxels max(x) is considered a differentiable function for deep learning, when the maximum value of the vector x is selected for regression at run-time.").
Regarding claim 6, Rajagopal teaches The system of claim 1. wherein training the AI model on the datasets comprises training generic source models having strong generalizability and transferability to yield application-specific target models having superior task performance in the target task. (Para. 54 see "although the training mechanisms described herein focus on training anew model 130. These training mechanisms can also be utilized to fine tune existing models 130 trained from other datasets. For example, in some instances, a model 130 might have been pre-trained using images of other objects or biological structures or from sections from other subjects or studies (e.g., human trials or murine experiments). In those cases, the models 130 can be used for transfer learning and retrained/validated using the images 135." Para. 93 see "the network trained with only strong supervision does a non-trivial job in grading some sextants, indicating some level of generalization in detecting cancerous manifestations in lesion and gland tissue.").
Regarding claim 7, Rajagopal teaches The system of claim 1. wherein training the AI model on the datasets comprises training the AI model to generate as its output, one or more of: a prediction of disease in a medical image; a prediction of no disease in a medical image; an image-level label not present in the source image; an organ or lesion marker not present in the source image; an organ or lesion bounding box not present in the source image; and an organ or lesion mask not present in the source image. (Para. 69 see "Data were extracted from both pathology and radiology reports. Biopsy pathology reports were split into individual results. Any result containing “MRI” was identified and Gleason score extracted by isolating numbers preceding and following a “+” character, which was converted to Gleason grade. Similarly, any result starting with text corresponding to a systematic/sextant biopsy site (left/right apex/mid/base/anterior) was identified, Gleason score extracted and converted to Gleason grade. Mid and anterior grade results were combined with maximum grade. The numbering of MR lesions was confirmed to match between pathology reports and radiology reports by radiologist review. A summary of pathology results is shown in Tables 1 and 2, including a breakdown of lesion PIRADS scores and groundtruth biopsy score percentages for the train and test groups. TABLE-US-00001 TABLE 1 Train distribution of groundtruths over 778 exams: Gleason Grade 0 (benign) 1 2 3 4 5 Lesion 569 (49.4%) 280 (24.3%) 202 (17.6%) 57 (5.0%) 20 (1.7%) 22 (1.9%) Sextant 2898 (62.1%) 1137 (24.4%) 405 (8.7%) 137 (3.1%) 43 (0.9%) 48 (1.0%) Gland lesion 309 (39.1%) 195 (25.0%) 183 (23.5%) 52 (6.7%) 20 (2.6%) 19 (2.4%) maximum Glad overall 110 (14.1%) 247 (31.7%) 263 (33.8%) 95 (12.2%) 31 (4.0%) 32 (4.1%) maximum." (Examiner note: the model determines a grade of cancer.)).
Claim 8 is rejected under the same analysis as claim 1 above.
Claim 9 is rejected under the same analysis as claim 2 above.
Claim 11 is rejected under the same analysis as claim 4 above.
Claim 12 is rejected under the same analysis as claim 5 above.
Claim 13 is rejected under the same analysis as claim 6 above.
Claim 14 is rejected under the same analysis as claim 7 above.
Claim 15 is rejected under the same analysis as claim 1 above.
Claim 16 is rejected under the same analysis as claim 2 above.
Claim 18 is rejected under the same analysis as claim 4 above.
Claim 19 is rejected under the same analysis as claim 5 above.
Claim 20 is rejected under the same analysis as claim 6 above.
Claim 21 is rejected under the same analysis as claim 7 above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3, 10, 17 are rejected under 35 U.S.C. 103 as being unpatentable over Rajagopal et al. (US 20230410301 A1), hereinafter Rajagopal, in view of Li et al. (US 20220165418 A1), hereinafter Li.
Regarding claim 3, Rajagopal teaches The system of claim 1.
Rajagopal does not teach wherein training the AI model on the datasets comprises executing unsupervised learning operations on the datasets via the AI model.
However, Li teaches wherein training the AI model on the datasets comprises executing unsupervised learning operations on the datasets via the AI model. (Para. 80 see "the machine learning algorithm can be provided with non-domain data, which leaves the algorithm to identify hidden structure amongst the cases (referred to as unsupervised learning). Sometimes, unsupervised learning is useful for identifying the features that are most useful for classifying raw data into separate cohorts." Para. 81 see "the machine learning algorithm is selected from the group consisting of a supervised, semi-supervised and unsupervised learning, such as, for example, a support vector machine (SVM), a Naïve Bayes classification, a random forest, an artificial neural network, a decision tree, a K-means, learning vector quantization (LVQ), self-organizing map (SOM), graphical model, regression algorithm (e.g., linear, logistic, multivariate), association rule learning, deep learning, dimensionality reduction and ensemble selection algorithms.").
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Rajagopal to incorporate the teachings of Li to train the AI model using unsupervised learning. Doing so would predictably train a more robust model by allowing the model to identify hidden patterns and train more cost-effectively by not requiring human-guided labeling of data.
Claim 10 is rejected under the same analysis as claim 3 above.
Claim 17 is rejected under the same analysis as claim 3 above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Courtiol et al. (US 20210271847 A1) discloses a method and apparatus of a device to classify and segment medical images.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALEXANDER J VAUGHN whose telephone number is (571) 272-5253. The examiner can normally be reached M-F 8:30-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ANDREW MOYER can be reached on (571) 272-9523. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALEXANDER JOSEPH VAUGHN/Examiner, Art Unit 2675
/EDWARD PARK/Primary Examiner, Art Unit 2675