DETAILED ACTION
Claims 1-20 are pending in this application.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 10/30/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are:
“Access component” in claim 1.
“Curation component” in claims 1, 3, 4, 6, and 8.
“Training component” in claims 1, 6 and 7.
“Cleaning component” in claim 9.
“device” in claims 10, 12, 13, and 15-18.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-2, 6-7, 10-11, 15-16 and 19-20 are rejected under 35 U.S.C. 102(a) as being anticipated by Wang (US 20200167930 A1).
Regarding claim 1 Wang discloses; A system, comprising:
a processor that executes computer-executable components stored in a non-transitory computer-readable memory, wherein the computer-executable components comprise (Wang, [0288] the system has a processor and a memory for storing executable instructions):
an access component that accesses a plurality of medical images and a suite of first deep learning neural networks (Wang, [0014] multiple medical images are sent to a CNN and CRF model (multiple deep learning networks) to be labeled, [0288] the machine learning modules and the images are stored on a memory and accessed by a processor, which is being interpreted by the examiner as analogous to the access component per applicant’s specification paragraph [0005]),
wherein the suite of first deep learning neural networks are pre-trained to perform respective inferencing tasks for inputted images depicting respective anatomies or generated by respective imaging modalities (Wang, [0132] the first set of CNNs are pre-trained as part of the initial learning step, [0069] the system trains the models to generate segmentations of medical images/anatomies);
a curation component that curates the plurality of medical images in preparation for training of a second deep learning neural network (Wang,[0130] a training set is generate to train a pair of networks, [0131] the training network is generated via making adjustments to the initial training data, which per [0013] of the applicant’s specification is analogous to the claimed “curation” of the data),
based on generating for each of the plurality of medical images a respective concatenated embedding that is composed of hidden feature maps extracted from the suite of first deep learning neural networks (Wang, [0130] to generate the training data feature vectors (concatenated embedding from hidden features) are concatenated to form training feature data);
and a training component that trains, after such curation, the second deep learning neural network on at least some of the plurality of medical images (Wang, [0132]-[0134] the first set of neural networks (CRF-Net) are trained, then data is generated which may be used to train the second set of neural networks (P-net)).
Regarding claim 2 Wang discloses; The system of claim 1, wherein the second deep learning neural network joins, after such training, the suite of first deep learning neural networks, such that the second deep learning neural network contributes to concatenated embeddings used to train future deep learning neural networks (Wang, [0134] once trained, the P-net and CRF-net are used to generate data to train another set of networks (R-net)).
Regarding claim 6 Wang discloses; The system of claim 1, wherein the curation component curates the plurality of medical images based on: separating the plurality of medical images into two or more clusters of medical images according to their concatenated embeddings (Wang, [0075] in an example training implementation of the system, the system trains the neural networks using a data set which is separated based on containing only feature brain and placenta images, and then after training a second portion of the medical image data only containing fetal lung images or maternal kidney images may be used to test the mode, [0130]-[0131] a training set of 100k samples was generated to train the models, as well as a normalization of the training data based on the features in the training data (concatenated embeddings), [0132]-[0133] to train the second set of networks a second cluster set of image data, which was a subset of the initial images that had user annotations simulated on them (features/embeddings), was used to train the second set of networks (R-Net and second C-Net));
and forming a training dataset that includes a first percentage of each of the two or more clusters of medical images (Wang, [0141]-[0142] the data was split into multiple sets based on labels/embeddings assigned to each group/cluster of images, where 624 images are used for training the models, 122 images were used for validation, and 179 were used for testing, given that the totals for the whole dataset and for each subset/cluster of the images is given this would be equivalent to a percentage of the set being split based on the embeddings/content/features and being used for each portion of the training and validation of the system),
wherein the training component trains the second deep learning neural network on the training dataset and not on a remainder of the plurality of medical images (Wang, [0141]-[0142] the data was split into multiple sets based on labels/embeddings assigned to each group/cluster of images, where 624 images are used for training the models, 122 images were used for validation, and 179 were used for testing).
Regarding claim 7 Wang discloses; The system of claim 6, wherein the training component validates the second deep learning neural network on the remainder of the plurality of medical images after training (Wang, [0141]-[0142] the data was split into multiple sets based on labels/embeddings assigned to each group/cluster of images, where 624 images are used for training the models, 122 images were used for validation of the second, non-pretrained set of models, and 179 were used for testing).
Regarding claim 10 Wang discloses; A computer-implemented method, comprising:
accessing, by a device operatively coupled to a processor (Wang, [0288] the system has a processor and a memory), a plurality of medical images and a suite of first deep learning neural networks (Wang, [0014] multiple medical images are sent to a CNN and CRF model (multiple deep learning networks) to be labeled, [0288] the machine learning modules and the images are stored on a memory and accessed by a processor),
wherein the suite of first deep learning neural networks are pre-trained to perform respective inferencing tasks for inputted images depicting respective anatomies or generated by respective imaging modalities (Wang, [0132] the first set of CNNs are pre-trained as part of the initial learning step, [0069] the system trains the models to generate segmentations of medical images/anatomies);
curating, by the device, the plurality of medical images in preparation for training of a second deep learning neural network (Wang,[0130] a training set is generate to train a pair of networks, [0131] the training network is generated via making adjustments to the initial training data, which per [0013] of the applicant’s specification is analogous to the claimed “curation” of the data),
based on generating for each of the plurality of medical images a respective concatenated embedding that is composed of hidden feature maps extracted from the suite of first deep learning neural networks (Wang, [0130] to generate the training data feature vectors (concatenated embedding from hidden features) are concatenated to form training feature data);
and training, by the device and after such curation, the second deep learning neural network on at least some of the plurality of medical images (Wang, [0132]-[0134] the first set of neural networks (CRF-Net) are trained, then data is generated which may be used to train the second set of neural networks (P-net)).
Regarding claim 11 Wang discloses; The computer-implemented method of claim 10, wherein the second deep learning neural network joins, after such training, the suite of first deep learning neural networks, such that the second deep learning neural network contributes to concatenated embeddings used to train future deep learning neural networks (Wang, [0134] once trained, the P-net and CRF-net are used to generate data to train another set of networks (R-net)).
Regarding claim 15 Wang discloses; The computer-implemented method of claim 10, wherein the curating comprises: separating, by the device, the plurality of medical images into two or more clusters of medical images according to their concatenated embeddings (Wang, [0075] in an example training implementation of the system, the system trains the neural networks using a data set which is separated based on containing only feature brain and placenta images, and then after training a second portion of the medical image data only containing fetal lung images or maternal kidney images may be used to test the mode, [0130]-[0131] a training set of 100k samples was generated to train the models, as well as a normalization of the training data based on the features in the training data (concatenated embeddings), [0132]-[0133] to train the second set of networks a second cluster set of image data, which was a subset of the initial images that had user annotations simulated on them (features/embeddings), was used to train the second set of networks (R-Net and second C-Net));
and forming, by the device, a training dataset that includes a first percentage of each of the two or more clusters of medical images (Wang, [0141]-[0142] the data was split into multiple sets based on labels/embeddings assigned to each group/cluster of images, where 624 images are used for training the models, 122 images were used for validation, and 179 were used for testing, given that the totals for the whole dataset and for each subset/cluster of the images is given this would be equivalent to a percentage of the set being split based on the embeddings/content/features and being used for each portion of the training and validation of the system),
wherein the device trains the second deep learning neural network on the training dataset and not on a remainder of the plurality of medical images (Wang, [0141]-[0142] the data was split into multiple sets based on labels/embeddings assigned to each group/cluster of images, where 624 images are used for training the models, 122 images were used for validation, and 179 were used for testing).
Regarding claim 16 Wang discloses; The computer-implemented method of claim 15, further comprising: validating, by the device, the second deep learning neural network on the remainder of the plurality of medical images after training (Wang, [0141]-[0142] the data was split into multiple sets based on labels/embeddings assigned to each group/cluster of images, where 624 images are used for training the models, 122 images were used for validation of the second, non-pretrained set of models, and 179 were used for testing).
Regarding claim 19 Wang discloses; A computer program product for facilitating training image curation via hidden feature concatenation, the computer program product comprising a non-transitory computer-readable memory having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to (Wang, [0288] the system has a processor and a memory for storing executable instructions):
access a plurality of medical images and a suite of first deep learning neural networks (Wang, [0014] multiple medical images are sent to a CNN and CRF model (multiple deep learning networks) to be labeled, [0288] the machine learning modules and the images are stored on a memory and accessed by a processor),
wherein the suite of first deep learning neural networks are pre-trained to perform respective inferencing tasks for inputted images depicting respective anatomies or generated by respective imaging modalities (Wang, [0132] the first set of CNNs are pre-trained as part of the initial learning step, [0069] the system trains the models to generate segmentations of medical images/anatomies);
curating, by the device, the plurality of medical images in preparation for training of a second deep learning neural network (Wang,[0130] a training set is generate to train a pair of networks, [0131] the training network is generated via making adjustments to the initial training data, which per [0013] of the applicant’s specification is analogous to the claimed “curation” of the data),
based on generating for each of the plurality of medical images a respective concatenated embedding that is composed of hidden feature maps extracted from the suite of first deep learning neural networks (Wang, [0130] to generate the training data feature vectors (concatenated embedding from hidden features) are concatenated to form training feature data);
and training, by the device and after such curation, the second deep learning neural network on at least some of the plurality of medical images (Wang, [0132]-[0134] the first set of neural networks (CRF-Net) are trained, then data is generated which may be used to train the second set of neural networks (P-net)).
Regarding claim 20 Wang discloses; The computer program product of claim 19, wherein the processor curates the plurality of medical images based on: embeddings (Wang, [0075] in an example training implementation of the system, the system trains the neural networks using a data set which is separated based on containing only feature brain and placenta images, and then after training a second portion of the medical image data only containing fetal lung images or maternal kidney images may be used to test the mode, [0130]-[0131] a training set of 100k samples was generated to train the models, as well as a normalization of the training data based on the features in the training data (concatenated embeddings), [0132]-[0133] to train the second set of networks a second cluster set of image data, which was a subset of the initial images that had user annotations simulated on them (features/embeddings), was used to train the second set of networks (R-Net and second C-Net));
and forming a training dataset that includes a first percentage of each of the two or more clusters of medical images (Wang, [0141]-[0142] the data was split into multiple sets based on labels/embeddings assigned to each group/cluster of images, where 624 images are used for training the models, 122 images were used for validation, and 179 were used for testing, given that the totals for the whole dataset and for each subset/cluster of the images is given this would be equivalent to a percentage of the set being split based on the embeddings/content/features and being used for each portion of the training and validation of the system),
wherein the processor trains the second deep learning neural network on the training dataset and not on a remainder of the plurality of medical images (Wang, [0141]-[0142] the data was split into multiple sets based on labels/embeddings assigned to each group/cluster of images, where 624 images are used for training the models, 122 images were used for validation, and 179 were used for testing).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 3-5 and 12-14 are rejected under 35 U.S.C. 103 as being unpatentable over Wang (US 20200167930 A1) in view of Cassidy,(Cassidy et al, Analysis of the ISIC image datasets: Usage, benchmarks and recommendations, Medical Image Analysis, Volume 75, Jan. 2022, doi: 10.1016).
Regarding claim 3 Wang fails to teach; The system of claim 1, wherein the curation component curates the plurality of medical images based on: identifying two or more medical images having concatenated embeddings that are within a threshold margin of similarity of each other;
and removing all but one of those two or more medical images from the plurality of medical images.
However, in the same field of endeavor, Cassidy teaches; wherein the curation component curates the plurality of medical images based on: identifying two or more medical images having concatenated embeddings that are within a threshold margin of similarity of each other (Cassidy Page 6 Column 1 paragraph 2- Column 2 paragraph 2, images were compared using a cosine similarity comparison between their vectors (embeddings) to determine similarity, where the closer the value is to 1, the more similar the images are, Table 8 shows that multiple images were removed as a result of each different data comparison and duplicate removal strategy);
and removing all but one of those two or more medical images from the plurality of medical images (Cassidy Page 6 Column 1 paragraph 2- Column 2 paragraph 2, images were compared using a cosine similarity comparison between their vectors (embeddings) to determine similarity, where the closer the value is to 1, the more similar the images are, Table 8 shows that multiple images were removed as a result of each different data comparison and duplicate removal strategy, therefore all “duplicate” or highly similar images to one image are deleted, and that original image remains in the dataset).
PNG
media_image1.png
110
344
media_image1.png
Greyscale
(Cassidy, page 6)
The combination of Wang and Cassidy would be obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Wang teaches a method of using multiple deep learning networks to train each other to classify medical images. The motivation to add the duplicate or overly similar data removal features of Cassidy are that removal of excessive duplicates may help provide a more balances and cleaned dataset for training allowing for less bias within the trained model (Cassidy, page 2, column 1-2).
Regarding claim 4 the combination of Wang and Cassidy teaches; The system of claim 1, wherein the curation component curates the plurality of medical images based on: identifying one or more medical images having concatenated embeddings whose mean pairwise similarities with concatenated embeddings of others of the plurality of medical images are below a threshold margin (Cassidy Page 6 Column 1 paragraph 2- Column 2 paragraph 2, images were compared using a cosine similarity comparison (mean pairwise similarity measure) between their vectors (embeddings) to determine similarity, where the closer the value is to 1, the more similar the images are, Table 8 shows that multiple images were removed as a result of each different data comparison and duplicate removal strategy, therefore all “duplicate” or highly similar images to one image are deleted, and that original image remains in the dataset, all similarity values must be equal to or less than one, therefore this would be analogous to the threshold being below a margin);
and removing those one or more medical images from the plurality of medical images (Cassidy Page 6 Column 1 paragraph 2- Column 2 paragraph 2, images were compared using a cosine similarity comparison (mean pairwise similarity measure) between their vectors (embeddings) to determine similarity, where the closer the value is to 1, the more similar the images are, Table 8 shows that multiple images were removed as a result of each different data comparison and duplicate removal strategy).
The combination of Wang and Cassidy would be obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Wang teaches a method of using multiple deep learning networks to train each other to classify medical images. The motivation to add the duplicate or overly similar data removal features of Cassidy are that removal of excessive duplicates may help provide a more balances and cleaned dataset for training allowing for less bias within the trained model (Cassidy, page 2, column 1-2).
Regarding claim 5 the combination of Wang and Cassidy teaches; The system of claim 4, wherein the plurality of medical images respectively correspond to modality classes, anatomy classes, or view classes (Cassidy, page 5, column 1 paragraphs 2-3, the datasets are classified based on type of skin lesion (anatomy) and labeled accordingly), and wherein the mean pairwise similarities are computed on a class-wise basis (Cassidy, Table 9 and Page column 2, paragraphs 2 and 3, both inter- and intra-class similarity comparisons are performed, therefore at least some of the pairwise comparisons are class-wise as well).
The combination of Wang and Cassidy would be obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Wang teaches a method of using multiple deep learning networks to train each other to classify medical images. The motivation to add the duplicate or overly similar data removal features of Cassidy are that removal of excessive duplicates may help provide a more balances and cleaned dataset for training allowing for less bias within the trained model (Cassidy, page 2, column 1-2).
Regarding claim 12 the combination of Wang and Cassidy teaches; The computer-implemented method of claim 10, wherein the curating comprises: identifying, by the device, two or more medical images having concatenated embeddings that are within a threshold margin of similarity of each other (Cassidy Page 6 Column 1 paragraph 2- Column 2 paragraph 2, images were compared using a cosine similarity comparison between their vectors (embeddings) to determine similarity, where the closer the value is to 1, the more similar the images are, Table 8 shows that multiple images were removed as a result of each different data comparison and duplicate removal strategy);
and removing, by the device, all but one of those two or more medical images from the plurality of medical images (Cassidy Page 6 Column 1 paragraph 2- Column 2 paragraph 2, images were compared using a cosine similarity comparison between their vectors (embeddings) to determine similarity, where the closer the value is to 1, the more similar the images are, Table 8 shows that multiple images were removed as a result of each different data comparison and duplicate removal strategy, therefore all “duplicate” or highly similar images to one image are deleted, and that original image remains in the dataset).
The combination of Wang and Cassidy would be obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Wang teaches a method of using multiple deep learning networks to train each other to classify medical images. The motivation to add the duplicate or overly similar data removal features of Cassidy are that removal of excessive duplicates may help provide a more balances and cleaned dataset for training allowing for less bias within the trained model (Cassidy, page 2, column 1-2).
Regarding claim 13 the combination of Wang and Cassidy teaches; The computer-implemented method of claim 10, wherein the curating comprises: identifying, by the device, one or more medical images having concatenated embeddings whose mean pairwise similarities with concatenated embeddings of others of the plurality of medical images are below a threshold margin(Cassidy Page 6 Column 1 paragraph 2- Column 2 paragraph 2, images were compared using a cosine similarity comparison (mean pairwise similarity measure) between their vectors (embeddings) to determine similarity, where the closer the value is to 1, the more similar the images are, Table 8 shows that multiple images were removed as a result of each different data comparison and duplicate removal strategy, therefore all “duplicate” or highly similar images to one image are deleted, and that original image remains in the dataset, all similarity values must be equal to or less than one, therefore this would be analogous to the threshold being below a margin);
and removing, by the device, those one or more medical images from the plurality of medical images (Cassidy Page 6 Column 1 paragraph 2- Column 2 paragraph 2, images were compared using a cosine similarity comparison (mean pairwise similarity measure) between their vectors (embeddings) to determine similarity, where the closer the value is to 1, the more similar the images are, Table 8 shows that multiple images were removed as a result of each different data comparison and duplicate removal strategy).
The combination of Wang and Cassidy would be obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Wang teaches a method of using multiple deep learning networks to train each other to classify medical images. The motivation to add the duplicate or overly similar data removal features of Cassidy are that removal of excessive duplicates may help provide a more balances and cleaned dataset for training allowing for less bias within the trained model (Cassidy, page 2, column 1-2).
Regarding claim 14 the combination of Wang and Cassidy teaches; The computer-implemented method of claim 13, wherein the plurality of medical images respectively correspond to modality classes, anatomy classes, or view classes, and wherein the mean pairwise similarities are computed on a class-wise basis (Cassidy, page 5, column 1 paragraphs 2-3, the datasets are classified based on type of skin lesion (anatomy) and labeled accordingly), and wherein the mean pairwise similarities are computed on a class-wise basis (Cassidy, Table 9 and Page column 2, paragraphs 2 and 3, both inter- and intra-class similarity comparisons are performed, therefore at least some of the pairwise comparisons are class-wise as well).
The combination of Wang and Cassidy would be obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Wang teaches a method of using multiple deep learning networks to train each other to classify medical images. The motivation to add the duplicate or overly similar data removal features of Cassidy are that removal of excessive duplicates may help provide a more balances and cleaned dataset for training allowing for less bias within the trained model (Cassidy, page 2, column 1-2).
Claims 8 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Wang (US 20200167930 A1) in view of Song (US 2020065656 A1).
Regarding claim 8 Wang discloses; The system of claim 1, wherein a first medical image in the plurality of medical images corresponds to a first ground-truth annotation (Wang, [0157] one of the testing subsets of data obtained had ground truth data provided, therefore there would be at least a first image with a ground truth),
Wang fails to teach;
wherein two or more second medical images in the plurality of medical images lack ground-truth annotations,
and wherein the curation component curates the plurality of medical images based on: identifying which of the two or more second medical images have concatenated embeddings that are within a threshold margin of, or that are in a same cluster as, that of the first medical image;
and assigning the first ground-truth annotation to such identified ones of the two or more second medical images.
However, in the same field of endeavor of machine learning, Song teaches; wherein two or more second medical images in the plurality of medical images lack ground-truth annotations (Song, [0021] multiple training inputs are assigned to a set of clusters based on their embeddings, but do not have a ground truth because a ground truth is assigned to each cluster/set of images based upon embeddings in the images),
and wherein the curation component curates the plurality of medical images based on: identifying which of the two or more second medical images have concatenated embeddings that are within a threshold margin of, or that are in a same cluster as, that of the first medical image (Song, [0021] multiple training inputs are assigned to a set of clusters based on their embeddings, but do not have a ground truth because a ground truth is assigned to each cluster/set of images based upon embeddings in the images, [0022]-[0023] the system has a training item which is what each training image is compared with to determine if it belongs in that cluster and can be assigned to that ground truth);
and assigning the first ground-truth annotation to such identified ones of the two or more second medical images (Song, [0022]-[0023] the system has a training item which is what each training image is compared with to determine if it belongs in that cluster and can be assigned to that ground truth).
The combination of Wang and Song would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Wang teaches a method of training multiple neural networks using medical images. The motivation for the addition of the ground truth classification of Song is that it would be beneficial for quickly generating ground truth data for images without the need for manual classification of the data in the event ground truth data was not provided. Further, this method would be obvious for one of ordinary skill in the art to apply to medical image data even though Song does not explicitly claim the use medical images because large amounts of data are required to train and validate medical image machine learning applications, and this method of Song would allow for faster generation of ground truth data from raw images, allowing for a larger training/validation set. (Wang, [0002]-[0006] and Song, [0001]-[0008])
Regarding claim 17 the combination of Wang and Song teaches; The computer-implemented method of claim 10, wherein a first medical image in the plurality of medical images corresponds to a first ground-truth annotation (Wang, [0157] one of the testing subsets of data obtained had ground truth data provided, therefore there would be at least a first image with a ground truth),
wherein two or more second medical images in the plurality of medical images lack ground-truth annotations, and wherein the curating comprises (Song, [0021] multiple training inputs are assigned to a set of clusters based on their embeddings, but do not have a ground truth because a ground truth is assigned to each cluster/set of images based upon embeddings in the images):
identifying, by the device, which of the two or more second medical images have concatenated embeddings that are within a threshold margin of, or that are in a same cluster as, that of the first medical image (Song, [0021] multiple training inputs are assigned to a set of clusters based on their embeddings, but do not have a ground truth because a ground truth is assigned to each cluster/set of images based upon embeddings in the images, [0022]-[0023] the system has a training item which is what each training image is compared with to determine if it belongs in that cluster and can be assigned to that ground truth);
and assigning, by the device, the first ground-truth annotation to such identified ones of the two or more second medical images (Song, [0022]-[0023] the system has a training item which is what each training image is compared with to determine if it belongs in that cluster and can be assigned to that ground truth).
The combination of Wang and Song would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Wang teaches a method of training multiple neural networks using medical images. The motivation for the addition of the ground truth classification of Song is that it would be beneficial for quickly generating ground truth data for images without the need for manual classification of the data in the event ground truth data was not provided. Further, this method would be obvious for one of ordinary skill in the art to apply to medical image data even though Song does not explicitly claim the use medical images because large amounts of data are required to train and validate medical image machine learning applications, and this method of Song would allow for faster generation of ground truth data from raw images, allowing for a larger training/validation set. (Wang, [0002]-[0006] and Song, [0001]-[0008])
Claims 9 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Wang (US 20200167930 A1) in view of Lyman (US 20220180986 A1).
Regarding claim 9 Wang fails to disclose; The system of claim 1, wherein the computer-executable components further comprise: a cleaning component that removes, via execution of a third deep learning neural network and prior to curation of the plurality of medical images, text, legends, or logos that are superimposed over respective ones of the plurality of medical images.
However, in the same field of endeavor of machine learning for medical image applications, Lyman teaches; a cleaning component that removes, via execution of a third deep learning neural network and prior to curation of the plurality of medical images, text, legends, or logos that are superimposed over respective ones of the plurality of medical images (Lyman, [0174] a de-identification function (cleaning component) is used to de-identify the medical images, where this function may use trained medical scan analysis functions or other types of trained models (third deep learning network), Figure 1 shows the plurality of different machine learning models, therefore there would be at least a first, a second and a third model,[0149] the De-identification function may remove text containing patient confidential information).
The combination of Wang and Lyman would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Wang teaches a method of generating and curating training data and training multiple deep neural networks. The motivation for the addition of the text removal and de-identification method of Lyman is that it allows the training data or medical image data to be de-identified to comply with proper treatment of patient health information. (Lyman, [0149])
Regarding claim 18 the combination of Wang and Lyman teaches; The computer-implemented method of claim 10, further comprising: removing, by the device, via execution of a third deep learning neural network, and prior to curation of the plurality of medical images, text, legends, or logos that are superimposed over respective ones of the plurality of medical images (Lyman, [0174] a de-identification function (cleaning component) is used to de-identify the medical images, where this function may use trained medical scan analysis functions or other types of trained models (third deep learning network), Figure 1 shows the plurality of different machine learning models, therefore there would be at least a first, a second and a third model,[0149] the De-identification function may remove text containing patient confidential information).
The combination of Wang and Lyman would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Wang teaches a method of generating and curating training data and training multiple deep neural networks. The motivation for the addition of the text removal and de-identification method of Lyman is that it allows the training data or medical image data to be de-identified to comply with proper treatment of patient health information. (Lyman, [0149])
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. In addition to the art cited below, for full listing of analogous prior as determined by the examiner please see the attached PTO-892 Notice of References Cited form.
Golden, (WO 2019103912 A1), teaches a method of training a system to identify and annotate medical images using multiple machine learning models.
Zhou, (US 20220328189 A1, teaches a method and system of using multiple machine learning models to annotate and generate diagnoses from medical image data and embedded features.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JORDAN M ELLIOTT whose telephone number is (703)756-5463. The examiner can normally be reached M-F 8AM-5PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.M.E./Examiner, Art Unit 2666 /Molly Wilburn/Primary Examiner, Art Unit 2666
Prosecution Insights