Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Notice to Applicants
This communication is in response to the Application filed on 08/15/2024.
Claims 1-20 are pending.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “the first machine learning-based system generating, based on the received medical image data, a plurality of image embedding vectors …” and “generating, by the second machine learning-based system, a feature vector …” in claim 1, 16 and 20, “a natural language processing system to generate a set of data representing findings …” in claim 11 and “a third machine learning-based system to generate natural language text ...” in claim 12.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Specification
The disclosure is objected to because of the following informalities:
[0037] and [0041]: “fourth machine learning-based system” should be “third machine learning-based system.” Please see claim 12-14.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 4-6, 10-11 and 19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 4 recites the limitation “the medical image feature” (line 2). It is unclear if “the medical image feature” is referring back to “the respective medical image feature” (at Claim 1 in line 9), “the indicated first medical image feature” (at Claim 3 in line 2) or something else. Clarification/explanation is required.
Claim 10 recites the limitation “the corresponding trial feature vectors” (line 14). There is insufficient antecedent basis for this limitation in the claim. It is unclear if “the corresponding trial feature vectors” is referring back to “a trial feature vector” (line 6) or something else. Clarification/explanation is required.
With respect to claim 19, arguments analogous to those presented for claim 4, are applicable.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim 1, 3, 4, 7-9, 12, 13, 15, 16, 18, 19 and 20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Zhou et al. (U.S. Publication No. 2022/0130499) (hereafter, "Zhou").
Regarding claim 1, Zhou teaches a computer implemented method for processing medical image data, the method comprising ([0020] The herein described VQA system can combine data presented via multiple modalities into a joint representation of all of the data. The joint representation is based on a mapping between the data derived from medical records and data derived from medical images; [0021] Referring to FIGS. 1, 2, 3, and 4, a multi-phase process of training a VQA system 100 is described): receiving medical image data representing a medical image at a first machine learning-based system; the first machine learning-based system ([0021] The image embedder unit 102 is operable to receive an image 202 as a training instance and employ a model to identify objects in the image 202) generating, based on the received medical image data, a plurality of image embedding vectors corresponding to a respective plurality of medical image features ([0037] The system 100 further includes an image embedder unit 102 for analyzing an image and extracting features from objects in the image; [0038] The image embedder unit 102 can receive an image 110 from and extract domain-specific features describing objects in the image 110. The image embedder unit 102 can further employ a computer vision model to detect and label domain specific-objects in the image 110. The image embedder unit 102 can receive the image 110 as an input and predict a class for each object contained in the image 110. The image embedder unit 102 can further label each object class. The image embedder unit 102 can generate a user image embedding vector to represent the extracted features and identified object classes; [0025] The image embedding vectors 204 and the text embedding vectors 208 are high-dimensional vectors that can be translated in a low-dimensional embedding space), each of the plurality of image embedding vectors relating to a different respective medical image feature and ([0038] The image embedder unit 102 can … extract domain-specific features describing objects in the image 110 … The image embedder unit 102 can receive the image 110 as an input and predict a class for each object contained in the image 110) comprising medical image feature data indicative of presence or absence of the respective medical image feature at each of a plurality of locations in the medical image ([0021] The image embedder unit 102 can employ a model that executes computer vision techniques on the image 202 for object detection. Object detection includes both image classification and object localization. Image classification includes predicting a class of one or more objects in the image 202. To perform image classification, the image embedder unit 102 receives the image 202 as an input and outputs a class label in the form of one or more integer values mapped to class values. Object localization includes identifying a location of the one or more identified objects in the image 202. To perform image classification, the image embedder unit 102 receives the image 202 as an input and outputs a class label in the form of one or more integer values mapped to class values. Object localization includes identifying a location of the one or more identified objects in the image 202. To perform object localization, the image embedder unit 102 can process the received image 202 and output one or more bounding boxes, which define a spatial relationship of the objects in the image 202); receiving, at a second machine learning-based system, an indication of a first medical image feature included in the medical image and ([0037] The system 100 includes a text embedder unit 104 for receiving a natural language text, for example, a search query from a user, and semantically analyzing the text; [0039] The text embedder unit 104 is operable to receive a query 112 in electronic format from a user computing device 114. The query 112 can be a question from a user requesting information about some aspect of the image, for example, “What is the most alarming part in this x-ray scan?” The text embedder unit 104 can apply a model that uses natural language processing (NLP) techniques to analyze to query 112 and determine a context of the query 112; [0023] The text embedder unit 104 can receive a text document 206 … The text embedder unit 104 can be trained to semantically analyze text from a particular domain (e.g., medical domain) by adjusting the weights and biases of the neural network; [0040] & [0046]) generating, by the second machine learning-based system, a feature vector based on the indication; and ([0023] The text embedder unit 104 can be trained to semantically analyze text from a particular domain (e.g., medical domain) by adjusting the weights and biases of the neural network. The text embedder unit 104 further uses the values of the weights and biases to generate a text embedding vector 208 to represent the semantic meaning of the text document 206; [0040] Upon determining a context, the text embedder unit 104 can generate a user text embedding vector. The user text embedding vector is a numeric representation of the respective words and phrases in the query 112 and denotes the query's semantic meaning) performing a comparison of the feature vector with the plurality of image embedding vectors and ([0041] The multi-modal encoder unit 106 is operable to determine a correlation between the user text embedding vector and the user image embedding vector. The multi-modal encoder unit 106 can map tokens described by the user text embedding vector to features in the user image embedding vector; [0028] the multi-modal encoder 212 can receive the image embedding vector 204 and be trained to recognize domain-specific features ... The masked-layer model can receive the context features patches 216 218 surrounding the masked feature 214 as inputs to generate a predicted masked feature 220 as to what the masked feature 214 should be; [0029] the multi-modal encoder 212 can also receive the text embedding vector 208 and be trained to derive a meaning from the vector ... The masked-layer model then uses the context tokens 224 226 surrounding the masked token 222 to generate a predicted masked token 228 as to what the masked token 222 should be; [0030] the multi-modal encoder 212 can be trained to determine whether a relationship exists between the predicted masked feature 220 and the predicted masked token 228; [0032] The multi-modal encoder 212 can employ the image-text matching model to match features from the image embedding vector 204 and tokens from the text embedding vector 208) identifying a first image embedding vector from among the plurality of image embedding vectors based on the comparison ([0048] a multi-modal encoder unit 106 can receive the image embedding vector 204 and the text embedding vector 208 ... the multi-modal encoder unit 106 can be implemented by a neural network that executes a model. During a training phase, the weights and biases of the neural network can be adjusted to cause the model to determine a correlation between an object in the image 202 and a natural language description in the text document 206. The multi-modal encoder unit 106 can write data structure to the image embedding vector 204 and the text embedding vector 208 to generate a mapping between the features of the image 202 and the respective portions of the text document 206 that describe the features. The data structure can be, for example, a mapping. For example, the multi-modal encoder unit 106 can write a function that associates a first portion of the image embedding vector 204 to a portion of the text embedding vector 208).
Regarding claim 3, Zhou teaches all the limitations of claim 1 above. Zhou teaches further comprising determining location data representing a location, in the medical image, of the indicated first medical image feature in the medical image data, based on the identified first image embedding vector ([0049] an answer unit 108 can receive an image 110 and a query 112 related to the image ... The answer unit 108 can be trained to classify objects in the image 110, and generate a user image embedding vector. The answer unit 108 can further be trained to semantically analyze the query 112 and determine which object(s) in the image 110 is the query 112 referring to. The answer unit 108 can further generate a user text embedding vector. The answer unit 108 can map the user text embedding vector to the object being referred to in the image 110. Based on the mapping, the answer unit 108 can generate a joint user text-image representation; [0003] extracting a domain-specific object feature from a first image data, wherein the feature describes an object in the first image data. A domain-specific semantic meaning of text data is determined; [0039] The query 112 can be a question from a user requesting information about some aspect of the image, for example, “What is the most alarming part in this x-ray scan?”. Determining "which object" or mapping text vectors to an object in an image inherently requires identifying its spatial location data within that image).
Regarding claim 4, Zhou teaches all the limitations of claim 3 above. Zhou teaches wherein the medical image feature data represents a segmentation map of the medical image feature, and ([0030] The image-text matching model can determine a semantic relationship between an object described in the image embedding vector 204 and the words described in the text embedding vector 208; [0031] The multi-modal encoder 212 can employ an attention mechanism that allows the encoder to have the ability to focus on a subset of tokens (or features). The attention module mechanism can be implemented on a two-dimensional convolutional layer of a neural network, and include a sigmoid function to generate a mask of the feature map of the embedding space. The attention mechanism can receive an a×b×c feature map as an input and outputs a 1×b×c as an output attention map; [0042] The answer unit 108 can generate an answer to the query 112. The answer unit 108 generates an answer token by token based at least in part on the user joint representation and the image-text representation 500 … the answer unit 108 can alter the image 110 to highlight a target object of the query 112) the determined location data comprises the segmentation map ([0041] The multi-modal encoder unit 106 is operable to determine a correlation between the user text embedding vector and the user image embedding vector. The multi-modal encoder unit 106 can map tokens described by the user text embedding vector to features in the user image embedding vector; [0031] The multi-modal encoder 212 can employ an attention mechanism that allows the encoder to have the ability to focus on a subset of tokens (or features). The attention module mechanism can be implemented on a two-dimensional convolutional layer of a neural network, and include a sigmoid function to generate a mask of the feature map of the embedding space. The attention mechanism can receive an a×b×c feature map as an input and outputs a 1×b×c as an output attention map) represented by the identified first image embedding vector ([0031] The attention module mechanism can be implemented on a two-dimensional convolutional layer of a neural network, and include a sigmoid function to generate a mask of the feature map of the embedding space. The attention mechanism can receive an a×b×c feature map as an input and outputs a 1×b×c as an output attention map; [0048] the multi-modal encoder unit 106 can write a function that associates a first portion of the image embedding vector 204 to a portion of the text embedding vector 208).
Regarding claim 7, Zhou teaches all the limitations of claim 1 above. Zhou teaches wherein the indication received at the second machine learning-based system comprises area data ([0026] The image embedder unit 102 and the text embedder unit 104 can transmit the image embedding vector 204 and the text embedding vector 208 to the second module 210, which includes a multi-modal encoder 212; [0027] The second phase is performed by the second module 210 which learns features found the image embedding vector 204 that relate to features in the text embedding vector 208, and vice-versa. The second module 210 includes a multi-modal encoder 212 for encoding a relationship between features from the image embedding vector 204 and the text embedding vector 208; [0023] The text embedder unit 104 can receive a text document 206 in electronic form as a training instance and derive a semantic meaning of the document ... The text document 206 describes at least a portion of the image 202) representing an area of the first medical image feature in the medical image, and ([0021] The image embedder unit 102 is operable to receive an image 202 as a training instance and employ a model to identify objects in the image 202. The image 202 can be, for example, an x-ray image, a CT scan; [0022] the image embedder unit 102 employs a trained artificial neural network to execute the model, for example, a region-based convolutional neural network (R-CNN) ... for image analysis. The R-CNN generally operates in three phases. First, the R-CNN analyzes the image 202, extracts independent regions in the image 202, and delineates the regions as candidate bounding boxes. Second, the R-CNN extracts features, for example, using a deep convolutional neural network, from each region. Third, a classifier, for example, a support vector machine (SVM), is used to analyze the features and predict a class for one or more objects in a region; [0027]) the comparison of the feature vector with the plurality of medical image embedding vectors is performed ([0026] The image embedder unit 102 and the text embedder unit 104 can transmit the image embedding vector 204 and the text embedding vector 208 to the second module 210, which includes a multi-modal encoder 212; [0027] The second phase is performed by the second module 210 which learns features found the image embedding vector 204 that relate to features in the text embedding vector 208, and vice-versa. The second module 210 includes a multi-modal encoder 212 for encoding a relationship between features from the image embedding vector 204 and the text embedding vector 208; [0032] The multi-modal encoder 212 can employ the image-text matching model to match features from the image embedding vector 204 and tokens from the text embedding vector 208) at least partly based on the area data ([0022] the R-CNN analyzes the image 202, extracts independent regions in the image 202, and delineates the regions as candidate bounding boxes. Second, the R-CNN extracts features, for example, using a deep convolutional neural network, from each region. Third, a classifier, for example, a support vector machine (SVM), is used to analyze the features and predict a class for one or more objects in a region; [0028] the multi-modal encoder 212 can receive the image embedding vector 204 and be trained to recognize domain-specific features. Various methods can be used to recognize the features, for example, the multi-modal encoder 212 can employ masked feature model ... the masked feature model is trained to recognize features of a particular domain, for example, the healthcare domain).
Regarding claim 8, Zhou teaches all the limitations of claim 1 above. Zhou teaches wherein the indication received at the second machine learning-based system comprises data representing a text prompt indicating the first medical image feature ([0037] The system 100 includes a text embedder unit 104 for receiving a natural language text, for example, a search query from a user, and semantically analyzing the text; [0039] The text embedder unit 104 is operable to receive a query 112 in electronic format from a user computing device 114. The query 112 can be a question from a user requesting information about some aspect of the image, for example, “What is the most alarming part in this x-ray scan?”; [0023] The text embedder unit 104 can receive a text document 206 … The text embedder unit 104 can be trained to semantically analyze text from a particular domain (e.g., medical domain) by adjusting the weights and biases of the neural network).
Regarding claim 9, Zhou teaches all the limitations of claim 1 above. Zhou teaches further comprising performing a training process to train the first machine learning-based system and the second machine learning-based system ([0021] a first phase of the training is illustrated. The first phase is executed on a first module 200, which includes an image embedder unit 102 and a text embedder unit 104. [0022] the image embedder unit 102 employs a trained artificial neural network to execute the model, for example, a region-based convolutional neural network (R-CNN), or other neural network appropriate for image analysis; [0023] The text embedder unit 104 can be trained to semantically analyze text from a particular domain (e.g., medical domain) by adjusting the weights and biases of the neural network).
Regarding claim 12, Zhou teaches all the limitations of claim 1 above. Zhou teaches further comprising inputting at least one of the plurality of image embedding vectors ([0035] The multi-modal encoder 212 can transmit the image-text representation 500 to an answer decoder 502; [0033] The multi-modal encoder 212 can generate an image-text representation 500 in the form of a high dimensional vector. The image-text representation 500 is based on a matching of a token (or set of tokens) and an image feature (or set of image features). The matching features from the image embedding vector 204 are mapped to matching tokens from text embedding vector 208; [0049] an answer unit 108 can receive an image 110 and a query 112 related to the image ... the answer unit 108 can be implemented by a neural network that executes a model. The answer unit 108 can be trained to classify objects in the image 110, and generate a user image embedding vector. The answer unit 108 can further be trained to semantically analyze the query 112 and determine which object(s) in the image 110 is the query 112 referring to) to a third machine learning-based system to generate natural language text describing a finding relating to the medical image data ([0035] The answer decoder 502 can receive a training query 112 and an image 110 as inputs and generate an answer prediction 504. The answer decoder 502 can be implemented by a neural network. The answer decoder 502 can further be in the form of a sequential generating model, such a long short-term memory (LTSM) ... The answer prediction 504 can be retrieved from a database 116 and provided in natural language; [0021] The image 202 can be, for example, an x-ray image; [0050] the answer unit 108 can determine whether the joint image-text representation 500 references an answer to the query 112 ... both the joint user image-text representation and the joint image-text representation 500 can be in the form of respective vectors that relate to a semantic meaning of each ... If the joint user image-text representation and the joint image-text representation 500 are within a threshold distance, the answer unit 108 can extract a natural language answer from the textual portion of the joint image-text representation 500).
Regarding claim 13, Zhou teaches all the limitations of claim 12 above. Zhou teaches further comprising performing a training method to train the third machine learning-based system ([0035] The third phase of the pre-training includes generating answers to queries. The multi-modal encoder 212 can transmit the image-text representation 500 to an answer decoder 502. The answer decoder 502 can receive a training query 112 and an image 110 as inputs and generate an answer prediction 504; [0032] The multi-modal encoder 212 can employ a classifier that is trained to determine whether a text matches an image or an object in an image; [0049] The answer unit 108 can be trained to classify objects in the image 110, and generate a user image embedding vector. The answer unit 108 can further be trained to semantically analyze the query 112 and determine which object(s) in the image 110 is the query 112 referring to).
Regarding claim 15, Zhou teaches all the limitations of claim 1 above. Zhou teaches wherein at least one of the plurality of medical image features comprises a medical abnormality ([0043] a query 112 may be targeted to an image of a liver suffering from hepatic encephalopathy (HE). HE can affect the functioning of the nervous system and the brain. The answer unit 108 can be trained to recognize related effects of a condition. In this situation, the answer unit 108 can determine whether either a nervous system is in the image 110. If so, the answer unit 108 can alter the nervous system image for highlighting purposes. The highlighting can be distinct from the liver highlighting. If the nervous system is not detected in the image 110, the answer unit 108 can retrieve an image of a nervous system and provide an image to the user. The retrieved nervous system image can include the effects of the HE).
With respect to claim 16, arguments analogous to those presented for claim 1, are applicable.
With respect to claim 18, arguments analogous to those presented for claim 3, are applicable.
With respect to claim 19, arguments analogous to those presented for claim 4, are applicable.
With respect to claim 20, arguments analogous to those presented for claim 1, are applicable.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 2 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Zhou et al. (U.S. Publication No. 2022/0130499) (hereafter, "Zhou") in view of Wang (NPL – “Unified medical image-text-label contrastive learning with continuous prompt”).
Regarding claim 2, Zhou teaches all the limitations of claim 1 above. Zhou does not expressly teach wherein the first image embedding vector is identified as having a highest degree of similarity with the feature vector from among the plurality of image embedding vectors.
However, Wang teaches wherein the first image embedding vector is identified as having a highest degree of similarity with the feature vector from among the plurality of image embedding vectors (page 8, line 2-4, we computed several Image-Text pairs with the largest similarity scores by encoding the images and corresponding sentence embeddings; page 5, line 15-20, The predicted similar score is also obtained by L2 normalize:
PNG
media_image1.png
32
112
media_image1.png
Greyscale
(3) where vi and tj represent the image embedding and text embedding.
PNG
media_image2.png
60
198
media_image2.png
Greyscale
(4) s indicates the medical semantic similarity).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of Zhou to incorporate the step/system of selecting the image-text pairs with the largest similarity scores using normalized vectors taught by Wang.
The suggestion/motivation for doing so would have been to improve the accuracy for cross-modal retrieval (page 8, line 2-8, we computed several Image-Text pairs with the largest similarity scores by encoding the images and corresponding sentence embeddings. We then evaluated our model’s accuracy using Precision@K to separately compute the accuracy in top K retrieval reports/sentences. The specific results are presented in Table3, which demonstrate that our Image-Text-Label Pre-training model achieved optimal results on the metrics of multiple sampled samples). Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predicted results. Therefore, it would have been obvious to combine Zhou and Wang to obtain the invention as specified in claim 2.
With respect to claim 17, arguments analogous to those presented for claim 2, are applicable.
Claim 5 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Zhou et al. (U.S. Publication No. 2022/0130499) (hereafter, "Zhou") in view of Lyman et al. (U.S. Publication No. 2024/0161035) (hereafter, "Lyman").
Regarding claim 5, Zhou teaches all the limitations of claim 4 above. Zhou does not expressly teach wherein the segmentation map indicates probabilities that the respective medical image feature is present at respective locations in the medical image.
However, Lyman teaches wherein the segmentation map indicates probabilities ([0163] The inferred output vector of the inference data 1370 can include a plurality of abnormality probabilities mapped to a pixel location of each of a plurality of cross-sectional image slices of the new medical scan. For example, the inferred output vector can indicate a set of probability matrices 1371, where each matrix in the set corresponds to one of the plurality of image slices of the medical scan … where each cell of each matrix corresponds to a pixel of the corresponding image slice, whose value is the abnormality probability of the corresponding pixel) that the respective medical image feature is present ([0164] A detection step 1372 can include determining if an abnormality is present in the medical scan based on the plurality of abnormality probabilities. Determining if an abnormality is present can include, for example, determining that a cluster of pixels in the same region of the medical scan correspond to high abnormality probabilities ... Determining if an abnormality is present can also include calculating a confidence score based on the abnormality probabilities and/or other data corresponding to the medical scan such as patient history data. The location of the detected abnormality can be determined in the detection step 1372 based on the location of the pixels with the high abnormality probabilities. The detection step can further include determining an abnormality region 1373, such as a two-dimensional subregion on one or more image slices that includes some or all of the abnormality) at respective locations in the medical image ([0163] The inferred output vector of the inference data 1370 can include a plurality of abnormality probabilities mapped to a pixel location of each of a plurality of cross-sectional image slices of the new medical scan).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of Zhou to incorporate the step/system of mapping an output vector containing abnormality probabilities to a pixel location across image slices taught by Lyman.
The suggestion/motivation for doing so would have been to reduce the prediction errors in abnormality location and classification data ([0157] The output feature vector can also include multiple values which can include abnormality location and/or classification data, diagnosis data, or other output; [0158] the learning algorithm 1350 utilized in conjunction with a neural network model can include determining the model parameter data 1355 corresponding to the neural network model, for example, by populating the weight vector with optimal weights that best reduce output error; [0159] This process can continue to repeat until the output error converges, the output error is within a certain error threshold, or another criterion is reached to determine the most recently updated weight vector and/or other model parameter data 1355 is optimal or otherwise determined for selection). Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predicted results. Therefore, it would have been obvious to combine Zhou and Lyman to obtain the invention as specified in claim 5.
Regarding claim 6, Zhou teaches all the limitations of claim 4 above. Zhou does not expressly teach further comprising: determining whether the medical image represents a medical abnormality, and in response to determining that the medical image represents a medical abnormality, displaying the segmentation map on a display device.
However, Lyman teaches further comprising: determining whether the medical image represents a medical abnormality ([0348] FIGS. 14A-14C present an embodiment of a multi-label medical scan analysis system 5002. The multi-label medical scan analysis system can be operable to train a multi-label model, and/or can utilize the multi-label model to generate inference data for new medical scans, indicating probabilities that each of a set of abnormality classes are present in the medical scan), and in response to determining that the medical image represents a medical abnormality, displaying the segmentation map on a display device ([0348] Heat maps for each of the set of abnormality can be generated based on probability matrices for display to via a display device; [0363] The probability matrix data generated as output of the inference function 5020 can be transmitted to the client device 120 for display via a display device ... the probability matrix data can be transmitted for use by another subsystem 101 and/or can be transmitted to the medical scan database to be mapped to the corresponding medical scan … the probability matrix data can be utilized by the multi-label medical scan analysis system 5002 to generate saliency maps, to generate region of interest data, to generate global probabilities for each class as illustrated in FIG. 14B, and/or to generate heat map visualization data as illustrated in FIG. 14C).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of Zhou to incorporate the step/system of processing medical scans, computing probabilities for various abnormality classes, and using probability matrices to generate heat maps or visual data on a display device taught by Lyman.
Motivation for this combination has been stated in claim 5.
Claim 10 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Zhou et al. (U.S. Publication No. 2022/0130499) (hereafter, "Zhou") in view of Syeda-Mahmood et al. (U.S. Patent No. 11,244,755) (hereafter, "Syeda-Mahmood").
Regarding claim 10, Zhou teaches all the limitations of claim 9 above. Zhou teaches wherein the training process comprises ([0021] a multi-phase process of training a VQA system 100): inputting first training data comprising a plurality of sets of medical image data to the first machine learning-based system ([0021] The first phase is executed on a first module 200, which includes an image embedder unit 102 ... The image embedder unit 102 is operable to receive an image 202 as a training instance and employ a model to identify objects in the image 202. The image 202 can be, for example, an x-ray image, a CT scan, a photographic image, or other medically related image data. The image embedder unit 102 can employ a model that executes computer vision techniques on the image 202 for object detection ... The image embedder unit 102 can be trained to detect objects from a particular domain (e.g., medical domain) by adjusting the weights and biases of the neural network; [0020] The joint representation is based on a mapping between the data derived from medical records and data derived from medical images) to generate, for each set of medical image data, a plurality of trial image embedding vectors; and ([0021] The image embedder unit 102 further uses the values of the weights and biases to generate an image embedding vector 204 to represent the identified objects) inputting second training data to the second machine learning-based system ([0021] The first phase is executed on a first module 200, which includes ... a text embedder unit 104; [0023] The text embedder unit 104 can receive a text document 206 in electronic form as a training instance and derive a semantic meaning of the document) to generate, for each set of medical image data, a trial feature vector ([0023] The text document 206 describes at least a portion of the image 202. For example, the text document 206 can be a transcription of a physician's impressions of an x-ray, where the x-ray is the image 202 ... The text embedder unit 104 can be trained to semantically analyze text from a particular domain (e.g., medical domain) by adjusting the weights and biases of the neural network. The text embedder unit 104 further uses the values of the weights and biases to generate a text embedding vector 208 to represent the semantic meaning of the text document 206), wherein the second training data comprises data representing a plurality of medical reports (The text embedder unit 104 can receive a text document 206 in electronic form as a training instance and derive a semantic meaning of the document. The text document 206 can be an electronic medical record, physician's notes, journal article or other textual document) … wherein the training process comprises jointly training ([0048] During a training phase, the weights and biases of the neural network can be adjusted to cause the model to determine a correlation between an object in the image 202 and a natural language description in the text document 206) the first machine learning-based system and the second machine learning-based system to minimize a loss function between the trial image embedding vectors and the corresponding trial feature vectors ([0048] a multi-modal encoder unit 106 can receive the image embedding vector 204 and the text embedding vector 208 ... the multi-modal encoder unit 106 can be implemented by a neural network that executes a model. During a training phase, the weights and biases of the neural network can be adjusted to cause the model to determine a correlation between an object in the image 202 and a natural language description in the text document 206. The multi-modal encoder unit 106 can write data structure to the image embedding vector 204 and the text embedding vector 208 to generate a mapping between the features of the image 202 and the respective portions of the text document 206 that describe the features. The data structure can be, for example, a mapping. For example, the multi-modal encoder unit 106 can write a function that associates a first portion of the image embedding vector 204 to a portion of the text embedding vector 208. The multi-modal encoder unit 106 can further use the mapping to generate a joint image-text representation 500 of the image 202 and the text document).
Zhou does not expressly teach wherein each medical report comprises data indicating presence, in a corresponding one of the sets of medical image data, of one of the plurality of medical image features, and each medical report comprises data representing an area, in the medical image represented by the corresponding set of medical image data, of the feature indicated as present.
However, Syeda-Mahmood teaches wherein each medical report comprises data indicating presence, in a corresponding one of the sets of medical image data, of one of the plurality of medical image features, and (column 4, lines 61-67 - column 5, lines 1-3, The improved automated computer tool and computer tool methodology provides a comprehensive approach to extracting the fine-grained finding labels from medical imaging reports, radiology reports, which implements a new descriptor for fine-grained finding labels utilizing valid combinations of findings and their characterization modifiers, i.e. terms that characterize attributes of the findings, e.g., positioning, laterality, severity, appearance characteristics, etc., found in medical imaging reports; column 31, lines 57-64, the illustrative embodiments provide mechanisms for computer executed automatic learning of fine-grained finding labels (FFLs) from medical imaging report data structures and automatic generation of descriptor data structures that can be used to train machine learning/deep learning models to identify instances of such FFLs or patterns representative of such FFLs in other textual and/or image input data) each medical report comprises data representing an area (column 19, lines 33-40, given a medical imaging report, and identifying an instance of natural language content matching a FFL of a fine-grained finding descriptor data structure, the trained ML/DL computer model can predict the location in a medical image of a corresponding structure, abnormality, etc. based on the learned associations of the FFL of the fine-grained finding descriptor data structure with medical image features; column 4, lines 63-67 - column 5, lines 1-3, extracting the fine-grained finding labels from medical imaging reports, radiology reports, which implements a new descriptor for fine-grained finding labels utilizing valid combinations of findings and their characterization modifiers, i.e. terms that characterize attributes of the findings, e.g., positioning, laterality, severity, appearance characteristics, etc., found in medical imaging reports), in the medical image represented by the corresponding set of medical image data, of the feature indicated as present (column 19, lines 33-40, given a medical imaging report, and identifying an instance of natural language content matching a FFL of a fine-grained finding descriptor data structure, the trained ML/DL computer model can predict the location in a medical image of a corresponding structure, abnormality, etc. based on the learned associations of the FFL of the fine-grained finding descriptor data structure with medical image features).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of Zhou to incorporate the step/system of extracting specific diagnostic terms from text and mapping them to image features and locating abnormalities on medical images taught by Syeda-Mahmood.
The suggestion/motivation for doing so would have been to improve the accuracy to identify findings in medical images for reducing the diagnostic errors and enhancing clinical efficiency (column 32, lines 5-7, more focused and accurate information is able to be provided to medical practitioners, which in turn reduces sources of error in treatment of patients; column 32, lines 34-38 & 42-45, Automated medical imaging report generation can greatly assist medical practitioners by providing improved computing tools that can quickly and accurately identify findings in medical images that should be brought to the attention of the medical practitioner and/or patient ... computing tools may be developed to perform automated preliminary reads of medical imaging data which can expedite clinical workflows, improve accuracy). Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predicted results. Therefore, it would have been obvious to combine Zhou and Syeda-Mahmood to obtain the invention as specified in claim 10.
Regarding claim 11, Zhou teaches all the limitations of claim 10 above. Zhou does not expressly teach further comprising inputting each of the medical reports to a natural language processing system to generate a set of data representing findings of the medical report, wherein the data representing the plurality of medical reports is the data representing the findings of the medical reports.
However, Syeda-Mahmood teaches further comprising inputting each of the medical reports to a natural language processing system (column 11, lines 3-7, the operation starts by performing natural language processing and computer textual analysis on a first corpus of medical imaging report data structures to extract core findings and core modifiers used in natural language content or text of medical imaging reports) to generate a set of data representing findings of the medical report, wherein the data representing the plurality of medical reports is the data representing the findings of the medical reports (column 11, lines 51-60, The medical imaging reports, e.g., radiology reports, are pre-processed to isolate the sections describing the findings and impression. Often, these are indicated by section headings found in medical imaging reports and thus, the pre-processing can use natural language processing to identify section headings and the terms in such section headings that are indicative of findings or impressions. The lexicon or vocabulary driven extraction process is then executed on the identified sections of the medical imaging reports).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of Zhou to incorporate the step/system of using natural language processing to isolate, analyze, and extract medical report findings taught by Syeda-Mahmood.
Motivation for this combination has been stated in claim 10.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Zhou et al. (U.S. Publication No. 2022/0130499) (hereafter, "Zhou") in view of Harzig et al. (U.S. Publication No. 2020/0294654) (hereafter, "Harzig").
Regarding claim 14, Zhou teaches all the limitations of claim 13 above. Zhou teaches wherein the training method comprises: inputting image embedding vectors generated by the first machine learning-based system based on sets of input medical image data ([0021] The image 202 can be, for example, an x-ray image … The image embedder unit 102 can be trained to detect objects from a particular domain (e.g., medical domain) ... The image embedder unit 102 further uses the values of the weights and biases to generate an image embedding vector 204 to represent the identified objects; [0026] The image embedder unit 102 and the text embedder unit 104 can transmit the image embedding vector 204 and the text embedding vector 208 to the second module 210, which includes a multi-modal encoder 212; [0035] The multi-modal encoder 212 can transmit the image-text representation 500 to an answer decoder 502; [0033] The image-text representation 500 is based on a matching of a token (or set of tokens) and an image feature (or set of image features). The matching features from the image embedding vector 204 are mapped to matching tokens from text embedding vector 208) to the third machine learning-based system to generate, for each set of input medical image data, trial natural language text describing a finding relating to the set of input medical image data; and ([0035] The answer decoder 502 can receive a training query 112 and an image 110 as inputs and generate an answer prediction 504. The answer decoder 502 can be implemented by a neural network. The answer decoder 502 can further be in the form of a sequential generating model, such a long short-term memory (LTSM) ... The answer prediction 504 can be retrieved from a database 116 and provided in natural language; [0021] The image 202 can be, for example, an x-ray image; [0050] the answer unit 108 can determine whether the joint image-text representation 500 references an answer to the query 112 ... both the joint user image-text representation and the joint image-text representation 500 can be in the form of respective vectors that relate to a semantic meaning of each ... If the joint user image-text representation and the joint image-text representation 500 are within a threshold distance, the answer unit 108 can extract a natural language answer from the textual portion of the joint image-text representation 500).
Zhou does not expressly teach training the third machine learning-based system to minimize a loss function between the trial natural language text and data representative of medical reports corresponding to the sets of input medical image data.
However, Harzig teaches training the third machine learning-based system to minimize a loss function between the trial natural language text and data representative of medical reports corresponding to the sets of input medical image data ([0026] After the image feature extraction model 240 has extracted feature representations of image content of the image 235, a text generation model 245 on a neural network is trained to predict words in the sentences of text report 220 sequentially by considering the extracted feature representations of image content of the image 235 and previously predicted words of the text generation model 245 to generate text reports 250. During the training of the text generation model 245, optimization loss may be calculated according to the loss of word predictions and sentence annotations, where the annotations are determined based on how likely a sentence being generated is describing a medical abnormality based on the sentence wise abnormality annotations provided by the sentence annotation model 225).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of Zhou to incorporate the step/system of receiving a medical image data, extracting features, and generating text sequentially while evaluating an optimization loss against reference text annotations by using a neural network text-generation model taught by Harzig.
The suggestion/motivation for doing so would have been to improve the clinical accuracy by assigning higher weights to sentences describing medical abnormalities ([0014] typical image captioning tasks usually regard words in a description equally important for optimizing a model. This does not appropriately apply to medical reports where certain sentences in a report may be worth more attention than others, particularly those regarding medical abnormality, and require higher accuracy. To address this situation, the present application describes a system that may integrate knowledge of medical abnormalities at the sentence level for learning a report generation model. Additionally, at least two approaches to identifying sentences describing medical abnormalities are illustrated that may further reduce the burden of labeling training data). Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predicted results. Therefore, it would have been obvious to combine Zhou and Harzig to obtain the invention as specified in claim 14.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL C. CHANG whose telephone number is (571)270-1277. The examiner can normally be reached Monday-Thursday and Alternate Fridays 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan S. Park can be reached at (571) 272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL C CHANG/Examiner, Art Unit 2669
/JOHN B STREGE/Primary Examiner, Art Unit 2669
Prosecution Insights