Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 04/07/2026 has been entered.
Response to Arguments
Regarding the rejection of claims under 35 U.S.C. 112(b), amendments to the claims have overcome the previous rejections, which are withdrawn.
Regarding the rejection of claims under 35 U.S.C. 101, Applicant submits that “the claims, as amended, provide a technological solution that is more than mere mental operations being performed by a computer. The use of a cross-attention encoder to determine the classification of objects in an image based, in part, on text associated with that image is more than merely using a computer to detect and classify objects, but rather are concrete improvements to the classification of objects in images and generation of associated text.”
Examiner respectfully disagrees. Examiner notes that classifying objects in an image using the image and corresponding textual data is a mental process, as described below; a person could review an image and associated text to determine the objects in the image. Examiner finds that using a “cross-attention encoder” in the performance of this mental process does not integrate the abstract idea into a practical application. The claim does not describe a particular type of cross-attention encoder, or a particular model to which the encoder belongs, or a particular method by which the cross-attention encoder determines the object classifications. Examiner further notes that use of a cross-attention encoder is not described as an improvement in the disclosure. Thus “using a cross-attention encoder” merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
The argument is therefore found unpersuasive.
Regarding the rejection of claims under 35 U.S.C. 103, Applicant’s arguments are directed towards amended claims that have not been previously examined. New grounds of rejection under 35 U.S.C. 103 are provided below.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-31 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Analysis is provided for the claims under the guidelines of MPEP 2106.
Regarding claim 1:
Step 1:
The claim recites “One or more processors, comprising.” Thus the claim is to a manufacture, which is a statutory category of invention.
Step 2A prong 1:
The element (bold only) “circuitry to use one or more neural networks to detect one or more objects in one or more images using a decoder of the one or more neural networks,” in its broadest reasonable interpretation, recites a mental process. This process could be performed using observation and judgment.
The element (bold only) “determine a classification of the one or more detected objects using a cross-attention encoder of the one or more neural networks and based on the one or more images and textual data corresponding to the one or more detected objects,“ in its broadest reasonable interpretation, recites a mental process. A person could classify objects in an image based on the images and associated textual data, using observation and judgment.
The element (bold only) “decode, using the decoder, an output of the cross-attention encoder to generate a saliency map, the saliency map indicating locations in the one or more images associated with the one or more detected objects, based on the classification,” in its broadest reasonable interpretation, recites a mental process. A saliency map highlights locations of interest in an image. A person could highlight locations associated with objects, based on a classification of the one or more objects in the one or more images, using both the image itself and textual data related to the image, using observation and judgment.
Thus, the claim recites an abstract idea.
Step 2A prong 2:
The element (bold only) “circuitry to use one or more neural networks to detect one or more objects in one or more images using a decoder of the one or more neural networks,” recites the use of neural networks at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
The element (bold only) “circuitry to use one or more neural networks to detect one or more objects in one or more images using a decoder of the one or more neural networks” recites the use of a decoder at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
The element (bold only) “determine a classification of the one or more detected objects using a cross-attention encoder of the one or more neural networks and based on the one or more images and textual data corresponding to the one or more detected objects” recites the use of a cross-attention encoder at a high level of generality. No particular method of cross-attention encoding or use of the resulting encoding is described. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
The element (bold only) “decode, using the decoder, an output of the cross-attention encoder to generate a saliency map, the saliency map indicating locations in the one or more images associated with the one or more detected objects, based on the classification” recites the use of a decoder at a high level of generality. No particular method of decoding or use of the resulting decoding is described. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
Thus, the additional elements merely recite the use of a computer as a tool to perform the abstract idea. Taken alone, the additional elements do not integrate the abstract idea into a practical application. Considering the elements together as an ordered combination adds nothing that is not present from examining the elements individually. The elements, individually or together, do not describe an improvement in the functioning of technology.
Step 2B:
The claim as a whole does not amount to significantly more than the recited judicial exception.
These additional claim elements recite mere instructions to apply the abstract idea:
(bold only) “circuitry to use one or more neural networks to detect one or more objects in one or more images using a decoder of the one or more neural networks”
(bold only) “circuitry to use one or more neural networks to detect one or more objects in one or more images using a decoder of the one or more neural networks”
(bold only) “determine a classification of the one or more detected objects using a cross-attention encoder of the one or more neural networks and based on the one or more images and textual data corresponding to the one or more detected objects”
(bold only) “decode, using the decoder, an output of the cross-attention encoder to generate a saliency map, the saliency map indicating locations in the one or more images associated with the one or more detected objects, based on the classification”
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 2:
For step 2A prong 1, claim 2 further limits claim 1 and the same elements in claim 2 still recite an abstract idea.
The element (bold only) “wherein the decoder is to generate the saliency map based on a prediction of one or more portions of the textual data, wherein the prediction uses the one or more images and the textual data” in its broadest reasonable interpretation, recites a mental process. A person could highlight locations of interest in an image based on a prediction produced from the images and associated text, using observation and judgment.
Thus the element adds to the abstract idea.
For step 2A prong 2, the element (bold only) “wherein the decoder is to generate the saliency map based on a prediction of one or more portions of the textual data, wherein the prediction uses the one or more images and the textual data” recites the use of a decoder at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
For step 2B, the claim as a whole does not amount to significantly more than the recited judicial exception. The additional element (bold only) “wherein the decoder is to generate the saliency map based on a prediction of one or more portions of the textual data, wherein the prediction uses the one or more images and the textual data” recites mere instructions to apply the abstract idea.
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 3:
For step 2A prong 1, claim 3 further limits claim 1 and the same elements in claim 3 still recite an abstract idea.
Step 2A prong 2:
The element “wherein a first portion and a second portions of the one or more neural networks are trained in parallel” recites simultaneous training of neural networks at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
The element “to encode features of the one or more images and the textual data to a shared latent space” recite a combined encoding at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
Step 2B:
The claim as a whole does not amount to significantly more than the recited judicial exception.
These additional claim elements recite mere instructions to apply the abstract idea:
“wherein a first portion and a second portions of the one or more neural networks are trained in parallel”
“to encode features of one or more images and the textual data to a shared latent space”
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 4:
For step 2A prong 1, claim 4 further limits claim 1 and the same elements in claim 4 still recite an abstract idea.
The element “wherein the textual data comprises textual descriptions of the one or more images” further limits the mental processes in claim 1 but they remain mental processes.
For step 2A prong 2, and step 2B, no further elements remain to be considered. The claim as a whole does not amount to significantly more than the recited judicial exception and is ineligible under 35 U.S.C. 101.
Regarding claim 5:
For step 2A prong 1, claim 5 further limits claim 1 and the same elements in claim 5 still recite an abstract idea.
For step 2A prong 2, the further element “wherein the one or more neural networks comprise a cross-attention encoder, wherein a query input to the cross-attention encoder comprises output from a second portion of the one or more neural networks, and wherein key and value input to the cross-attention encoder comprises output from a first portion of the recites the application of a cross-attention encoder at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
For step 2B, the claim as a whole does not amount to significantly more than the recited judicial exception. The additional element “wherein the one or more neural networks comprise a cross- attention encoder, wherein a query input to the cross-attention encoder comprises output from a second portion of the one or more neural networks, and wherein key and value input to the cross-attention encoder comprises output from a first portion of therecites mere instructions to apply the abstract idea.
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 6:
For step 2A prong 1, claim 6 further limits claim 1 and the same elements in claim 6 still recite an abstract idea.
For step 2A prong 2, the further element “wherein the one or more neural networks comprise a decoder to generate a saliency map based, at least in part, on output of a cross-attention encoder” recites the application, at a high level of generality, of a cross-attention encoder to the abstract idea of creating a saliency map. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
For step 2B, the claim as a whole does not amount to significantly more than the recited judicial exception. The additional element “wherein the one or more neural networks comprise a decoder to generate a saliency map based, at least in part, on output of a cross-attention encoder” recites mere instructions to apply the abstract idea.
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 7:
For step 2A prong 1, claim 7 further limits claim 1 and the same elements in claim 7 still recite an abstract idea.
The element “wherein the textual data represents a textual document” further limits the mental processes in claim 1 but they remain mental processes.
The element (bold only) “wherein an output of the one or more neural networks comprises a classification of a condition depicted in the one or more images and described in the textual document,” in its broadest reasonable interpretation, recites a mental process. A person could examine an image and a textual description of the image and identify a class of condition depicted in the image, using observation, evaluation, and judgement.
For step 2A prong 2, the element (bold only) “wherein an output of the one or more neural networks comprises a classification of a condition depicted in the one or more images and described in the textual document” recites the use of neural networks at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
For step 2B, the claim as a whole does not amount to significantly more than the recited judicial exception. The additional element (bold only) “wherein an output of the one or more neural networks comprises a classification of a condition depicted in the one or more images and described in the textual document” recites mere instructions to apply the abstract idea.
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 8:
For step 2A prong 1, claim 8 further limits claim 1 and the same elements in claim 8 still recite an abstract idea.
The element (bold only) “wherein an output of the one or more neural networks comprises information identifying a condition depicted in the one or more images,” in its broadest reasonable interpretation, recites a mental process. A person could examine an image identify a condition depicted in the image, using observation, evaluation, and judgement.
For step 2A prong 2, the element (bold only) “wherein an output of the one or more neural networks comprises information identifying a condition depicted in the one or more images” recites the use of neural networks at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
For step 2B, the claim as a whole does not amount to significantly more than the recited judicial exception. The additional element (bold only) “wherein an output of the one or more neural networks comprises information identifying a condition depicted in the one or more images” recites mere instructions to apply the abstract idea.
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 9:
For step 1, the claim recites “A system, comprising: one or more processors to.” Thus the claim is to a machine, which is a statutory category of invention. The claim is otherwise analogous to claim 1 and is rejected by the same arguments.
Regarding claim 10:
For step 2A prong 1, claim 10 further limits claim 9 and the same elements in claim 10 still recite an abstract idea.
Step 2A prong 2:
The element “wherein a first portion and a second portions of the one or more neural networks are trained in parallel” recites simultaneous training of neural networks at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
The element “wherein the second portion of the one or more neural networks is taught during training to provide information for training the first portion of the one or more neural network” recites training a neural network with the output of another neural network at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
Step 2B:
The claim as a whole does not amount to significantly more than the recited judicial exception.
These additional claim elements recite mere instructions to apply the abstract idea:
“wherein a first portion and a second portions of the one or more neural networks are trained in parallel”
“wherein the second portion of the one or more neural networks is taught during training to provide information for training the first portion of the one or more neural network”
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 11:
For step 1, the claim recites “A system, comprising: one or more processors to.” Thus the claim is to a machine, which is a statutory category of invention. The claim is otherwise analogous to claim 2 and is rejected by the same arguments.
Regarding claim 12:
For step 2A prong 1, claim 10 further limits claim 9 and the same elements in claim 10 still recite an abstract idea.
The element “wherein the textual data comprises textual descriptions of the one or more images” further limits the mental processes in claim 1 but they remain mental processes.
For step 2A prong 2, and step 2B, no further elements remain to be considered. The claim as a whole does not amount to significantly more than the recited judicial exception and is ineligible under 35 U.S.C. 101.
Regarding claim 13:
For step 2A prong 1, claim 10 further limits claim 9 and the same elements in claim 10 still recite an abstract idea.
For step 2A prong 2, the further element “wherein the one or more neural networks comprise a cross-attention encoder, wherein a query input to the cross-attention encoder comprises output from a second portion of the one or more neural networks, and wherein key and value input to the cross-attention encoder comprises output from a first portion of the recites the application of a cross-attention encoder at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
For step 2B, the claim as a whole does not amount to significantly more than the recited judicial exception. The additional element “wherein the one or more neural networks comprise a cross- attention encoder, wherein a query input to the cross-attention encoder comprises output from a second portion of the one or more neural networks, and wherein key and value input to the cross-attention encoder comprises output from a first portion of therecites mere instructions to apply the abstract idea.
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 14:
For step 2A prong 1, claim 10 further limits claim 9 and the same elements in claim 10 still recite an abstract idea.
The element (bold only) “wherein the one or more neural networks comprise a decoder to generate information indicative of a region of an image” in its broadest reasonable interpretation, recites a mental process. This process could be performed using observation and judgment.
For step 2A prong 2, the further element (bold only) “wherein the one or more neural networks comprise a decoder to generate information indicative of a region of an image” recites a neural network acting as a decoder at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
For step 2B, the claim as a whole does not amount to significantly more than the recited judicial exception. The additional element “(bold only) “wherein the one or more neural networks comprise a decoder to generate information indicative of a region of an image”” recites mere instructions to apply the abstract idea.
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 15:
For step 2A prong 1, claim 10 further limits claim 9 and the same elements in claim 10 still recite an abstract idea.
The element (bold only) “wherein an output of the one or more neural networks comprises a classification of a condition depicted in an image,” in its broadest reasonable interpretation, recites a mental process. A person could examine an image and identify a class of condition depicted in the image, using observation, evaluation, and judgement.
For step 2A prong 2, the element (bold only) “wherein an output of the one or more neural networks comprises a classification of a condition depicted in an image” recites the use of neural networks at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
For step 2B, the claim as a whole does not amount to significantly more than the recited judicial exception. The additional element (bold only) “wherein an output of the one or more neural networks comprises a classification of a condition depicted in an image” recites mere instructions to apply the abstract idea.
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 16:
For step 2A prong 1, claim 10 further limits claim 9 and the same elements in claim 10 still recite an abstract idea.
The element “wherein the one or more images comprise a diagnostic image and the textual data comprises a diagnostic report corresponding to the diagnostic image” further limits the mental processes in claim 1 but they remain mental processes.
For step 2A prong 2, and step 2B, no further elements remain to be considered. The claim as a whole does not amount to significantly more than the recited judicial exception and is ineligible under 35 U.S.C. 101.
Regarding claim 17:
Step 1:
The claim recites “One or more processors, comprising.” Thus the claim is to a manufacture, which is a statutory category of invention.
Step 2A prong 1:
The element (bold only) “circuitry to use one or more neural networks to infer a condition of one or more detected objects in one or more images using a decoder of the one or more neural networks,” in its broadest reasonable interpretation, recites a mental process. This process could be performed using observation and judgment.
The element (bold only) “determine a classification of the one or more detected objects using a cross-attention encoder of the one or more neural networks and based on the one or more images and textual data corresponding to the one or more detected objects,“ in its broadest reasonable interpretation, recites a mental process. A person could classify objects in an image based on the images and associated textual data. This process could be performed using observation and judgment.
The element (bold only) “decode, using the decoder, an output of the cross-attention encoder to generate a saliency map, the saliency map indicating locations in the one or more images associated with the one or more detected objects, based on the classification,” in its broadest reasonable interpretation, recites a mental process. A saliency map highlights locations of interest in an image. A person could highlight locations associated with objects, based on a classification of the one or more objects in the one or more images, using both the image itself and textual data related to the image, using observation and judgment.
Thus, the claim recites an abstract idea.
Step 2A prong 2:
The element (bold only) “circuitry to use one or more neural networks to infer a condition of one or more objects in one or more images using a decoder of the one or more neural networks,” recites the use of neural networks at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
The element (bold only) “circuitry to use one or more neural networks to infer a condition of one or more objects in one or more images using a decoder of the one or more neural networks” recites the use of a decoder at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
The element (bold only) “determine a classification of the one or more detected objects using a cross-attention encoder of the one or more neural networks and based on the one or more images and textual data corresponding to the one or more detected objects” recites the use of a cross-attention encoder at a high level of generality. No particular method of cross-attention encoding or use of the resulting encoding is described. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
The element (bold only) “decode, using the decoder, an output of the cross-attention encoder to generate a saliency map, the saliency map indicating locations in the one or more images associated with the one or more detected objects, based on the classification” recites the use of a decoder at a high level of generality. No particular method of decoding or use of the resulting decoding is described. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
Thus, the additional elements merely recite the use of a computer as a tool to perform the abstract idea. Taken alone, the additional elements do not integrate the abstract idea into a practical application. Considering the elements together as an ordered combination adds nothing that is not present from examining the elements individually. The elements, individually or together, do not describe an improvement in the functioning of technology.
Step 2B:
The claim as a whole does not amount to significantly more than the recited judicial exception.
These additional claim elements recite mere instructions to apply the abstract idea:
(bold only) “circuitry to use one or more neural networks to infer a condition of one or more objects in one or more images using a decoder of the one or more neural networks”
(bold only) “circuitry to use one or more neural networks to infer a condition of one or more objects in one or more images using a decoder of the one or more neural networks”
(bold only) “determine a classification of the one or more detected objects using a cross-attention encoder of the one or more neural networks and based on the one or more images and textual data corresponding to the one or more detected objects”
(bold only) “decode, using the decoder, an output of the cross-attention encoder to generate a saliency map, the saliency map indicating locations in the one or more images associated with the one or more detected objects, based on the classification”
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 18:
For step 2A prong 1, claim 18 further limits claim 17 and the same elements in claim 18 still recite an abstract idea.
For step 2A prong 2, the further element “wherein a first portion of the one or more neural networks is trained to encode features of the one or more images and a second portion of the one or more neural networks is trained to encode features of the textual data” recites feature extraction at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
For step 2B, the claim as a whole does not amount to significantly more than the recited judicial exception. The additional element “wherein a first portion of the one or more neural networks is trained to encode features of the one or more images and a second portion of the one or more neural networks is trained to encode features of the textual data” recites mere instructions to apply the abstract idea.
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 19:
For step 2A prong 1, claim 19 further limits claim 18 and the same elements in claim 19 still recite an abstract idea.
For step 2A prong 2, the element “wherein the first portion of the one or more neural networks, and the second portion of the one or more neural networks, encode their respective inputs to a common latent space” recite a combined encoding at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
For step 2B, the claim as a whole does not amount to significantly more than the recited judicial exception. The additional element (bold only) “wherein the first portion of the one or more neural networks, and the second portion of the one or more neural networks, encode their respective inputs to a common latent space” recites mere instructions to apply the abstract idea.
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 20:
For step 2A prong 1, claim 20 further limits claim 17 and the same elements in claim 20 still recite an abstract idea.
For step 2A prong 2, the further element “wherein the one or more neural networks are is trained based, at least in part, on output of a cross-attention encoder using, as input to the cross-attention encoder, output of an image encoder and output of a language encoder” recites the application, at a high level of generality, of a cross-attention encoder to multi-modal input. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
For step 2B, the claim as a whole does not amount to significantly more than the recited judicial exception. The additional element “wherein the one or more neural networks are is trained based, at least in part, on output of a cross-attention encoder using, as input to the cross-attention encoder, output of an image encoder and output of a language encoder” recites mere instructions to apply the abstract idea.
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 21:
For step 2A prong 1, claim 21 further limits claim 17 and the same elements in claim 21 still recite an abstract idea.
The element “wherein the one or more images comprises diagnostic images and the textual data comprises diagnostic reports corresponding to the diagnostic images” further limits the mental processes in claim 1 but they remain mental processes.
For step 2A prong 2, and step 2B, no further elements remain to be considered. The claim as a whole does not amount to significantly more than the recited judicial exception and is ineligible under 35 U.S.C. 101.
Regarding claim 22:
For step 2A prong 1, claim 22 further limits claim 17 and the same elements in claim 22 still recite an abstract idea.
The element “wherein the inferred condition comprises information indicative of an area of interest in the one or more images” further limits the mental processes in claim 1 but they remain mental processes.
For step 2A prong 2, and step 2B, no further elements remain to be considered. The claim as a whole does not amount to significantly more than the recited judicial exception and is ineligible under 35 U.S.C. 101.
Regarding claim 23:
For step 2A prong 1, claim 23 further limits claim 17 and the same elements in claim 23 still recite an abstract idea.
Step 2A prong 2:
The element “wherein a first portion of the one or more neural networks is trained to encode features of the one or more images and a second portion of the one or more neural networks is trained to encode features of the textual data” recites feature extraction at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
The element “wherein the first portion of the neural network one or more neural networks, after training, is capable of inferring the information condition independently of the second portion” recites independent neural network inference at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
Step 2B:
The claim as a whole does not amount to significantly more than the recited judicial exception.
These additional claim elements recite mere instructions to apply the abstract idea:
“wherein a first portion of the one or more neural networks is trained to encode features of the one or more images and a second portion of the one or more neural networks is trained to encode features of the textual data”
“wherein the first portion of the neural network one or more neural networks, after training, is capable of inferring the information condition independently of the second portion”
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 24:
Step 1:
The claim recites “A method, comprising.” Thus the claim is to a process, which is a statutory category of invention.
Step 2A prong 1:
The element (bold only) “using one or more neural networks to diagnose a condition depicted in a diagnostic image using a decoder of the one or more neural networks,” in its broadest reasonable interpretation, recites a mental process. This process could be performed using observation and judgment.
The element (bold only) “determine a classification of the condition using a cross-attention encoder of the one or more neural networks and based on the diagnostic image and textual data corresponding to the condition,“ in its broadest reasonable interpretation, recites a mental process. A person could classify a condition depicted in an image based on the images and associated textual data, using observation and judgement.
The element (bold only) “decode, using the decoder, an output of the cross-attention encoder to generate a saliency map, the saliency map indicating locations in the diagnostic image associated with the condition, based on the classification and a prediction of one or more portions of a set of diagnostic reports, wherein the prediction uses the diagnostic image and the textual data” in its broadest reasonable interpretation, recites a mental process. A saliency map highlights locations of interest in an image. A person could highlight locations of interest in an image based on a classification of the one or more objects in the one or more images, using the image itself, the textual data related to the image, and a prediction produced from a set of diagnostic reports, using observation and judgment.
Thus, the claim recites an abstract idea.
Step 2A prong 2:
The element (bold only) “using one or more neural networks to diagnose a condition depicted in a diagnostic image using a decoder of the one or more neural networks” recites the use of neural networks at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
The element (bold only) “using one or more neural networks to diagnose a condition depicted in a diagnostic image using a decoder of the one or more neural networks” recites the use of a decoder at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
The element (bold only) “determine a classification of the condition using a cross-attention encoder of the one or more neural networks and based on the one or more images and textual data corresponding to the condition” recites the use of a cross-attention encoder at a high level of generality. No particular method of cross-attention encoding or use of the resulting encoding is described. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
The element (bold only) “decode, using the decoder, an output of the cross-attention encoder to generate a saliency map, the saliency map indicating locations in the diagnostic image associated with the condition, based on the classification and a prediction of one or more portions of a set of diagnostic reports, wherein the prediction uses the diagnostic image and the textual data” recites the use of a decoder at a high level of generality. No particular method of decoding or use of the resulting decoding is described. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
Thus, the additional elements merely recite the use of a computer as a tool to perform the abstract idea. Taken alone, the additional elements do not integrate the abstract idea into a practical application. Considering the elements together as an ordered combination adds nothing that is not present from examining the elements individually. The elements, individually or together, do not describe an improvement in the functioning of technology.
Step 2B:
The claim as a whole does not amount to significantly more than the recited judicial exception.
These additional claim elements recite mere instructions to apply the abstract idea:
(bold only) “using one or more neural networks to diagnose a condition depicted in a diagnostic image using a decoder of the one or more neural networks”
(bold only) “using one or more neural networks to diagnose a condition depicted in a diagnostic image using a decoder of the one or more neural networks”
(bold only) “determine a classification of the condition using a cross-attention encoder of the one or more neural networks and based on the one or more images and textual data corresponding to the condition”
(bold only) “decode, using the decoder, an output of the cross-attention encoder to generate a saliency map, the saliency map indicating locations in the diagnostic image associated with the condition, based on the classification and a prediction of one or more portions of a set of diagnostic reports, wherein the prediction uses the diagnostic image and the textual data”
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 25:
For step 2A prong 1, claim 25 further limits claim 24 and the same elements in claim 25 still recite an abstract idea.
The element “wherein a first portion of the one or more neural networks is trained in parallel with a second portion of the one or more neural networks” recites simultaneous training of neural networks at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
The element “wherein the second portion of the one or more neural networks is trained to encode features of the set of diagnostic reports” recites feature extraction at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
Step 2B:
The claim as a whole does not amount to significantly more than the recited judicial exception.
These additional claim elements recite mere instructions to apply the abstract idea:
“wherein a first portion of the one or more neural networks is trained in parallel with a second portion of the one or more neural networks”
“wherein the second portion of the one or more neural networks is trained to encode features of the set of diagnostic reports”
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 26:
For step 2A prong 1, claim 26 further limits claim 25 and the same elements in claim 26 still recite an abstract idea.
For step 2A prong 2, the element “wherein the first and second portions of the one or more neural networks are trained to encode features of the diagnostic image and the textual to a shared latent space” recites a combined encoding at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
For step 2B, the claim as a whole does not amount to significantly more than the recited judicial exception. The additional element “wherein the first and second portions of the one or more neural networks are trained to encode features of the diagnostic image and the textual to a shared latent space” recites mere instructions to apply the abstract idea.
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 27:
For step 2A prong 1, claim 27 further limits claim 24 and the same elements in claim 27 still recite an abstract idea.
For step 2A prong 2, the further element “providing, as input to a cross-attention encoder, a query input comprising output from a language encoder, and key and value input comprising output from an image encode” recites the application of a cross-attention encoder at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
For step 2B, the claim as a whole does not amount to significantly more than the recited judicial exception. The additional element “providing, as input to a cross-attention encoder, a query input comprising output from a language encoder, and key and value input comprising output from an image encode” recites mere instructions to apply the abstract idea.
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 28:
For step 2A prong 1, claim 28 further limits claim 24 and the same elements in claim 28 still recite an abstract idea.
For step 2A prong 2, the element “training a language encoder of the one or more neural networks to encode features of the diagnostic reports to a latent space shared with output of an image encoder” recites a combined encoding at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
For step 2B, the claim as a whole does not amount to significantly more than the recited judicial exception. The additional element “training a language encoder of the one or more neural networks to encode features of the diagnostic reports to a latent space shared with output of an image encoder” recites mere instructions to apply the abstract idea.
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 29:
For step 2A prong 1, claim 29 further limits claim 24 and the same elements in claim 29 still recite an abstract idea.
The element (bold only) “decoding output of an encoder to generate information summarizing the condition,” in its broadest reasonable interpretation, recites a mental process. This process could be performed using observation and judgment.
For step 2A prong 2, the element (bold only) “decoding output of an encoder to generate information summarizing the condition,” recites a combined encoding at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
For step 2B, the claim as a whole does not amount to significantly more than the recited judicial exception. The additional element (bold only) “decoding output of an encoder to generate information summarizing the condition” recites mere instructions to apply the abstract idea.
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 30:
For step 2A prong 1, claim 30 further limits claim 24 and the same elements in claim 30 still recite an abstract idea.
The element (bold only) “wherein the one or more neural networks comprises a decoder to generate information indicative of a region in the diagnostic image that depicts the condition,” in its broadest reasonable interpretation, recites a mental process. This process could be performed using observation and judgment.
For step 2A prong 2, the element (bold only) “wherein the one or more neural networks comprises a decoder to generate information indicative of a region in the diagnostic image that depicts the condition,” recites a neural network acting as a decoder at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
For step 2B, the claim as a whole does not amount to significantly more than the recited judicial exception. The additional element (bold only) “wherein the one or more neural networks comprises a decoder to generate information indicative of a region in the diagnostic image that depicts the condition” recites mere instructions to apply the abstract idea.
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 31:
For step 2A prong 1, claim 31 further limits claim 24 and the same elements in claim 31 still recite an abstract idea.
The element (bold only) “wherein diagnoses of the condition comprises identifying one or more categories of conditions determined, by the one or more neural networks, to be associated with a region of the diagnostic image,” in its broadest reasonable interpretation, recites a mental process. This process could be performed using observation and judgment.
For step 2A prong 2, the element (bold only) “wherein diagnoses of the condition comprises identifying one or more categories of conditions determined, by the one or more neural networks, to be associated with a region of the diagnostic image,” recites using neural network acting at a high level of generality. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
For step 2B, the claim as a whole does not amount to significantly more than the recited judicial exception. The additional element (bold only) “wherein diagnoses of the condition comprises identifying one or more categories of conditions determined, by the one or more neural networks, to be associated with a region of the diagnostic image” recites mere instructions to apply the abstract idea.
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1–4, 6, 9–12, and 14 rejected under 35 U.S.C. 103 as being unpatentable over Zhou et al., “Learning Deep Features for Discriminative Localization,” 2015, arXiv:1512.04150v1 (hereafter Zhou) in view of Jain et al., US Pre-Grant Publication No. 2022/0256175 (hereafter Jain).
Regarding claim 1 and analogous claim 9:
Zhou teaches:
(bold only) “One or more processors, comprising circuitry to use one or more neural networks to: detect one or more objects in one or more images using a decoder of the one or more neural networks;”: Zhou, section 1.1, paragraph 5, “There has been a number of recent works [29, 14, 4, 33] that visualize the internal representation learned by CNNs in an attempt to better understand their properties. Zeiler et al [29] use deconvolutional networks to visualize what patterns activate each unit. Zhou et al. [33] show that CNNs learn object detectors [detect one or more objects in one or more images using a decoder of the one or more neural networks] while being trained to recognize scenes, and demonstrate that the same network can perform both scene recognition and object localization in a single forward-pass.”
(bold only) “determine a classification of the one or more detected objects using a cross-attention encoder of the one or more neural networks and based on the one or more images and textual data corresponding to the one or more detected objects”: Zhou, section 2., paragraph 3, “For a given image, let fk(x, y) represent the activation of unit k in the last convolutional layer at spatial location (x, y). Then, for unit k, the result of performing global average pooling, Fk is Σx, y fk(x, y). Thus, for a given class c, the input to the softmax, Sc, is Σx, y wck Fk where wck is the weight corresponding to class c for unit k. Essentially, wck indicates the importance of Fk for class c [determine a classification of the one or more detected objects using … one or more neural networks and based on the one or more images … corresponding to the one or more detected objects].”
(bold only) “decode, using the decoder, an output of the cross-attention encoder to generate a saliency map, the saliency map indicating locations in the one or more images associated with the one or more detected objects, based on the classification”: Zhou, section 2, paragraph 1, “In this section, we describe the procedure for generating class activation maps (CAM) using global average pooling (GAP) in CNNs. A class activation map for a particular category indicates the discriminative image regions used by the CNN to identify that category (e.g., Fig. 3) [generate a saliency map, the saliency map indicating locations in the one or more images associated with the one or more detected objects, based on the classification].”
Zhou does not explicitly teach:
(bold only) “One or more processors, comprising circuitry to use one or more neural networks detect one or more objects in one or more images using a decoder of the one or more neural networks”
(bold only) “determine a classification of the one or more detected objects using a cross-attention encoder of the one or more neural networks and based on the one or more images and textual data corresponding to the one or more detected objects”
(bold only) “decode, using the decoder, an output of the cross-attention encoder to generate a saliency map, the saliency map indicating locations in the one or more images associated with the one or more detected objects, based on the classification”
Jain teaches:
(bold only) “One or more processors, comprising circuitry to use one or more neural networks detect one or more objects in one or more images using a decoder of the one or more neural networks”: Jain, paragraph 0049, “The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g. , a processor core , a microprocessor , an ASIC , an FPGA , a controller , a microcontroller , etc. ) and can be one processor or a plurality of processors that are operatively connected [one or more processors, comprising circuitry].”
(bold only) “determine a classification of the one or more detected objects using a cross-attention encoder of the one or more neural networks and based on the one or more images and textual data corresponding to the one or more detected objects”: Jain, paragraph 0021, “In some implementations , at least one of the frame-level encoder model or the segment-level encoder model can be a multimodal encoder based at least in part on associated text . For instance , in addition to encoding the video data and/or representations thereof, the encoder(s) (e.g., the lower-level encoder and/or the higher-level encoder) can be cross-modal encoders that additionally fuse the video data and/or representations thereof with associated text data , such as, for example, captioning data for the video and/or query data descriptive of a user query representing a user's search for videos and/or, more particularly , content depicted within the videos. For instance, in the encoder (s) , the input modality pairs can have cross-attention [cross-attention encoder], such as visual-caption/transcripts, visual-query [images and textual data], and/or transcript/query attention. In some implementations , the associated text can be encoded (e.g. , by a text encoder model, such as a text transformer).”
(bold only) “decode, using the decoder, an output of the cross-attention encoder to generate a saliency map, the saliency map indicating locations in the one or more images associated with the one or more detected objects, based on the classification”: Jain, paragraph 0070, “In some cases , the machine learned model (s) can be configured to perform a task that includes encoding input data for reliable and/or efficient transmission or storage (and/or corresponding decoding) [decode, using the decoder, an output of the cross-attention encoder].”
Jain and Zhou are analogous arts as they are both related to machine learning model design. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the cross-modal data encoding of Jain with the teachings of Zhou to arrive at the present invention, in order to combine data from multiple modalities to improve the contextual quality of data encoding, as stated in Jain, paragraph 0085, “Cross-modal transformer 534 then produces contextualized segment representations 540 based on cross-attention between the encoded segment representations 525 from video data encoder 532 and encoded associated text data from text encoder model 536.”
Regarding claim 2 and analogous claim 11:
Zhou as modified by Jain teaches “The one or more processors of claim 1.”
Zhou further teaches (bold only) “wherein the decoder is to generate the saliency map based on a prediction of one or more portions of the textual data, wherein the prediction uses the one or more images and the textual data”: Zhou, section 2., paragraph 3, “For a given image, let fk(x, y) represent the activation of unit k in the last convolutional layer at spatial location (x, y). Then, for unit k, the result of performing global average pooling, Fk is Σx, y fk(x, y). Thus, for a given class c, the input to the softmax, Sc, is Σx, y wck Fk where wck is the weight corresponding to class c for unit k. Essentially, wck indicates the importance of Fk for class c. Finally the output of the softmax for class c, Pc is given by
PNG
media_image1.png
28
67
media_image1.png
Greyscale
. Here we ignore the bias term: we explicitly set the input bias of the softmax to 0 as it has little to no impact on the classification performance. By plugging Fk = Σx, y fk(x, y) into the class score, Sc, we obtain
PNG
media_image2.png
87
160
media_image2.png
Greyscale
We define Mc as the class activation map for class c, where each spatial element is given by
PNG
media_image3.png
37
187
media_image3.png
Greyscale
Thus, Sc = Σx, y Mc(x, y), and hence Mc(x, y) directly indicates the importance of the activation at spatial grid (x, y) leading to the classification of an image to class c.”
Jain further teaches (bold only) “wherein the decoder is to generate the saliency map based on a prediction of one or more portions of the textual data, wherein the prediction uses the one or more images and the textual data”: Jain, paragraph 0021, “In some implementations , at least one of the frame-level encoder model or the segment-level encoder model can be a multimodal encoder based at least in part on associated text . For instance , in addition to encoding the video data and/or representations thereof, the encoder(s) (e.g., the lower-level encoder and/or the higher-level encoder) can be cross-modal encoders that additionally fuse the video data and/or representations thereof with associated text data , such as, for example, captioning data for the video and/or query data descriptive of a user query representing a user's search for videos and/or, more particularly , content depicted within the videos. For instance, in the encoder (s), the input modality pairs can have cross-attention, such as visual-caption/transcripts, visual-query [a prediction of one or more portions of the textual data, wherein the prediction uses the one or more images and the textual data], and/or transcript/query attention. In some implementations , the associated text can be encoded (e.g. , by a text encoder model, such as a text transformer)”; Jain, paragraph 0070, “In some cases , the machine learned model (s) can be configured to perform a task that includes encoding input data for reliable and/or efficient transmission or storage (and/or corresponding decoding).”
Jain and Zhou are combinable for the rationale given under claim 1.
Regarding claim 3:
Zhou as modified by Jain teaches “The one or more processors of claim 1.”
Jain further teaches “wherein a first portion and a second portion of the one or more neural networks are trained in parallel to encode features of the one or more images and the textual data to a shared latent space”: Jain, paragraph 0021, “In some implementations, at least one of the frame-level encoder model or the segment-level encoder model can be a multimodal encoder based at least in part on associated text . For instance , in addition to encoding the video data and/or representations thereof, the encoder(s) (e.g., the lower-level encoder and/or the higher-level encoder) can be cross-modal encoders that additionally fuse the video data and/or representations thereof with associated text data , such as, for example, captioning data for the video and/or query data descriptive of a user query representing a user's search for videos and/or, more particularly , content depicted within the videos. For instance, in the encoder (s), the input modality pairs can have cross-attention, such as visual-caption/transcripts, visual-query [encode features of the one or more images and the textual data to a shared latent space], and/or transcript/query attention. In some implementations , the associated text can be encoded (e.g. , by a text encoder model, such as a text transformer)”; Jain, paragraph 0039, “The hierarchical video encoders as described herein can be jointly trained on both tasks in a multitask learning configuration [a first portion and a second portion of the one or more neural networks are trained in parallel]. The hierarchical ( e.g. , and cross - attentional ) encoders as described herein can be beneficial for these tasks , as the two tasks can require understanding semantics of a video at differing temporal resolutions , and the models described herein can model short - range and long - range video semantics.”
Jain and Zhou are combinable are the rationale given under claim 1.
Regarding claim 4 and analogous claim 12:
Zhou as modified by Jain teaches “The one or more processors of claim 1.”
Jain further teaches “wherein the textual data comprises textual descriptions of the one or more images”: Jain, paragraph 0021, “In some implementations, at least one of the frame-level encoder model or the segment-level encoder model can be a multimodal encoder based at least in part on associated text . For instance , in addition to encoding the video data and/or representations thereof, the encoder(s) (e.g., the lower-level encoder and/or the higher-level encoder) can be cross-modal encoders that additionally fuse the video data and/or representations thereof with associated text data [wherein the textual data comprises textual descriptions of the one or more images], such as, for example, captioning data for the video and/or query data descriptive of a user query representing a user's search for videos and/or, more particularly , content depicted within the videos. For instance, in the encoder (s), the input modality pairs can have cross-attention, such as visual-caption/transcripts, visual-query, and/or transcript/query attention. In some implementations , the associated text can be encoded (e.g. , by a text encoder model, such as a text transformer).”
Jain and Zhou are combinable are the rationale given under claim 1.
Regarding claim 6:
Zhou as modified by Jain teaches “The one or more processors of claim 1.”
Zhou further teaches (bold only) “wherein the one or more neural networks comprise a decoder to generate a saliency map based, at least in part, on output of a cross-attention encoder”: Zhou, section 2., paragraph 3, “For a given image, let fk(x, y) represent the activation of unit k in the last convolutional layer at spatial location (x, y) [output of a … encoder]. Then, for unit k, the result of performing global average pooling, Fk is Σx, y fk(x, y). Thus, for a given class c, the input to the softmax, Sc, is Σx, y wck Fk where wck is the weight corresponding to class c for unit k. Essentially, wck indicates the importance of Fk for class c. Finally the output of the softmax for class c, Pc is given by
PNG
media_image1.png
28
67
media_image1.png
Greyscale
. Here we ignore the bias term: we explicitly set the input bias of the softmax to 0 as it has little to no impact on the classification performance. By plugging Fk = Σx, y fk(x, y) into the class score, Sc, we obtain
PNG
media_image2.png
87
160
media_image2.png
Greyscale
We define Mc as the class activation map for class c, where each spatial element is given by
PNG
media_image3.png
37
187
media_image3.png
Greyscale
Thus, Sc = Σx, y Mc(x, y), and hence Mc(x, y) directly indicates the importance of the activation at spatial grid (x, y) leading to the classification of an image to class c [the one or more neural networks … generate a saliency map].”
Jain further teaches (bold only) “wherein the one or more neural networks comprise a decoder to generate a saliency map based, at least in part, on output of a cross-attention encoder”: Jain, paragraph 0021, “In some implementations , at least one of the frame-level encoder model or the segment-level encoder model can be a multimodal encoder based at least in part on associated text . For instance , in addition to encoding the video data and/or representations thereof, the encoder(s) (e.g., the lower-level encoder and/or the higher-level encoder) can be cross-modal encoders that additionally fuse the video data and/or representations thereof with associated text data , such as, for example, captioning data for the video and/or query data descriptive of a user query representing a user's search for videos and/or, more particularly , content depicted within the videos. For instance, in the encoder (s) , the input modality pairs can have cross-attention [cross-attention encoder], such as visual-caption/transcripts, visual-query, and/or transcript/query attention. In some implementations , the associated text can be encoded (e.g. , by a text encoder model, such as a text transformer)”; Jain, paragraph 0070, “In some cases , the machine learned model (s) can be configured to perform a task that includes encoding input data for reliable and/or efficient transmission or storage (and/or corresponding decoding) [comprise a decoder].”
Jain and Zhou are combinable are the rationale given under claim 1.
Regarding claim 10:
Zhou as modified by Jain teaches “The system of claim 9.”
Jain further teaches “wherein a first portion and a second portion of the one or more neural networks are trained in parallel, and wherein the second portion of the one or more neural networks is taught during training to provide information for training the first portion of the one or more neural networks”: Jain, paragraph 0021, “In some implementations, at least one of the frame-level encoder model or the segment-level encoder model can be a multimodal encoder based at least in part on associated text . For instance , in addition to encoding the video data and/or representations thereof, the encoder(s) (e.g., the lower-level encoder and/or the higher-level encoder) can be cross-modal encoders that additionally fuse the video data and/or representations thereof with associated text data , such as, for example, captioning data for the video and/or query data descriptive of a user query representing a user's search for videos and/or, more particularly , content depicted within the videos. For instance, in the encoder (s), the input modality pairs can have cross-attention [the second portion of the one or more neural networks is taught during training to provide information for training the first portion of the one or more neural networks], such as visual-caption/transcripts, visual-query, and/or transcript/query attention. In some implementations , the associated text can be encoded (e.g. , by a text encoder model, such as a text transformer)”; Jain, paragraph 0039, “The hierarchical video encoders as described herein can be jointly trained on both tasks in a multitask learning configuration [a first portion and a second portion of the one or more neural networks are trained in parallel]. The hierarchical ( e.g. , and cross - attentional ) encoders as described herein can be beneficial for these tasks , as the two tasks can require understanding semantics of a video at differing temporal resolutions, and the models described herein can model short - range and long - range video semantics.”
Jain and Zhou are combinable are the rationale given under claim 9.
Regarding claim 14:
Zhou as modified by Jain teaches “The system of claim 9.”
Zhou further teaches “wherein the one or more neural networks comprise a decoder to generate information indicative of a region of an image”: Zhou, section 2, paragraph 1, “In this section, we describe the procedure for generating class activation maps (CAM) using global average pooling (GAP) in CNNs. A class activation map for a particular category indicates the discriminative image regions used by the CNN to identify that category (e.g., Fig. 3) [comprise a decoder to generate information indicative of a region of an image].”
Claim 5 and analogous claim 13 rejected under 35 U.S.C. 103 as being unpatentable Zhou as modified by Jain in view of Wei et al., “Multi-Modality Cross Attention Network for Image and Sentence Matching,” 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, doi: 10.1109/CVPR42600.2020.01095 (hereafter Wei).
Zhou as modified by Jain teaches “The one or more processors of claim 1.”
Zhou as modified by Jain does not explicitly teach “wherein the one or more neural networks comprise a cross-attention encoder, wherein a query input to the cross-attention encoder comprises output from a second portion of the one or more neural networks, and wherein key and value input to the cross-attention encoder comprises output from a first portion of the one or more neural networks.”
Wei teaches “wherein the one or more neural networks comprise a cross-attention encoder, wherein a query input to the cross-attention encoder comprises output from a second portion of the one or more neural networks, and wherein key and value input to the cross-attention encoder comprises output from a first portion of the one or more neural networks”: Wei, section 3.3, “for image I with the fine grained representation R = {r1, r2, ..., rk} [showing that R is an output representation of the first portion of the network] […] The BERT consists of multiple Transformer units, and its output E = {e1, e2, ..., en} naturally includes the intra-modality information [showing that E is an output representation of the second portion of the network]”; Wei, section 3.4, “In this section, we introduce how to model both the inter-modality and intra-modality relationships in a unified model with our Cross-Attention Module [comprise a cross-attention encoder] […]Here, the query [a query input to the cross-attention encoder comprises output from the second portion of the one or more neural networks], key and value [key and value input to the cross-attention encoder comprises output from the first portion of the one or more neural networks] for the fragments are formed with the following equations:
PNG
media_image4.png
217
404
media_image4.png
Greyscale
”
Wei and Zhou are analogous arts as they are both related to image analysis. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the multi-modal cross-attention of Wei with the teachings of Zhou to arrive at the present invention, in order to improve model performance, as stated in Wei, Abstract, “In the proposed MM-CA, we design a novel cross-attention mechanism, which is able to exploit not only the intra-modality relationship within each modality, but also the inter-modality relationship between image regions and sentence words to complement and enhance each other for image and sentence matching. Extensive experimental results on two standard benchmarks including Flickr30K and MS-COCO demonstrate that the proposed model performs favorably against state-of-the-art image and sentence matching methods.”
Claims 7-8, 15–22, 24–26, and 28–31 rejected under 35 U.S.C. 103 as being unpatentable over Zhou as modified by Jain in view of PanQiao et al., “Image and Text Fusion for Character-based Breast Cancer Classification,” 2018, 2018 IEEE 20th International Conference on High Performance Computing and Communications (hereafter PanQiao).
Regarding claim 7:
Zhou as modified by Jain teaches “The one or more processors of claim 1.”
Jain further teaches “wherein the textual data represents a textual document”: Jain, paragraph 0021, “In some implementations , at least one of the frame-level encoder model or the segment-level encoder model can be a multimodal encoder based at least in part on associated text . For instance , in addition to encoding the video data and/or representations thereof, the encoder(s) (e.g., the lower-level encoder and/or the higher-level encoder) can be cross-modal encoders that additionally fuse the video data and/or representations thereof with associated text data , such as, for example, captioning data for the video [the textual data represents a textual document] and/or query data descriptive of a user query representing a user's search for videos and/or, more particularly , content depicted within the videos. For instance, in the encoder (s) , the input modality pairs can have cross-attention, such as visual-caption/transcripts, visual-query], and/or transcript/query attention. In some implementations , the associated text can be encoded (e.g. , by a text encoder model, such as a text transformer).”
Jain and Zhou are combinable are the rationale given under claim 1.
Zhou as modified by Jain does not explicitly teach “and wherein an output of the one or more neural networks comprises a classification of a condition depicted in the one or more images and described in the textual document.”
PanQiao teaches “and wherein an output of the one or more neural networks comprises a classification of a condition depicted in the one or more images and described in the textual document”: PanQiao, section I, “Character-based feature representation method is used to fully capture the complicated matching relations between image and sentence and are fully captured in our proposed f-CNN. We validate the effectiveness of f-CNNs on the classification tasks for breast cancer [output of the one or more neural networks comprises a classification of a condition], and demonstrate that f-CNNs can achieve performances superior to the state-of-the-art approaches by letting image and the composed fragments of the sentence meet and interact at different levels”; PanQiao, section III. A., “Our task is to extract and mine the semantic interactions of radiology images and reports [a classification of a condition depicted in the one or more images and described in the textual document], and generate fusion feature vector with better performance. We will train our model on a training set of N images and N corresponding sentences that describe their content (Figure 1).”
PanQiao and Zhou as modified by Jain are analogous arts as both are related to the analysis of image-text pairs. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the medical condition classification of PanQiao to the teachings of Zhou as modified by Jain to arrive at the present invention, in order to leverage machine learning to improve medical treatment, as stated in PanQiao, section I, “The efficient analysis of images and electronic health records (EHRs) is significant for improving reliability of laboratory results, which is used to help assessing disease risks and monitoring the treatment.”
Regarding claim 8 and analogous claim 15:
Zhou as modified by Jain teaches “The one or more processors of claim 1.”
Zhou as modified by Jain does not explicitly teach “wherein an output of the one or more neural networks comprises information identifying a condition depicted in the one or more images.”
PanQiao teaches “wherein an output of the one or more neural networks comprises information identifying a condition depicted in the one or more images e”: PanQiao, section I, “Character-based feature representation method is used to fully capture the complicated matching relations between image and sentence and are fully captured in our proposed f-CNN. We validate the effectiveness of f-CNNs on the classification tasks for breast cancer [information identifying a condition], and demonstrate that f-CNNs can achieve performances superior to the state-of-the-art approaches by letting image and the composed fragments of the sentence meet and interact at different levels”; PanQiao, section III. A., “Our task is to extract and mine the semantic interactions of radiology images and reports [output of the one or more neural networks comprises information identifying a condition depicted in the one or more images], and generate fusion feature vector with better performance. We will train our model on a training set of N images and N corresponding sentences that describe their content (Figure 1).”
PanQiao and Zhou as modified by Jain are analogous arts as both are related to the analysis of image-text pairs. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the medical condition classification of PanQiao to the teachings of Zhou as modified by Jain to arrive at the present invention, in order to leverage machine learning to improve medical treatment, as stated in PanQiao, section I, “The efficient analysis of images and electronic health records (EHRs) is significant for improving reliability of laboratory results, which is used to help assessing disease risks and monitoring the treatment.”
Regarding claim 16:
Zhou as modified by Jain teaches “The one or more processors of claim 1.”
Zhou as modified by Jain does not explicitly teach “wherein the one or more images comprise a diagnostic image and the textual data comprises a diagnostic report corresponding to the diagnostic image.”
PanQiao teaches “wherein the one or more images comprise a diagnostic image and the textual data comprises a diagnostic report corresponding to the diagnostic image”: PanQiao, Section V. A., “The experimental datasets in this paper came from a major hospital in Shanghai, including medical documents of clinical breast cases and related mammography image [image data comprises a diagnostic]. The document report mainly consists of two parts: one is basic personal information of the patient, the other is the description of x-ray images, the diagnostic opinion of the doctor and the result of pathologic diagnosis [textual data comprises a diagnostic report corresponding to the diagnostic image].”
PanQiao and Zhou as modified by Jain are analogous arts as both are related to the analysis of image-text pairs. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the medical condition classification of PanQiao to the teachings of Zhou as modified by Jain to arrive at the present invention, in order to leverage machine learning to improve medical treatment, as stated in PanQiao, section I, “The efficient analysis of images and electronic health records (EHRs) is significant for improving reliability of laboratory results, which is used to help assessing disease risks and monitoring the treatment.”
Regarding claim 17:
Zhou teaches:
(bold only) “determine a classification of the one or more detected objects using a cross-attention encoder of the one or more neural networks and based on the one or more images and textual data corresponding to the one or more detected objects”: Zhou, section 2., paragraph 3, “For a given image, let fk(x, y) represent the activation of unit k in the last convolutional layer at spatial location (x, y). Then, for unit k, the result of performing global average pooling, Fk is Σx, y fk(x, y). Thus, for a given class c, the input to the softmax, Sc, is Σx, y wck Fk where wck is the weight corresponding to class c for unit k. Essentially, wck indicates the importance of Fk for class c [determine a classification of the one or more detected objects using … one or more neural networks and based on the one or more images … corresponding to the one or more detected objects].”
(bold only) “decode, using the decoder, an output of the cross-attention encoder to generate a saliency map, the saliency map indicating locations in the one or more images associated with the one or more detected objects, based on the classification”: Zhou, section 2, paragraph 1, “In this section, we describe the procedure for generating class activation maps (CAM) using global average pooling (GAP) in CNNs. A class activation map for a particular category indicates the discriminative image regions used by the CNN to identify that category (e.g., Fig. 3) [generate a saliency map, the saliency map indicating locations in the one or more images associated with the one or more detected objects, based on the classification].”
Zhou does not explicitly teach:
“One or more processors, comprising: circuitry to use one or more neural networks to infer a condition of one or more detected objects in one or more images using a decoder of the one or more neural networks”
(bold only) “determine a classification of the one or more detected objects using a cross-attention encoder of the one or more neural networks and based on the one or more images and textual data corresponding to the one or more detected objects”
(bold only) “decode, using the decoder, an output of the cross-attention encoder to generate a saliency map, the saliency map indicating locations in the one or more images associated with the one or more detected objects, based on the classification”
Jain teaches:
(bold only) “One or more processors, comprising: circuitry to use one or more neural networks to”: Jain, paragraph 0049, “The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller , a microcontroller , etc. ) and can be one processor or a plurality of processors that are operatively connected [one or more processors, comprising circuitry].”
(bold only) “determine a classification of the one or more detected objects using a cross-attention encoder of the one or more neural networks and based on the one or more images and textual data corresponding to the one or more detected objects”: Jain, paragraph 0021, “In some implementations , at least one of the frame-level encoder model or the segment-level encoder model can be a multimodal encoder based at least in part on associated text . For instance , in addition to encoding the video data and/or representations thereof, the encoder(s) (e.g., the lower-level encoder and/or the higher-level encoder) can be cross-modal encoders that additionally fuse the video data and/or representations thereof with associated text data , such as, for example, captioning data for the video and/or query data descriptive of a user query representing a user's search for videos and/or, more particularly , content depicted within the videos. For instance, in the encoder (s) , the input modality pairs can have cross-attention [cross-attention encoder], such as visual-caption/transcripts, visual-query [images and textual data], and/or transcript/query attention. In some implementations , the associated text can be encoded (e.g. , by a text encoder model, such as a text transformer).”
(bold only) “decode, using the decoder, an output of the cross-attention encoder to generate a saliency map, the saliency map indicating locations in the one or more images associated with the one or more detected objects, based on the classification”: Jain, paragraph 0070, “In some cases , the machine learned model (s) can be configured to perform a task that includes encoding input data for reliable and/or efficient transmission or storage (and/or corresponding decoding) [decode, using the decoder, an output of the cross-attention encoder].”
Jain and Zhou are analogous arts as they are both related to machine learning model design. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the cross-modal data encoding of Jain with the teachings of Zhou to arrive at the present invention, in order to combine data from multiple modalities to improve the contextual quality of data encoding, as stated in Jain, paragraph 0085, “Cross-modal transformer 534 then produces contextualized segment representations 540 based on cross-attention between the encoded segment representations 525 from video data encoder 532 and encoded associated text data from text encoder model 536.”
PanQiao teaches “use one or more neural networks to infer a condition of one or more detected objects in one or more images using a decoder of the one or more neural networks”: PanQiao, section I, “Character-based feature representation method is used to fully capture the complicated matching relations between image and sentence and are fully captured in our proposed f-CNN [using a decoder of the one or more neural networks]. We validate the effectiveness of f-CNNs on the classification tasks for breast cancer [a condition], and demonstrate that f-CNNs can achieve performances superior to the state-of-the-art approaches by letting image and the composed fragments of the sentence meet and interact at different levels”; PanQiao, section III. A., “Our task is to extract and mine the semantic interactions of radiology images and reports [e one or more neural networks to infer a condition of one or more objects in one or more images], and generate fusion feature vector with better performance. We will train our model on a training set of N images and N corresponding sentences that describe their content (Figure 1).”
PanQiao and Zhou as modified by Jain are analogous arts as both are related to the analysis of image-text pairs. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the medical condition classification of PanQiao to the teachings of Zhou as modified by Jain to arrive at the present invention, in order to leverage machine learning to improve medical treatment, as stated in PanQiao, section I, “The efficient analysis of images and electronic health records (EHRs) is significant for improving reliability of laboratory results, which is used to help assessing disease risks and monitoring the treatment.”
Regarding claim 18:
Zhou as modified by Jain and PanQiao teaches “The one or more processors of claim 17.”
Jain further teaches “wherein a first portion of the one or more neural networks is trained to encode features of the one or more images and a second portion of the one or more neural networks is trained to encode features of the textual data”: Jain, paragraph 0021, “In some implementations , at least one of the frame-level encoder model or the segment-level encoder model can be a multimodal encoder based at least in part on associated text. For instance , in addition to encoding the video data [a first portion of the one or more neural networks is trained to encode features of the one or more images] and/or representations thereof, the encoder(s) (e.g., the lower-level encoder and/or the higher-level encoder) can be cross-modal encoders that additionally fuse the video data and/or representations thereof with associated text data [a second portion of the one or more neural networks is trained to encode features of the textual data] , such as, for example, captioning data for the video and/or query data descriptive of a user query representing a user's search for videos and/or, more particularly , content depicted within the videos. For instance, in the encoder (s) , the input modality pairs can have cross-attention, such as visual-caption/transcripts, visual-query, and/or transcript/query attention. In some implementations , the associated text can be encoded (e.g. , by a text encoder model, such as a text transformer).”
Jain and Zhou are combinable are the rationale given under claim 17.
Regarding claim 19:
Zhou as modified by Jain and PanQiao teaches “The one or more processors of claim 18.”
Jain further teaches “wherein the first portion of the one or more neural networks, and the second portion of the one or more neural networks, encode their respective inputs to a common latent space”: Jain, paragraph 0021, “In some implementations, at least one of the frame-level encoder model or the segment-level encoder model can be a multimodal encoder based at least in part on associated text . For instance , in addition to encoding the video data and/or representations thereof, the encoder(s) (e.g., the lower-level encoder and/or the higher-level encoder) can be cross-modal encoders that additionally fuse the video data and/or representations thereof with associated text data [wherein the first portion of the one or more neural networks, and the second portion of the one or more neural networks, encode their respective inputs to a common latent space], such as, for example, captioning data for the video and/or query data descriptive of a user query representing a user's search for videos and/or, more particularly , content depicted within the videos. For instance, in the encoder (s), the input modality pairs can have cross-attention, such as visual-caption/transcripts, visual-query, and/or transcript/query attention. In some implementations , the associated text can be encoded (e.g. , by a text encoder model, such as a text transformer).”
Jain and Zhou are combinable are the rationale given under claim 17.
Regarding claim 20:
Zhou as modified by Jain and PanQiao teaches “The one or more processors of claim 17.”
Jain further teaches “wherein the one or more neural networks are is trained based, at least in part, on output of a cross-attention encoder using, as input to the cross-attention encoder, output of an image encoder and output of a language encoder”: Jain, paragraph 0021, “In some implementations , at least one of the frame-level encoder model or the segment-level encoder model can be a multimodal encoder based at least in part on associated text . For instance , in addition to encoding the video data and/or representations thereof [output of an image encoder], the encoder(s) (e.g., the lower-level encoder and/or the higher-level encoder) can be cross-modal encoders that additionally fuse the video data and/or representations thereof with associated text data , such as, for example, captioning data for the video and/or query data descriptive of a user query representing a user's search for videos and/or, more particularly , content depicted within the videos. For instance, in the encoder (s) , the input modality pairs can have cross-attention [a cross-attention encoder], such as visual-caption/transcripts, visual-query, and/or transcript/query attention. In some implementations , the associated text can be encoded (e.g. , by a text encoder model, such as a text transformer) [output of a language encoder]”; Jain, paragraph 0039, “The hierarchical video encoders as described herein can be jointly trained on both tasks in a multitask learning configuration [the one or more neural networks are is trained based, at least in part, on output of a cross-attention encoder]. The hierarchical ( e.g. , and cross - attentional ) encoders as described herein can be beneficial for these tasks , as the two tasks can require understanding semantics of a video at differing temporal resolutions , and the models described herein can model short - range and long - range video semantics.”
Jain and Zhou are combinable for the rationale given under claim 17.
Regarding claim 21:
Zhou as modified by Jain and PanQiao teaches “The one or more processors of claim 17.”
PanQiao further teaches “wherein the one or more images comprise diagnostic images and the textual data comprises diagnostic reports corresponding to the diagnostic images”: PanQiao, Section V. A., “The experimental datasets in this paper came from a major hospital in Shanghai, including medical documents of clinical breast cases and related mammography image [the one or more images comprise diagnostic images]. The document report mainly consists of two parts: one is basic personal information of the patient, the other is the description of x-ray images, the diagnostic opinion of the doctor and the result of pathologic diagnosis [textual data comprises diagnostic reports corresponding to the diagnostic images].”
PanQiao and Zhou as modified by Jain are combinable for the rationale given under claim 17.
Regarding claim 22:
Zhou as modified by Jain and PanQiao teaches “The one or more processors of claim 17.”
Zhou further teaches (bold only) “wherein the inferred condition comprises information indicative of an area of interest in the one or more images”: Zhou, section 2, paragraph 1, “In this section, we describe the procedure for generating class activation maps (CAM) using global average pooling (GAP) in CNNs. A class activation map for a particular category indicates the discriminative image regions used by the CNN to identify that category (e.g., Fig. 3) [wherein the inferred condition comprises information indicative of an area of interest in the one or more images].”
PanQiao further teaches (bold only) “wherein the inferred condition comprises information indicative of an area of interest in the one or more images”: PanQiao, section III. A., “Our task is to extract and mine the semantic interactions of radiology images and reports [inferred condition], and generate fusion feature vector with better performance. We will train our model on a training set of N images and N corresponding sentences that describe their content (Figure 1)”; PanQiao, section III. C. 1, “Moreover, as most convolutional models [25], we consider the convolution unit with a local ‘receptive field’ [an area of interest in the one or more images] and shared weights to adequately model the rich structures for word composition and intermodal interaction.”
PanQiao and Zhou as modified by Jain are combinable for the rationale given under claim 17.
Regarding claim 24:
Zhou teaches:
“A method, comprising: using one or more neural networks to”: Zhou, section 2, paragraph 1, “In this section, we describe the procedure for generating class activation maps (CAM) using global average pooling (GAP) in CNNs [A method, comprising: using one or more neural networks to]. A class activation map for a particular category indicates the discriminative image regions used by the CNN to identify that category (e.g., Fig. 3)
(bold only) “determine a classification of the condition using a cross-attention encoder of the one or more neural networks and based on the diagnostic image and textual data corresponding to the condition”: Zhou, section 2., paragraph 3, “For a given image, let fk(x, y) represent the activation of unit k in the last convolutional layer at spatial location (x, y). Then, for unit k, the result of performing global average pooling, Fk is Σx, y fk(x, y). Thus, for a given class c, the input to the softmax, Sc, is Σx, y wck Fk where wck is the weight corresponding to class c for unit k. Essentially, wck indicates the importance of Fk for class c [determine a classification … using … the one or more neural networks and based on the … image ].”
(bold only) “decode, using the decoder, an output of the cross-attention encoder to generate a saliency map, the saliency map indicating locations in the diagnostic image associated with the condition, based on the classification and a prediction of one or more portions of a set of diagnostic reports, wherein the prediction uses the diagnostic image and the textual data”: Zhou, section 2, paragraph 1, “In this section, we describe the procedure for generating class activation maps (CAM) using global average pooling (GAP) in CNNs. A class activation map for a particular category indicates the discriminative image regions used by the CNN to identify that category (e.g., Fig. 3) [generate a saliency map, the saliency map indicating locations in the … image…, based on the classification and a prediction of one or more portions of a set of … reports, wherein the prediction uses the image].”
Zhou does not explicitly teach:
“diagnose a condition depicted in a diagnostic image using a decoder of the one or more neural networks”
(bold only) “determine a classification of the condition using a cross-attention encoder of the one or more neural networks and based on the diagnostic image and textual data corresponding to the condition”
(bold only) “decode, using the decoder, an output of the cross-attention encoder to generate a saliency map, the saliency map indicating locations in the diagnostic image associated with the condition, based on the classification and a prediction of one or more portions of a set of diagnostic reports, wherein the prediction uses the diagnostic image and the textual data”
Jain teaches:
(bold only) “determine a classification of the condition using a cross-attention encoder of the one or more neural networks and based on the one or more images and textual data corresponding to the condition”: Jain, paragraph 0021, “In some implementations , at least one of the frame-level encoder model or the segment-level encoder model can be a multimodal encoder based at least in part on associated text . For instance , in addition to encoding the video data and/or representations thereof, the encoder(s) (e.g., the lower-level encoder and/or the higher-level encoder) can be cross-modal encoders that additionally fuse the video data and/or representations thereof with associated text data , such as, for example, captioning data for the video and/or query data descriptive of a user query representing a user's search for videos and/or, more particularly , content depicted within the videos. For instance, in the encoder (s) , the input modality pairs can have cross-attention [cross-attention encoder], such as visual-caption/transcripts, visual-query, and/or transcript/query attention. In some implementations , the associated text can be encoded (e.g. , by a text encoder model, such as a text transformer).”
(bold only) “decode, using the decoder, an output of the cross-attention encoder to generate a saliency map, the saliency map indicating locations in the diagnostic image associated with the condition, based on the classification and a prediction of one or more portions of a set of diagnostic reports, wherein the prediction uses the diagnostic image and the textual data”: Jain, paragraph 0070, “In some cases , the machine learned model (s) can be configured to perform a task that includes encoding input data for reliable and/or efficient transmission or storage (and/or corresponding decoding) [decode, using the decoder, an output of the cross-attention encoder].”
Jain and Zhou are analogous arts as they are both related to machine learning model design. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the cross-modal data encoding of Jain with the teachings of Zhou to arrive at the present invention, in order to combine data from multiple modalities to improve the contextual quality of data encoding, as stated in Jain, paragraph 0085, “Cross-modal transformer 534 then produces contextualized segment representations 540 based on cross-attention between the encoded segment representations 525 from video data encoder 532 and encoded associated text data from text encoder model 536.”
PanQiao teaches “diagnose a condition depicted in a diagnostic image using a decoder of the one or more neural networks,” (bold only) “determine a classification of the condition using a cross-attention encoder of the one or more neural networks and based on the diagnostic image and textual data corresponding to the condition,” and (bold only) “decode, using the decoder, an output of the cross-attention encoder to generate a saliency map, the saliency map indicating locations in the diagnostic image associated with the condition, based on the classification and a prediction of one or more portions of a set of diagnostic reports, wherein the prediction uses the diagnostic image and the textual data”: PanQiao, section 1, paragraph 6, “In this paper, we propose a novel fusion convolutional neural network (f-CNN) framework [using a decoder of the one or more neural network] to make full use of the image and text resources. Trained on a set of image and sentence pairs, the proposed f-CNNs are able to mine the semantic interactions of radiology images [diagnostic image] and reports [textual data corresponding to the condition] and mutually enhance feature learning process [diagnose a condition depicted in a diagnostic image]”; PanQiao, Section V. A., “The experimental datasets in this paper came from a major hospital in Shanghai, including medical documents of clinical breast cases and related mammography image [associated with the condition][of the condition]. The document report mainly consists of two parts: one is basic personal information of the patient, the other is the description of x-ray images, the diagnostic opinion of the doctor and the result of pathologic diagnosis [portions of a set of diagnostic reports].”
PanQiao and Zhou as modified by Jain are analogous arts as both are related to the analysis of image-text pairs. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the medical condition classification of PanQiao to the teachings of Zhou as modified by Jain to arrive at the present invention, in order to leverage machine learning to improve medical treatment, as stated in PanQiao, section I, “The efficient analysis of images and electronic health records (EHRs) is significant for improving reliability of laboratory results, which is used to help assessing disease risks and monitoring the treatment.”
Regarding claim 25:
Zhou as modified by Jain and PanQiao teaches “The method of claim 24.”
Jain further teaches:
“wherein a first portion of the one or more neural networks is trained in parallel with a second portion of the one or more neural networks“: Jain, paragraph 0039, “The hierarchical video encoders as described herein can be jointly trained on both tasks in a multitask learning configuration [a first portion and a second portion of the one or more neural networks are trained in parallel]. The hierarchical ( e.g., and cross - attentional ) encoders as described herein can be beneficial for these tasks , as the two tasks can require understanding semantics of a video at differing temporal resolutions , and the models described herein can model short - range and long - range video semantics.”
(bold only) “and wherein the second portion of the one or more neural networks is trained to encode features of the set of diagnostic reports”: Jain, paragraph 0021, “In some implementations, at least one of the frame-level encoder model or the segment-level encoder model can be a multimodal encoder based at least in part on associated text . For instance , in addition to encoding the video data and/or representations thereof, the encoder(s) (e.g., the lower-level encoder and/or the higher-level encoder) can be cross-modal encoders that additionally fuse the video data and/or representations thereof with associated text data [the second portion of the one or more neural networks is trained to encode features of the set of … reports], such as, for example, captioning data for the video and/or query data descriptive of a user query representing a user's search for videos and/or, more particularly , content depicted within the videos. For instance, in the encoder (s), the input modality pairs can have cross-attention, such as visual-caption/transcripts, visual-query, and/or transcript/query attention. In some implementations , the associated text can be encoded (e.g. , by a text encoder model, such as a text transformer)”;
Jain and Zhou are combinable for the rationale given under claim 24.
PanQiao teaches (bold only) “and wherein the second portion of the one or more neural networks is trained to encode features of the set of diagnostic reports”: PanQiao, Section V. A., “The experimental datasets in this paper came from a major hospital in Shanghai, including medical documents of clinical breast cases and related mammography image. The document report mainly consists of two parts: one is basic personal information of the patient, the other is the description of x-ray images, the diagnostic opinion of the doctor and the result of pathologic diagnosis [the set of diagnostic reports].”
PanQiao and Zhou as modified by Jain are combinable for the rationale given under claim 24.
Regarding claim 26:
Zhou as modified by Jain and PanQiao teaches “The method of claim 25.”
Jain further teaches (bold only) “wherein the first and second portions of the one or more neural networks are trained to encode features of the diagnostic image and the textual data to a shared latent space”: Jain, paragraph 0021, “In some implementations, at least one of the frame-level encoder model or the segment-level encoder model can be a multimodal encoder based at least in part on associated text . For instance , in addition to encoding the video data and/or representations thereof, the encoder(s) (e.g., the lower-level encoder and/or the higher-level encoder) can be cross-modal encoders that additionally fuse the video data and/or representations thereof with associated text data , such as, for example, captioning data for the video and/or query data descriptive of a user query representing a user's search for videos and/or, more particularly , content depicted within the videos. For instance, in the encoder (s), the input modality pairs can have cross-attention, such as visual-caption/transcripts, visual-query [encode features of the diagnostic image and the textual data to a shared latent space], and/or transcript/query attention. In some implementations , the associated text can be encoded (e.g. , by a text encoder model, such as a text transformer)”; Jain, paragraph 0039, “The hierarchical video encoders as described herein can be jointly trained on both tasks in a multitask learning configuration [a first portion and a second portion of the one or more neural networks are trained in parallel]. The hierarchical ( e.g. , and cross - attentional ) encoders as described herein can be beneficial for these tasks , as the two tasks can require understanding semantics of a video at differing temporal resolutions , and the models described herein can model short - range and long - range video semantics.”
Jain and Zhou are combinable are the rationale given under claim 24.
PanQiao further teaches (bold only) “wherein the first and second portions of the one or more neural networks are trained to encode features of the diagnostic image and the textual data to a shared latent space”: PanQiao, Section V. A., “The experimental datasets in this paper came from a major hospital in Shanghai, including medical documents of clinical breast cases and related mammography image. The document report mainly consists of two parts: one is basic personal information of the patient, the other is the description of x-ray images [the diagnostic image], the diagnostic opinion of the doctor and the result of pathologic diagnosis.”
PanQiao and Zhou as modified by Jain are combinable for the rationale given under claim 24.
Regarding claim 28:
Zhou as modified by Jain and PanQiao teaches “The method of claim 24.”
Jain further teaches (bold only) “training a language encoder of the one or more neural networks to encode features of the diagnostic reports to a latent space shared with output of an image encoder”: Jain, paragraph 0021, “In some implementations, at least one of the frame-level encoder model or the segment-level encoder model can be a multimodal encoder based at least in part on associated text . For instance , in addition to encoding the video data and/or representations thereof, the encoder(s) (e.g., the lower-level encoder and/or the higher-level encoder) can be cross-modal encoders that additionally fuse the video data and/or representations thereof with associated text data , such as, for example, captioning data for the video and/or query data descriptive of a user query representing a user's search for videos and/or, more particularly , content depicted within the videos. For instance, in the encoder (s), the input modality pairs can have cross-attention, such as visual-caption/transcripts, visual-query [encode features of the … reports to latent space shared with output of an image encoder], and/or transcript/query attention. In some implementations , the associated text can be encoded (e.g. , by a text encoder model, such as a text transformer)”; Jain, paragraph 0039, “The hierarchical video encoders as described herein can be jointly trained on both tasks in a multitask learning configuration. The hierarchical ( e.g. , and cross - attentional ) encoders as described herein can be beneficial for these tasks , as the two tasks can require understanding semantics of a video at differing temporal resolutions , and the models described herein can model short - range and long - range video semantics.”
Jain and Zhou are combinable are the rationale given under claim 24.
PanQiao further teaches (bold only) “training a language encoder of the one or more neural networks to encode features of the diagnostic reports to a latent space shared with output of an image encoder”: PanQiao, Section V. A., “The experimental datasets in this paper came from a major hospital in Shanghai, including medical documents of clinical breast cases and related mammography image. The document report mainly consists of two parts: one is basic personal information of the patient, the other is the description of x-ray images, the diagnostic opinion of the doctor and the result of pathologic diagnosis [the diagnostic reports].”
PanQiao and Zhou as modified by Jain are combinable for the rationale given under claim 24.
Regarding claim 29:
Zhou as modified by Jain and PanQiao teaches “The method of claim 24.”
PanQiao further teaches “decoding output of an encoder to generate information summarizing the condition”: PanQiao, section III. A., “Our task is to extract and mine the semantic interactions of radiology images and reports, and generate fusion feature vector with better performance. We will train our model on a training set of N images and N corresponding sentences that describe their content (Figure 1). Given this set of correspondences, we train the weights of a neural network to output a high score when a compatible image-sentence pair is fed through the network, and low score otherwise [decoding output of an encoder to generate information summarizing the condition].”
PanQiao and Zhou as modified by Jain are combinable for the rationale given under claim 24.
Regarding claim 30:
Zhou as modified by Jain and PanQiao teaches “The method of claim 24.”
Zhou further teaches (bold only) “wherein the one or more neural networks comprises a decoder to generate information indicative of a region in the diagnostic image that depicts the condition”: Zhou, section 2, paragraph 1, “In this section, we describe the procedure for generating class activation maps (CAM) using global average pooling (GAP) in CNNs. A class activation map for a particular category indicates the discriminative image regions used by the CNN to identify that category (e.g., Fig. 3) [wherein the one or more neural networks comprises a decoder to generate information indicative of a region in the … image].”
PanQiao further teaches (bold only) “wherein the one or more neural networks comprises a decoder to generate information indicative of a region in the diagnostic image that depicts the condition”: PanQiao teaches (bold only) “training a language encoder of the one or more neural networks to encode features of the diagnostic reports to a latent space shared with output of an image encoder”: PanQiao, Section V. A., “The experimental datasets in this paper came from a major hospital in Shanghai, including medical documents of clinical breast cases and related mammography image [the diagnostic image that depicts the condition]. The document report mainly consists of two parts: one is basic personal information of the patient, the other is the description of x-ray images, the diagnostic opinion of the doctor and the result of pathologic diagnosis.”
PanQiao and Zhou as modified by Jain are combinable for the rationale given under claim 24.
Regarding claim 31:
Zhou as modified by Jain and PanQiao teaches “The method of claim 24.”
Zhou further teaches (bold only) “wherein diagnoses of the condition comprises identifying one or more categories of conditions determined, by the one or more neural networks, to be associated with a region of the diagnostic image”: Zhou, section 2, paragraph 1, “In this section, we describe the procedure for generating class activation maps (CAM) using global average pooling (GAP) in CNNs. A class activation map for a particular category indicates the discriminative image regions used by the CNN to identify that category (e.g., Fig. 3) [determined, by the one or more neural networks, to be associated with a region of the … image].”
PanQiao further teaches (bold only) “wherein diagnoses of the condition comprises identifying one or more categories of conditions determined, by the one or more neural networks, to be associated with a region of the diagnostic image”: PanQiao, section III. A., “Our task is to extract and mine the semantic interactions of radiology images and reports, and generate fusion feature vector with better performance. We will train our model on a training set of N images and N corresponding sentences that describe their content (Figure 1). Given this set of correspondences, we train the weights of a neural network to output a high score when a compatible image-sentence pair is fed through the network, and low score otherwise. Once the training is complete, the evaluation will score all image-sentence pairs, sort images/sentences in order of decreasing score and record the location of a ground truth result in the list [identifying one or more categories of conditions determined, by the one or more neural networks, to be associated with a region of the diagnostic image].”
PanQiao and Zhou as modified by Jain are combinable for the rationale given under claim 24.
Claims 23 and 27 rejected under 35 U.S.C. 103 as being unpatentable Zhou as modified by Jain and PanQiao in view of Wei.
Regarding claim 23:
Zhou as modified by Jain and PanQiao teaches “The one or more processors of claim 17.”
Jain further teaches “wherein a first portion of the one or more neural networks is trained to encode features of the one or more images and a second portion of the one or more neural networks is trained to encode features of the textual data”: Jain, paragraph 0021, “In some implementations , at least one of the frame-level encoder model or the segment-level encoder model can be a multimodal encoder based at least in part on associated text. For instance , in addition to encoding the video data [a first portion of the one or more neural networks is trained to encode features of the one or more images] and/or representations thereof, the encoder(s) (e.g., the lower-level encoder and/or the higher-level encoder) can be cross-modal encoders that additionally fuse the video data and/or representations thereof with associated text data [a second portion of the one or more neural networks is trained to encode features of the textual data] , such as, for example, captioning data for the video and/or query data descriptive of a user query representing a user's search for videos and/or, more particularly , content depicted within the videos. For instance, in the encoder (s) , the input modality pairs can have cross-attention, such as visual-caption/transcripts, visual-query, and/or transcript/query attention. In some implementations , the associated text can be encoded (e.g. , by a text encoder model, such as a text transformer).”
Jain and Zhou are combinable are the rationale given under claim 17.
Zhou as modified by Jain and PanQiao does not explicitly teach “and wherein the first portion of the neural network one or more neural networks, after training, is capable of inferring the information condition independently of the second portion.”
Wei teaches “and wherein the first portion of the neural network one or more neural networks, after training, is capable of inferring the information condition independently of the second portion.”: Wei, Fig. 2,
PNG
media_image5.png
383
787
media_image5.png
Greyscale
[showing that the first portion of the neural network, top, through its self-attention module, can infer the information independently from the combination of first and second portions used in the cross-attention module]; Wei, section 3.1, “As shown in Figure 2, we get two pairs of embeddings for the given image-sentence pair (i0, c0) and (i1, c1), which are used for image and sentence matching.”
Wei and Zhou are analogous arts as they are both related to image analysis. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the multi-modal cross-attention of Wei with the teachings of Zhou to arrive at the present invention, in order to improve model performance, as stated in Wei, Abstract, “In the proposed MM-CA, we design a novel cross-attention mechanism, which is able to exploit not only the intra-modality relationship within each modality, but also the inter-modality relationship between image regions and sentence words to complement and enhance each other for image and sentence matching. Extensive experimental results on two standard benchmarks including Flickr30K and MS-COCO demonstrate that the proposed model performs favorably against state-of-the-art image and sentence matching methods.”
Regarding claim 27:
Zhou as modified by Jain and PanQiao teaches “The method of claim 24.”
Zhou as modified by Jain and PanQiao does not explicitly teach “providing, as input to a cross-attention encoder, a query input comprising output from a language encoder, and key and value input comprising output from an image encode.”
Wei teaches “providing, as input to a cross-attention encoder, a query input comprising output from a language encoder, and key and value input comprising output from an image encoder”: Wei, section 3.3, “for image I with the fine grained representation R = {r1, r2, ..., rk} [showing that R is an output representation from an image encoder] […] The BERT consists of multiple Transformer units, and its output E = {e1, e2, ..., en} naturally includes the intra-modality information [showing that E includes an output representation from a language encoder]”; Wei, section 3.4, “In this section, we introduce how to model both the inter-modality and intra-modality relationships in a unified model with our Cross-Attention Module [a cross-attention encoder] […] Here, the query [a query input comprising output from a language encoder], key and value [key and value input comprising output from an image encoder] for the fragments are formed with the following equations:
PNG
media_image4.png
217
404
media_image4.png
Greyscale
”
Wei and Zhou are analogous arts as they are both related to image analysis. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the multi-modal cross-attention of Wei with the teachings of Zhou to arrive at the present invention, in order to improve model performance, as stated in Wei, Abstract, “In the proposed MM-CA, we design a novel cross-attention mechanism, which is able to exploit not only the intra-modality relationship within each modality, but also the inter-modality relationship between image regions and sentence words to complement and enhance each other for image and sentence matching. Extensive experimental results on two standard benchmarks including Flickr30K and MS-COCO demonstrate that the proposed model performs favorably against state-of-the-art image and sentence matching methods.”
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Min et al., “TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detection,” 2019, arXiv:1908.05786v1, discloses a method of constructing a saliency map for images using an encoder-decoder network.
Evangelopoulos et al., “Multimodal Saliency and Fusion for Movie Summarization Based on Aural, Visual, and Textual Attention,” 2013, IEEE Transactions on Multimedia, Vol. 15, No. 7, discloses an attention-based method of generating saliency across multiple modalities including images and text.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VINCENT SPRAUL whose telephone number is (703) 756-1511. The examiner can normally be reached M-F 9:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MICHAEL HUNTLEY can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/VAS/ Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129
Prosecution Insights