DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Reissue Applications
This application seeks to reissue US Patent No. 12,300,007 (“the ‘007 patent”). This application is filed to add an inventor that the applicant states was inadvertently omitted from the ‘007 patent. Inventorship error is correctable by reissue under 35 U.S.C. § 251. See MPEP 1412.04 II. Claims 1-20 of the ‘007 patent have not been amended and are currently pending.
For reissue applications filed before September 16, 2012, all references to 35 U.S.C. 251 and 37 CFR 1.172, 1.175, and 3.73 are to the law and rules in effect on September 15, 2012. Where specifically designated, these are “pre-AIA ” provisions.
For reissue applications filed on or after September 16, 2012, all references to 35 U.S.C. 251 and 37 CFR 1.172, 1.175, and 3.73 are to the current provisions.
Applicant is reminded of the continuing obligation under 37 CFR 1.178(b), to timely apprise the Office of any prior or concurrent proceeding in which Patent No. 12,300,007 is or was involved. These proceedings would include any trial before the Patent Trial and Appeal Board, interferences, reissues, reexaminations, supplemental examinations, and litigation.
Applicant is further reminded of the continuing obligation under 37 CFR 1.56, to timely apprise the Office of any information which is material to patentability of the claims under consideration in this reissue application.
These obligations rest with each individual associated with the filing and prosecution of this application for reissue. See also MPEP §§ 1404, 1442.01 and 1442.04.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Claim 1 is the narrowest independent claim and will be analyzed below. Claims 4 and 13 recite similar inventions, but are broader in scope. The analysis of claim 1 also applies to these claims.
Claim 1 includes the following limitations that fall under the “mental processes” grouping of abstract ideas:
“receiving first image data” and “generating second image data” by cropping the first image data according to an aspect ratio of a display
“generating … first text data representing a first description of first content of the first image data” as well as “second text data representing a second description of second content of the second image data”
causing “the second image data to be displayed” based on a comparison of the similarity of representations of the text data
These limitations describe that an input image is cropped, and text descriptions are generated that describe both images. These text descriptions are compared, and if they are sufficiently similar, the cropped image is displayed. This could reasonably be performed by a human by observing the two images, and writing and comparing captions describing the images. This only involves the steps of observation, evaluation, and judgment. See MPEP 2106.04(a)(2)(III). Therefore these limitations in claim 1 amount to an abstract idea.
Claim 1 also includes limitations that fall under “mathematical concepts” grouping of abstract ideas. These limitations are: generating “vector representations” of the text data; and “determining a first cosine similarity score” between representations of the text descriptions. The specification describes an example of a “vector representation” as a mathematical or a “numerical representation that quantifies the linguistic and/or semantic information” of text, which is “learned based on their distributional properties in large samples of language data (e.g., the training data for the natural language encoder.)” ‘007 patent at 9:28-62. A person skilled in this art would understand this to involve purely mathematical operations on the text data, based on a trained dataset. As such, this limitation involves an abstract idea. Similarly, calculating a “cosine similarity score” is likewise an abstract idea is only a mathematical calculation on a numerical value (i.e., the vector representations), thus is also an abstract idea. See MPEP 2106.04(a)(2)(I)(C).
Therefore claim 1 recites a combination of “mental processes” and “mathematical concepts” abstract ideas. Claims 4 and 13 recite similar combinations of abstract ideas.
This judicial exception is not integrated into a practical application. Certain limitations in claim 1 fall outside of the above-described abstract idea. These are: performing the method on a computing device and causing the image to be displayed on a display device; using a “captioning model” to generate the text descriptions; and using an “encoder” to generate the vector representations.
For a limitation to integrate the judicial exception into a practical application requires more than merely using generic computing elements to implement the abstract idea. See MPEP 2106.04(d)(I). Therefore the limitations that require use of a general purpose computing devices and display devices are insufficient to render the claims eligible under §101.
The claims also require using a “captioning model” to generate text descriptions. The specification of the ’007 shows that these steps involve nothing more than invoking generic and well-known computer algorithms. The specification describes that “any image captioning model may be used”, and gives examples of neural networks and other known models. See, e.g., the ‘007 patent at 6:48-53 and 7:52-8:31. Because the captioning model is an existing algorithm that is used in its ordinary capacity, this limitation amounts to using a generic computing element as a tool to perform an existing process. See MPEP 2106.04(d)(I) and 2106.05(f). This limitation therefore does not integrate the abstract idea into a practical application.
The claim limitation relating to using an “encoder” to generate the vector representations also merely invokes a generic tool to produce a mathematical representation of text. The ‘007 patent specification describes that “any desired natural language encoder (e.g., Word2vec, CLIP, etc.)” can be used to generate these vectors. See ‘007 patent at 9:28-62. Similar to above, this limitation in claim 1 recites the use of a known algorithm, in its ordinary capacity, as a tool to perform an existing process. MPEP 2106.05(f). The claims include no limitations describing a novel manner in which the vector representations are generated. This limitation also fails to integrate the abstract idea into a practical application.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. For reasons that are similar to the analysis given above, the limitations that fall outside of the abstract ideas amount to nothing more than well understood, routine, and conventional in the field. The “captioning model” and “vector representations” limitations are described in the specification as known in the prior art. See above. Similarly, the POSITA would recognize that the generic computing elements that are recited in the claims are well known. Therefore the claims include no limitations outside of the abstract idea that amount to significantly more than the abstract idea. See MPEP 2106.05(d).
The dependent claims all recite further abstract ideas or merely invoke known tools in their ordinary capacity. For example, claim 2 requires the use of the SPICE model to generate graph data. The POSITA would understand SPICE to be a standard computing tool, and the claim only uses it in a known manner to generate the graph data. Therefore this claim does not integrate the abstract idea into a practical application. See MPEP 2106.05(f). Claim 3 further relates to generating vector representations and determining cosine similarity scores, which is ineligible for reasons similar to reasons given above with respect to claim 1. Claims 5 and 6 recite comparing sets of words in the two graphs, which is a further mental process abstract idea.
Claims 10 and 20 relate to using two different1 types of encoders on the first and second text descriptions to determine similarity between images in two different ways. These similarity scores are then input into a neural network to generate an overall similarity score between the uncropped and the cropped image. This only entails repeating the mental processes that are recited in claim 4. While the repeated steps use a different type of analysis (that is, a different encoder), the steps still fall under the same abstract ideas as they do in claim 4. Outside of this abstract idea are the claim limitations requiring inputting data into a neural network to produce a result. Similar to the analysis above, this only uses a generic computing function in its standard capacity to produce an output. Therefore this does not amount to an integration of the abstract idea into a practical application, nor does it amount to significantly more.
The remaining claims recite further observation and evaluation, or mathematical operations being performed based on this information, and as such do not amount to a practical application of, or significantly more than, the abstract ideas.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4, 11-13, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang, US 20210056663 in view of Gonsalves, US 20240054748.
Claim 1: Zhang discloses a computer-implemented method, comprising:
receiving first image data representing a first image (Image data representing a first, full-size image is received by a crop generation system. ¶25.);
generating second image data representing a portion of the first image that is generated by cropping the first image data according to a first aspect ratio (A number of crop candidates is generated by cropping the first image data. ¶26. The cropping is performed to generate cropped images that have a particular aspect ratio. ¶¶ 26 and 42. Based on an evaluation of the cropped image to the original image, one of the crop candidates is selected as the final output image, i.e. second image data. ¶30);
generating, based at least in part on a similarity score, first computer- executable instructions to cause the second image data to be displayed (Each crop candidate is evaluated based on a comparison with the source image, and how well the crop candidate preserves visual content and composition of the original image. ¶¶18 and 29. The crop candidate with the highest evaluation score is selected and is displayed. ¶30.).
Zhang fails to disclose that the crop criteria relates to an aspect ratio of a first display of a target device, and displaying the second image data on the first display of the target device.
In addition, while there are various disclosed methods by which the crop candidates are scored (see Zhang ¶ 30), Zhang does not determine similarity in the same way that is recited in this claim. Therefore Zhang fails to disclose the following limitations:
generating, by inputting the first image data into an image captioning model, first text data representing a first description of first content of the first image data;
generating, using a first encoder, a first vector representation of the first text data;
generating, by inputting the second image data into the image captioning model, second text data representing a second description of second content of the second image data;
generating, using the first encoder, a second vector representation of the second text data;
determining a first cosine similarity score between the first vector representation and the second vector representation
Gonsalves discloses:
cropping a source image into second image data, wherein cropping is performed according to an aspect ratio of a first display of a target device, and displaying the second image data on the first display of the target device (An image is cropped to change the aspect ratio to fit a target display device. ¶¶ 5 and 30);
generating, by inputting the first image data into an image captioning model, first text data representing a first description of first content of the first image data; generating, using a first encoder, a first vector representation of the first text data (Text captions describing the content of the image are generated. See Gonsalves ¶¶20-21, and “Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks,” by Li, X. et al., (incorporated by reference into Gonsalves), page 7 and 11. After the captions are generated, the image and text are encoded into a vector representation. Gonsalves ¶¶20-21.);
generating, by inputting the second image data into the image captioning model, second text data representing a second description of second content of the second image data; generating, using the first encoder, a second vector representation of the second text data (The above-described process to generate captions and vector encodings is also used for the cropped images. Gonsalves ¶22.);
determining a first cosine similarity score between the first vector representation and the second vector representation (The encoded vector representations (i.e., the embeddings) of the cropped image and the source image are compared to determine similarity. Gonsalves ¶¶ 21-22. Similarity is determined as a cosine metric. Id. See also “Term-weighting Approaches in Automatic Text Retrieval” by G. Salton and C. Buckley., incorporated by reference by Gonsalves.).
It would have been obvious to a skilled artisan before the effective filing date of the claimed invention to modify Zhang with teachings in Gonsalves. The POSITA would have recognized that Zhang would be improved by scoring the candidate crop images based on the routines described in Gonsalves, which would enable the system to “find important objects within an image so that cropping could be performed more efficiently.” Gonsalves ¶3 (internal omissions removed).
Claim 4 is rendered obvious by the Zhang-Gonsalves combination for reasons given above in the rejection of claim 1. This combination discloses a method comprising:
receiving first image data representing a first image; receiving second image data representing a second image comprising a first subset of pixels of the first image data (Zhang ¶¶ 25-26. See above.);
generating, using an image captioning model executed by at least one computing device, first text data describing the first image; generating, using the image captioning model, second text data describing the second image (Gonsalves ¶¶ 20-22. See above.);
generating first data representing the first text data; generating second data representing the second text data (Gonsalves ¶¶ 20-22. See above.);
determining a third data representing a degree of similarity between the first data and the second data (Gonsalves ¶¶ 21-22.); and
generating first computer-executable instructions configured to cause the second image data to be displayed on a display based at least in part on the third data (Zhang ¶¶ 18 and 29-30. See above.).
Claim 11: Zhang-Gonsalves renders obvious:
receiving a first input describing a first aspect ratio of an output display (The desired aspect ratio of the cropped image is known, thus this ratio is received. Gonsalves ¶¶ 30 and 32.);
generating a plurality of cropped images from the first image data by iterating a window of the first aspect ratio over a plurality of positions overlaying the first image data, wherein each position of the plurality of positions corresponds to one of the plurality of cropped images (A candidate pool of cropped images is produced. Zhang ¶16. In a Zhang-Gonsalves combination, this candidate pool will be of the desired output aspect ratio.);
generating, for a first cropped image of the plurality of cropped images, a first score representing a first degree of similarity between the first cropped image and the first image; generating, for a second cropped image of the plurality of cropped images, a second score representing a second degree of similarity between the second cropped image and the first image; and selecting the first cropped image from among the plurality of cropped images based on the first score and the second score (The multiple crop candidates are scored for similarity to the input image, and a first image is selected based on this score. Zhang ¶¶ 18 and 29-30.).
Claim 12: Zhang-Gonsalves discloses that the third data represents a similarity between first content of the first image and second content of the second image (Zhang ¶29).
Claim 13: see rejection of claim 4. Zhang-Gonsalves further discloses a system comprising: at least one processor; and at least one non-transitory computer-readable memory storing instructions that, when executed by the at least one processor, are effective to program the at least one processor to perform the method of claim 4 (see e.g. Zhang Fig. 7 and its description.).
Claim 17: see rejection of claim 11.
Claims 2, 3, 5, 7, 9, 14, 16, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang, Gonsalves, and Anderson et al., “SPICE: Semantic Propositional Image Caption Evaluation” (arXiv:1607.08822, publicly available on arXiv on 07/29/2016).
Claim 2: the Zhang-Gonsalves combination does not disclose generating first graph data representing the first text data using a semantic propositional image caption evaluation (SPICE) model, wherein a node in the first graph data represents a word of the first text data; and generating second graph data representing the second text data using SPICE.
But Anderson discloses generating first graph data representing the first text data using a semantic propositional image caption evaluation (SPICE) model, wherein a node in the first graph data represents a word of the first text data (Anderson §§ 2.2-3.1, pages 4-7).
It would have been obvious to a skilled artisan before the effective filing date of the claimed invention to modify Zhang-Gonsalves with teachings found in Anderson, the rationale being to produce improved image captions. When modified in this manner, the POSITA would have found it likewise obvious to have the method include generating second graph data representing the second text data using SPICE, as Zhang-Gonsalves involves comparing first and second images using first and second sets of captions (see rejection of claim 1).
Claim 3: the Zhang-Gonsalves-Anderson combination discloses:
generating, by the first encoder, the first vector representation at least in part by generating a first embedding (Gonsalves ¶¶20-22) for a first node in the first graph data, the first node representing a first word (Anderson pg. 4-7);
generating, by the first encoder, the second vector representation at least in part by generating a second embedding for a second node in the second graph data, the second node representing a second word (Gonsalves ¶¶20-22 and Anderson pg. 4-7. See also the rejection of claim 2, which described why generating multiple graphs for the multiple images would have been the natural result of modifying Zhang-Gonsalves with Anderson); and
determining the first cosine similarity score based at least in part by determining a cosine similarity between the first embedding and the second embedding (Gonsalves ¶¶ 21-22.).
Claims 5 and 14: see rejection of claim 2.
Claims 7 and 16: see rejection of claim 3.
Claims 9 and 19: the Zhang-Gonsalves-Anderson combination renders obvious the steps of: receiving third image data representing a third image comprising a second subset of pixels of the first image data different from the first subset; generating, using the image captioning model, third text data describing the third image data; generating, using a first encoder and the third text data, fourth data representing the third text data; determining fifth data representing a degree of similarity between the first data and the fourth data; and selecting the second image data for output based on a comparison of the third data and the fifth data (Zhang discloses comparing multiple candidate cropped images to an input image to determine saliency scores, and comparing these scores to select a candidate. See Zhang Abstract. When Zhang is modified using teachings found in Gonsalves and Anderson, the invention that is recited in claim 9 is rendered obvious for reasons discussed in the rejection of claims 1, 2, and 4, above).
Claims 6 and 15 rejected under 35 U.S.C. 103 as being unpatentable over Zhang, Gonsalves, and Anderson in view of Xie, US 20230394855.
Claims 6 and 15: the Zhang-Gonsalves-Anderson combination fails to disclose: determining, using the first graph data and the second graph data, a first set of words present in the first graph data that are also present in the second graph data; and generating the third data based at least in part on the first set of words.
Xie discloses determining, using first graph data and second graph data, a first set of words present in the first graph data that are also present in the second graph data; and generating third data based at least in part on the first set of words (Two sets of SPICE graphs are extracted and are compared to produce a comparison score. ¶¶ 29-30 and 46.).
It would have been obvious to a skilled artisan before the effective filing date of the claimed invention to modify Zhang-Gonsalves-Anderson with teachings found in Xie, the rationale being to produce improved comparison scores for the two images. See Xie ¶29. While Xie relates to determining similarity between an image and a text description, the POSITA would have been motivated by this disclosure to compare the caption graphs of Zhang-Gonsalves-Anderson using this method, as this would have produced a more accurate determination of the similarity of the images.
Claims 8 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang, Gonsalves, and Anderson in view of Gray, US 20100309225.
Claims 8 and 18: the Zhang-Gonsalves-Anderson combination fails to disclose: determining first keypoints in the first image data using a keypoint detection model; determining second keypoints in the second image data using the keypoint detection model; determining a ratio of a number of the second keypoints to a number of the first keypoints; and determining the third data based at least in part on the ratio.
Gray discloses: determining first keypoints in first image data using a keypoint detection model; determining second keypoints in second image data using the keypoint detection model; determining a ratio of a number of the second keypoints to a number of the first keypoints; and determining third data based at least in part on the ratio (A candidate image is compared with a query image to determine similarity score based on the number of matching keypoints. ¶31.).
It would have been obvious to a skilled artisan before the effective filing date of the claimed invention to modify Zhang-Gonsalves-Anderson with teachings found in Gray, the rationale being to produce improved comparison scores for the two images.
Claims 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang-Gonsalves-Anderson in view of Gandhi, US 20230394391.
Claims 10 and 20: the Zhang-Gonsalves-Anderson combination discloses that the first data and the second data are generated using a first encoder (see rejection of claim 1).
Zhang-Gonsalves-Anderson combination does not disclose:
generating, using a second encoder executed by the at least one computing device, fourth data representing the first text data; generating, using the second encoder and the second text data, fifth data representing the second text data; determining sixth data representing a degree of similarity between the fourth data and the fifth data; inputting the third data and the sixth data into a neural network; and generating, by the neural network, output data indicating a semantic similarity between the first image data and the second image data.
However, Gandhi discloses using a first and second encoder on first and second extracted text to generate data indicative of the similarity between the first and second text (Different word embedding models are used on two sets of text to produce a similarity score. ¶¶ 13-15 and 30.).
This teaching in Gandi would have suggested to the POSITA to modify Zhang-Gonsalves-Anderson to arrive at the invention that is recited in claim 10. Gandi provides clear motivation for using multiple word embedding models when determining a similarity score between two sets of text: doing so produces more accurate similarity scores. See Gandhi ¶30. When modifying Zhang-Gonsalves-Anderson combination in this manner, the POSITA would have found it obvious to produce the claimed fourth, fifth, and sixth data using a different word embedding model than was used to generate the first, second, and third data, as this would have improved accuracy.
Zhang-Gonsalves-Anderson-Gandhi does not disclose inputting the third data and the sixth data into a neural network; and generating, by the neural network, output data indicating a semantic similarity between the first image data and the second image data. However, official notice is taken that this was well known in the art before the effective filing date of the claimed invention. For example, it was widely practiced to use a neural network to analyze various items of input that describe the same data, and to output a result of that analysis. Therefore it would have been obvious to the POSITA to modify Zhang-Gonsalves-Anderson-Gandhi to do this, the rationale being to produce a more accurate final similarity score relating to the two images. See also Gandhi ¶41, which would have provided further suggestion to modify Zhang-Gonsalves-Anderson-Gandhi to use a neural network in determining the semantic similarity score.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT J HANCE whose telephone number is (571)270-5319. The examiner can normally be reached M-F 11:00am-7:00pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Fuelling can be reached at (571) 270-1367. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROBERT J HANCE/Reexamination Specialist, Art Unit 3992
/CHARLES R CRAVER/Reexamination Specialist, Art Unit 3992
/M.F/Supervisory Patent Examiner, Art Unit 3992
1 The claims do not explicitly recite that the “second encoder” is different from the first. However, the POSITA, upon reading the specification, would conclude that the BRI of the claim is limited to this. See e.g. the ‘007 patent at 5:60-6:25.
Prosecution Insights