DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is in responsive to communication(s):
Application filed on 5/6/2024 with effective filing date of 5/6/2024.
The status of the claims is summarized as below:
Claims 1-20 are pending.
Claims 1, and 11 are independent claims.
Information Disclosure Statement
The information disclosure statement(s) filed on 5/6/2024 comply/complies with the provisions of 37 C.F.R. § 1.97, 1.98, and MPEP § 609, and therefore has/have been placed in the application file. The information referred to therein has/have been considered as to the merits.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 6, 10-13, 16, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Santos Moraes et al. (US Pub 20200175063, hereinafter Santos Moraes), in view of Shioya et al. (US Pub 20170091224, hereinafter Shioya).
Per claim 1, Santos Moraes teaches:
A method for improving a textual description, comprising: (abstract: method of augmenting image caption through contextual graph of the entities included in the image and their relationships);
receiving an image and alt-text, and the alt-text has been generated based on the image; ([0026-0027] Fig. 2 shows at step 206 the method checks to see if an existing ALT text/descript of an image exists or not, and at step 208 a descript is generated through deep learning if ALT text does not exist);
extracting, from the alt-text, a description of an object that is included in the image; ([0027-0028] Fig. 2 shows at step 212, description from step 206-208 is used to identify entities/objects in the image (extract description of an object) using natural language processing; also see [0034-0036] Fig. 4 where keywords such as “John Doe”, “arrival gate”, “94-year-old mother” are extracted);
…
when the relevance fails to meet a relevance threshold, generating modified alt-text by removing the description of the object from the alt-text ([0030-0031] Fig. 2 shows at step 222-226, an augmented caption graph is generated using the both the contextual graph from step 220 and caption graph from 218, then relevance weights are assigned to the augmented caption graph; a relevance weight corresponding to overlap between the caption graph and contextual graph is assigned; [0032-0033] a natural language description of the image is generated using the augmented caption graph for the portion of the graph with relevance weights above a predetermined threshold weight (relevance fails to meet a threshold), thus the entities/objects in the original caption graph with lower weights are removed from the modified description; in the example shown in Fig. 3, all of the original caption entity nodes E1-E5 are above the threshold of 0.5; also see [0034-0036] Fig. 4).
Although Santos Moraes teaches using existing caption/generated description to identify objects in an image, and textual description of the accompanying documents to augment the caption to generate augmented caption graph, and modify the description based on parts of the graph with assigned relevance weights above a predefined threshold, i.e. using relevance weights with respect to a threshold to determine whether entities/objects from an image should be included in the description of the image; Santo Moraes does not explicitly teach estimating a relevance of an object detected based from the image.
However, Shioya teaches estimating relevance of the objects detected from the image and removal of the irrelevant objects:
… detecting in the image, using the description, where the object is located; estimating a relevance of the object; and … ([0024-0026] Fig. 3 shows at step 300-302, objects from a received image is identified/detected, including location of the object as shown in Fig. 11; [0030] Fig. 3 further shows at step 304, the text tag created from the description of the image is used to determine if each text tag describe one of the object based on object tag, and a list of object tag from the textual description is emphasized as shown in Fig. 4; [0040-0041] the image modification program further determines a relevance of each object in the object tag list 109 from Fig. 4, where the determination is based on if the object is in focus in the image, which is used to determine if the object should be masked/excluded from the image as shown in Fig. 5 and Fig. 7; see also Fig. 7);
Shioya and Santos Moraes are analogous art because Shioya also teaches method of modification based on extract objects from image and description. Therefore, it would have been obvious to one of ordinary skills in art before the effective filing date, having the teachings of Santos Moraes and Shioya before him/her, to modify the teachings of Santos Moraes to include the teachings of Shioya so that objects detected from the image that are determined not to be relevant can be removed from the description. One would be motivated to make the combination, with a reasonable expectation of success, because it would provide alternatives for generation of image alt-text/description by using object detections in image to determine relevance of the objects so that irrelevant objects can be removed from the textual description, which would further improve accuracy of the image alt-text/description.
Per claim 2, Santos Moraes-Shioya teach all the limitations of claim 1, and they further teach:
wherein the image comprises a website image. (Santos Moraes: [0026] Fig. 2 shows at step 202 a document with image was received, where the document can be web pages, and the image a webpage image).
Per claim 3, Santos Moraes-Shioya teach all the limitations of claim 1, and they further teach:
wherein the modified alt-text is presented to a user when the user navigates to a web page that includes the image. (Santo Moraes: [0033, 0026] Fig. 2 shows at step 226-228, a modified description was generated and saved in association with the image and corresponding document, and displayed at alt text associated with the image, where the document can be a webpage).
Per claim 6, Santos Moraes-Shioya teach all the limitations of claim 1, and they further teach:
wherein the extracting of the description of the objection is performed using a Question-Answering (QA) Large Language Model (LLM). (Santos Moraes: [0027-0028] step 212 from Fig. 2 for extracting objects from description is performed utilizing natural language processing and deep learning).
Per claim 10, Santos Moraes-Shioya teach all the limitations of claim 1, and they further teach:
wherein estimating a relevance of the object comprises determining a blur score for the object. (Shioya: [0040-0041] a relevance of an object in the image is determined based on if the object is in focus in the image (blur score)).
Per claim 11, claim 11 is a non-transitory medium claim ([0040, 0052] Fig. 6 storage device 65) that includes limitations that are substantially the same as claim 1, and is likewise rejected.
Per claim 12-13, 16, 20, claims 12-13, 16, 20 include limitations that are substantially the same as claims 2-3, 6, and 10 respectively, and are likewise rejected.
Claims 4-5, 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Santos Moraes, in view of Shioya, and Barkan et al. (US Pub 20220269895, hereinafter Barkan).
Per claim 4, Santos Moraes-Shioya teach all the limitations of claim 1, but they don’t explicitly teach “wherein estimating a relevance of the object comprises obtaining respective relevance scores for each relevance measure in a group of relevance measures”.
However, Barkan teaches:
wherein estimating a relevance of the object comprises obtaining respective relevance scores for each relevance measure in a group of relevance measures. (abstract [0041-0044] Fig. 5 shows at step 508-516: an image with multiple objects are used for detection of objects, where heatmaps are generated for each object, and relevant score for each object are determined).
Barkan and Santos Moraes-Shioya are analogous art because Barkan also teaches method of objection detection in images. Therefore, it would have been obvious to one of ordinary skills in art before the effective filing date, having the teachings of Santos Moraes-Shioya and Barkan before him/her, to modify the teachings of Santos Moraes-Shioya to include the teachings of Barkan so that objects can also be detected in images using heatmaps. One would be motivated to make the combination, with a reasonable expectation of success, because it would provide alternatives for object detections in images, where a plurality of objects can be detected with relevance score for each to determine the relevant/irrelevant objects in the image, thereby improving accuracy of the object detection in images.
Per claim 5, Santos Moraes-Shioya teach all the limitations of claim 1, but they don’t explicitly teach “wherein estimating a relevance of the object comprises generating a respective heat map for each relevance measure in a group of relevance measures and, based on the heat map, generating a respective score for each relevance measure and comparing the scores to respective thresholds to determine the estimated relevance of the object”.
However, Barkan teaches:
wherein estimating a relevance of the object comprises generating a respective heat map for each relevance measure in a group of relevance measures and, based on the heat map, generating a respective score for each relevance measure and comparing the scores to respective thresholds to determine the estimated relevance of the object. (abstract [0041-0044] Fig. 5 shows at step 508-516: an image with multiple objects are used for detection of objects, where heatmaps are generated for each object, and relevant score for each object are determined; [0051-0052] relevancy scores are determined and object relevancy are ranked based on a threshold as shown in Fig. 10).
Barkan and Santos Moraes-Shioya are analogous art because Barkan also teaches method of objection detection in images. Therefore, it would have been obvious to one of ordinary skills in art before the effective filing date, having the teachings of Santos Moraes-Shioya and Barkan before him/her, to modify the teachings of Santos Moraes-Shioya to include the teachings of Barkan so that objects can also be detected in images using heatmaps. One would be motivated to make the combination, with a reasonable expectation of success, because it would provide alternatives for object detections in images, where a plurality of objects can be detected with relevance score for each to determine the relevant/irrelevant objects in the image, thereby improving accuracy of the object detection in images.
Per claim 14-15, claims 14-15 includes limitations that are substantially the same as claim 4-5, and are likewise rejected.
Claims 7, 17 are rejected under 35 U.S.C. 103 as being unpatentable over Santos Moraes, in view of Shioya, and Kuo et al. (US Pub 20250384650, hereinafter Kuo).
Per claim 7, Santos Moraes-Shioya teach all the limitations of claim 1, but they don’t explicitly teach “wherein the detecting is performed using a zero-shot semantic segmentation (ZSSS) process”.
However, Kuo teaches:
wherein the detecting is performed using a zero-shot semantic segmentation (ZSSS) process. (abstract [0046-0047] detection of object in image can be based on frozen VLM while combining self-training for zero-shot semantic segmentation).
Kuo and Santos Moraes-Shioya are analogous art because Kuo also teaches method of objection detection in images. Therefore, it would have been obvious to one of ordinary skills in art before the effective filing date, having the teachings of Santos Moraes-Shioya and Kuo before him/her, to modify the teachings of Santos Moraes-Shioya to include the teachings of Kuo so that objects can also be detected in images using method via zero-shot semantic segmentation process. One would be motivated to make the combination, with a reasonable expectation of success, because it would provide alternatives for object detections in images, where zero-shot detection can also alleviate the challenges by learning to detect novel categories not present in training data (Kuo [0047]), thereby improving accuracy of the object detection in images.
Per claim 17, claim 17 includes limitations that are substantially the same as claim 7, and is likewise rejected.
Claims 8-9, 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Santos Moraes, in view of Shioya, and Huang (US Pat 11457138, hereinafter Huang).
Per claim 8, Santos Moraes-Shioya teach all the limitations of claim 1, but they don’t explicitly teach “wherein estimating a relevance of the object comprises determining a centrality score for the object, and the centrality score is based on a center of mass (COM) of the object”.
However, Huang teaches:
wherein estimating a relevance of the object comprises determining a centrality score for the object, and the centrality score is based on a center of mass (COM) of the object. (Col 8 line 4 – 29: object detection can be performed using a central weight map to determine a confidence map of an object).
Huang and Santos Moraes-Shioya are analogous art because Huang also teaches method of objection detection in images. Therefore, it would have been obvious to one of ordinary skills in art before the effective filing date, having the teachings of Santos Moraes-Shioya and Huang before him/her, to modify the teachings of Santos Moraes-Shioya to include the teachings of Huang so that objects can also be detected in images using central weight map and depth image. One would be motivated to make the combination, with a reasonable expectation of success, because it would provide alternatives for object detections in images, where an object in a center of the image may be detected more easily by using the central weight map (Huang col 8 line 25-29), thereby improving accuracy of the object detection in images.
Per claim 9, Santos Moraes-Shioya teach all the limitations of claim 1, but they don’t explicitly teach “wherein estimating a relevance of the object comprises determining a depth score for the object, and the depth score is based on semantic segmentation pixels identified as part of the detecting”.
However, Huang teaches:
wherein estimating a relevance of the object comprises determining a depth score for the object, and the depth score is based on semantic segmentation pixels identified as part of the detecting. (Col 8 line 4 – 29: object detection can be performed using a depth image to determine a confidence map of an object; col 16 line 4-22: depth image is semantically labeled to label object in the depth image during training).
Huang and Santos Moraes-Shioya are analogous art because Huang also teaches method of objection detection in images. Therefore, it would have been obvious to one of ordinary skills in art before the effective filing date, having the teachings of Santos Moraes-Shioya and Huang before him/her, to modify the teachings of Santos Moraes-Shioya to include the teachings of Huang so that objects can also be detected in images using central weight map and depth image. One would be motivated to make the combination, with a reasonable expectation of success, because it would provide alternatives for object detections in images, where an object close to the camera may be detected more easily by using the depth image (Huang col 8 line 25-29), thereby improving accuracy of the object detection in images.
Per claim 18-19, claims 18-19 include limitations that are substantially the same as claims 8-9 respectively, and are likewise rejected.
Conclusion
The examiner requests, in response to this Office action, support by shown for language added to any original claims on amendment and any new claims. That is, indicate support for newly added claim language by specifically pointing to page(s) and line no(s) in the specification and/or drawing figure(s). This will assist the examiner in prosecuting the application.
When responding to this office action, Applicant is advised to clearly point out the patentable novelty which he or she thinks the claims present, in view of the state of the art disclosed by the references cited or the objections made. He or she must also show how the amendments avoid such references or objections, See 37 CFR 1.111(c).
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PHOEBE X PAN whose telephone number is (571)270-7794. The examiner can normally be reached M-F 9am-6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Fred Ehichioya can be reached at (571) 272-4034. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PHOEBE X PAN/Examiner, Art Unit 2179
/IRETE F EHICHIOYA/Supervisory Patent Examiner, Art Unit 2179
Prosecution Insights