DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This Office Action is FINAL.
This office action is in responsive to communication(s):
Amendment filed on 8/6/2026.
Application filed on 5/6/2024 with effective filing date of 5/6/2024.
The status of the claims is summarized as below:
Claims 1-20 are pending.
Claims 1, and 11 are independent claims.
In the amendment, claims 1, 4-5, 8-11, 14-15, 18-20 are amended.
Response to Arguments
The examiner acknowledges the amendment made to claim(s) 1, 4-5, 8-11, 14-15, 18-20 in the amendment filed on 8/6/2026.
Applicant’s arguments filed 8/6/2026 have been fully considered but they are directed to newly amended language which is now rejected with a new ground of rejection in light of newly cited art Ruan.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 6, 10-13, 16, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Santos Moraes et al. (US Pub 20200175063, hereinafter Santos Moraes), in view of Shioya et al. (US Pub 20170091224, hereinafter Shioya), and Ruan et al. (US Pub 20170351934, hereinafter Ruan).
Per claim 1, Santos Moraes teaches:
A method for improving a textual description, comprising: (abstract: method of augmenting image caption through contextual graph of the entities included in the image and their relationships);
receiving an image and alt-text, wherein the alt-text has been generated based on the image; ([0026-0027] Fig. 2 shows at step 206 the method checks to see if an existing ALT text/descript of an image exists or not, and at step 208 a description is generated through deep learning if ALT text does not exist);
extracting, from the alt-text, using the descriptions of object that are included in the image; ([0027-0028] Fig. 2 shows at step 212, description from step 206-208 is used to identify entities/objects in the image (extract description of an object) using natural language processing; also see [0034-0036] Fig. 4 where keywords such as “John Doe”, “arrival gate”, “94-year-old mother” are extracted);
…
when the one object is determined to be decorative, generating modified alt-text by removing a description of the one object from the alt-text while retaining descriptions corresponding to remaining objects that are determined not to be decorative. ([0030-0031] Fig. 2 shows at step 222-226, an augmented caption graph is generated using the both the contextual graph from step 220 and caption graph from 218, then relevance weights are assigned to the augmented caption graph; a relevance weight corresponding to overlap between the caption graph and contextual graph is assigned; [0032-0033] a natural language description of the image is generated using the augmented caption graph for the portion of the graph with relevance weights above a predetermined threshold weight (objects not decorative), thus the entities/objects in the original caption graph with lower weights are removed from the modified description (objects determined to be decorative); in the example shown in Fig. 3, all of the original caption entity nodes E1-E5 are above the threshold of 0.5; also see [0034-0036] Fig. 4).
Although Santos Moraes teaches using existing caption/generated description to identify objects in an image, and textual description of the accompanying documents to augment the caption to generate augmented caption graph, and modify the description based on parts of the graph with assigned relevance weights above a predefined threshold, i.e. using relevance weights with respect to a threshold to determine whether entities/objects from an image should be included in the description of the image; Santo Moraes does not explicitly teach using descriptions to detect objects in the image: “detecting in the image, using the descriptions, where the objects are located”.
However, Shioya teaches a method of estimating relevance of the objects detected from the image and removal of the irrelevant objects:
… detecting in the image, using the descriptions, where the objects are located; ([0024-0027] Fig. 3 shows at step 300-302, objects tags or metadata from the image can be used to identify and location objects in an input image, i.e. object tags may include information about the position of the object in the image);
Shioya and Santos Moraes are analogous art because Shioya also teaches method of modification based on extract objects from image and description. Therefore, it would have been obvious to one of ordinary skills in art before the effective filing date, having the teachings of Santos Moraes and Shioya before him/her, to modify the teachings of Santos Moraes to include the teachings of Shioya so that objects included in the description can be identified/detected from the image information such as object tags/metadata. One would be motivated to make the combination, with a reasonable expectation of success, because it would provide alternatives for locating objects in an image by utilizing existing image description such as object tags and metadata, which would improve the speed of object detection/identification in an image for the purpose of alt-text modification/generation.
Additionally, Santos Moraes-Shioya do not explicitly teach estimating contextual relevance of each object in the image to determine if each object is decorative based on a contextual relevance threshold: “estimating a contextual relevance of each object with respect to a semantic context of each object in the image; determining that one object among the objections is decorative when a contextual relevance of the one object fails to meet a contextual relevance threshold”.
However, Ruan teaches:
estimating a contextual relevance of each object with respect to a semantic context of each object in the image; ([0075-0078] Fig. 9 shows at step S60, a contextual relevance score for each recognition target region from an input image is computed; [0040] a plurality of target regions may be extracted from the input image; also see Fig. 3, Fig. 4A and associated paragraphs);
determining that one object among the objections is decorative when a contextual relevance of the one object fails to meet a contextual relevance threshold, and ([0076-0078] the relevance score for the target region is computed, where the relevance score indicates a level of interest a person has in the region/object; if the relevance score does not meet a threshold value, then region/object is deemed uninteresting/irrelevant).
Ruan and Santos Moraes-Shioya are analogous art because Ruan also teaches method of image object recognition in an input image. Therefore, it would have been obvious to one of ordinary skills in art before the effective filing date, having the teachings of Santos Moraes-Shioya and Ruan before him/her, to modify the teachings of Santos Moraes-Shioya to include the teachings of Ruan so that each identified region with an object can include a computed relevance score to determine if the region/object is decorative/irrelevant or not. One would be motivated to make the combination, with a reasonable expectation of success, because it would provide alternatives for determining if an object included in the image is not of any interest/irrelevant to a user so that it can be removed from the al-text/description, which would further improve accuracy of the image alt-text/description.
Per claim 2, Santos Moraes-Shioya-Ruan teach all the limitations of claim 1, and they further teach:
wherein the image comprises a website image. (Santos Moraes: [0026] Fig. 2 shows at step 202 a document with image was received, where the document can be web pages, and the image a webpage image).
Per claim 3, Santos Moraes-Shioya-Ruan teach all the limitations of claim 1, and they further teach:
wherein the modified alt-text is presented to a user when the user navigates to a web page that includes the image. (Santo Moraes: [0033, 0026] Fig. 2 shows at step 226-228, a modified description was generated and saved in association with the image and corresponding document, and displayed at alt text associated with the image, where the document can be a webpage).
Per claim 6, Santos Moraes-Shioya-Ruan teach all the limitations of claim 1, and they further teach:
wherein the extracting of the description of the objection is performed using a Question-Answering (QA) Large Language Model (LLM). (Santos Moraes: [0027-0028] step 212 from Fig. 2 for extracting objects from description is performed utilizing natural language processing and deep learning).
Per claim 10, Santos Moraes-Shioya-Ruan teach all the limitations of claim 1, and they further teach:
wherein comparing the contextual relevance of the one object comprises determining a blur score for the object. (Shioya: [0040-0041] a relevance of an object in the image is determined based on if the object is in focus in the image (blur score)).
It would have been obvious to one of ordinary skills in art before the effective filing date, to modify the combination of claim 1 to utilize the additional teachings of Shioya so that contextual relevance of an identified object on an image can be made based on if the object is in focus/blurred or not. One would be motivated to make the combination, with a reasonable expectation of success, because it would provide alternatives for determining if an object included in the image is irrelevant or not so that it can be removed from the al-text/description, which would further improve accuracy of the image alt-text/description.
Per claim 11, claim 11 is a non-transitory storage medium claim ([0040, 0052] Fig. 6 storage device 65) that includes limitations that are substantially the same as claim 1, and is likewise rejected.
Per claim 12-13, 16, 20, claims 12-13, 16, 20 include limitations that are substantially the same as claims 2-3, 6, and 10 respectively, and are likewise rejected.
Claims 4-5, 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Santos Moraes, in view of Shioya, Ruan, and Barkan et al. (US Pub 20220269895, hereinafter Barkan).
Per claim 4, Santos Moraes-Shioya-Ruan teach all the limitations of claim 1, but they don’t explicitly teach “wherein comparing the contextual a relevance of the one object comprises obtaining respective contextual relevance scores for each relevance measure in a group of relevance measures”.
However, Barkan teaches:
wherein comparing the contextual a relevance of the one object comprises obtaining respective contextual relevance scores for each relevance measure in a group of relevance measures. (abstract [0041-0044] Fig. 5 shows at step 508-516: an image with multiple objects are used for detection of objects, where heatmaps are generated for each object, and relevant score for each object are determined).
Barkan and Santos Moraes-Shioya-Ruan are analogous art because Barkan also teaches method of objection detection in images. Therefore, it would have been obvious to one of ordinary skills in art before the effective filing date, having the teachings of Santos Moraes-Shioya-Ruan and Barkan before him/her, to modify the teachings of Santos Moraes-Shioya-Ruan to include the teachings of Barkan so that objects can also be detected in images using heatmaps. One would be motivated to make the combination, with a reasonable expectation of success, because it would provide alternatives for object detections in images, where a plurality of objects can be detected with relevance score for each to determine the relevant/irrelevant objects in the image, thereby improving accuracy of the object detection in images.
Per claim 5, Santos Moraes-Shioya-Ruan teach all the limitations of claim 1, but they don’t explicitly teach “wherein comparing the contextual relevance of the one object comprises generating a respective heat map for each contextual relevance measure in a group of contextual relevance measures and, based on the heat map, generating a respective score for each contextual relevance measure and comparing the scores to respective contextual relevance thresholds to determine the estimated contextual relevance of the one object”.
However, Barkan teaches:
wherein comparing the contextual relevance of the one object comprises generating a respective heat map for each contextual relevance measure in a group of contextual relevance measures and, based on the heat map, generating a respective score for each contextual relevance measure and comparing the scores to respective contextual relevance thresholds to determine the estimated contextual relevance of the one object. (abstract [0041-0044] Fig. 5 shows at step 508-516: an image with multiple objects are used for detection of objects, where heatmaps are generated for each object, and relevant score for each object are determined; [0051-0052] relevancy scores are determined and object relevancy are ranked based on a threshold as shown in Fig. 10).
Barkan and Santos Moraes-Shioya-Ruan are analogous art because Barkan also teaches method of objection detection in images. Therefore, it would have been obvious to one of ordinary skills in art before the effective filing date, having the teachings of Santos Moraes-Shioya-Ruan and Barkan before him/her, to modify the teachings of Santos Moraes-Shioya-Ruan to include the teachings of Barkan so that objects can also be detected in images using heatmaps. One would be motivated to make the combination, with a reasonable expectation of success, because it would provide alternatives for object detections in images, where a plurality of objects can be detected with relevance score for each to determine the relevant/irrelevant objects in the image, thereby improving accuracy of the object detection in images.
Per claim 14-15, claims 14-15 includes limitations that are substantially the same as claim 4-5, and are likewise rejected.
Claims 7, 17 are rejected under 35 U.S.C. 103 as being unpatentable over Santos Moraes, in view of Shioya, Ruan, and Kuo et al. (US Pub 20250384650, hereinafter Kuo).
Per claim 7, Santos Moraes-Shioya-Ruan teach all the limitations of claim 1, but they don’t explicitly teach “wherein the detecting is performed using a zero-shot semantic segmentation (ZSSS) process”.
However, Kuo teaches:
wherein the detecting is performed using a zero-shot semantic segmentation (ZSSS) process. (abstract [0046-0047] detection of object in image can be based on frozen VLM while combining self-training for zero-shot semantic segmentation).
Kuo and Santos Moraes-Shioya-Ruan are analogous art because Kuo also teaches method of objection detection in images. Therefore, it would have been obvious to one of ordinary skills in art before the effective filing date, having the teachings of Santos Moraes-Shioya-Ruan and Kuo before him/her, to modify the teachings of Santos Moraes-Shioya-Ruan to include the teachings of Kuo so that objects can also be detected in images using method via zero-shot semantic segmentation process. One would be motivated to make the combination, with a reasonable expectation of success, because it would provide alternatives for object detections in images, where zero-shot detection can also alleviate the challenges by learning to detect novel categories not present in training data (Kuo [0047]), thereby improving accuracy of the object detection in images.
Per claim 17, claim 17 includes limitations that are substantially the same as claim 7, and is likewise rejected.
Claims 8-9, 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Santos Moraes, in view of Shioya, Ruan, and Huang (US Pat 11457138, hereinafter Huang).
Per claim 8, Santos Moraes-Shioya-Ruan teach all the limitations of claim 1, but they don’t explicitly teach “wherein comparing the contextual relevance of the one object comprises determining a centrality score for the one object, and the centrality score is based on a center of mass (COM) of the object”.
However, Huang teaches:
wherein comparing the contextual relevance of the one object comprises determining a centrality score for the one object, and the centrality score is based on a center of mass (COM) of the object. (Col 8 line 4 – 29: object detection can be performed using a central weight map to determine a confidence map of an object).
Huang and Santos Moraes-Shioya-Ruan are analogous art because Huang also teaches method of objection detection in images. Therefore, it would have been obvious to one of ordinary skills in art before the effective filing date, having the teachings of Santos Moraes-Shioya-Ruan and Huang before him/her, to modify the teachings of Santos Moraes-Shioya-Ruan to include the teachings of Huang so that objects can also be detected in images using central weight map and depth image. One would be motivated to make the combination, with a reasonable expectation of success, because it would provide alternatives for object detections in images, where an object in a center of the image may be detected more easily by using the central weight map (Huang col 8 line 25-29), thereby improving accuracy of the object detection in images.
Per claim 9, Santos Moraes-Shioya-Ruan teach all the limitations of claim 1, but they don’t explicitly teach “wherein comparing the contextual relevance of the one object comprises determining a depth score for the one object, and the depth score is based on semantic segmentation pixels identified as part of the detecting”.
However, Huang teaches:
wherein comparing the contextual relevance of the one object comprises determining a depth score for the one object, and the depth score is based on semantic segmentation pixels identified as part of the detecting. (Col 8 line 4 – 29: object detection can be performed using a depth image to determine a confidence map of an object; col 16 line 4-22: depth image is semantically labeled to label object in the depth image during training).
Huang and Santos Moraes-Shioya-Ruan are analogous art because Huang also teaches method of objection detection in images. Therefore, it would have been obvious to one of ordinary skills in art before the effective filing date, having the teachings of Santos Moraes-Shioya-Ruan and Huang before him/her, to modify the teachings of Santos Moraes-Shioya-Ruan to include the teachings of Huang so that objects can also be detected in images using central weight map and depth image. One would be motivated to make the combination, with a reasonable expectation of success, because it would provide alternatives for object detections in images, where an object close to the camera may be detected more easily by using the depth image (Huang col 8 line 25-29), thereby improving accuracy of the object detection in images.
Per claim 18-19, claims 18-19 include limitations that are substantially the same as claims 8-9 respectively, and are likewise rejected.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The examiner requests, in response to this Office action, support by shown for language added to any original claims on amendment and any new claims. That is, indicate support for newly added claim language by specifically pointing to page(s) and line no(s) in the specification and/or drawing figure(s). This will assist the examiner in prosecuting the application.
When responding to this office action, Applicant is advised to clearly point out the patentable novelty which he or she thinks the claims present, in view of the state of the art disclosed by the references cited or the objections made. He or she must also show how the amendments avoid such references or objections, See 37 CFR 1.111(c).
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PHOEBE X PAN whose telephone number is (571)270-7794. The examiner can normally be reached M-F 9am-6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Fred Ehichioya can be reached at (571) 272-4034. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PHOEBE X PAN/Examiner, Art Unit 2179
/IRETE F EHICHIOYA/Supervisory Patent Examiner, Art Unit 2179
