Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments filed July 8, 2026 have been fully considered.
Please see newly incorporated reference Xu below which teaches a transformer encoder receiving OCR text, bounding-box coordinates for the OCR words, and image-region embeddings and generating contextualized representations from those combined modalities (Xu, Fig. 2; pp. 2-4).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Taylor (20070140595) in view of Xu (“LayoutLM: Pre-training of Text and Layout for Document Image Understanding”), in view of Liu (20190205643), and in further view of Sharifi (20220188321).
Regarding claim 1, Taylor teaches a computer-implemented method of processing images. Taylor receives, using a computer system, image data comprising a plurality of street-scene or storefront images, including multiple views and panoramic images (Taylor, Figs. 2-4; pars. 48, 56-60).
determining attributes associated with the images by processing the images with an OCR system configured to recognize text segments in the images. The recognized attributes include street addresses and business names (pars. 15-16, 47-50, 59-61).
Taylor teaches updating map data associated with a plurality of locations based on the OCR results. The OCR results refine or update a map-data database, including the locations of addresses and businesses identified from the images (pars. 28, 53, 57-60; Fig. 2).
Taylor does not teach that the model has a main encoder configured to generate multimodal embeddings from multimodal inputs comprising the images, recognized text segments, and detection boxes associated with the text segments.
Xu teaches a transformer encoder that generates contextualized representations from multimodal inputs, using a bounding box for each region (Fig. 2 and section 2.1-2.4 and 3.5).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Taylor’s to use a transformer based multimodal encoder as taught by Xu. The reason is to predictably improve the quality of recognition.
Taylor and Xu do not teach that the machine learned model further comprises a plurality of task-specific heads configured to determine the plurality of attributes based on the multimodal embeddings.
Liu teaches a multitask deep neural network having a shared network trunk and multiple side branches. (Fig. 4 and pars. 35-38, 41-43).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to include in Taylor and Xu a plural task specific prediction heads as taught by Liu. The reason is to improve efficiency by doing multiple tasks at once.
Taylor, Xu, and Liu do not expressly teach determining one or more entities associated with the plurality of attributes and generating attribute data comprising the plurality of attributes associated with the entities.
Sharifi teaches this in pars. 44-52, 65-67 and Fig. 2.
It would have been obvious to a person of ordinary skill in the art prior to the effective filing date of the invention to incorporate in Taylor, Xu and Liu an entity association technique as taught by Sharifi. The reason is to discover entities from image-recognition attributes.
Regarding claim 2, see Xu, Fig. 2 and section 2.1-2.3.
Regarding claim 3, Taylor teaches confidence values associated with OCR hypotheses and detected text, and Sharifi teaches calibrated backend confidence scores indicating the accuracy of OCR or textual visual-recognition labels (see Taylor, pars. 35, 39-48 and Sharifi, pars. 45-47).
Regarding claim 4, see Xu, Fig. 2 and section 2.3.
Regarding claim 5, see Xu sections 2.1-2.4.
Regarding claim 6, see Xu, Fig. 2 and section 2.3-2.4.
Regarding claim 7, see Taylor, pars. 15-16, 45-48.
Regarding claim 8, see Liu, Fig. 3 and pars. 33-34, 41-43.
Regarding claim 9, see the rejection of claim 1 and also Liu teaches a multitask model comprising a shared network trunk and plural task-specific side branches. The trunk generates feature representations from images, and the branches determine different classifications and attributes from those representations (Fig. 4 and pars. 35-38, 41-43).
Regarding claim 10, see Taylor, pars. 28, 43-45, 53 and Figs. 2-4.
Regarding claim 11, see Taylor, Fig. 1B and pars. 19-22, 28, 46-47, 53.
Regarding claim 12, Taylor teaches images of buildings and storefronts captured as street-level images from a vehicle or pedestrian perspective, including panoramic images along a street and views taken in driving directions substantially parallel to the ground plane (pars. 3, 15, 38-40).
Regarding claim 13, see Liu pars. 34, 40-44.
Regarding claim 14, Taylor teaches training or operating on OCR-derived text from images, Xu teaches OCR-derived word tokens and detection boxes for those tokens, and Sharifi teaches confidence scores associated with OCR-derived labels (Taylor, pars. 48-50; Xu, Fig. 2 and section 2.3 and Sharifi, pars. 45-47).
Regarding claims 15-20, see the rejection of claims 1-2 and 9 above.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HADI AKHAVANNIK whose telephone number is (571)272-8622. The examiner can normally be reached 9 AM - 5 PM Monday to Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HADI AKHAVANNIK/ Primary Examiner, Art Unit 2676
