DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are currently pending (claims 8-20 withdrawn) in U.S. Patent Application No. 18/666,519 and an Office action on the merits follows.
Election/Restrictions
Applicant’s election without traverse of Group I (see requirement mailed 04/13/2026), in the reply filed on 05/06/2026 is acknowledged. Claims not directed to Group I, i.e. directed to nonelected Group(s) II and/or III are withdrawn from further consideration pursuant to 37 CFR 1.142(b), and consistent with the provided claim status indicator(s) “(withdrawn)” provided in the most recent claim set filed 05/06/2026 (MPEP 714 with reference to 37 CFR 1.121(c)). Applicant may have intended for claims currently withdrawn to remain pending in view of potential rejoinder/rejoinder practice as described in MPEP § 821.04, however for clarity of record purposes Examiner notes that no claims are understood by the Examiner to be “linking claim(s)” as defined in MPEP § 809, since none are (A) genus claim(s) linking species claims, or (B) subcombination claims linking plural combinations. While Groups I and II were drawn to species (they may also/alternatively be subs useable together in view of Fig. 6 – see the requirement at page 3, similar test for distinctness), a genus by definition does not possess the specific characteristics of any individual species. Claim 7 for example, inherits the specifics of claim 1, i.e. that use of a projection matrix so as to project first visual features into a language decoder embedding space, and further requires an additional high resolution encoder (common to Group II), but does not require the specific feature of Group II that is the use of that cross-attention layer to produce key-value pairs from the high resolution vision encoder output. Accordingly, Examiner best understands claim 7 to correspond to a combination claim drawn under Group I and not a linking claim, despite that recited second encoder. See 806.05(c) sub-section I example: ABsp (claim 7, wherein feature A is the additional/second vision encoder) / Bsp (claim 1, wherein feature Bsp is the projection matrix applied to first vision encoder output). In the interest of compact prosecution, Applicant is invited to contact the Examiner undersigned in the event that Applicant intends for and/or understands rejoinder (potentially eliminating a need for Divisional filings otherwise) to be warranted/appropriate, and/or if Applicant’s inventive concept/ improvement falls outside of Group I as drawn so as to include subject matter of any non-elected grouping.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
1. Claims 1, 3-4 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Hu et al. “BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions” (Thirty-Eighth AAAI Conference understood to be held late February 2024, with cited document published e.g. March 2024, and arxiv versions e.g. Aug-Dec 2023, prior to Applicant’s 16 May 2024 EFD – cited in the 04/13/2026 PTO-892), in view of Mishra et al. “OCR-VQA: Visual Question Answering by Reading Text in Images” (Cited by Applicant, NPL Citation No. 6) and Pham et al. “ViOCRVQA: Novel Benchmark Dataset and Vision Reader for Visual Question Answering by Understanding Vietnamese Text in Images” (29 April 2024).
As to claim 1, Hu discloses a computer-implemented method comprising:
extracting, utilizing a vision-language model (see Figs. 1d and 2 wherein architecture as a whole comprises a vision encoder in conjunction with that LLM producing output text) comprising a projection matrix (Figs. 1 and 2d “Projection” portion adjacent to Q-Former, also established in LLaVA, see 1c) and a language decoder (Fig. 2d LLM), a first set of visual features from a digital image depicting text-rich content (Fig. 1d BLIVA “image” input to vision encoder, and resultant visual features as vision encoder output, in further view of page 1, Abs describing the input image(s), e.g. “scenes with text-rich context. To improve upon them, the present study introduces BLIVA: an augmented version of InstructBLIP with Visual Assistant. BLIVA incorporates the query embeddings from InstructBLIP and also directly projects encoded patch embeddings into the LLM, a technique inspired by LLaVA”, page 4 (2259) Experiment “we evaluate our model…. on 10 OCR-related tasks”, page 7 Qualitative Analysis “We use real-life scene images, movie posters, webpages, and memes to demonstrate our model’s performance regarding interaction with humans based on text-rich images. The examples are in Appendix of arXiv version”);
projecting the first set of visual features into an embedding space of the language decoder utilizing the projection matrix comprising parameters learned from digital images (Figs. 1d and 2, Projection portion with output therefrom fed to LLM/decoder, illustrated with the fire symbol, not frozen/locked (e.g. for the case that the ‘learned’ recited concerns not simply taking a pre-trained and subsequently frozen model – interpretation under BRI includes both scenarios); see Figs of Hu reproduced in part below) with at least a threshold probability of depicting text-rich content (see page 1 Abs cited above, as Hu concerns input images explicitly disclosed as being “text-rich” even if silent regarding any associated ‘threshold’ for establishing/deeming an image to be text rich/dense, regarding said threshold, Hu discloses drawing from the OCR-VQA dataset, which corresponds to a document like context (as opposed to scenes with sparse and/or no text), generally expecting (implicitly if not inherent) a minimum amount of text content (e.g. under the loosest interpretation, i.e. a very low threshold, the training images are not scenes with no or sparse text because they are in the context of OCR));
PNG
media_image1.png
448
344
media_image1.png
Greyscale
extracting, utilizing the vision-language model (Figs. 1d and 2 more specifically from the vision encoder, that is of the model at large), a second set of visual features from the digital image depicting the text-rich content (Fig. 2, any subset of the vision encoder output, named as ‘a second’ – claim does require these (either set) to be derived from any specific vision encoder, or any vision encoder distinct from another, i.e. ‘a second set’ may distinguish a subset of those features output from the same vision encoder – as Applied ‘first’ and ‘second’ sets may also distinguish between those embeddings/feature outputs fed to the Q-Former portion, vs. those fed to the Projection portion of the model); and
generating, from the first set of visual features projected into the embedding space of the language decoder (Figs. 1d and 2 vision encoder output features/embeddings fed to Projection for subsequent input to LLM) and from the second set of visual features (Figs. 1d and 2 vision encoder outputs fed to Q-Former, in view of Q-Former output similarly fed to LLM, thereby LLM considers both – see page 3 Method “In particular, Figure 2 illustrates that our model incorporates a vision tower, which encodes visual representations from the input image into encoded patch embeddings. Subsequently, it is sent separately to the Q-former to extract refined learned query embeddings, and to the projection layer, allowing the LLM to grasp the rich visual knowledge. We concatenate the two types of embeddings and feed them directly to the LLM. These combined visual embeddings are appended immediately after the question text embedding to serve as the final input to the LLM. During inference, we employed beam search to select the best-generated output”), a predicted text phrase from the text-rich content depicted in the digital image utilizing the language decoder of the vision-language model to process the digital image (Figs. 1d and 2 “Output Text” from LLM, page 7 Qualitative Analysis “BLIVA showcases exceptional OCR capabilities, paired with a robust localization ability that accurately identifies texts and objects within images”) according to parameters learned from ground truth text phrases (LLM while frozen, was previously/pre- trained/learned from ground truth text phrases, page 1 Introduction “Our model is initialized from a pre-trained InstructBLIP and an encoded patch projection layer trained from scratch. Following (Zhu et al. 2023; Liu et al. 2023a), we further demonstrate a two-stage training paradigm. We begin by pre-training the patch embeddings projection layer. Subsequently, with the instruction tuning data, we fine-tune both the Q-former and the patch embeddings projection layer. During this phase, we maintain both the image encoder and LLM in a frozen state. We adopt this approach based on two findings from our experiments: firstly, unfreezing the vision encoder results in catastrophic forgetting of prior knowledge; secondly, training the LLM concurrently didn’t bring improvement but brought significant training complexity”, see also page 6 Table 5 for that implementation involving fine-tuning LLM, page 4 Thumbnails dataset “After retrieving all the YouTube thumbnails, we created the annotation file with the following fields: ”video id” representing the unique identification for a specific YouTube video, ”question” representing the human-made question based on the text and image in the thumbnail, ”video classes” representing the 11 video categories, ”answers” representing the ground truth answer, and ”video link” representing the URL link for each YouTube video”, etc.; Examiner notes this mapping is not to imply that the ‘learned parameters’ need concern only those of the LLM, as identified in the mapping above, the Projection layer of Hu is pre-trained (but not frozen), and both the Q-Former and Projection layer portions are fine-tuned, and while not the most preferred embodiment of Hu, Hu entertains fine-tuning the LLM also (Table 6)) page 5 (2260) Table 1 description “Note that our work follows InstructBLIP which incorporated OCR-VQA in its training dataset, thus inevitably making OCR-VQA evaluation not zero-shot”; see also BLIP-2 cited by Applicant NPL Citation No. 24, InstructBLIP Citation no. 42, and Mishra et al. OCR-VQA Citation No. 6, of the 09/26/2024 IDS).
PNG
media_image2.png
734
1320
media_image2.png
Greyscale
While Hu discloses text-rich samples in an OCR-VQA context, Hu fails to explicitly disclose that ground truth values are obtained from/generated using an optical character recognition model. Hu discloses that ground truth (GT) values are added to metadata fields (page 4 Thumbnails dataset), but the Examiner does not understand this to imply that ground truth values of Hu must be metadata derived, precluding any alternative means for deriving ground truth values, be that to include human/manual annotation and/or use of an OCR model to derive text labels.
As an additional consideration however, the OCR-VQA dataset (Mishra et al.) referenced in Hu (also Applicant’s NPL Citation No. 6) contains approximately 207,572 images of book covers and 1 million question-answer pairs, and text concerning author names, titles, and genres, that is understood by the Examiner to be derived from metadata in part at Stage 1 (with reference to Iwana et al.) and more explicitly so at Stage 2 “The ground truth answer is obtained using meta-data made available with every book title”. Mishra however further discloses use of Tesseract for text block extraction, and additional OCR in Section B, storing each block index and an OCR result of a best performing OCR method as part of a validation dataset – thereby at least suggesting the obvious nature of utilizing an OCR model to derive validating (even if in the context of a fine-tuning stage) information/data sets. Accordingly, Mishra’s OCR-VQA referenced in Hu is understood to at least suggest Hu being modifiable so as to rely on any of e.g. Tesseract, CRNN and/or VGG text spotter to derive validating and/or ground truth information, and accordingly Mishra at least suggests the obvious nature of ground truth values obtained from/generated using an optical character recognition model (as e.g. an alternative to human/manual labelling routinely implemented to generate supervised training samples – see e.g. Pham et al. “ViOCRVQA: Novel Benchmark Dataset and Vision Reader for Visual Question Answering by Understanding Vietnamese Text in Images” Fig 2 human annotators and training, development and test data sets).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date, to modify the system and method of Hu so as to utilize OCR derived information as a substitute for e.g. otherwise manually labeled ground truth information, as taught/suggested by Mishra/Pham and readily recognized by POSITA, that deriving ground truth values by such a means would involve no more than a simple substitution of one known ground truth labeling technique/element for another, so as to predictably obtain samples for a supervised training, in no manner rendering Hu inoperable and further characterized by a reasonable expectation of success (see MPEP 2143 Rationale A and/or B).
As to claim 3, Hu in view of Mishra and Pham teaches/suggests the method of claim 1.
Hu in view of Mishra and Pham teaches/suggests the method further comprising projecting the first set of visual features into the embedding space of the language decoder by utilizing the projection matrix (Hu/BLIVA Figs 1d and 2 output from frozen vision encoder fed to Projection layer) comprising parameters learned from a subset of digital images from among the digital images with at least the threshold probability of depicting text-rich content (Fig. 1d wherein the Projection layer is illustrated as being trained from scratch and/or finetuned (even if pre-trained, not frozen – see notes above in the mapping for claim 1)), wherein the subset of digital images corresponds to a set of text-rich image classifications (see Examiner notes from claim 1 above, with respect to Hu’s explicit “text rich” disclosure).
As to claim 4, Hu in view of Mishra and Pham teaches/suggests the method of claim 3.
Hu in view of Mishra and Pham teaches/suggests the method further comprising generating the ground truth text phrases using the optical character recognition model to process the subset of digital images corresponding to the set of text-rich image classifications (see Mishra as applied, e.g. Tesseract, CRNN and/or VGG text spotter to derive validating and/or ground truth information, in that proposed modification for the case of claim 1).
As to claim 6, Hu in view of Mishra and Pham teaches/suggests the method of claim 1.
Hu in view of Mishra and Pham teaches/suggests the method further comprising determining a text phrase prompt that instructs the vision-language model to generate the predicted text phrase from the digital image (Hu Fig. 2 ‘user instruction’/prompt used to generate those associated text embeddings, e.g. “What is this image about?”, and common to VQA more broadly, Examiner notes that permissible interpretation for the recited determining includes a determination that may be made by a user/operator interacting with the model); and
generating the predicted text phrase utilizing the vision-language model to process the text phrase prompt and the digital image (Hu Fig. 2 LLM output e.g. “The image depicts the famous Hollywood sign …”, also common to VQA with any multi-modal/vision encoder comprising LLM).
2. Claims 2 and 5 are rejected under 35 U.S.C. 103 as being unpatentable over Hu et al. “BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions”, in view of Mishra et al. “OCR-VQA: Visual Question Answering by Reading Text in Images”, Pham et al. “ViOCRVQA: Novel Benchmark Dataset and Vision Reader for Visual Question Answering by Understanding Vietnamese Text in Images”, and Florencio et al. (US 2021/0133438 A1).
As to claim 2, Hu in view of Mishra and Pham teaches/suggests the method of claim 1.
Hu in view of Mishra and Pham fails to explicitly disclose the method comprising determining the digital images with at least the threshold probability of depicting text-rich content by utilizing an image text detection model to determine probabilities of the digital images depicting text-rich content and instead features disclosure suggesting those training images utilized and/or images evaluated at inference, are characterized as text-rich based on characteristic if non-specific threshold probability of depicting text-rich content (Hu is explicitly in the context of handling Text-Rich VQA), while falling silent regarding any explicitly determined probability in that regard.
Florencio however evidences the obvious nature of determining digital images with at least the threshold probability of desired content/features (text-rich content in the context of Hu) by utilizing an image text detection model to determine probabilities of the digital images depicting specific content (Fig. 11, initial clustering and established list of forms at 1110, [0012] “Disclosed embodiments also include systems, methods and interfaces for clustering forms and for selecting forms to use for developing ground truth for training models that can be used to process and identify content contained within form”, [0071] “In some examples, form clustering may be used to separate the forms into separate types of forms and/or into clusters based on forms of a same type but that have other similarities in attributes that vary between the forms, and/or based on clusters of forms based on form source, or form completeness, content, etc. In some examples, rather than using automatic form clusters, the type of form may be an input parameter by the user”, [0072], [0109] “As shown in FIG. 11, a process (1100) may be used for prioritizing the forms shown to the user to select user-assisted labeling and for training models to use for labeling other forms. Initially, the system clusters groups of forms/documents/or form sections (1110). The clustering may be performed, as described throughout to identify similar groups of forms or form sections to use for selecting training data. The clustered groupings may be prioritized to identify the groups/groupings of forms or form data that have the highest scores (e.g., frequency of similar elements) to be used for training data”, [0163] “the system identifies a select subset of the forms in the cluster to be used for selecting ground truth to train a model (act 3620). This process may include identifying a predetermined percentage of the initial plurality of forms that are identified and/or a predetermined percentage of forms in a specific cluster of forms. (e.g., 5%, 10%, 20%, 30% or another percentage)”, etc.,).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date, to further modify the system and method of Hu in view of Mishra and Pham so as to implement a clustering and content score based ranking in the determination of ground truth samples as taught/suggested by Florencio, the motivation as similarly taught/suggested therein and known to POSITA more broadly, that such a scoring/ ranking would enable the selection of training samples with characteristics most similar to those expected at inference, thereby facilitating a training for accurate decision boundaries/predictions, while potentially reducing computational costs, the possibility of overfitting, and/or any incorporation of less/irrelevant training samples (resulting in less accurate decision boundaries/model training) characterized by lower scores.
As to claim 5, Hu in view of Mishra and Pham teaches/suggests the method of claim 1.
Hu in view of Mishra and Pham fails to explicitly disclose the clustering and selecting as recited in claim 5.
Florencio however evidences the obvious nature of determining digital images with at least the threshold probability of depicting desired content/features (e.g. text-rich content in the case of Hu) (see Florencio Fig. 11, [0012], [0071], [0072], [0109] as identified above for the case of claim 4);
clustering the digital images according to image classifications (Florencio [0071] “In some examples, form clustering may be used to separate the forms into separate types of forms and/or into clusters based on forms of a same type but that have other similarities in attributes that vary between the forms, and/or based on clusters of forms based on form source, or form completeness, content, etc.”); and
selecting a subset of the digital images from clusters corresponding to text-rich image classifications (Florencio [0163] “the system identifies a select subset of the forms in the cluster to be used for selecting ground truth to train a model (act 3620). This process may include identifying a predetermined percentage of the initial plurality of forms that are identified and/or a predetermined percentage of forms in a specific cluster of forms. (e.g., 5%, 10%, 20%, 30% or another percentage)”, etc.,).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date, to further modify the system and method of Hu in view of Mishra and Pham so as to implement that probability ranking/scoring, clustering, and subset selection of training samples as disclosed in Florencio, for those same reason(s) taught/suggested therein and that motivation/rationale as identified above for the case of claim 4 (e.g. such curated training data can be used to create an accurate distribution for samples expected during inference while culling lower scoring samples that may otherwise reduce model accuracy).
3. Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Hu et al. “BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions”, in view of Mishra et al. “OCR-VQA: Visual Question Answering by Reading Text in Images”, Pham et al. “ViOCRVQA: Novel Benchmark Dataset and Vision Reader for Visual Question Answering by Understanding Vietnamese Text in Images”, and Hong et al. “CogAgent: A Visual Language Model for GUI Agents” (arxiv v2 21 Dec 2023).
As to claim 7, Hu in view of Mishra and Pham teaches/suggests the method of claim 1.
Hu in view of Mishra and Pham fails to explicitly disclose the method wherein extracting the second set of visual features comprises utilizing a high-resolution vision encoder to extract high-resolution visual features at a resolution higher than the low-resolution visual features; and
generating the predicted text phrase from the low-resolution visual features and the high-resolution visual features.
With regards to whether or not that vision encoder of Hu is low-resolution, while potentially failing to explicitly disclose the language ‘low resolution’, it is the Examiner’s understanding that the vision encoder of Hu is indeed low resolution, in view of the manner in which their model is initialized from a pre-trained InstructBLIP (Dai et al. 2023, Applicant’s IDS Citation No. 42) which discloses at page 9 Section 3.5 “InstructBLIP maintains the same image resolution (224×224) as used in instruction tuning and keeps the visual encoder frozen during finetuning”, and which is further derived (unless the Examiner is mistaken, but at any rate Dai’s disclosure is sufficient evidence) from Radford’s CLIP-ViT (Applicant’s IDS NPL Citation No. 2) – which at page 37 discloses the model(s) trained on and receiving 224x224 and 336x336 pixel input images (low resolution) respectively.
Hong further evidences the obvious nature of extracting a second set of visual features comprises utilizing a high-resolution vision encoder to extract high-resolution visual features at a resolution higher than the low-resolution visual features (page 4, Fig. 2 left side high-resolution image encoder receiving 1120x1120 input image, “1. At a modest resolution such as 224 × 224, images can depict most objects and layouts effectively, yet the resolution falls short in rendering text with clarity. Hence, our new high-resolution module should emphasize text-related features, which are vital for understanding
GUIs… As shown in Fig. 2, the high-resolution cross-module acts as a new branch for higher-resolution input, which accepts images of size 1120 × 1120 pixels in our implementation. Different from the original low-resolution input branch, the high-resolution cross-module adopts a much smaller pre-trained vision encoder (visual encoder of EVA2-CLIP-L [35] in our implementation”); and
generating the predicted text phrase from the low-resolution visual features and the high-resolution visual features (Fig. 2 target text from original/frozen VLM on right side, in view of both low resolution image features from low res encoder/MLP adapter (MLP adapter of Hong is also likely a projection layer alternative – in terms of its function), and high resolution features, see Hong Fig. 2 reproduced below).
PNG
media_image3.png
544
464
media_image3.png
Greyscale
It would have been obvious to a person of ordinary skill in the art, before the effective filing date, to further modify the combination of Hu in view of Mishra and Pham so as to implement a high resolution encoder and consider also high resolution image features in producing a predicted/target text as taught/suggested by Hong, the motivation as similarly taught/suggested therein and known to POSITA more broadly, that such high resolution image features may better capture characteristics of text otherwise missed by solely relying upon low resolution features.
Additional References
Prior art made of record and not relied upon that is considered pertinent to applicant's disclosure:
Additionally cited references (see attached PTO-892) otherwise not relied upon above have been made of record in view of the manner in which they evidence the general state of the art.
Inquiry
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IAN L LEMIEUX whose telephone number is (571)270-5796. The examiner can normally be reached Mon - Fri 9:00 - 6:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached on 571-272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/IAN L LEMIEUX/Primary Examiner, Art Unit 2669
Prosecution Insights