DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s amendment filed on 08/21/2026 has been entered and made of record. Claims 1, 5. 8, 15 and 19 are amended. No New Claim was added. No Claims were cancelled. Claims 1-20 are pending.
Applicant’s remarks in view of the newly presented amendments have been considered but are not found to be persuasive for at least the following reasons:
The applicant argues on page 12-13 of the remarks filed the previous cited prior art of Wu et al. ("VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks") in view of Guo et al. US PG-Pub(US 20250272959 A1) do not explicitly disclose the newly amended limitation of generating, by merging the first set of visual features of the digital image with the second set of visual features of the digital image at a dimensionality of the second set of visual features, a set of combined visual features. The Examiner agrees as the previously cited prior art fails to disclose this newly amended limitation. However, after further search and consideration, the newly discovered art of Zhao et al. ("MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning") would disclose generating, by merging the first set of visual features of the digital image with the second set of visual features of the digital image at a dimensionality of the second set of visual features, a set of combined visual features in Page 5, Section 3.2 Multi-Granularity Vision Flow, Paragraph 1-2 and Fig. 3(b). Please see the updated claim rejections under 35 USC § 103 below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-4, 7-9, 13-17 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. ("VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks") in view of Zhao et al. ("MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning").
Regarding Claim 1, Wu teaches a computer-implemented method(Fig.2 shows architecture of VisionLLM) comprising:
generating, utilizing a first visual encoder, low-resolution visual features of a digital image(Page 4, 3 VisionLLM v2, Paragraph 1, “The overall architecture of VisionLLM v2 is depicted in Figure 2. It mainly consists of four parts: (1) an image encoder and a region encoder that encode the image-level and region-level information;”, in this section of the prior art an image encoder is used to extract image level features from the image and Page 4, 3 VisionLLM v2, Paragraph 3 discloses the visual features extracted both low and high resolution features.); generating, utilizing a second visual encoder, a second set of visual features of the digital image; (Page 4, 3 VisionLLM v2, Paragraph 3, “For an image input, we utilize a pre-trained vision foundation model, such as CLIP [141], to extract image features. Recognizing that current vision models operate the images at a low resolution, we adopt the dynamic resolution approach [31] to process the input images. Specifically, the input image is first automatically matched to an optimal aspect ratio from a predefined ratio set. Subsequently, the image is scaled up to a higher resolution based on the selected aspect ratio and divided into P square patches, each whose resolutions are 336×336. These local patches, along with a 336×336 global image Iglobal, are processed by the image encoder to capture both holistic scenes and fine-grained details, resulting in image features Fimg ∈ R576(P+1)× C.”, In this section of the prior art, the CLIP encoder extracts image features that are scaled up to be high resolution visual image features.)
determining, utilizing a visual-text encoder, a text string corresponding to the text of the digital image; (Page 4, 3 VisionLLM v2, Paragraph 3, “For an image input, we utilize a pre-trained vision foundation model, such as CLIP [141], to extract image features.”, In this section of the prior art, the CLIP encoder extracts image and text features that are scaled up to be high resolution visual image features.);and generating, for a query directed to the text of the digital image, a response from the first set of visual features, the second set of visual features, and the text string utilizing a large language model. (Page 4, VisionLLM v2, Paragraph 2, “Tokenization. VisionLLM v2 is flexible for handling multimodal input. (1) For text prompts, we employ the text tokenizer to tokenize them into distinct vocabulary indices, which can be further processed by LLM and result in the text features Ftext ∈ RL× C, where L denotes the length of input text, and C is the channel dimension of LLM.” In this section of the prior art a tokenization is performed and as seen in figure 2 the text tokenizer’s output is input into a large language model in order to generate a response based on the text tokens.)
Wu does not explicitly teach generating, by merging the first set of visual features of the digital image with the second set of visual features of the digital image at a dimensionality of the second set of visual features, a set of combined visual features
Zhao teaches generating, by merging the first set of visual features of the digital image with the second set of visual features of the digital image at a dimensionality of the second set of visual features, a set of combined visual features (Page 5, Section 3.2 Multi-Granularity Vision Flow, “Hybrid Vision Encoders As depicted in Fig. 3(b), MG-LLaVA initially processes images at two different resolutions: low-resolution VL and high-resolution VH. In the low-resolution branch, we follow the LLaVA-1.5 [31] to utilize a CLIP-pretrained ViT [57] denoted as fL V to derive low resolution features EL ∈ RN× C,”, as disclosed in this section and in shown in Fig. 3(b) the prior art processes the digital images at a low and high resolution using 2 encoders and in Section 3.2, “Conv-Gate Fusion Combining both low and high-resolution features as inputs results in a doubling of the visual tokens to be processed, which is computationally ineffective. Moreover, the distinct architectures of ViT and ConvNeXt lead to a discrepancy between EL and EH, requiring a careful fusion process. Inspired from [14], we implement a lightweight Conv-Gate fusion network that facilitates feature aggregation while maintaining a single resolution’s token count, as shown in Fig. 3(c).”, discloses these features are combined to form a combined set of visual features.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Wu with Zhao in order to combining the high and low resolution into a set of combined visual features. One skilled in the art would have been motivated to modify Wu in this manner in order to capture fine grained details, which are then fused with base visual features through a Conv-Gate fusion network. (Zhao, Abstract)
Regarding Claim 2, the combination of Wu and Zhao teach the computer-implemented method of claim 1, where Wu further teaches wherein generating the second set of visual features comprises generating visual features that have a lower resolution than the first set of visual features. Page 4, 3 VisionLLM v2, Paragraph 3, “For an image input, we utilize a pre-trained vision foundation model, such as CLIP [141], to extract image features. Recognizing that current vision models operate the images at a low resolution, we adopt the dynamic resolution approach [31] to process the input images. Specifically, the input image is first automatically matched to an optimal aspect ratio from a predefined ratio set. Subsequently, the image is scaled up to a higher resolution based on the selected aspect ratio and divided into P square patches, each whose resolutions are 336×336. These local patches, along with a 336×336 global image Iglobal, are processed by the image encoder to capture both holistic scenes and fine-grained details, resulting in image features Fimg ∈ R576(P+1)× C.”, In this section of the prior art, the CLIP encoder extracts image features that are scaled up to be high resolution visual image features.)
Regarding Claim 3, the combination of Wu and Zhao teach the computer-implemented method of claim 1, where Wu further teaches further comprising: determining, utilizing the visual-text encoder, text location information for the text string within the digital image; (Page 4, 3 VisionLLM v2, Paragraph 3, “For an image input, we utilize a pre-trained vision foundation model, such as CLIP [141], to extract image features.”, In this section of the prior art, the CLIP encoder extracts image and text features that are scaled up to be high resolution visual image features.); and generating the response from the first set of visual features, the second set of visual features, the text string, and the text location information utilizing the large language model. (Page 4, VisionLLM v2, Paragraph 2, “Tokenization. VisionLLM v2 is flexible for handling multimodal input. (1) For text prompts, we employ the text tokenizer to tokenize them into distinct vocabulary indices, which can be further processed by LLM and result in the text features Ftext ∈ RL× C, where L denotes the length of input text, and C is the channel dimension of LLM.” In this section of the prior art a tokenization is performed and as seen in figure 2 the text tokenizer’s output is input into a large language model in order to generate a response based on the text tokens.)
Regarding Claim 4, the combination of Wu and Zhao teach the computer-implemented method of claim 1, where Wu further teaches wherein generating the response comprises prompting the large language model with tokens for the first set of visual features, the second set of visual features, the text string, and the query. (See Figure 2 where, text tokens and image encoded features are input into a large language model in order to generate a text response pertaining to the image.)
Regarding Claim 7, the combination of Wu and Zhao teach the computer-implemented method of claim 6, where Wu further teaches wherein generating the response comprises prompting the large language model with the visual tokens, the text tokens, and the query tokens to generate the response for the query. (See Figure 2 where, text tokens and image encoded features are input into a large language model in order to generate a text response pertaining to the image.)
Regarding Claim 8, Wu teaches a system(Fig. 2) comprising: a memory component and one or more processing devices coupled to the memory component, the one or more processing devices to perform operations (Page 22, A More results, Paragraph 3 discloses using GPU which inherently would be coupled to a memory to perform the tasks of image analysis.) comprising: generating, utilizing a first visual encoder, low-resolution visual features of a digital image(Page 4, 3 VisionLLM v2, Paragraph 1, “The overall architecture of VisionLLM v2 is depicted in Figure 2. It mainly consists of four parts: (1) an image encoder and a region encoder that encode the image-level and region-level information;”, in this section of the prior art an image encoder is used to extract image level features from the image and Page 4, 3 VisionLLM v2, Paragraph 3 discloses the visual features extracted both low and high resolution features.); generating, utilizing a second visual encoder, high-resolution visual features of the digital image,
wherein the high-resolution visual features have a higher resolution than the low-resolution visual features; (Page 4, 3 VisionLLM v2, Paragraph 3, “For an image input, we utilize a pre-trained vision foundation model, such as CLIP [141], to extract image features. Recognizing that current vision models operate the images at a low resolution, we adopt the dynamic resolution approach [31] to process the input images. Specifically, the input image is first automatically matched to an optimal aspect ratio from a predefined ratio set. Subsequently, the image is scaled up to a higher resolution based on the selected aspect ratio and divided into P square patches, each whose resolutions are 336×336. These local patches, along with a 336×336 global image Iglobal, are processed by the image encoder to capture both holistic scenes and fine-grained details, resulting in image features Fimg ∈ R576(P+1)× C.”, In this section of the prior art, the CLIP encoder extracts image features that are scaled up to be high resolution visual image features.)and generating, for a query directed to text within the digital image, a response based on the visual tokens. (Page 4, VisionLLM v2, Paragraph 2, “Tokenization. VisionLLM v2 is flexible for handling multimodal input. (1) For text prompts, we employ the text tokenizer to tokenize them into distinct vocabulary indices, which can be further processed by LLM and result in the text features Ftext ∈ RL× C, where L denotes the length of input text, and C is the channel dimension of LLM.” In this section of the prior art a tokenization is performed and as seen in figure 2 the text tokenizer’s output is input into a large language model in order to generate a response based on the text tokens.)
Wu does not explicitly teach generating, by combining the high-resolution visual features and the low-resolution visual features at a dimensionality of the low-resolution visual features, a set of combined visual features for the digital image
Zhao teaches generating, by combining the high-resolution visual features and the low-resolution visual features at a dimensionality of the low-resolution visual features, a set of combined visual features for the digital image (Page 5, Section 3.2 Multi-Granularity Vision Flow, “Hybrid Vision Encoders As depicted in Fig. 3(b), MG-LLaVA initially processes images at two different resolutions: low-resolution VL and high-resolution VH. In the low-resolution branch, we follow the LLaVA-1.5 [31] to utilize a CLIP-pretrained ViT [57] denoted as fL V to derive low resolution features EL ∈ RN× C,”, as disclosed in this section and in shown in Fig. 3(b) the prior art processes the digital images at a low and high resolution using 2 encoders and in Section 3.2, “Conv-Gate Fusion Combining both low and high-resolution features as inputs results in a doubling of the visual tokens to be processed, which is computationally ineffective. Moreover, the distinct architectures of ViT and ConvNeXt lead to a discrepancy between EL and EH, requiring a careful fusion process. Inspired from [14], we implement a lightweight Conv-Gate fusion network that facilitates feature aggregation while maintaining a single resolution’s token count, as shown in Fig. 3(c).”, discloses these features are combined to form a combined set of visual features.)
Regarding Claim 9, the combination of Wu and Zhao teach the system of claim 8, where Wu further teaches wherein the operations further comprise: generating a prompt comprising instructions to determine a text string corresponding to the text within the digital image and generating the text string from the prompt utilizing a large language model(Page 4, VisionLLM v2, Paragraph 2, “Tokenization. VisionLLM v2 is flexible for handling multimodal input. (1) For text prompts, we employ the text tokenizer to tokenize them into distinct vocabulary indices, which can be further processed by LLM and result in the text features Ftext ∈ RL× C, where L denotes the length of input text, and C is the channel dimension of LLM.” In this section of the prior art a tokenization is performed and as seen in figure 2 the text tokenizer’s output is input into a large language model in order to generate a response based on the text tokens.);
Guo teaches adjusting parameters of the projection layer to reduce a measure of loss determined by comparing the text string with a ground truth text string for the text within the digital image (¶[0052], “The training loss may in one embodiment be configured to be an auto-regressive training objective. The target values for the visual language processor's learning function may be configured to be the desired responses. [0053] The learning function quantifies the visual language processor's prediction error and adjusts the visual language processor's parameters to minimize the difference between its predictions (responses) and the correct responses.”, ¶[0052]¶[0053] disclose determining a training loss by comparing the language processors response and the correct/ground truth response.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Wu with Guo in order to adjust the loss of the layers of the network. One skilled in the art would have been motivated to modify Wu in this manner in order to improve the visual language processor's performance on the trained tasks. (Guo, ¶[0053])
Regarding Claim 13, the combination of Wu and Zhao teach the system of claim 8, where Wu further teaches wherein the operations further comprise: determining, utilizing a visual-text encoder, a text string and text location information corresponding to the text within the digital image(Page 4, 3 VisionLLM v2, Paragraph 3, “For an image input, we utilize a pre-trained vision foundation model, such as CLIP [141], to extract image features.”, In this section of the prior art, the CLIP encoder extracts image and text features that are scaled up to be high resolution visual image features.); and generating the response based on the visual tokens, the text string, the text location information, and the query. (Page 4, VisionLLM v2, Paragraph 2, “Tokenization. VisionLLM v2 is flexible for handling multimodal input. (1) For text prompts, we employ the text tokenizer to tokenize them into distinct vocabulary indices, which can be further processed by LLM and result in the text features Ftext ∈ RL× C, where L denotes the length of input text, and C is the channel dimension of LLM.” In this section of the prior art a tokenization is performed and as seen in figure 2 the text tokenizer’s output is input into a large language model in order to generate a response based on the text tokens.)
Regarding Claim 14, the combination of Wu and Zhao teach the system of claim 8, where Wu further teaches wherein generating the response comprises prompting a large language model with the visual tokens and tokens for the query. (See Figure 2 where, text tokens and image encoded features are input into a large language model in order to generate a text response pertaining to the image.)
Regarding Claim 15, Wu teaches a non-transitory computer-readable medium storing executable instructions that, when executed by a processing device, cause the processing device to perform operations comprising: (Page 22, A More results, Paragraph 3 discloses using GPU which inherently would be coupled to a memory to perform the tasks of image analysis.) comprising: generating, utilizing a high-resolution visual encoder and a low-resolution visual encoder, a set of visual features for a digital image; (Page 4, 3 VisionLLM v2, Paragraph 3, “For an image input, we utilize a pre-trained vision foundation model, such as CLIP [141], to extract image features. Recognizing that current vision models operate the images at a low resolution, we adopt the dynamic resolution approach [31] to process the input images. Specifically, the input image is first automatically matched to an optimal aspect ratio from a predefined ratio set. Subsequently, the image is scaled up to a higher resolution based on the selected aspect ratio and divided into P square patches, each whose resolutions are 336×336. These local patches, along with a 336×336 global image Iglobal, are processed by the image encoder to capture both holistic scenes and fine-grained details, resulting in image features Fimg ∈ R576(P+1)× C.”, In this section of the prior art, the CLIP encoder extracts image features that are scaled up to be high resolution visual image features.)
determining, utilizing a visual-text encoder to extract text information from the digital image, a text string identifying text within the digital image (Page 4, VisionLLM v2, Paragraph 2, “Tokenization. VisionLLM v2 is flexible for handling multimodal input. (1) For text prompts, we employ the text tokenizer to tokenize them into distinct vocabulary indices, which can be further processed by LLM and result in the text features Ftext ∈ RL× C, where L denotes the length of input text, and C is the channel dimension of LLM.”, this section of the prior art discloses tokenizing the text in order to extract textual information from the image.)generating, utilizing a large language model, a response for a query directed to the text based on the visual tokens. (Page 4, VisionLLM v2, Paragraph 2, “Tokenization. VisionLLM v2 is flexible for handling multimodal input. (1) For text prompts, we employ the text tokenizer to tokenize them into distinct vocabulary indices, which can be further processed by LLM and result in the text features Ftext ∈ RL× C, where L denotes the length of input text, and C is the channel dimension of LLM.” In this section of the prior art a tokenization is performed and as seen in figure 2 the text tokenizer’s output is input into a large language model in order to generate a response based on the text tokens.)
Wu does not explicitly teach generating, by merging the first set of visual features of the digital image with the second set of visual features of the digital image at a dimensionality of the second set of visual features, a set of combined visual features
Zhao teaches generating, by merging the first set of visual features of the digital image with the second set of visual features of the digital image at a dimensionality of the second set of visual features, a set of combined visual features (Page 5, Section 3.2 Multi-Granularity Vision Flow, “Hybrid Vision Encoders As depicted in Fig. 3(b), MG-LLaVA initially processes images at two different resolutions: low-resolution VL and high-resolution VH. In the low-resolution branch, we follow the LLaVA-1.5 [31] to utilize a CLIP-pretrained ViT [57] denoted as fL V to derive low resolution features EL ∈ RN× C,”, as disclosed in this section and in shown in Fig. 3(b) the prior art processes the digital images at a low and high resolution using 2 encoders and in Section 3.2, “Conv-Gate Fusion Combining both low and high-resolution features as inputs results in a doubling of the visual tokens to be processed, which is computationally ineffective. Moreover, the distinct architectures of ViT and ConvNeXt lead to a discrepancy between EL and EH, requiring a careful fusion process. Inspired from [14], we implement a lightweight Conv-Gate fusion network that facilitates feature aggregation while maintaining a single resolution’s token count, as shown in Fig. 3(c).”, discloses these features are combined to form a combined set of visual features.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Wu with Zhao in order to combining the high and low resolution into a set of combined visual features. One skilled in the art would have been motivated to modify Wu in this manner in order to capture fine grained details, which are then fused with base visual features through a Conv-Gate fusion network. (Zhao, Abstract)
Regarding Claim 17, the combination of Wu and Zhao teach the non-transitory computer-readable medium of claim 16, where Wu further teaches wherein the operations further comprise: generating, from the text information, text tokens for the text string identifying the text within the digital image; and generating, utilizing the large language model, the response for the query from the text tokens. (See Figure 2 where, text tokens and image encoded features are input into a large language model in order to generate a text response pertaining to the image.)
Regarding Claim 19, the combination of Wu and Zhao teach the non-transitory computer-readable medium of claim 15, where Zhao further teaches wherein generating the set of combined visual features comprises merging outputs of a first fully connected layer within the high-resolution visual encoder with outputs of a second fully connected layer within the low-resolution visual encoder (Page 5, Section 3.2 Multi-Granularity Vision Flow, “Hybrid Vision Encoders As depicted in Fig. 3(b), MG-LLaVA initially processes images at two different resolutions: low-resolution VL and high-resolution VH. In the low-resolution branch, we follow the LLaVA-1.5 [31] to utilize a CLIP-pretrained ViT [57] denoted as fL V to derive low resolution features EL ∈ RN× C,”, as disclosed in this section and in shown in Fig. 3(b) the prior art processes the digital images at a low and high resolution using 2 encoders and in Section 3.2, “Conv-Gate Fusion Combining both low and high-resolution features as inputs results in a doubling of the visual tokens to be processed, which is computationally ineffective. Moreover, the distinct architectures of ViT and ConvNeXt lead to a discrepancy between EL and EH, requiring a careful fusion process. Inspired from [14], we implement a lightweight Conv-Gate fusion network that facilitates feature aggregation while maintaining a single resolution’s token count, as shown in Fig. 3(c).”, discloses these features are combined to form a combined set of visual features.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Wu with Zhao in order to combining the high and low resolution into a set of combined visual features. One skilled in the art would have been motivated to modify Wu in this manner in order to capture fine grained details, which are then fused with base visual features through a Conv-Gate fusion network. (Zhao, Abstract)
Regarding Claim 20, the combination of Wu and Zhao teach the non-transitory computer-readable medium of claim 15, where Wu further teaches wherein the operations further comprise: determining, utilizing the visual-text encoder, text location information for the text string within the digital image; (Page 4, 3 VisionLLM v2, Paragraph 3, “For an image input, we utilize a pre-trained vision foundation model, such as CLIP [141], to extract image features.”, In this section of the prior art, the CLIP encoder extracts image and text features that are scaled up to be high resolution visual image features.); and generating the response from the first set of visual features, the second set of visual features, the text string, and the text location information utilizing the large language model. (Page 4, VisionLLM v2, Paragraph 2, “Tokenization. VisionLLM v2 is flexible for handling multimodal input. (1) For text prompts, we employ the text tokenizer to tokenize them into distinct vocabulary indices, which can be further processed by LLM and result in the text features Ftext ∈ RL× C, where L denotes the length of input text, and C is the channel dimension of LLM.” In this section of the prior art a tokenization is performed and as seen in figure 2 the text tokenizer’s output is input into a large language model in order to generate a response based on the text tokens.)
Claims 5-6 are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. ("VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks") in view of Zhao et al. ("MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning") in view of Guo et al. US PG-Pub(US 20250272959 A1).
Regarding Claim 5, while the combination of Wu and Zhao teach the computer-implemented method of claim 1, they do not explicitly teach generating, utilizing a projection layer to transform the set of combined visual features, visual tokens for the digital image;
Guo teaches generating, utilizing a projection layer to transform the set of combined visual features, visual tokens for the digital image; ([0037], “The visual-language connector 212 may comprise a multi-layer perceptron configured to project the image-level features Z.sub.v and the region-level features Z.sub.r of the image into text embeddings H.sub.v and H.sub.r, respectively, and these are merged with embeddings for a text prompt in a language encoder 204 (e.g., a Large Language Model).”, ¶[0037] discloses using a projection layer to generate text embeddings related to the input image. ¶[0050] “The language encoder 204 is configured (trained) to tokenize and transform text inputs into word embeddings. The image-level features Z.sub.v and the region-level image features Z.sub.r are input into the embedding space of the language encoder 204 along with the embedding of the text input prompt.”, ¶[0050] discloses that the image features are being tokenized into word embeddings. )
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Wu and Zhao with Guo in order to combining the high and low resolution into a set of combined visual features and using a projection layer to generate tokens. One skilled in the art would have been motivated to modify Wu and Zhao in this manner in order to improve the visual language processor's performance on the trained tasks. (Guo, ¶[0053])
Regarding Claim 6, the combination of Wu, Zhao and Guo teach the computer-implemented method of claim 5, where Wu further teaches further comprising: generating, utilizing a text tokenizer, text tokens for the text of the digital image; (Page 4, 3 VisionLLM v2, Paragraph 2, discloses using a text tokenizer to generate text tokens for the digital image); and generating, utilizing the text tokenizer, query tokens for the query directed to the text of the digital image. (Page 4, VisionLLM v2, Paragraph 2, “Tokenization. VisionLLM v2 is flexible for handling multimodal input. (1) For text prompts, we employ the text tokenizer to tokenize them into distinct vocabulary indices, which can be further processed by LLM and result in the text features Ftext ∈ RL× C, where L denotes the length of input text, and C is the channel dimension of LLM.” In this section of the prior art a tokenization is performed and as seen in figure 2 the text tokenizer’s output is input into a large language model in order to generate a response based on the text tokens.)
Claims 10-11, 16 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. ("VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks") in view of Zhao et al. ("MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning") in view of Rimchala et al. US PG-Pub(US 20250140012 A1).
Regarding Claim 10, the combination of Wu and Zhao teach the system of claim 8, where Wu further teaches wherein the operations further comprise: generating a prompt comprising instructions to determine text location information for the text within the digital image (Page 4, 3 VisionLLM v2, Paragraph 4, “For a visual prompt, we employ binary masks to flexibly represent the visual prompts, such as point, box, scribble, and mask. To extract the region embedding, we first concatenate the binary mask with the input image along the channel dimension and then process it with three convolutional layers to downsample by a factor of 14 (see appendix for more details).”, this section of the prior art discloses generating a prompt to extract text regions from the input image.); generating the text location information from the prompt utilizing a large language model (Fig. 2 shows the region encoder output being input into a large language model to generate text location information and a text response.);
However, Wu and Zhao do not explicitly teach adjusting parameters of the projection layer to reduce a measure of loss determined by comparing the text location information with ground truth text location information for the text within the digital image.
Rimchala teaches adjusting parameters of the projection layer to reduce a measure of loss determined by comparing the text location information with ground truth text location information for the text within the digital image. ([0109] “Note that the process of using reference values for key-value pairs is used for stage 2 training (FIG. 3B), not for stage 1 training (FIG. 3A). In contrast, the sub-method for stage 1 training (FIG. 3A) uses the digital image and corresponding optical character recognition text extracted from the digital image.[0110] Step 304 includes generating a loss function by comparing the output to the reference output. The loss function may be instructions to adjust one or more tunable aspects of the projection network model, such as hyperparameters, or other adjustable aspects of the machine learning model used for the projection network model. Thus, generating the loss function may include executing instructions that modify the aspects of the projection model in a manner that may cause the projection network to generate a new output that is closer in value or values to the reference output.”, ¶[0109] discloses comparing the text values extracted with a reference/ground truth value and ¶[0110] discloses updating the parameters of the projection network based on a loss generated from the comparison of the text vs ground truth text.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Wu and Zhao with Rimchala in order to adjust the parameters of the projection layer based on the predicted text location. One skilled in the art would have been motivated to modify Wu and Zhao in this manner in order to generate a new output that is closer in value or values to the reference output. (Rimchala, ¶[0110])
Regarding Claim 11, while the combination of Wu and Zhao teach the system of claim 8, they do not explicitly teach wherein the operations further comprise: generating a prompt comprising instructions to determine plain text and text location information for the text within the digital image; parsing the digital image to generate the plain text and the text location information from the prompt utilizing a large language model; and adjusting parameters of the projection layer to reduce a measure of loss determined by: comparing the plain text with ground truth text for the text within the digital image; and comparing the text location information with ground truth text location information for the text within the digital image.
Rimchala teaches wherein the operations further comprise: generating a prompt(¶[0044] discloses generating a prompt and figure 2 shows the process of generating the prompt in further detail) comprising instructions to determine plain text and text location information for the text within the digital image([0033] More specifically, the layout text vector (112) is a vector that encodes at least one word in the text of the digital image and also encodes a position of the at least one word in the layout of the digital image. For example, the layout text vector (112) may describe the numerical values of a table shown in the digital image (102) and further describe the arrangement of the numerical values of the table within the digital image (102).);; parsing the digital image to generate the plain text and the text location information from the prompt utilizing a large language model(¶[0031] discloses determining the location and position of the text within the digital image using a LLM.) and adjusting parameters of the projection layer to reduce a measure of loss determined by: comparing the plain text with ground truth text for the text within the digital image and comparing the text location information with ground truth text location information for the text within the digital image. ([0109] “Note that the process of using reference values for key-value pairs is used for stage 2 training (FIG. 3B), not for stage 1 training (FIG. 3A). In contrast, the sub-method for stage 1 training (FIG. 3A) uses the digital image and corresponding optical character recognition text extracted from the digital image.[0110] Step 304 includes generating a loss function by comparing the output to the reference output. The loss function may be instructions to adjust one or more tunable aspects of the projection network model, such as hyperparameters, or other adjustable aspects of the machine learning model used for the projection network model. Thus, generating the loss function may include executing instructions that modify the aspects of the projection model in a manner that may cause the projection network to generate a new output that is closer in value or values to the reference output.”, ¶[0109] discloses comparing the text values extracted with a reference/ground truth value and ¶[0110] discloses updating the parameters of the projection network based on a loss generated from the comparison of the text vs ground truth text.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Wu and Zhao with Rimchala in order to adjust the parameters of the projection layer based on the predicted text location. One skilled in the art would have been motivated to modify Wu and Zhao in this manner in order to generate a new output that is closer in value or values to the reference output. (Rimchala, ¶[0110])
Regarding Claim 16, the combination of Wu and Zhao teach the non-transitory computer-readable medium of claim 15, where Wu further teaches wherein the operations further comprise: generating the response for the query utilizing the large language model from the visual tokens and tokens for the query; (Page 4, VisionLLM v2, Paragraph 2, “Tokenization. VisionLLM v2 is flexible for handling multimodal input. (1) For text prompts, we employ the text tokenizer to tokenize them into distinct vocabulary indices, which can be further processed by LLM and result in the text features Ftext ∈ RL× C, where L denotes the length of input text, and C is the channel dimension of LLM.” In this section of the prior art a tokenization is performed and as seen in figure 2 the text tokenizer’s output is input into a large language model in order to generate a response based on the text tokens.)
However, Wu and Zhao do not explicitly teach adjusting parameters of the large language model to reduce a measure of loss determined by comparing the response with a ground truth response for the query.
Rimchala teaches adjusting parameters of the large language model to reduce a measure of loss determined by comparing the response with a ground truth response for the query. ([0109] “Note that the process of using reference values for key-value pairs is used for stage 2 training (FIG. 3B), not for stage 1 training (FIG. 3A). In contrast, the sub-method for stage 1 training (FIG. 3A) uses the digital image and corresponding optical character recognition text extracted from the digital image.[0110] Step 304 includes generating a loss function by comparing the output to the reference output. The loss function may be instructions to adjust one or more tunable aspects of the projection network model, such as hyperparameters, or other adjustable aspects of the machine learning model used for the projection network model. Thus, generating the loss function may include executing instructions that modify the aspects of the projection model in a manner that may cause the projection network to generate a new output that is closer in value or values to the reference output.”, ¶[0109] discloses comparing the text values extracted with a reference/ground truth value and ¶[0110] discloses updating the parameters of the projection network based on a loss generated from the comparison of the text vs ground truth text.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Wu and Zhao with Rimchala in order to adjust the parameters of the projection layer based on the predicted text location. One skilled in the art would have been motivated to modify Wu and Zhao in this manner in order to generate a new output that is closer in value or values to the reference output. (Rimchala, ¶[0110])
Regarding Claim 18, while the combination of Wu and Zhao teach the non-transitory computer-readable medium of claim 16, they do not explicitly teach wherein the operations further comprise adjusting parameters of the projection layer to reduce the measure of loss determined by comparing the response with the ground truth response for the query.
Rimchala teaches wherein the operations further comprise adjusting parameters of the projection layer to reduce the measure of loss determined by comparing the response with the ground truth response for the query. ([0109] “Note that the process of using reference values for key-value pairs is used for stage 2 training (FIG. 3B), not for stage 1 training (FIG. 3A). In contrast, the sub-method for stage 1 training (FIG. 3A) uses the digital image and corresponding optical character recognition text extracted from the digital image.[0110] Step 304 includes generating a loss function by comparing the output to the reference output. The loss function may be instructions to adjust one or more tunable aspects of the projection network model, such as hyperparameters, or other adjustable aspects of the machine learning model used for the projection network model. Thus, generating the loss function may include executing instructions that modify the aspects of the projection model in a manner that may cause the projection network to generate a new output that is closer in value or values to the reference output.”, ¶[0109] discloses comparing the text values extracted with a reference/ground truth value and ¶[0110] discloses updating the parameters of the projection network based on a loss generated from the comparison of the text vs ground truth text.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Wu and Zhao with Rimchala in order to adjust the parameters of the projection layer based on the predicted text location. One skilled in the art would have been motivated to modify Wu and Zhao in this manner in order to generate a new output that is closer in value or values to the reference output. (Rimchala, ¶[0110])
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. ("VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks") in view of Zhao et al. ("MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning") in view of Chen et al. ("TextDiffuser: Diffusion Models as Text Painter").
Regarding Claim 12, while the combination of Wu and Zhao teach the system of claim 8, they do not explicitly teach wherein the operations further comprise: generating a prompt comprising instructions to reconstruct a layout of the text within the digital image; generating a textual layout of the text within the digital image from the prompt utilizing a large language model; and adjusting parameters of the projection layer to reduce a measure of loss determined by comparing the textual layout with a ground truth layout for the text within the digital image.
Chen teaches wherein the operations further comprise: generating a prompt comprising instructions to reconstruct a layout of the text within the digital image (Page 2, Introduction, Paragraph 3, “Figure 1 illustrates the application of TextDiffuser in generating accurate and coherent text images using text prompts alone or text template images. Additionally, TextDiffuser is capable of performing text inpainting2 to reconstruct incomplete images with text.”, in this section of the prior art, reconstruction is performed to reconstruct the layout of text in the image.) generating a textual layout of the text within the digital image from the prompt utilizing a large language model(Page 3, 3.1 Stage1: Layout Generation, Paragraph 1, “In this stage, the objective is to utilize bounding boxes to determine the layout of keywords (enclosed with quotes specified by user prompts). Inspired by Layout Transformer [20], we utilize the Trans former architecture to obtain the layout of keywords. Formally, we denote the tokenized prompt as P = (p0,p1,...,pL−1), where L means the maximum length of tokens. Following LDM [67], we use CLIP [60] and two linear layers to encode the sequence as CLIP(P) ∈ RL×d, where d is the dimension of latent space. To distinguish the keywords against others, we design a keyword embedding Key(P) ∈ RL×d with two entries (i.e., keywords and non-keywords). Furthermore, we encode the width of keywords with an embedding layer Width(P) ∈ RL×d.”, this section of the prior art discloses how the layout is generated using an LLM.); and adjusting parameters of the projection layer to reduce a measure of loss determined by comparing the textual layout with a ground truth layout for the text within the digital image. (Figure 2 shows the network is trained based on denoising and character aware loss in order to adjust the parameters of the network.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Wu and Zhao with Chen in order to generate a layout of the text in the image. One skilled in the art would have been motivated to modify Wu and Zhao in this manner in order to create high-quality text images using text prompts alone or together with text template images, and conduct text inpainting to reconstruct incomplete images with text. (Chen, Abstract)
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HAN D HOANG whose telephone number is (571)272-4344. The examiner can normally be reached Monday-Friday 8-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JOHN M VILLECCO can be reached at 571-272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HAN HOANG/Primary Examiner, Art Unit 2661
Prosecution Insights