DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1 – 20 are rejected under 35 U.S.C. 103 as being unpatentable over Song et al. (See Machine Translation for CN 117273107 A; hereafter referred to as Song) in view of Xia et al. (See Machine Translation for CN 117313740 A; hereafter referred to as Xia).
Regarding Claim 1, Song teaches:
A method for generating a text description for an image (Song, page 2, para 5, “a training method for a text generation model is provided”), comprising:
generating a first feature of the image by a visual encoder, used to train a conversion model and a language model (Song, page 7, para 3 – 5, “The visual encoder can be used to process the input image to obtain image coding features, and the query converter can be used to process the image coding features to obtain image visual features… using the feature extraction unit to extract sample visual features of the sample image may include: using a visual encoder to extract image encoding features of the sample image; and using a query converter to process the image encoding features and the query token to obtain the sample visual characteristics… the sample image features are input into the large language model to obtain predicted text corresponding to the sample image”);
converting the first feature into a second feature by the conversion model, wherein the first feature and the second feature correspond to different feature spaces (Song, page 8, para 7, “As shown in FIG. 4, in process 400, the input image 401 can be input into the visual encoder 402 to obtain image encoding features of the input image. The image encoding features can then be given as input to the query converter 403 as key (K) or value (V). The query converter 403 outputs sample visual features of the input image 401 based on the input keys (K), values (V), and query vectors (Q).. Before inputting the features into the large language model 405, image embedding can be performed on the mapped sample visual features, so that the features input into the large language model 405 conform to the input format of the large language model.”); and
generating, by the language model, a text description for the image based on the second feature (Song, page 5, step S206, “the sample image features are input into the large language model to obtain predicted text corresponding to the sample image”; Song, page 8, para 7, “A large language model can be used to process the mapped sample visual features to obtain a text generation result for the input image 401”).
However, Song does not explicitly recite:
wherein a text encoder semantically aligned with the visual encoder is used to train a conversion model and a language model;
In the same field of endeavor, Xia teaches:
wherein a text encoder semantically aligned with the visual encoder is used to train a conversion model and a language model (Xia, page 3, para 14, “The generation model module is configured to add a feature conversion unit and a text generation unit to the target image-text semantic alignment model, and generate a language model to be trained based on the addition results”; Xia, page 8, para 3, “the image encoding The unit specifically refers to the unit that can encode image data, and is used to convert the image data into image encoding vectors to characterize the characteristics of the image data in the form of vectors; accordingly, the image-text alignment unit specifically refers to the unit that can encode the image data and The text data matching unit is used to enable the image-text semantic alignment model to learn the ability to find image data matching text”);
Song and Xia are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Song with the invention of Xia to make the invention that uses a text encoder that is semantically aligned with the visual encoder to train a conversion model and a language model; doing to can yield predictable result of improved performance by consuming less time and improving speed of generating the text description for the input image (Xiao, page 2); thus one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 2, Song in view of Xia teaches the method according to claim 1, wherein generating, by the language model, a text description for the image based on the second feature comprises:
acquiring a prompt content for the image, wherein the prompt content is used to indicate a type of the text description (Song, page 8, para 5, “the model 302 can be used to process the input image 301 to obtain the description text 303 of the input image 301... Among them, model 302 may be a BLIP-2 model. Then, the description text 303 and the prompt information 304 can be input into the large language model 305 together, where the prompt information 304 is used to indicate the style of the text 306 output by the large language model 305”); and
generating, by the language model, the text description for the image based on the second feature and the prompt content (Song, page 8, para 5, “The large language model 305 outputs text 306 of a specified style based on the description text 303 and the prompt information 304”).
Regarding Claim 3, Song in view of Xia teaches the method according to claim 1, further comprising:
generating a first text feature of a training text by the text encoder (Xia, page 8, last para, “the text data in the image-text pair data is input to the text feature extraction sub-unit in the image-text alignment unit. The unit performs processing; generates image features corresponding to the image data and text features corresponding to the text data according to the processing results”);
converting the first text feature into a second text feature by the conversion model, wherein the first text feature and the second text feature correspond to different feature spaces (Xia, page 10, para 5, “the original A feature conversion unit is added to the model structure to convert the dimensions and meaning of vectors, so that the converted vectors can be understood and processed by the text processing unit. Correspondingly, the text generation unit specifically refers to a processing unit that can generate text description information based on the vector output by the feature conversion unit”);
generating, by the language model, a training text description based on the second text feature (Xia, page 10, para 6, “the language model to be trained specifically refers to a language model with a new structure formed by adding a feature conversion unit and a text generation unit on the basis of the target image-text semantic alignment model. This model has the ability to output text based on the input image”); and
training the conversion model and the language model based on a loss between the training text and the training text description (Xia, page 8, last para, “calculates a loss value based on the image features and the text features, and calculates the initial value based on the loss value. Parameters of the image-text semantic alignment model are adjusted until the target image-text semantic alignment model that satisfies the first training stop condition is obtained.”; page 10, para 6, “when generating the language model to be trained based on the target image and text semantic alignment model, the target image and text semantic alignment model is actually used as the basic model, and a feature conversion unit and a text generation unit are added on this basis”).
Regarding Claim 4, Song in view of Xia teaches the method according to claim 3, wherein features of a training image in a training image-text pair generated by the visual encoder and features of a training text in the training image-text pair generated by the text encoder satisfy similarity conditions (Xia, page 10, para 5, “image-text comparison loss is used, and based on the simulation box mask, the output of each learnable query (learnable query) calculates pairwise similarity with the CLS tokens of the text output. The learnable query and the text are invisible to each other, maximizing the mutual information between the image representation and the text representation”).
Regarding Claim 5, Song in view of Xia teaches the method according to claim 3, wherein training the conversion model and the language model comprises:
adjusting parameters in the conversion model and the language model based on a loss between the training text and the training text description, wherein in adjusting process, parameters in the visual encoder and the text encoder are unchanged (Xia, page 10, para 1, “adjust the parameters of the image and text semantic alignment model based on the calculation results, and so on, until a better image and text alignment ability is obtained. After a strong target image-text semantic alignment model is established, it can be used to subsequently build a language model”); and
determining completion of the training of the conversion model and the language model in response to the loss between the training text and the training text description satisfying a convergence condition (Xia, page 9, para 1 – 2, “According to the loss value, the image and text alignment unit and image coding unit in the initial image and text semantic alignment model are adjusted., to achieve the target image-text semantic alignment model that satisfies the training stop condition… in order to ensure that the model has a high image-text alignment capability, the loss value can be calculated based on image features and text features, and then the parameters of the initial image-text semantic alignment model can be adjusted based on the loss value. If the image-text semantic alignment model after parameter adjustment does not meet the first training stop condition, you can select new image-text pair data, and then perform the above training until the target image-text semantic alignment model that meets the first training stop condition is obtained”).
Regarding Claim 6, Song in view of Xia teaches the method according to claim 4, wherein the visual encoder is a visual encoding sub-model in a text-image alignment model, and the text encoder is a text encoding sub-model in the text-image alignment model (Xia, page 8, last para, “The image encoding vector is input to the image feature extraction sub-unit in the image-text alignment unit for processing, and the text data in the image-text pair data is input to the text feature extraction sub-unit in the image-text alignment unit”).
Regarding Claim 7, Song in view of Xia teaches the method according to claim 3, further comprising:
generating a first image feature of a training image in a training image-text pair by an image encoder semantically aligned with the text encoder (Xia, page 8, para 1, “the process of using the image-text pair data set to train the initial image-text semantic alignment model, in order to prepare the basic model for the language model, the initial image-text semantic alignment model including the image coding unit and the image-text alignment unit can be combined Carry out training”);
converting the first image feature into a second image feature by the conversion model, wherein the first image feature and the second image feature correspond to different feature spaces (Song, page 7, para 4, “using a query converter to process the image encoding features and the query token to obtain the sample visual characteristics. Among them, Q-Former can extract the features most relevant to the image from the image encoding features.”; Xia, page 8, para 3, “the image encoding The unit specifically refers to the unit that can encode image data, and is used to convert the image data into image encoding vectors to characterize the characteristics of the image data in the form of vectors; accordingly, the image-text alignment unit specifically refers to the unit that can encode the image data and The text data matching unit is used to enable the image-text semantic alignment model to learn the ability to find image data matching text”);
generating, by the language model, a fine-tuned text description for the training image based on the second image feature (Song, page 3, para 3, “the ability of a large language model can be utilized to quickly and efficiently generate corresponding text content for a given image, thereby quickly obtaining a multi-modal training data set for instruction fine-tuning of the text generation model”; Song, page 5, last para, “Using the method for training a text generation model provided by embodiments of the present disclosure, the ability of a large language model can be used to quickly and efficiently generate corresponding text content for a given image, thereby quickly obtaining instructions for the text generation model. Fine-tuned multi-modal training dataset”; Xia, page 6, para 4, “large models only require a small number of samples to fine-tune the pre-trained model and can be used in different tasks”); and
fine tuning the image encoder based on a loss between the training text in the training image-text pair and the fine-tuned text description (Xia, page 9, para 1, “the loss value specifically refers to the loss value obtained after calculating the image features and text features based on the preset loss function. According to the loss value, the image and text alignment unit and image coding unit in the initial image and text semantic alignment model are adjusted., to achieve the target image-text semantic alignment model that satisfies the training stop condition”).
Regarding Claim 8, Song in view of Xia teaches the method according to claim 7, wherein generating, by the language model, a fine-tuned text description for the training image based on the second image feature comprises:
acquiring a training prompt content for the training image-text pair, wherein the training prompt content is used to indicate a type of the fine-tuned text description (Song, page 6, para 9, “use the pre-trained second text generation model to process the sample image to obtain a text description of the sample image; input the text description of the sample image together with the prompt information A large language model is used to obtain sample text corresponding to the sample image, wherein the prompt information specifies the style of the sample text”; Xia, page 16, para 1); and
generating, by the language model, the fine-tuned text description for the training image based on the second image feature and the training prompt content (Song, page 6, para 9, “A large language model is used to obtain sample text corresponding to the sample image, wherein the prompt information specifies the style of the sample text. In some examples, the same prompt information is used for all sample images, so that a sample data set for a specified text generation task can be obtained”).
Regarding Claim 9, Song in view of Xia teaches the method according to claim 1, wherein the feature space comprises at least one of a feature size and a feature space distribution (Song, page 6, para 4, “image feature extractor extracts image features and maps the image features to a text embedding space”; Song, page 7, para 2, S204, “feature extraction unit to extract sample visual features of the sample image… image features can be extracted using edge detection, corner detection, texture analysis, color histogram, etc... deep learning-based methods can also be used to extract image features”).
Regarding Claim 10, Song in view of Xia teaches the method according to claim 1, further comprising:
acquiring the image (Song, page 2, para 6, “sample data acquisition unit configured to acquire a sample data set, wherein the sample data set includes a sample image and a sample corresponding to the sample image”);
wherein acquiring the image comprises:
acquiring a video (Xia, page 7, para 2, “as video resources… if the user uploads a video clip”); and
extracting the image from a video frame of the video (Xia, page 7, para 2, “as video resources, image resources, text resources, etc... if the user uploads a video clip, a frame can be intercepted from the video clip as an image for subsequent processing”).
Regarding Claim 11, Song teaches:
An electronic device, comprising:
a processor (Song, page 9, para 11, “at least one processor”); and
a memory coupled with the processor, wherein the memory has instructions stored therein, and the instructions, when executed by the processor (Song, page 9, para 11, “a memory communicatively connected to the at least one processor; wherein the memory stores information that can be executed by the at least one processor. instructions, which are executed by the at least one processor to enable the at least one processor to perform methods”), cause the electronic device to:
generate a first feature of the image by a visual encoder, used to train a conversion model and a language model (Song, page 7, para 3 – 5, “The visual encoder can be used to process the input image to obtain image coding features, and the query converter can be used to process the image coding features to obtain image visual features… using the feature extraction unit to extract sample visual features of the sample image may include: using a visual encoder to extract image encoding features of the sample image; and using a query converter to process the image encoding features and the query token to obtain the sample visual characteristics… the sample image features are input into the large language model to obtain predicted text corresponding to the sample image”);
convert the first feature into a second feature by the conversion model, wherein the first feature and the second feature correspond to different feature spaces (Song, page 8, para 7, “As shown in FIG. 4, in process 400, the input image 401 can be input into the visual encoder 402 to obtain image encoding features of the input image. The image encoding features can then be given as input to the query converter 403 as key (K) or value (V). The query converter 403 outputs sample visual features of the input image 401 based on the input keys (K), values (V), and query vectors (Q).. Before inputting the features into the large language model 405, image embedding can be performed on the mapped sample visual features, so that the features input into the large language model 405 conform to the input format of the large language model.”); and
generate, by the language model, a text description for the image based on the second feature (Song, page 5, step S206, “the sample image features are input into the large language model to obtain predicted text corresponding to the sample image”; Song, page 8, para 7, “A large language model can be used to process the mapped sample visual features to obtain a text generation result for the input image 401”).
However, Song does not explicitly recite:
wherein a text encoder semantically aligned with the visual encoder is used to train a conversion model and a language model;
In the same field of endeavor, Xia teaches:
wherein a text encoder semantically aligned with the visual encoder is used to train a conversion model and a language model (Xia, page 3, para 14, “The generation model module is configured to add a feature conversion unit and a text generation unit to the target image-text semantic alignment model, and generate a language model to be trained based on the addition results”; Xia, page 8, para 3, “the image encoding The unit specifically refers to the unit that can encode image data, and is used to convert the image data into image encoding vectors to characterize the characteristics of the image data in the form of vectors; accordingly, the image-text alignment unit specifically refers to the unit that can encode the image data and The text data matching unit is used to enable the image-text semantic alignment model to learn the ability to find image data matching text”);
Song and Xia are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Song with the invention of Xia to make the invention that uses a text encoder that is semantically aligned with the visual encoder to train a conversion model and a language model; doing to can yield predictable result of improved performance by consuming less time and improving speed of generating the text description for the input image (Xiao, page 2); thus one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 12, Song in view of Xia teaches the device according to claim 11, wherein the instructions causing the electronic device to generate, by the language model, a text description for the image based on the second feature further cause the electronic device to:
acquire a prompt content for the image, wherein the prompt content is used to indicate a type of the text description (Song, page 8, para 5, “the model 302 can be used to process the input image 301 to obtain the description text 303 of the input image 301... Among them, model 302 may be a BLIP-2 model. Then, the description text 303 and the prompt information 304 can be input into the large language model 305 together, where the prompt information 304 is used to indicate the style of the text 306 output by the large language model 305”); and
generate, by the language model, the text description for the image based on the second feature and the prompt content (Song, page 8, para 5, “The large language model 305 outputs text 306 of a specified style based on the description text 303 and the prompt information 304”).
Regarding Claim 13, Song in view of Xia teaches the device according to claim 11, further comprising:
generate a first text feature of a training text by the text encoder (Xia, page 8, last para, “the text data in the image-text pair data is input to the text feature extraction sub-unit in the image-text alignment unit. The unit performs processing; generates image features corresponding to the image data and text features corresponding to the text data according to the processing results”);
convert the first text feature into a second text feature by the conversion model, wherein the first text feature and the second text feature correspond to different feature spaces (Xia, page 10, para 5, “the original A feature conversion unit is added to the model structure to convert the dimensions and meaning of vectors, so that the converted vectors can be understood and processed by the text processing unit. Correspondingly, the text generation unit specifically refers to a processing unit that can generate text description information based on the vector output by the feature conversion unit”);
generate, by the language model, a training text description based on the second text feature (Xia, page 10, para 6, “the language model to be trained specifically refers to a language model with a new structure formed by adding a feature conversion unit and a text generation unit on the basis of the target image-text semantic alignment model. This model has the ability to output text based on the input image”); and
train the conversion model and the language model based on a loss between the training text and the training text description (Xia, page 8, last para, “calculates a loss value based on the image features and the text features, and calculates the initial value based on the loss value. Parameters of the image-text semantic alignment model are adjusted until the target image-text semantic alignment model that satisfies the first training stop condition is obtained.”; page 10, para 6, “when generating the language model to be trained based on the target image and text semantic alignment model, the target image and text semantic alignment model is actually used as the basic model, and a feature conversion unit and a text generation unit are added on this basis”).
Regarding Claim 14, Song in view of Xia teaches the device according to claim 13, wherein features of a training image in a training image-text pair generated by the visual encoder and features of a training text in the training image-text pair generated by the text encoder satisfy similarity conditions (Xia, page 10, para 5, “image-text comparison loss is used, and based on the simulation box mask, the output of each learnable query (learnable query) calculates pairwise similarity with the CLS tokens of the text output. The learnable query and the text are invisible to each other, maximizing the mutual information between the image representation and the text representation”).
Regarding Claim 15, Song in view of Xia teaches the device according to claim 13, wherein the instructions causing the electronic device to train the conversion model and the language model further cause the electronic device to:
adjust parameters in the conversion model and the language model based on a loss between the training text and the training text description, wherein in adjusting process, parameters in the visual encoder and the text encoder are unchanged (Xia, page 10, para 1, “adjust the parameters of the image and text semantic alignment model based on the calculation results, and so on, until a better image and text alignment ability is obtained. After a strong target image-text semantic alignment model is established, it can be used to subsequently build a language model”); and
determine completion of the training of the conversion model and the language model in response to the loss between the training text and the training text description satisfying a convergence condition (Xia, page 9, para 1 – 2, “According to the loss value, the image and text alignment unit and image coding unit in the initial image and text semantic alignment model are adjusted., to achieve the target image-text semantic alignment model that satisfies the training stop condition… in order to ensure that the model has a high image-text alignment capability, the loss value can be calculated based on image features and text features, and then the parameters of the initial image-text semantic alignment model can be adjusted based on the loss value. If the image-text semantic alignment model after parameter adjustment does not meet the first training stop condition, you can select new image-text pair data, and then perform the above training until the target image-text semantic alignment model that meets the first training stop condition is obtained”).
Regarding Claim 16, Song in view of Xia teaches the device according to claim 14, wherein the visual encoder is a visual encoding sub-model in a text-image alignment model, and the text encoder is a text encoding sub-model in the text-image alignment model (Xia, page 8, last para, “The image encoding vector is input to the image feature extraction sub-unit in the image-text alignment unit for processing, and the text data in the image-text pair data is input to the text feature extraction sub-unit in the image-text alignment unit”).
Regarding Claim 17, Song in view of Xia teaches the device according to claim 13, the instructions further cause the electronic device to:
generate a first image feature of a training image in a training image-text pair by an image encoder semantically aligned with the text encoder (Xia, page 8, para 1, “the process of using the image-text pair data set to train the initial image-text semantic alignment model, in order to prepare the basic model for the language model, the initial image-text semantic alignment model including the image coding unit and the image-text alignment unit can be combined Carry out training”);
convert the first image feature into a second image feature by the conversion model, wherein the first image feature and the second image feature correspond to different feature spaces (Song, page 7, para 4, “using a query converter to process the image encoding features and the query token to obtain the sample visual characteristics. Among them, Q-Former can extract the features most relevant to the image from the image encoding features.”; Xia, page 8, para 3, “the image encoding The unit specifically refers to the unit that can encode image data, and is used to convert the image data into image encoding vectors to characterize the characteristics of the image data in the form of vectors; accordingly, the image-text alignment unit specifically refers to the unit that can encode the image data and The text data matching unit is used to enable the image-text semantic alignment model to learn the ability to find image data matching text”);
generate, by the language model, a fine-tuned text description for the training image based on the second image feature (Song, page 3, para 3, “the ability of a large language model can be utilized to quickly and efficiently generate corresponding text content for a given image, thereby quickly obtaining a multi-modal training data set for instruction fine-tuning of the text generation model”; Song, page 5, last para, “Using the method for training a text generation model provided by embodiments of the present disclosure, the ability of a large language model can be used to quickly and efficiently generate corresponding text content for a given image, thereby quickly obtaining instructions for the text generation model. Fine-tuned multi-modal training dataset”; Xia, page 6, para 4, “large models only require a small number of samples to fine-tune the pre-trained model and can be used in different tasks”); and
fine tune the image encoder based on a loss between the training text in the training image-text pair and the fine-tuned text description (Xia, page 9, para 1, “the loss value specifically refers to the loss value obtained after calculating the image features and text features based on the preset loss function. According to the loss value, the image and text alignment unit and image coding unit in the initial image and text semantic alignment model are adjusted., to achieve the target image-text semantic alignment model that satisfies the training stop condition”).
Regarding Claim 18, Song in view of Xia teaches the device according to claim 17, wherein the instructions causing the electronic device to generate, by the language model, a fine-tuned text description for the training image based on the second image feature further cause the electronic device to:
acquire a training prompt content for the training image-text pair, wherein the training prompt content is used to indicate a type of the fine-tuned text description (Song, page 6, para 9, “use the pre-trained second text generation model to process the sample image to obtain a text description of the sample image; input the text description of the sample image together with the prompt information A large language model is used to obtain sample text corresponding to the sample image, wherein the prompt information specifies the style of the sample text”; Xia, page 16, para 1); and
generate, by the language model, the fine-tuned text description for the training image based on the second image feature and the training prompt content (Song, page 6, para 9, “A large language model is used to obtain sample text corresponding to the sample image, wherein the prompt information specifies the style of the sample text. In some examples, the same prompt information is used for all sample images, so that a sample data set for a specified text generation task can be obtained”).
Regarding Claim 19, Song in view of Xia teaches the device according to claim 11, wherein the feature space comprises at least one of a feature size and a feature space distribution (Song, page 6, para 4, “image feature extractor extracts image features and maps the image features to a text embedding space”; Song, page 7, para 2, S204, “feature extraction unit to extract sample visual features of the sample image… image features can be extracted using edge detection, corner detection, texture analysis, color histogram, etc... deep learning-based methods can also be used to extract image features”).
Regarding Claim 20, Song teaches:
A non-transitory computer-readable storage medium, having computer-executable instructions stored therein, wherein the computer-executable instructions, when executed by a processor, cause the processor to (Song, page 9, para 12, “a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according”):
generate a first feature of the image by a visual encoder, used to train a conversion model and a language model (Song, page 7, para 3 – 5, “The visual encoder can be used to process the input image to obtain image coding features, and the query converter can be used to process the image coding features to obtain image visual features… using the feature extraction unit to extract sample visual features of the sample image may include: using a visual encoder to extract image encoding features of the sample image; and using a query converter to process the image encoding features and the query token to obtain the sample visual characteristics… the sample image features are input into the large language model to obtain predicted text corresponding to the sample image”);
convert the first feature into a second feature by the conversion model, wherein the first feature and the second feature correspond to different feature spaces (Song, page 8, para 7, “As shown in FIG. 4, in process 400, the input image 401 can be input into the visual encoder 402 to obtain image encoding features of the input image. The image encoding features can then be given as input to the query converter 403 as key (K) or value (V). The query converter 403 outputs sample visual features of the input image 401 based on the input keys (K), values (V), and query vectors (Q).. Before inputting the features into the large language model 405, image embedding can be performed on the mapped sample visual features, so that the features input into the large language model 405 conform to the input format of the large language model.”); and
generate, by the language model, a text description for the image based on the second feature (Song, page 5, step S206, “the sample image features are input into the large language model to obtain predicted text corresponding to the sample image”; Song, page 8, para 7, “A large language model can be used to process the mapped sample visual features to obtain a text generation result for the input image 401”).
However, Song does not explicitly recite:
wherein a text encoder semantically aligned with the visual encoder is used to train a conversion model and a language model;
In the same field of endeavor, Xia teaches:
wherein a text encoder semantically aligned with the visual encoder is used to train a conversion model and a language model (Xia, page 3, para 14, “The generation model module is configured to add a feature conversion unit and a text generation unit to the target image-text semantic alignment model, and generate a language model to be trained based on the addition results”; Xia, page 8, para 3, “the image encoding The unit specifically refers to the unit that can encode image data, and is used to convert the image data into image encoding vectors to characterize the characteristics of the image data in the form of vectors; accordingly, the image-text alignment unit specifically refers to the unit that can encode the image data and The text data matching unit is used to enable the image-text semantic alignment model to learn the ability to find image data matching text”);
Song and Xia are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Song with the invention of Xia to make the invention that uses a text encoder that is semantically aligned with the visual encoder to train a conversion model and a language model; doing to can yield predictable result of improved performance by consuming less time and improving speed of generating the text description for the input image (Xiao, page 2); thus one of the ordinary skill in the art would have been motivated to combine the references.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20260004475 A1 IMAGE PROCESSING METHOD AND APPARATUS, DEVICE, MEDIUM, AND PROGRAM PRODUCT: this application disclose an image processing method and apparatus, a device, and a medium. The method includes: obtaining a target image to be processed; inputting the target image to a pre-trained image-text model, a model loss of the image-text model including an image loss, and the image loss being constructed according to a first sample image and a second sample image that is obtained by converting a first sample text configured for describing the first sample image; and obtaining a target text configured for describing the target image and generated by the image-text model. In technical solutions of the embodiments of this application, the generated target text can describe the target image as accurately as possible, thereby ensuring accuracy of the target text.
US 20250329144 A1 METHOD FOR SUBDIVIDED REPRESENTATION REINFORCEMENT OF IMAGE/TEXT REPRESENTATION VECTOR THROUGH ATTRIBUTE VALUE OF OBJECT IN IMAGE-LANGUAGE ALIGNMENT MODEL: The method for training an image-language alignment model, according to an embodiment of the present invention, generates, in an input image, object-specific representation vectors of the image, generates, in an input text, object-specific representation vectors of the text, and uses the generated object-specific representation vectors so as to train an image-language align model through a contrast loss function.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VAISALI RAO KOPPOLU whose telephone number is (571)270-0273. The examiner can normally be reached Monday - Friday 8:30 - 5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
VAISALI RAO. KOPPOLU
Examiner
Art Unit 2664
/VAISALI RAO KOPPOLU/Examiner of Art Unit 2664
Prosecution Insights