DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on December 2, 2025 has been entered.
Response to Arguments
Applicant’s arguments with respect to claim(s) 8-15 and 17-20 have been considered but are moot because new grounds of rejection are made in view of Li (see citation below).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 8-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Price (US 20190377987A1) in view of Li (see citation below).
As per Claim 8, Price teaches a method for image generation, comprising: obtaining training data including an image embedding (image feature vector) for a training image and a text embedding (caption feature vector) for a caption describing the training image (training the retrieval machine learning system 204 using contrastive loss technique, domain-specific encoders compute an image feature vector, such as by using convolutional neural network, and caption feature vector, such as by using recurrent neural network based text encoder, [0044]). The caption sequence is generated by a form of LSTM model. The output of the caption sequence depends on the previously generated word and on the context/hidden state. At training time the word fed to the state is the ground truth word; at test time, the word fed to the state is the predicted word. The image features (mapped to the dimensions of the word embeddings) serve as the initial word fed to the state at t=0 [0061]. The discriminability loss, which can be treated as an error signal, is sent to the caption generation machine learning system 202 and then the caption generation machine learning system 202 can use that discriminability loss to adjust internal parameters of the caption generation machine learning system 202 to improve future captions generated by the caption generation machine learning system 202 [0032]. Thus, it would have been obvious to one of ordinary skill in the art that this means that image feature vector (image embedding) is fed to caption generation machine learning system 202, then caption generation machine learning system generates a descriptive embedding of training image based on the image feature vector (image embedding) [0044, 0061, 0032]. Thus, Price teaches training, using the training data, an adapter network of a machine learning model to generate a descriptive embedding of the training image based on the image embedding [0044, 0061, 0032].
However, Price does not teach generating the descriptive embedding by performing an attention process to translate an image embedding into a text embedding space. However, Li teaches generating the descriptive embedding by performing an attention process to translate an image embedding into a text embedding space (our work follows the classic direction to learn a joint embedding space for sentences and images, pre-training using image-text data pairs, they mainly employ self-attention mechanism found in transformer models and set up training objectives to learn cross-modal representations from a concatenated-sequence of visual region features and language token embeddings, p. 643, left column; our goal is to infer the similarity between a full sentence and a whole image by mapping image regions and the text descriptions into a common embedding space, for the image part, we begin with image regions and their features generated by the bottom-up attention model, for the text caption part, in addition to learning an embedding for the sentence, finally, the whole model is trained with joint optimization of matching and generation objectives, p. 644, right column, 1st paragraph).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Price to include generating the descriptive embedding by performing an attention process to translate an image embedding into a text embedding space as suggested by Li. It is well-known in the art that this allows the decoder to selectively focus on relevant parts of the input sequence when generating the output sequence, which improves performance by giving the decoder to ability to look back and specific parts of the input sequence, leading to more accurate and contextually appropriate outputs.
9. As per Claim 9, Price teaches wherein: the descriptive embedding has a same number of dimensions as the text embedding [0037, 0061].
10. As per Claim 10, Price teaches further comprising: encoding the training image using an image encoder to obtain the image embedding; and encoding the caption using a text encoder to obtain the text embedding [0044].
11. As per Claim 11, Price teaches further comprising: computing a translation loss based on the descriptive embedding and the text embedding, wherein the adapter network is trained based on the translation loss in a first training phase [0044, 0061, 0032].
12. Claim(s) 12 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Price (US 20190377987A1) and Li (see citation below) in view of Xu 1 (US 20180144458A1) and Cho (US 20250014148A1).
13. As per Claim 12, Price and Li are relied upon for the teachings as discussed above relative to Claim 11.
However, Price and Li do not teach further comprising: obtaining a second training image and a second image embedding for the second training image, wherein the second training image depicts a target element; generating a training embedding based on the second image embedding; generating a training composite image based on the training embedding using an image generation model. However, Xu 1 teaches further comprising: obtaining a second training image and a second image embedding for the second training image, wherein the second training image depicts a target element (target object 80 is isolated (extracted) from the background to obtain the training view, [0119], Augmented Reality system in which an object, such as a three dimensional computer graphics object, is rendered on a display device in response to the derived position and pose orientation of the target object, [0157], (insert models of 3D objects into an image or video, so that they are rendered with the correct perspective and appear to have been part of the original scene, [0087], M-dimensional depth-based descriptors 88 are computed for predefined object views of a target object 80, each descriptor encodes the appearance of the target object 80 from a particular view, each of the depth-based descriptors may be comprised of characteristics that together help to uniquely identify the 3D target object from a particular view, [0153]); generating a training embedding based on the second image embedding (object detection and pose estimation, 3D Descriptor Extractor 84 computes M-dimensional depth-based 3D descriptors for predefined object views, as illustrated by matrix 88, matrix 88 pictorially illustrates the extracted training view descriptors, which are used to identify a target object in a test image and determine its pose, each 3D descriptor may include labels identifying the training view (and target object) from which it was extracted, and its associated pose information, [0121], in a SIFT application, feature points of target objects are first extracted from a set of training images, [0093], training set to train the SIFT, [0104]); generating a training composite image based on the training embedding using an image generation model [0121, 0119, 0157, 0087].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Price and Li to include obtaining a second training image and a second image embedding for the second training image, wherein the second training image depicts a target element; generating a training embedding based on the second image embedding; generating a training composite image based on the training embedding using an image generation model because Xu 1 suggests that this way, models of 3D objects can be inserted into an image or video, so that they are rendered with the correct perspective and appear to have been part of the original scene [0087].
However, Price, Li, and Xu 1 do not teach computing an adapter loss based on the training composite image, wherein the adapter network is trained based on the adapter loss in a second training phase. However, Cho teaches computing an adapter loss based on the training composite image, wherein the adapter network is trained based on the adapter loss in a second training phase (train the first artificial neural network to minimize the loss between the information on the position and size of a specific object in the original training image, which is ground truth, and the information on the position and size of the training foreground image to be placed within the training background image output from first artificial neural network, [0086]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Price, Li, and Xu 1 to include computing an adapter loss based on the training composite image, wherein the adapter network is trained based on the adapter loss in a second training phase because Cho suggests that this way, the loss is minimized, which optimizes the position and size of the foreground image to be placed within the background image [0086].
14. As per Claim 14, Price, Li, and Xu 1 do not teach further comprising: computing an image generation loss; and fine-tuning the image generation model based on the image generation loss during a third training phase. However, Cho teaches further comprising: computing an image generation loss; and fine-tuning the image generation model based on the image generation loss during a third training phase (train the second artificial neural network to minimize the loss between the original training image, which is the ground truth, and the composite image output from the second artificial neural network, [0087].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Price, Li, and Xu 1 to include computing an image generation loss; and fine-tuning the image generation model based on the image generation loss during a third training phase because Cho suggests that this way, the loss is minimized, which optimizes the composite image [0087].
15. Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Price (US 20190377987A1), Li (see citation below), Xu 1 (US 20180144458A1), and Cho (US 20250014148A1) in view of Xu 2 (US 20240153093A1).
Price, Li, Xu 1, and Cho are relied on for teachings discussed above relative to Claim 12.
However, Price, Li, Xu 1, and Cho do not teach wherein: the image generation model is frozen during the second training phase. However, Xu 2 teaches wherein: the image generation model is frozen during the second training phase (parameters of the pre-trained diffusion model 110 are frozen and not updated during training of the diffusion-based segmentation system 100, [0026], during open-vocabulary panoptic segmentation training, the parameters of the image encoder and diffusion model 120 are unchanged (frozen) and parameters of the MLP are fine-tuned, [0039]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Price, Li, Xu 1, and Cho so that the image generation model is frozen during the second training phase as suggested by Xu 2. It is well-known in the art that this creates the right embedding that achieves the identify the concept we are trying to teach the model, instead of updating the model itself to generate images of a certain concept with a fixed embedding that has no meaning at all.
16. Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Price (US 20190377987A1), Li (see citation below), Xu 1 (US 20180144458A1), and Cho (US 20250014148A1) in view of LeGendre (US 20240212106A1).
Price, Li, Xu 1, and Cho are relied upon for teachings discussed relative to Claim 14.
However, Price, Li, Xu 1, and Cho do not teach further comprising: applying a first augmentation to the training image and a second augmentation to the second training image, wherein the image generation model is fine-tuned based on the first augmentation and the second augmentation. However, LeGendre teaches further comprising: applying a first augmentation to the training image and a second augmentation to the second training image, wherein the image generation model is fine-tuned based on the first augmentation and the second augmentation (to make the ground truth alpha generation model robust to small misalignments between the cluttered clean plate and input images, slight spatial perturbations may be added to the backgrounds during training, and background images with slight adjustments may be added as well, standard data augmentation techniques may be employed to improve model generalization (cropping, and so forth), [0109]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Price, Li, Xu 1, and Cho to include applying a first augmentation to the training image and a second augmentation to the second training image, wherein the image generation model is fine-tuned based on the first augmentation and the second augmentation because LeGrendre suggests that this trains the generation model better by simulating small misalignments during training, so that the generation model outputs quality images even if there are small misalignments in the input images [0109].
17. Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu 1 (US 20180144458A1) in view of Li (see citation below).
Xu 1 teaches a system for image generation (insert models of 3D objects into an image or video, so that they are rendered with the correct perspective and appear to have been part of the original scene, [0087]), comprising: one or more processors; one or more memory components coupled with the one or more processors (all of the above-described methods and embodiments may be implemented in a computing device, integrating these computing devices with electronic memory stores, [0163]); an adapter network trained to generate a descriptive embedding based on an image (object detection and pose estimation, 3D Descriptor Extractor 84 computes M-dimensional depth-based 3D descriptors for predefined object views, as illustrated by matrix 88, matrix 88 pictorially illustrates the extracted training view descriptors, which are used to identify a target object in a test image and determine its pose, each 3D descriptor may include labels identifying the training view (and target object) from which it was extracted, and its associated pose information, [0121], in a SIFT application, feature points of target objects are first extracted from a set of training images, [0093], training set to train the SIFT, [0104]); and an image generation model trained to generate a composite image based on the descriptive embedding and an additional image ([0121, 0093, 0104], target object 80 is isolated (extracted) from the background to obtain the training view, [0119], Augmented Reality system in which an object, such as a three dimensional computer graphics object, is rendered on a display device in response to the derived position and pose orientation of the target object, [0157], [0087]).
However, Xu 1 does not teach generating the descriptive embedding by performing an attention process to translate an image embedding into a text embedding space. However, Li teaches generating the descriptive embedding by performing an attention process to translate an image embedding into a text embedding space (our work follows the classic direction to learn a joint embedding space for sentences and images, pre-training using image-text data pairs, they mainly employ self-attention mechanism found in transformer models and set up training objectives to learn cross-modal representations from a concatenated-sequence of visual region features and language token embeddings, p. 643, left column; our goal is to infer the similarity between a full sentence and a whole image by mapping image regions and the text descriptions into a common embedding space, for the image part, we begin with image regions and their features generated by the bottom-up attention model, for the text caption part, in addition to learning an embedding for the sentence, finally, the whole model is trained with joint optimization of matching and generation objectives, p. 644, right column, 1st paragraph). This would be obvious for the reasons given in the rejection for Claim 8.
18. Claim(s) 18-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu 1 (US 20180144458A1) and Li (see citation below) in view of Menapace (US 20240307783A1).
19. As per Claim 18, Xu 1 and Li are relied upon for the teachings as discussed above relative to Claim 17.
However, Xu 1 and Li do not teach wherein: the adapter network comprises a convolutional layer, an attention block, and a multilayer perceptron. However, Menapace teaches wherein: the adapter network comprises a convolutional layer [0196], an attention block [0089], and a multilayer perceptron [0056].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Xu 1 and Li so that the adapter network comprises a convolutional layer as suggested by Menapace because it is well-known in the art that multiple convolutional layers are used to learn increasingly complex features, with each layer building upon the features learned by the previous layer. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Xu 1 so that the adapter network comprises an attention block as suggested by Menapace because it is well-known in the art that this is needed to determine the relative importance of each component in a sequence relative to the other components in that sequence. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Xu 1 so that the adapter network comprises a multilayer perceptron as suggested by Menapace because it is well-known in the art that it can handle complex, non-linear data and learn intricate patterns, making it a powerful tool for various machine learning tasks.
20. As per Claim 19, Xu 1 and Li do not teach wherein: the image generation model comprises a diffusion model that is conditioned on the descriptive embedding. However, Menapace teaches wherein: the image generation model comprises a diffusion model that is conditioned on the descriptive embedding (animation model produces states based on user-provided conditioning signals for states and action that are rendered by the synthesis model, diffusion-based animation model 120 predicts noise applied to the noisy states conditioned on known partial states (including pose but not position) and actions with the respective masks, diffusion step k of the diffusion model and framerate, [0042]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Xu 1 and Li so that the image generation model comprises a diffusion model that is conditioned on the descriptive embedding because Menapace suggests that this way, it can model human interactions with the environment [0030, 0042].
21. Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu 1 (US 20180144458A1) and Li (see citation below) in view of Price (US 20190377987A1).
Xu 1 and Li are relied upon for the teachings as discussed above relative to Claim 17.
However, Xu 1 and Li do not teach further comprising: an image encoder trained to generate an image embedding of the image, wherein the adapter network is trained to generate the descriptive embedding based on the image embedding. However, Price teaches further comprising: an image encoder trained to generate an image embedding of the image [0044], wherein the adapter network is trained to generate the descriptive embedding based on the image embedding [0044, 0061, 0032].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Xu 1 and Li to include image encoder trained to generate an image embedding of the image, wherein the adapter network is trained to generate the descriptive embedding based on the image embedding because Price suggests that this generates captions for images that accurately and concisely describe the images [0003, 0044, 0061, 0032].
Allowable Subject Matter
22. Claims 1-7 are allowed.
23. Claim 16 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Prior Art of Record
Li, Kunpeng; Image-Text Embedding Learning via Visual and Textual Semantic Reasoning; January 2023; IEEE Transactions on Pattern Analysis and Machine Intelligence; Vol. 45; p. 641-654; https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9706340
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONI HSU whose telephone number is (571)272-7785. The examiner can normally be reached M-F 10am-6:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JH
/JONI HSU/Primary Examiner, Art Unit 2611