Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 04/27/2026 has been entered.
Response to Amendment
This is in response to applicant’s amendment/response filed on 04/27/2026, which has been entered and made of record. Claims 1-5, 7-12, 15-19 have been amended. No claim has been cancelled. No claim has been added. Claims 1-20 are pending in the application.
Response to Arguments
Applicant’s arguments on 04/27/2026 have been fully considered but are moot because the arguments do not apply to any of the references being used in the current rejection.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-6, 8-13, 15-19 are rejected under 35 U.S.C. 103 as being unpatentable over Yuan et al. (US Pub 2023/0260164 A1) in view of Fan et al. (US Pub 2025/0131753 A1).
As to claim 1, Yuan discloses a processor, comprising: one or more circuits to use one or more neural networks to generate one or more images from text based, at least in part, on a first set of one or more images with one or more first captions (Yuan, Fig. 2, Fig. 5, ¶0004, “The same cross-modal encoder can then encode a text phrase to obtain a text phrase representation. An image search component is used to select a search image from the candidate images by comparing each of the search image representations to the text phrase representation. A search image is used as guidance along with the text phrase to generate a target image.” The search image is a first image without text indicating content. ¶0071, “identifying a training set comprising a set of images and a set of captions corresponding to the images, encoding the images using an image encoder to produce encoded images, encoding the captions using a text encoder to produce encoded text, computing a multi-modal loss function based on the encoded images and the encoded text, the multi-modal loss function comprising at least one image loss term, at least one text loss term, and at least one cross-modal term, and training the image encoder and the text encoder based on the multi-modal loss function.” ¶0075, “Image features are adjusted according to captions via back propagation, and vice versa.”).
Yuan does not disclose a loss calculated based, at least in part, on comparing the one or more first captions to one or more second captions generated for the first set of one or more images using the one or more neural networks; and a prediction, generated using an encoder, of one or more portions of a second set of one or more images without the one or more first captions.
Fan teaches a loss calculated based, at least in part, on comparing the one or more first captions to one or more second captions generated for the first set of one or more images using the one or more neural networks (Fan, ¶0063, “the image difference captioning system 102 utilizes the large language machine-learning model 510 to generate a target caption 512. To illustrate, the image difference captioning system 102 utilizes the large language machine-learning model 510 to generate the target caption 512 based on the text descriptions of the first digital image 500 and the second digital image 502. The target caption 512 thus includes a text-based description of a difference between text descriptions of digital images.” ¶0064, “the image difference captioning system 102 determines a loss 514 based on the predicted caption 506 and the target caption 512. For example, the image difference captioning system 102 compares the predicted caption 506 to the target caption 512 by comparing feature representations corresponding to the predicted caption 506 and the target caption 512. To illustrate, the image difference captioning system 102 determines the feature representations of the corresponding captions by accessing the feature maps from the cross-modal neural network 504 and/or the large language machine-learning model 510 (e.g., in the feature space of the corresponding large language machine-learning models). Alternatively, the image difference captioning system 102 determines the loss 514 by directly comparing the predicted caption 506 and the target caption 512 (e.g., utilizing a large language machine-learning model to compare phrases and/or another neural network). The image difference captioning system 102 utilizes the loss 514 to train (e.g., learn parameters of) the linear projection layer of the cross-modal neural network 504 to ensure that the linear projection layer accurately converts feature maps to a feature space of the large language machine-learning model of the cross-modal neural network 504.”); and
a prediction, generated using an encoder, of one or more portions of a second set of one or more images without the one or more first captions (Fan, ¶0072, “FIG. 7B illustrates a predicted difference caption 714 generated by a contrastive learning neural network encoder in a cross-modal neural network. As illustrated, the image difference captioning system 102 generates the predicted difference caption 714 that exactly matches the target caption 712.” ¶0073, “The image difference captioning system 102 determines a target caption 720 describing the material replacement in the first digital image 716 to obtain the second digital image 718. Furthermore, FIG. 7C illustrates a first predicted difference caption 722 generated by a transformer-based neural network encoder in a cross-modal neural network and a second predicted difference caption 724 generated by a contrastive learning neural network encoder in a cross-modal neural network.”).
Yuan and Fan are considered to be analogous art because all pertain to naural language processing. It would have been obvious before the effective filing date of the claimed invention to have modified Yuan with the features of “a loss calculated based, at least in part, on comparing the one or more first captions to one or more second captions generated for the first set of one or more images using the one or more neural networks; and a prediction, generated using an encoder, of one or more portions of a second set of one or more images without the one or more first captions.” as taught by Fan. The claim would have been obvious because the technique for improving a particular class of devices was part of the ordinary capabilities of a person of ordinary skill in the art, in view of the teaching of the technique for improvement in other situations.
As to claim 2, claim 1 is incorporated and the combination of Yuan and Fan discloses the prediction is a text indicating content of the one or more portions of the second set of one or more images, and is a caption embedded using the encoder (Yuan, ¶0071, “identifying a training set comprising a set of images and a set of captions corresponding to the images, encoding the images using an image encoder to produce encoded images, encoding the captions using a text encoder to produce encoded text, computing a multi-modal loss function based on the encoded images and the encoded text, the multi-modal loss function comprising at least one image loss term, at least one text loss term, and at least one cross-modal term, and training the image encoder and the text encoder based on the multi-modal loss function.” Fan, ¶0072, “FIG. 7B illustrates a predicted difference caption 714 generated by a contrastive learning neural network encoder in a cross-modal neural network. As illustrated, the image difference captioning system 102 generates the predicted difference caption 714 that exactly matches the target caption 712.” ¶0073, “The image difference captioning system 102 determines a target caption 720 describing the material replacement in the first digital image 716 to obtain the second digital image 718. Furthermore, FIG. 7C illustrates a first predicted difference caption 722 generated by a transformer-based neural network encoder in a cross-modal neural network and a second predicted difference caption 724 generated by a contrastive learning neural network encoder in a cross-modal neural network.”).
As to claim 3, claim 1 is incorporated and the combination of Yuan and Fan discloses the one or more first captions are compared to the one or more second captions generated for the first set of one ore more images based, at least in part, on the one or more neural networks calculating the loss (Fan, ¶0063, “the image difference captioning system 102 utilizes the large language machine-learning model 510 to generate a target caption 512. To illustrate, the image difference captioning system 102 utilizes the large language machine-learning model 510 to generate the target caption 512 based on the text descriptions of the first digital image 500 and the second digital image 502. The target caption 512 thus includes a text-based description of a difference between text descriptions of digital images.” ¶0064, “the image difference captioning system 102 determines a loss 514 based on the predicted caption 506 and the target caption 512. For example, the image difference captioning system 102 compares the predicted caption 506 to the target caption 512 by comparing feature representations corresponding to the predicted caption 506 and the target caption 512. To illustrate, the image difference captioning system 102 determines the feature representations of the corresponding captions by accessing the feature maps from the cross-modal neural network 504 and/or the large language machine-learning model 510 (e.g., in the feature space of the corresponding large language machine-learning models). Alternatively, the image difference captioning system 102 determines the loss 514 by directly comparing the predicted caption 506 and the target caption 512 (e.g., utilizing a large language machine-learning model to compare phrases and/or another neural network). The image difference captioning system 102 utilizes the loss 514 to train (e.g., learn parameters of) the linear projection layer of the cross-modal neural network 504 to ensure that the linear projection layer accurately converts feature maps to a feature space of the large language machine-learning model of the cross-modal neural network 504.”).
As to claim 4, claim 1 is incorporated and the combination of Yuan and Fan discloses the one or more neural networks generate the one or more images based, at least in part, the prediction of on one or more second captions of the one or more portions of the second set (Yuan, ¶0071, “identifying a training set comprising a set of images and a set of captions corresponding to the images, encoding the images using an image encoder to produce encoded images, encoding the captions using a text encoder to produce encoded text, computing a multi-modal loss function based on the encoded images and the encoded text, the multi-modal loss function comprising at least one image loss term, at least one text loss term, and at least one cross-modal term, and training the image encoder and the text encoder based on the multi-modal loss function.” ¶0075, “Image features are adjusted according to captions via back propagation, and vice versa. After the completion of multi-modal contrastive training, cross-modal encoder 505 can be directly applied to or fine-tuned for cross-modal search 500.”).
As to claim 5, claim 1 is incorporated and the combination of Yuan and Fan discloses the first set of the one or more images comprise one or more ground truth images and the one or more ground truth images are compared to the one or more second set of one or more image based, at least in part, on one or more loss operations (Yuan, Fig.3, ¶0044, “The example shown includes text phrase 300, search images 305, ground truth image 310, and target image 315.” ¶0045, “ground truth image 310 is a validation ground truth image.” ¶0060, “training component 420 computes a generator contrastive learning loss based on the target image and an original image, where the parameters of image generation network 440 are updated based on the generator contrastive learning loss. In some examples, training component 420 computes a discriminator contrastive learning loss based on the target image and an original image. In some examples, training component 420 updates parameters of the discriminator network based on the discriminator contrastive learning loss.” ¶0071, “cross-modal encoder 505 (or a retrieval module including cross-modal encoder 505) is pre-trained on cross-modal search tasks using contrastive learning.”).
As to claim 6, claim 5 is incorporated and the combination of Yuan and Fan discloses the one or more loss operations include a visual reconstruction loss, a contrastive caption loss, or a generative caption loss (Yuan, ¶0060, “training component 420 computes a generator contrastive learning loss based on the target image and an original image, where the parameters of image generation network 440 are updated based on the generator contrastive learning loss. In some examples, training component 420 computes a discriminator contrastive learning loss based on the target image and an original image. In some examples, training component 420 updates parameters of the discriminator network based on the discriminator contrastive learning loss.” ¶0071, “cross-modal encoder 505 (or a retrieval module including cross-modal encoder 505) is pre-trained on cross-modal search tasks using contrastive learning.”).
As to claim 8, the combination of Yuan and Fan discloses a system, comprising: one or more processors to use one or more neural networks to generate one or more images from text based, at least in part, on: a first set of one or more images with one or more first captions; a loss calculated based, at least in part, on comparing the one or more first captions to one or more second captions generated for the first set of one or more images using the one or more neural networks; and a prediction, generated using an encoder, of one or more portions of a second set of one or more images without the one or more first captions (See claim 1 for detailed analysis.).
As to claim 9, claim 8 is incorporated and the combination of Yuan and Fan discloses the prediction is a text indicating content of the one or more portions of a second set of one or more images, and is a caption embedded using the encoder (See claim 2 for detailed analysis.).
As to claim 10, claim 8 is incorporated and the combination of Yuan and Fan discloses the one or more first captions are compared to the one or more second captions generated for the first set of one or more images based, at least in part, on the one or more neural networks calculating the loss (See claim 3 for detailed analysis.).
As to claim 11, claim 8 is incorporated and the combination of Yuan and Fan discloses the one or more neural networks generate the one or more images based, at least in part, on the prediction of one or more second captions of the one or more portions of the second set (See claim 4 for detailed analysis.).
As to claim 12, claim 8 is incorporated and the combination of Yuan and Fan discloses the first set of the one or more images comprise one or more ground truth images and the one or more ground truth images are compared to the one or more second set of one or more images based, at least in part, on one or more loss operations. (See claim 5 for detailed analysis.).
As to claim 13, claim 12 is incorporated and the combination of Yuan and Fan discloses the one or more loss operations include a visual reconstruction loss, a contrastive caption loss, or a generative caption loss (See claim 6 for detailed analysis.).
As to claim 15, the combination of Yuan and Fan discloses a method, comprising: using one or more neural networks to generate one or more images from text based, at least in part, on a first set of one or more images with one or more first captions; a loss calculated based, at least in part, on comparing the one or more first captions to one or more second captions generated for the first set of one or more images using the one or more neural networks; and a prediction, generated using an encoder, of one or more portions of a second set of one or more images without the one or more first captions (See claim 1 for detailed analysis.).
As to claim 16, claim 15 is incorporated and the combination of Yuan and Fan discloses the prediction is a text indicating content of the one or more portions of a second set of one or more images, and is a caption embedded using the encoder (See claim 2 for detailed analysis.).
As to claim 17, claim 15 is incorporated and the combination of Yuan and Fan discloses the one or more first captions are compared to the one or more second captions generated for the first set of one or more images based, at least in part, on the one or more neural networks calculating the loss (See claim 3 for detailed analysis.).
As to claim 18, claim 15 is incorporated and the combination of Yuan and Fan discloses the one or more neural networks generate the one or more images based, at least in part, on the prediction of one or more second captions of the one or more portions of the second set (See claim 4 for detailed analysis.).
As to claim 19, claim 15 is incorporated and the combination of Yuan and Fan discloses the first set of the one or more images comprise one or more ground truth images and the one or more ground truth images are compared to the one or more second set of one or more images based, at least in part, on one or more loss operations. (See claim 5 for detailed analysis.).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 7, 14, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Yuan et al. (US Pub 2023/0260164 A1), Fan et al. (US Pub 2025/0131753 A1) in view of Cho et al. (US Pub 2023/0153522 A1).
As to claim 7, claim 1 is incorporated and Yuan discloses the one or more circuits are to cause the one or more neural networks to be trained based, at least in part, on using one or more visual encoders to embed one or more captions (Yuan, ¶0071, “training cross-modal encoder 505 includes identifying a training set comprising a set of images and a set of captions corresponding to the images, encoding the images using an image encoder to produce encoded images, encoding the captions using a text encoder to produce encoded text, computing a multi-modal loss function based on the encoded images and the encoded text”).
Yuan does not discloses comparing the one or more captions with one or more ground truth captions.
Cho teaches the one or more neural networks to be trained based, at least in part, on using one or more visual encoders to embed one or more captions and comparing the one or more captions with one or more ground truth captions (Cho, ¶0004, “Image captioning is an NLP task of generating a textual description (i.e., a caption) of an image. Words in a caption can be used to index an image so that it can be retrieved from an image search database. Existing deep learning based approaches for image captioning train an image-conditioned language model on an image-caption dataset. For example, an image captioning model can be trained by maximizing likelihood over ground truth captions, then maximizing n-gram based metrics between predicted captions and ground truth captions.”).
Yuan and Cho are considered to be analogous art because all pertain to naural language processing. It would have been obvious before the effective filing date of the claimed invention to have modified Yuan with the features of “comparing the one or more captions with one or more ground truth captions” as taught by Cho. The claim would have been obvious because the technique for improving a particular class of devices was part of the ordinary capabilities of a person of ordinary skill in the art, in view of the teaching of the technique for improvement in other situations.
As to claim 14, claim 8 is incorporated and the combination of Yuan and Cho discloses the one or more processors are to cause the one or more neural networks to be trained based, at least in part, on using one or more visual encoders to embed one or more captions and comparing the one or more captions with one or more ground truth captions (See claim 7 for detailed analysis.).
As to claim 20, claim 15 is incorporated and the combination of Yuan and Cho discloses causing the one or more neural networks to be trained based, at least in part, on using one or more visual encoders to embed one or more captions and comparing the one or more captions with one or more ground truth captions (See claim 7 for detailed analysis.).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Liu (US 2021/0097691 A1) discloses backpropagation can utilize a derivative of a loss function with respect to output generated by a statistical model.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YU CHEN whose telephone number is (571)270-7951. The examiner can normally be reached on M-F 8-5 PST Mid-day flex.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached on 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YU CHEN/Primary Examiner, Art Unit 2613
