Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
The instant application is a nonprovisional of and claims priority under 35 U.S.C. 119 to U.S. provisional application no. 63/301,978, filed January 21, 2022.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 4/18/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Preliminary Amendment
The preliminary amendment filed on 4/18/2025 has been received.
Claims 1-20 have been cancelled.
Claims 21-40 have been added.
Claims 21-40 are pending.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 21-40 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claims 21, 31, and 40 recite “re-training the multi-modal model using the updated training dataset; and generating, by the re-trained multi-modal model, the text description in response to the input image.”
However, the specification of the instant application has not mention “re-training the multi-modal model”
Claims 22-30 are rejected by virtue of their dependency upon claim 21.
Claims 32-39 are rejected by virtue of their dependency upon claim 31.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 21, 24, 25, 26, 27, 28, 31, 34, 35, 36, 37, 38, and 40 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 2, 3, 4, 5, 6, 11, 12, 13, 14, 15, 16, 20, and 21 of U.S. Patent No. 12,288,380 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because of the minor differences as shown below.
19/183,526
17/745,634
21. A method of generating a text description for an input image, the method comprising:
training a multi-modal model implemented one or more hardware processors using a training dataset of images and noisy text captions; loading, from the trained multi-modal model, a first set of parameters and a second set of parameters into an image-grounded text decoder and an image-grounded text encoder, respectively;
generating, by the image-grounded text decoder, a predicted text description based on a training image from the training dataset;
generating, by the image-grounded text encoder, a filtering decision based on the training image, a corresponding noisy text caption and the predicted text description; updating the training dataset including removing the corresponding noisy text caption depending on the filter decision; re-training the multi-modal model using the updated training dataset;
and generating, by the re-trained multi-modal model, the text description in response to the input image.
1. A method of generating enhanced vison-language training data, the method comprising: receiving, from a communication interface, a first training dataset of image-text pairs and a second training dataset of annotated image-text pairs; fine-tuning an image-grounded text decoder and an image-grounded text encoder using the second training dataset of annotated image-text pairs;
generating, by the fine-tuned image-grounded text decoder, a predicted text based on a training image from the first training dataset;
generating, by the fine-tuned image-grounded text encoder, a filtering decision based on the training image and the predicted text;
adding the training image and the predicted text to form a third training dataset of image- text pairs depending on the filter decision;
and training a vision-language model using the third training dataset of image-text pairs.
24. The method of claim 22, wherein the image-grounded text decoder is finetuned by: generating a predicted text in response to an image in an annotated image-text pair; and computing a language modeling loss comparing the predicted text with an annotated text paired with the image.
2. The method of claim 1, wherein the image-grounded text decoder is finetuned by: generating a predicted text in response to an image in the second training dataset; and computing a language modeling loss comparing the predicted text with an annotated text paired with the image.
25. The method of claim 22, wherein the image-grounded text encoder is finetuned by: generating a text encoding of a text from an annotated image-text pair; generating an image encoding of an image pairing the text;
computing an image-text contrastive loss based on a positive pair of the text encoding and the image encoding, and negative pairs of the image encoding paired with other text encodings.
3. The method of claim 1, wherein the image-grounded text encoder is finetuned by: generating a text encoding of a text from the second training dataset; generating an image encoding of an image pairing the text from the second training dataset;
and computing an image-text contrastive loss based on a positive pair of the text encoding and the image encoding, and negative pairs of the image encoding paired with other text encodings.
26. The method of claim 22, wherein the image-grounded text encoder is finetuned by: generating a binary classification indicating whether a text and an image from an annotated image-text pair are a match; and computing an image-text matching loss comparing the binary classification and a ground truth.
4. The method of claim 1, wherein the image-grounded text encoder is finetuned by: generating a binary classification indicating whether a text and an image from the second training dataset are a match; and computing an image-text matching loss comparing the binary classification and a ground truth.
27. The method of claim 21, wherein the filter decision is generated by a binary classification indicating whether the training image and the predicted text description matches.
5. The method of claim 1, wherein the filter decision is generated by generating a binary classification indicating whether an input image and an input text matches.
28. The method of claim 27, wherein updating the training dataset further includes: adding the training image and the predicted text description as a training pair when the binary classification indicates a match.
6. The method of claim 5, wherein the third training dataset is formed by adding the input image and the input text as a training pair when the binary classification indicates the input image and the input text match.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PARDIS SOHRABY whose telephone number is (571)270-0809. The examiner can normally be reached Monday - Friday 9 am till 6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PARDIS SOHRABY/ Examiner, Art Unit 2664 /GANDHI THIRUGNANAM/Primary Examiner, Art Unit 2672
Prosecution Insights