DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP §§ 706.02(l)(1) - 706.02(l)(3) for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being anticipated over claims 1-12 of U.S. Patent No. 11,166,064.
Current Application 19/376,409
Patent # 12,489,956
A pipeline configured to: receive a video;
split the video into semantically consistent video clips;
apply cross-modality teacher models to the video clips; and
enable a student captioning model to learn from the teacher models, the student captioning model including a vision branch and a text branch configured to leverage inputs of multimodal data to the teacher models, wherein during training of the student captioning model a gradient propagation from the text branch to the vision branch is blocked.
2. The pipeline of claim 1, wherein the pipeline is configured to use automatic speech recognition (ASR) to automatically annotate video data with subtitles.
3. The pipeline of claim 1, wherein the teacher models have different pretraining weights and are configured to receive different input information.
4. The pipeline of claim 1, wherein the pipeline is configured to fine-tune a subset of the video clips using a retrieval model where an accurate caption of each of the video clips can be manually selected.
5. The pipeline of claim 4, wherein the subset is configured to be split to get training samples and testing samples.
6. The pipeline of claim 4, wherein after the fine-tuning of the subset, the pipeline is configured to use the retrieval model to select an accurate caption as an annotation.
7. The pipeline of claim 1, wherein the pipeline is configured to scale-up a dataset of a plurality of the videos to high-quality video-caption pairs.
8. The pipeline of claim 7, wherein the dataset is a publicly available dataset.
9. The pipeline of claim 1, wherein the multimodal data comprises textual video description, subtitles, and individual video frames.
10. The pipeline of claim 1, wherein the vision branch is configured to be used to extract large language model (LLM) compatible video embedding and the text branch includes a text encoder and a text Q-former configured to extract text embedding with fixed length to bridge the video and text embeddings.
11. A method of using a pipeline comprising the steps of: receiving a video;
splitting the video into semantically consistent video clips;
applying cross-modality teacher models to the video clips; and
enabling a student captioning model to learn from the teacher models, the student captioning model including a vision branch and a text branch leveraging inputs of multimodal data to the teacher models, wherein during training of the student captioning model a gradient propagation from the text branch to the vision branch is blocked.
12. The method of claim 11, wherein the pipeline uses automatic speech recognition (ASR) to automatically annotate video data with subtitles.
13. The method of claim 11, wherein the teacher models have different pretraining weights and receive different input information.
14. The method of claim 11, wherein the pipeline fine-tunes a subset of the video clips using a retrieval model where an accurate caption of each of the video clips can be manually selected.
15. The method of claim 14, wherein the subset is split to get training samples and testing samples.
16. The method of claim 14, wherein after the fine-tuning of the subset, the pipeline uses the retrieval model to select an accurate caption as an annotation.
17. The method of claim 11, wherein the pipeline scales-up a dataset of a plurality of the videos to high-quality video-caption pairs.
18. The method of claim 17, wherein the dataset is a publicly available dataset.
19. The method of claim 11, wherein the multimodal data comprises textual video description, subtitles, and individual video frames.
20. A non-transitory computer readable medium storing program code, which when executed, is operative to cause a pipeline to perform the steps of: receiving a video;
splitting the video into semantically consistent video clips;
applying cross-modality teacher models to the video clips; and
enabling a student captioning model to learn from the teacher models, the student captioning model including a vision branch and a text branch leveraging inputs of multimodal data to the teacher models, wherein during training of the student captioning model a gradient propagation from the text branch to the vision branch is blocked.
A pipeline configured to automatically annotate video data with subtitles, the pipeline configured to: curate a plurality of high-resolution videos from a publicly available dataset;
split the videos into semantically consistent video clips; and
apply multiple cross-modality teacher models to obtain captions for each of said video clips; and
enable a student captioning model to learn on the dataset to distill knowledge from the teacher models, the student captioning model including a vision branch and a text branch configured to leverage inputs of multimodal data to the teacher models, wherein during training of the student captioning model a gradient propagation from the text branch to the vision branch is blocked.
2. The pipeline of claim 1, wherein the pipeline is configured to use automatic speech recognition (ASR) to automatically annotate the video data with subtitles.
3. The pipeline of claim 1, wherein the teacher models have different pretraining weights and are configured to receive different input information.
4. The pipeline of claim 1, wherein the pipeline is configured to fine-tune a subset of the video clips using a retrieval model where an accurate caption of each of the video clips can be manually selected.
5. The pipeline of claim 4, wherein the subset is configured to be split to get training samples and testing samples.
6. The pipeline of claim 4, wherein after the fine-tuning of the subset, the pipeline is configured to use the retrieval model on an entirety of the dataset to select an accurate caption as an annotation.
7. The pipeline of claim 1, wherein the pipeline is configured to scale-up the dataset to high-quality video-caption pairs.
8. The pipeline of claim 7, wherein the multimodal data comprises textual video description, subtitles, and individual video frames.
11. The pipeline of claim 1, wherein the vision branch is configured to be used to extract large language model (LLM) compatible video embedding and the text branch includes a text encoder and a text Q-former configured to extract text embedding with fixed length to bridge the video and text embeddings.
12. A method of using a pipeline to automatically annotate video data with subtitles, the method comprising the steps of:
curating a plurality of high-resolution videos from a publicly available dataset;
splitting the videos into semantically consistent video clips;
applying multiple cross-modality teacher models to obtain captions for each said video clip; and enabling a student captioning model to learn on the dataset to distill knowledge from the teacher models, the student captioning model including a vision branch and a text branch leveraging inputs of multimodal data to the teacher models, wherein during training of the student captioning model a gradient propagation from the text branch to the vision branch is blocked.
13. The method of claim 12, wherein the pipeline uses automatic speech recognition (ASR) to automatically annotate the video data with subtitles.
14. The method of claim 12, wherein the teacher models have different pretraining weights and receive different input information.
15. The method of claim 12, wherein the pipeline fine-tunes a subset of the video clips using a retrieval model where an accurate caption of each of the video clips can be manually selected.
16. The method of claim 15, wherein the subset is split to get training samples and testing samples.
17. The method of claim 15, wherein after the fine-tuning of the subset, the pipeline uses the retrieval model on an entirety of the dataset to select an accurate caption as an annotation.
18. The method of claim 12, wherein the pipeline uses inputs of multimodal data to the teacher models and scales-up the dataset to high-quality video-caption pairs.
19. The method of claim 18, wherein the multimodal data comprises textual video description, subtitles, and individual video frames.
20. A non-transitory computer readable medium storing program code, which when executed, is operative to cause a pipeline to perform the steps of:
curating a plurality of high-resolution videos from a publicly available dataset;
splitting the videos into semantically consistent video clips;
applying multiple cross-modality teacher models to obtain captions for each said video clip; and enabling a student captioning model to learn on the dataset to distill knowledge from the teacher models, the student captioning model includes a vision branch and a text branch leveraging inputs of multimodal data to the teacher models, wherein during training of the student captioning model, a gradient propagation from the text branch to the vision branch is blocked.
Although the claims at issue are not identical, they are not patentably distinct from each other because at least one examined application claim is not patentable distinct from the reference claims(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YASSIN ALATA whose telephone number is (571)270-5683. The examiner can normally be reached Mon-Fri 7-4 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Nasser Goodarzi can be reached at 571-272-4195. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YASSIN ALATA/Primary Examiner, Art Unit 2426
Prosecution Insights