DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 6, 9, 14, and 17 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Wang (LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing).
Regarding claim 1:
Wang discloses: a method comprising: receiving an input including digital media assets, an input visual timeline, and text input (Wang sections 4-4.2. Wang discloses a UI having 3 components: language augmented video gallery, video editing timeline, and video editing agent. The video galley displays footage with language descriptions, the timeline contains the master timeline for editing; and the agent receives chat messages (Wang sec. 4), Wang further discloses that users select clips from the gallery and add them to the timeline (Wang sec. 4.1). Wang also discloses that the selected videos are displays on the editing timeline and users can preview the combined clip sequence (Wang sec. 4.2 and FIG. 3));
the text input indicating a natural language instruction for a modification to the input visual timeline using the digital media assets (Wang further discloses that the video editing agent is chat based; users interact using free form language; the agent interprets editing objectives and plans actions (Wang section 4.3). Wang also discloses an LLM based sequencing orders clips based on a user provided or LLM generated storyline; users can rearrange or remove timeline clips (Wang section 4.2.1). Wang also discloses that storyboarding arranges timeline clips according to user narrative guidance, including the example of transitioning from dog videos to cat videos (Wang section 5.2.5));
generating a first set of tokens for the digital media assets (Wang discloses sampling frames, captioning them using LLaVA, using GPT-4 to generate visual narrations (titles/summaries), and assigns each video a unique numeric ID (Wang 5.2.1 and table 1). Wang further teaches embedding visual narrations of each video using OpenAI text embeddings and ranks videos by query similarity, (Wang 5.2.2). Wang also teaches that LLM prompt contents are processed as BPE tokens: “the tokens are byte pair encoding (BPE) … token utilized by LLM such as GPT-4 (Wang 5.1.1). Therefore, under BRI Wang teaches or at least suggests generating a first set of tokens for the digital media assets because Wang generates video IDs, titles, summaries, visual narrations, embeddings, and BPE tokenized prompt content corresponding to the digital media assets);
a second set of tokens for the input visual timeline (Wang teaches providing information corresponding to the input visual timeline to the LLM. Wang discloses that LAVE’s storyboarding function has as input timeline video + optical guidance, and produces storyboard + video order, with updates to agent chat + timeline (Wang Table 1). Wang further teaches that storyboarding affects the videos in the timeline and that the prompt begins with a function instruction, followed by user narrative guidance and visual narrations of the timeline videos. The output includes a “storyboard” and “video_ids” identifying the sequence (Wang section 5.2.5). Wang also states that GPT-4 prompt tokens are BPE tokens (Wang 5.1.1)),
and a third set of tokens for the text input (Wang teaches that recent conversation history and the latest user input are appended to form the complete prompt sent to the LLM, and that the system retains up to 6000 tokens of message history. Wang then identifies those tokens as BPE tokens used by GPT-4 (Wang 5.1.1));
processing, by a large language model, the first set of tokens, the second set of tokens, and the third set of tokens to generate an output set of tokens (Wang teaches that LAVE primarily uses GPT-4 for LLM inferences and can process descriptions from about 40 videos in a single LLM call (Wang 5.3). Wang further teaches that the storyboarding prompt includes functions instruction, user narrative guidance, and visual narrations of timeline videos, and that the output Is structured JSON with storyboard text and video IDs (Want 5.2.5). Wang also teaches that the prompt is sent to the LLM for generation and that the LLM tokens are BPE tokens (Wang 5.1.1)); and
generating a reconstructed visual timeline using the output set of tokens (Wang teaches that the storyboarding output includes “video_ids” mapping to a list of Video IDs indicating the sequence, and that once execution is complete, the video order on the timeline is updated according to those IDs (Wang 5.2.5). Wang further teaches using ffmpeg to synthesize the outcome of users’ video edits (Wang 5.3)).
Regarding claim 6:
Wang further teaches: wherein the natural language instruction for the modification to the input visual timeline references one or more digital media assets of the digital media assets using at least one of: an identifier value, positional information within the input visual timeline, and a description of content of the one or more digital media assets (Wang’s storyboarding function receives user provided narrative guidance and affects only videos in the timeline. The guidance is used to arrange the timeline clips (Wang section 5.2.5). Wang also teaches content description references, e.g., dog’s videos, cat’s videos, from indoor to outdoor, city landscape, food and drinks, and night social gathering (Wang. Sections 4.3.2 and 5.2.5)).
Regarding claims 9 and 17: the claims limitations are similar to claim 1; therefore, rejected in the same manner.
Regarding claim 14: the claim limitations are similar to claim 6; therefore, rejected in the same manner.
Allowable Subject Matter
Claims 2-5,7-8,10-13,15-16 and 18-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Relevant art not relied upon
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Lin et al. (Video-LLaVA: Learning United Visual Representation by Alignment Before Projection) teaches utilizing a LanguageBind encoder to pre-bind visual signals into the language feature space. To enable a LLM to comprehend both images and videos simultaneously.
Gonsalves (US 20250045336) teaches: A large language model (LLM) is used to broaden a search for supporting material for a media project. The LLM is provided with contextual material from the media project and optional grounding material and generates a list of types of supporting material. The list is provided to a machine-learning-based encoder, which encodes the list items into embedding space vectors.
Panagopoulou (US 20240370718) teaches: generating a multi-modal task output to a text instruction relating to inputs of multiple different modalities (e.g., text, audio, video, 3D) comprises receiving, via a data interface, a first input of a first modality, a second input of a second modality and the text instruction relating to the first and the second inputs; encoding, by a first multimodal encoder adapted for the first modality, the first input of the first modality into a first encoded representation conditioned on the text instruction; encoding, by a second multimodal encoder adapted for the second modality, the second input of the second modality into a second encoded representation conditioned on the text instruction; and generating, by a neural network based language model, the multi-modal task output based on an input combining the first encoded representation, the second encoded representation, and the text instruction.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WASSIM MAHROUKA whose telephone number is (571)272-2945. The examiner can normally be reached Monday-Thursday 8:00-5:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen Koziol can be reached at (408) 918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WASSIM MAHROUKA/Primary Examiner, Art Unit 2665
Prosecution Insights