DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 5/1/26 has been entered.
Response to Arguments
Applicant's arguments filed 5/1/26 have been fully considered but they are not persuasive.
Regarding claim 15, Applicant argues that the cited combination of references does not disclose or suggest grouping coherent segments into chunks before selecting a summary style followed by generating summary prompts based on both the chunked segments and the selected style, and that Liu does not disclose the claimed sequence in which chunking is performed before summary style selection and prompt generation. Liu operates on textual data and may apply summarization techniques, a pipeline in which coherent segments derived from temporally aligned multimodal insights are first grouped into chunks, followed by selection of a summary style, that only then are used to generate summary prompts based on those chunked segments nor any dependency in which chunk formation constrains or precedes selection of a summary style for use in prompt generation (Arguments, pg. 10-13). Examiner respectfully disagrees as claim 15 does not recite the use of coherent segments, chunking nor the use of a summary style as argued by Applicant.
Schalkwyk discloses the amended language of “generating an aggregated timeline of the audio insights and the visual insights by temporally aligning the audio insights and the visual insights by temporally aligning the audio insights and the visual insights” and “after generating the plurality of extractive summary sentences, generating a summary prompt by combining the plurality of extractive summary sentences into the summary prompt” as presented in the rejection below.
Applicant’s arguments with respect to the rejection of claims 1 and 9 (Arguments, pg. 10-13) have been fully considered and are persuasive. The rejections are withdrawn.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
1. Claims 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over Schalkwyk et al US 2023/0281248 A1 (“Schalkwyk”) in view of and Liu et al US 2024/0177084 (“Liu”) and Lin US 2017/0357877 A1 (“Lin”)
Per claim 15, Schalkwyk discloses a method for generating a summary of multimedia content, the method comprising:
accessing the multimedia content, the multimedia content comprising audio content and visual content (a system 100 includes a user 2 viewing a content feed 120 played back on a computing/user device 10 through a media player application 150. \… In the example shown, the content feed 120 includes a recorded instructional cooking video played back on the computing device 10 for the user 2 to view and interact with. While examples herein depict the content feed 120 as an audio-visual (AV) feed (e.g., a video) …, para. [0028]);
applying one or more audio processing models to the audio content to thereby generate audio insights from the audio content, the audio insights comprising at least one of (i) a coherent transcript that comprises textual representations of spoken utterances contained in the audio content and speaker identifications for the spoken utterances, the coherent transcript being generated using one or more speech-to-text models of the one or more audio processing models or (ii) nonspeech sound labels corresponding to nonspeech sounds, the nonspeech sound labels being generated using one or more nonspeech sound models of the one or more audio processing models (the ASR module 230 and/or the diarization module 220 (or some other component of the application 150) may index a transcription 310 of the audio data 122 using the time-stamped speaker labels 226 predicted for each segment 222 obtained from the diarization results 224. As shown in FIG. 2, the transcription 310 for the content feed 120 may be indexed by speaker to associate portions of the transcript 202 with the respective speaker …, para. [0039]);
applying one or more image processing models to the visual content to thereby generate visual insights from the visual content, the visual insights including at least one of (i) text visualized in the visual content, the text being identified by one or more OCR (Optical Character Recognition) models of the one or more image processing models, (ii) object labels for objects visualized in the visual content, the object labels being generated by one or more object identification models of the one or more image processing models, and (iii) identity labels for people represented in the visual content, the people being identified by one or more facial recognition models of the one or more image processing models (para. [0035]; para. [0037]; generate diarization results 224 that include a corresponding speaker label 226 assigned to each segment 222 using a probability model (e.g., a probabilistic generative model) based on the audio data 122 (and optionally the image data 124).… the diarization module 220 may simultaneously execute a face tracking routine to identify which participant is speaking during which segment 222 …, para. [0038]-[0039]);
generating an aggregated timeline of the audio insights and the visual insights by temporally aligning the audio insights and the visual insights by temporally aligning the audio insights and the visual insights (para. [0038]-[0039]; The transcription 310 of the utterances 123 for inclusion in the structured document 300 also includes alignment information 315. The alignment information 315 provides an alignment between each word 312 (FIG. 3) of a plurality of words 312, 312a-n (FIG. 3) of the transcription 310 and a corresponding audio segment 222 of the audio data 122 that indicates a time when the corresponding word was recognized, para. [0041], time alignment information of words and audio as aggregated timeline);
generating a plurality of extractive summary sentences from the aggregated timeline (fig. 2; fig. 4, element 310; para. [0041]; the generator 250 receives the transcription 310, creator-provided text 320 recognized in the one or more image frames 125, and the corresponding alignment information 315, 322 and generates the structured document 300 …, para. [0044]; para. [0045]-[0046]; para. [0050]; para. [0053]-[0054], structured document of aligned transcription as including summary sentences/segments/portions);
after generating the plurality of extractive summary sentences, generating a summary prompt by combining the plurality of extractive summary sentences into the summary prompt (fig. 2, elements 180, 300; fig. 3; para. [0036]; the large language model 180 is configured to receive the semantically-rich, structured document 300 and the query 112 issued by the user 2 as input …, para. [0047], received input/prompt of structured document into large language model 180 as generated summary prompt)
Schalkwyk does not explicitly disclose providing the summary prompt to a model trained to generate summaries from summary prompts or obtaining a summary from the model based on the plurality of extractive summary sentences that is received in response to providing the summary prompt to the model
However, these features are taught by Liu:
providing the summary prompt to a model trained to generate summaries from summary prompts (para. [0072]; An example prompt for this type of transcript summarization is provided below, para. [0080]; Only return a one-paragraph summary—do not include any titles or headings, para. [0081]; If the method determines that the combined summary information is useful, the combined summarized transcripts are provided to the LLM to create 1212 a short summary of the combined summarized transcripts …, para. [0084]); and
obtaining a summary from the model based on the plurality of extractive summary sentences that is received in response to providing the summary prompt to the model (para. [0087])
Schalkwyk in view of Liu does not explicitly disclose generating an aggregated timeline of the audio insights and the visual insights by removing duplicate visual insights when generating the visual insights from the visual content or by removing duplicate audio insights when generating audio insights from the audio content
However, this feature is taught by Lin (para. [0026]; The “diversity” as used herein pertains to a completeness of the important digital images … The curation application 102 can remove duplicate ones of the representative digital images 112 based on the determined diversity …, para. [0038]; the curation application 102 can implement a face detection algorithm 204 that detects one or more faces in each of the digital images 106 that include at least one face … The importance rating 110 is a rating that indicates the importance of a digital image 106 in the context of the event type, such as a digital image that includes faces of one or more persons who are important to an event (e.g., the bride and groom …, para. [0041]; para. [0077])
It would have been obvious to one of ordinary skill in the art before the effective filing of the invention to combine the teachings of Liu with the method of Schalkwyk in arriving at the missing features of Schalkwyk, as well as to combine the teachings of Lin with the method of Schalkwyk in view of Liu in arriving at the missing features of Schalkwyk in view of Liu because such combination would have resulted in identifying important information of a meeting according to the amount of detail preferred by a user (Liu, para. [0062]; para. [0086]), as well as in ensuring a collection of visual insights is not oversized (Lin, para. [0001]; para. [0026]; para. [0038]; para. [0041]).
Per claim 16, Schalkwyk in view of Liu and Lin discloses the method of claim 15,
Liu discloses wherein generating the summary prompt further comprises identifying a selected summary style that is selected from a plurality of different summary styles and including an identification of the selected summary style in the summary prompt (para. [0080]-[0084]; method 1200 may also request the LLM to create 1214 a longer summary of the combined summarized transcripts. The summaries of different lengths give users different options for the amount of detail they want in a summary., para. [0086]).
Per claim 17, Schalkwyk in view of Liu and Lin discloses the method of claim 15,
Liu discloses wherein the model comprises a large language model (LLM) (para. [0084]).
Per claim 18, Schalkwyk in view of Liu and Lin discloses the method of claim 15,
Schalkwyk discloses wherein the method further comprises generating the audio insights by at least performing speech-to-text and diarization processing on the audio content (the ASR module 230 and/or the diarization module 220 (or some other component of the application 150) may index a transcription 310 of the audio data 122 using the time-stamped speaker labels 226 predicted for each segment 222 obtained from the diarization results 224. As shown in FIG. 2, the transcription 310 for the content feed 120 may be indexed by speaker to associate portions of the transcript 202 with the respective speaker …, para. [0039]).
Per claim 19, Schalkwyk in view of Liu and Lin discloses the method of claim 15,
Schalkwyk discloses wherein the method further comprises generating the visual insights by (i) performing facial recognition and object recognition on the visual content (para. [0031]; para. [0037]; generate diarization results 224 that include a corresponding speaker label 226 assigned to each segment 222 using a probability model (e.g., a probabilistic generative model) based on the audio data 122 (and optionally the image data 124).… the diarization module 220 may simultaneously execute a face tracking routine to identify which participant is speaking during which segment 222 …, para. [0038]-[0039]; para. [0042]), and
Lin discloses (ii) removing duplicate visual insights identified when performing facial recognition and object recognition on the visual content (para. [0026]; The curation application 102 can remove duplicate ones of the representative digital images 112 based on the determined diversity …, para. [0038]; para. [0041])
Per claim 20, Schalkwyk in view of Liu and Lin discloses the method of claim 16,
Schalkwyk discloses wherein the multimedia content comprises streaming content and wherein generating the aggregated timeline comprises generating the aggregated timeline for a portion of the multimedia content of a predetermined duration of time (para. [0031]; para. [0038]; para. [0041]; para. [0059]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. See PTO 892 form.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to OLUJIMI A ADESANYA whose telephone number is (571)270-3307. The examiner can normally be reached Monday-Friday 8:30-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Richemond Dorvil can be reached at 571-272-7602. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/OLUJIMI A ADESANYA/Primary Examiner, Art Unit 2658
Prosecution Insights