DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on March 13th, 2024 was reviewed and the listed references were noted.
Drawings
The drawings are objected to because: For Figure 7, “Storage 1106” should read “Storage Device 1106”, as described in the specification.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities:
For Paragraph [0075], fix spacing in “…at a particular timestamp…”
Appropriate correction is required.
Claim Rejections - 35 USC § 103
Claims 1-3, 8, 11-12, 14-15, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Caba Heilbron et al. (US 2024/0127857 – hereinafter referred to as “Caba” – w/ Effective Filing Date (EFD) of October 17, 2022).
Regarding Claim 1, Caba discloses “A computer-implemented method comprising: “ (Caba, Paragraph [0159], discloses: “a method 1800 for video segmentation and video segment selection and editing, in accordance with embodiments of the present invention.”); and “extracting, from a digital video, a set of audio features defining changes in audio content throughout the digital video and a set of video features defining changes in video content throughout the digital video; determining a set of break points for segmenting the digital video into sections based on the set of audio features and the set of video features; generating, for the digital video, a segmented video transcript comprising separated transcript sections according to the set of break points; and generating a video break from the segmented video transcript” (Caba, Paragraph [0050], discloses: “a video is ingested by detecting various features (e.g., a transcript), identifying boundaries for a video segmentation based on detected sentences and words, detecting active speakers using audio and/or video tracks and assigning detected speakers to corresponding portions of the transcript, and segmenting the transcript by paragraph.”). From Caba, boundaries are regarded the same way that breakpoints are defined to generate video breaks for the user to observe. Therefore, it would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the video segmentation techniques seen in Caba to improve the video segmentation method described in Claim 1 in the same way. By using these techniques, the video segmentation method can effectively achieve generating video breaks as Caba describes utilizing both audio and video characteristics from the video given to the system to generate an effective transcript that takes these different aspects into account that both have their specific nuances. Therefore, it would be obvious to use the video segmentation techniques seen in Caba to achieve the method described in Claim 1
Regarding Claim 2, Caba discloses “The computer-implemented method of claim 1” (Caba, Paragraphs [0159] & [0050], please refer to the above-described analysis for Claim 1); and “wherein extracting the set of audio features comprises extracting features indicating one or more of a topic change in the audio content, a sentence break in the audio content, a speech start in the audio content, or a speech stop in the audio content.” (Caba, Paragraph [0003], discloses “The transcript is used to identify boundaries for sentence segments, and if there are any non-speech segments (e.g., longer than a designated duration) between sentence segments, the boundaries for the sentence segments are retimed based on voice or audio activity. Each sentence segment is divided into word segments, and if there are any non-speech segments (e.g., longer than a designated duration) between word segments, the boundaries for the word segments are retimed based on voice or audio activity.”; Paragraph [0089] also discloses: “In some cases, transcript 195 and/or detected transcript segments are associated with the video's timeline, and transcript segments are associated with corresponding time ranges. In some embodiments, any known topical segmentation technique (semantic analysis, natural language processing, applying a language model) is used to partition or otherwise identify portions of the video likely to contain similar topics, and detected speech segments are associated with a score that represents how likely the speech segment ends a topical segment. Additionally or alternatively, transcript segmentation component 170 partitions or otherwise identifies paragraphs of transcript 195 as described in more detail below.”). Therefore, it would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to choose from a finite number of solutions that can be evidently seen in Caba to easily extract a set of audio features. By using any of these solutions like the sentence breaks and topic changes obtained from the audio extraction technique seen in Caba, one of ordinary skill in the art can get specific aspects of audio that can alter the way the transcript is generated from what is seen in from the video and audio track. Having a multitude of solutions allows one of ordinary skill in the art to choose either one that would be obvious to try to ensure the transcript is accurate to the video contents. Therefore, it would be obvious for one of ordinary skill in the art to use the potential solutions shown in Cuba to achieve the method described in Claim 2.
Regarding Claim 3, Caba discloses “The computer-implemented method of claim 1” (Caba, Paragraphs [0159] & [0050], please refer to the above-described analysis for Claim 1); and “wherein generating the segmented video transcript comprises using a large language model to generate predicted breaks in the digital video according to guidance parameters including one or more of timestamp formatting for the segmented video transcript, a stated role for the large language model, an indicated number of segments in the segmented video transcript, or a maximum segment length for segments in the segmented video transcript” (Caba, Paragraph [0087] and Figure 1A (see below), discloses: “At a high level, video ingestion tool 160 (e.g., feature extraction component(s) 162) detects, extracts, or otherwise determines various features (e.g., transcript 195, linguistic features, speakers, faces, audio classifications, visually similar scenes, visual artifacts, video objects or actions, audio events) from a video, for example, using one or more machine learning models, natural language processing, digital signal processing, and/or other techniques. In some embodiments, feature extraction component(s) 162 include one or more machine learning models for each of a plurality of categories of feature to detect. As such, video ingestion tool 160 and/or corresponding feature extraction component(s) 162 extract, generate, and/or store a representation of detected features (e.g., facets) in each category, corresponding feature ranges where the detected features are present, and/or corresponding confidence levels.”;
PNG
media_image1.png
711
508
media_image1.png
Greyscale
Paragraph [0134] further discloses: “In some embodiments, each sentence segment is mapped to its constituent word segments using timestamps from the transcript. In an example embodiments, the timestamps for the boundaries of the word segments are used to detect whether there are any gaps between word segments (non-speech segments), and the boundaries for word segments adjacent to a gap are retimed into the gap based on voice or audio activity. For each word segment (e.g., adjacent to a detected gap), a neighborhood of a designated length or duration (e.g., 0.1 seconds) (e.g. within an adjacent gap) is searched for the location at which voice or audio activity signal is a minimum, and the boundary is adjusted to that location. In some embodiments, short gaps that are less than a designated duration are closed, extending one or more of the adjacent word segment boundaries to a location in or across the gap where the voice or audio activity signal is a minimum.”; Finally, Paragraph [0135] further discloses about Figure 1A: “As such, a representation of the resulting (e.g., retimed) sentence and/or word boundaries (e.g., video/audio timestamps) are stored and used to snap a selection to the closest corresponding boundaries. Returning to FIG. 1A, in some embodiments, video segmentation component 180 stores a representation of video segmentation 196 defined by the boundaries of the word and/or sentence segments using one or more data structures. In an example implementation, video segments of a video segmentation(s) 196 are identified by values that represent, or references to, timeline locations (e.g., boundary locations, IDs, etc.), segment durations, separations between boundaries (e.g., snap points), and/or other representations. In some cases, a single copy of a video and/or a representation of boundary locations for one or more segmentations are maintained. Additionally or alternatively, the video file is broken up into fragments at boundary locations of video segments from the (e.g., default) video segmentation for efficiency purposes.”). Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to utilize large language models seen in Caba to ensure proper guidance for generating predicted video breaks, like determining when someone has talked for long and stopped at a specific sentence. Using the large language model allows the method to effectively interpret different aspects of audio to make effective segmentation and stylistic decisions to make it easier for users to digest the video transcript that gets outputted with the claimed invention. Therefore, it would be obvious for one of ordinary skill in the art to use the large language model with various solutions seen in Caba to achieve the method described in Claim 3.
Regarding Claim 8, Caba discloses “A system comprising:” (Caba, Paragraph [0183], discloses: “Embodiments described herein support video segmentation, speaker diarization, transcript paragraph segmentation, video navigation, video or transcript editing, and/or video playback. In various embodiments, the components described herein refer to integrated components of a system. The integrated components refer to the hardware architecture and software framework that support functionality using the system. The hardware architecture refers to physical components and interrelationships thereof and the software framework refers to software providing functionality that can be implemented with hardware embodied on a device.”); “at least one processor; and at least one non-transitory computer-readable storage medium storing instructions that, when executed by the at least one processor, cause the system to:” (Caba, Paragraph [0156], discloses: For instance, in some embodiments, various functions are carried out by a processor executing instructions stored in memory. In some cases, the methods are embodied as computer-usable instructions stored on computer storage media. In some implementations, the methods are provided by a standalone application, a service or hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few.”); “extract, from a digital video, a set of audio features defining changes in audio content throughout the digital video and a set of video features defining changes in video content throughout the digital video; determine, utilizing a break point prediction model to process the set of audio features and the set of video features, a set of break points for segmenting the digital video into sections; generate a segmented video transcript comprising separated transcript sections according to the set of break points” (Caba, Paragraph [0050], discloses: “a video is ingested by detecting various features (e.g., a transcript), identifying boundaries for a video segmentation based on detected sentences and words, detecting active speakers using audio and/or video tracks and assigning detected speakers to corresponding portions of the transcript, and segmenting the transcript by paragraph.”); “and provide, for display on a client device, a video break notification suggesting a timestamp of the digital video for inserting a break based on the segmented video transcript” (Caba, Paragraph [0142] and Figure 10 (see below), discloses: “In FIG. 10, transcript interface 1010 presents a diarized transcript segmented into paragraphs (e.g., based on change in speaker, change in topic). Depending on the embodiment, transcript interface 1010 presents each paragraph of transcript text 1015 (e.g., paragraph 1030a, paragraph 1030b) with a representation of a visualization of the person speaking that paragraph (e.g., a representative speaker thumbnail for that speaker, such as speaker thumbnail 1020), and/or a visualization of one or more video thumbnails of the video segment corresponding to that paragraph (e.g., video thumbnail 1025). In some embodiments, transcript interface 1010 accepts input selecting some transcript text 1015 (e.g., clicking or tapping and dragging along the transcript), snaps the selected transcript text to word and/or sentence boundaries, and/or snaps a selection of a corresponding video segment to corresponding boundaries. As such, transcript interface 1010 uses a text-based selection to define a corresponding video segment. In some embodiments, transcript interface 1010 accepts input identifying a text-based command (e.g., cut, copy, paste), and in response, executes a corresponding video editing operation on the video segment, as described in more detail below with respect to FIG. 11. In some embodiments, transcript interface 1010 annotates transcript text with an indication of corresponding portions of the video where various features were detected, displays a visual representation of detected non-speech audio or pauses (e.g., as sound bars), and/or displays video thumbnails corresponding to each line of transcript text in a timeline view below each line of text in the transcript (e.g., as a thumbnail bar), as described in greater detail below with respect to FIGS. 11-13.”;
PNG
media_image2.png
626
963
media_image2.png
Greyscale
Paragraph [0147] and Figure 11 (see below) discloses: “In some embodiments, upon receiving selection 1110 (and/or some subsequent input such as a right click on selection 1110), transcript interface 1100 displays a menu 1120 of options that include video editing operations. For example, create clip option 1130 takes the selected portion of the transcript, splits the corresponding video segment (e.g., including a corresponding portion of the audio and video tracks) from the loaded video project into a separate video clip, and/or adds the video clip to the user's media library. Cut option 1140 removes the corresponding video segment from the loaded video project and places it in the clipboard. Copy option 1150 leaves corresponding video segment in the loaded video project and places it in the clipboard. Paste option 1160 pastes a previously copied video segment into a location of the loaded video project corresponding to position cursor 1165 in the transcript (and/or pastes over the video segment correspond to selection 1110). Note that position cursor 1165 is displayed with a timestamp of the corresponding position of the video. Delete option 1170 deletes the corresponding video segment from the loaded video project (e.g., removing a corresponding portion of the video track, audio track, and rescript). New edit from source media option 1180 opens up a new video project with the video segment corresponding to selection 1110.”).
PNG
media_image3.png
625
924
media_image3.png
Greyscale
Accordingly, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the video segmentation system and user interface on the client device as seen in Caba to improve the system described in Claim 8. By using the video segmentation methods executed by a processor described in Caba to extract audio and video features while also generating breakpoints, the claimed system can perform its stated function efficiently and accurately. The disclosed video transcript interface seen in Caba to notify the user of identified video breaks is uniquely similar to what is disclosed in the drawings for the claimed invention, and offers a multitude of options for a user to insert breakpoints when necessary to segment the video accordingly. Thus, it would have been obvious for one of ordinary skill in the art to have used the video segmentation system and transcript interface on a client device seen in Caba to achieve the same system described in Claim 8.
Regarding Claim 11, Caba discloses “The system of claim 8, further comprising instructions that, when executed by the at least one processor, cause the system to determine the set of break points by:” (Caba, Paragraphs [0183], [0156], [0050], [0142], and [0147] as well as Figures 10 and 11; please refer to the above-described analysis for Claim 8); “determining, from the set of audio features, audio timestamps within the digital video for audio-based potential break points; determining, from the set of video features, video timestamps within the digital video for video-based potential break points” (Caba, Paragraph [0003], discloses: “In an example embodiment, an audio track from a video is transcribed, generating a transcript that identifies sentences, words, and timestamps representing when in the video each word is spoken.”; Paragraph [0135] further discloses: “As such, a representation of the resulting (e.g., retimed) sentence and/or word boundaries (e.g., video/audio timestamps) are stored and used to snap a selection to the closest corresponding boundaries. Returning to FIG. 1A, in some embodiments, video segmentation component 180 stores a representation of video segmentation 196 defined by the boundaries of the word and/or sentence segments using one or more data structures. In an example implementation, video segments of a video segmentation(s) 196 are identified by values that represent, or references to, timeline locations (e.g., boundary locations, IDs, etc.), segment durations, separations between boundaries (e.g., snap points), and/or other representations.”); and “aligning the audio-based potential break points with the video-based potential break points based on comparing the audio timestamps and the video timestamps.” (Caba, Paragraph [0078], discloses: “In some embodiments, video segmentation component 180 identifies candidate boundaries for video segments based on sentences boundaries and word boundaries in transcript 195. In FIG. 1A, video segmentation component 180 includes sentence segmentation component 182 that identifies sentence segments from transcript 195. In an example embodiment, video segmentation component 180 includes gap closing component 186 that retimes boundaries of the sentence segments based on voice or audio activity (e.g., closing non-speech silence gaps between sentences, expanding sentence boundaries to a location within a threshold duration where voice or audio activity is a minimum). Word segmentation component 182 segments the sentence segments into word segments based on transcript 195, and in some embodiments, gap closing component 186 retimes boundaries of the word segments based on voice or audio activity. The resulting boundaries can be thought of as audio cuts in embodiments in which they are derived at least in part using the audio track (e.g., the transcript is generated from the audio track, so the sentence and/or word boundaries are detected from the audio track).”). Accordingly, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to use the known audio and video extraction and identification techniques seen in Caba to improve the claimed video segmentation system to yield proper timing with the video and audio breakpoints for the video transcript upon further review from the user. These techniques disclosed by Caba allow for the user to click through different parts of the video transcript and go directly to where the audio and video go over that specific segment. Therefore, it would be obvious for one of ordinary skill in the art to use the video and audio techniques seen in Caba to achieve the same system described in Claim 11.
Regarding Claim 12, Caba discloses “The system of claim 8, further comprising instructions that, when executed by the at least one processor, cause the system to generate the segmented video transcript” (Caba, Paragraphs [0183], [0156], [0050], [0142], and [0147] as well as Figures 10 and 11; please refer to the above-described analysis for Claim 8); by utilizing a heuristic model to generate transcript sections having at least a threshold length (Caba, Paragraph [0077], discloses: “In some embodiments, transcript segmentation component 170 segments transcript 195 to make the transcript easier to read, understand, and interact with. In FIG. 1A, transcript segmentation component 170 includes sentence segmentation component 172 that identifies sentence segments from transcript 195, sentence embedding component 174 that generates sentence embeddings for each sentence segment (or accesses previously generated sentence embeddings), diarization and pause segmentation component 176 that segments transcript 195 at each speaker change (and optionally at speaker pauses, such as those longer than a designated length or duration), and paragraph segmentation component 178 breaks long paragraphs (e.g., longer than a designated length or duration) into multiple smaller paragraphs at sentence boundaries using dynamic programming to minimize a cost function that penalizes candidate segmentations based on divergence from a target paragraph length, that rewards candidate segmentations that group semantically similar sentences into a common paragraph, and/or that penalizes candidate segmentations that include candidate paragraphs with long pauses (e.g., longer than a normalized length or duration).”; Paragraph [0089], discloses: “In some embodiments, feature extraction component(s) 162 extract transcript 195 and/or linguistic features from an audio track associated with a video. In an example implementation, any known speech-to-text algorithm is applied to the audio track to generate a transcript of speech, detect speech segments (e.g., corresponding to words, sentences, utterances of continuous speech separated by audio gaps, etc.), detect non-speech segments (e.g., pauses, silence, or non-speech audio), and/or the like. In some embodiments, voice or audio activity detection is applied (e.g., to the audio track, to detected non-speech segments) to detect and/or categorize segments of the audio track with non-word human sounds (e.g., laughter, audible gasps, etc.). In some cases, transcript 195 and/or detected transcript segments are associated with the video's timeline, and transcript segments are associated with corresponding time ranges. In some embodiments, any known topical segmentation technique (semantic analysis, natural language processing, applying a language model) is used to partition or otherwise identify portions of the video likely to contain similar topics, and detected speech segments are associated with a score that represents how likely the speech segment ends a topical segment. Additionally or alternatively, transcript segmentation component 170 partitions or otherwise identifies paragraphs of transcript 195”). Accordingly, it would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the model seen in Caba to determine the length of speech segments between speakers and topical segments on specific subject matter to improve the system described in Claim 12.
Regarding Claim 14, Caba discloses “The system of claim 8, further comprising instructions that, when executed by the at least one processor, cause the system to” (Caba, Paragraphs [0183], [0156], [0050], [0142], and [0147] as well as Figures 10 and 11; please refer to the above-described analysis for Claim 8); generate the set of break points by implementing a global optimization model for selecting a number of break points that satisfy at least a threshold confidence score (Caba, Paragraph [0004], discloses “the faces in the video that correspond to the active voice at each moment in the video are identified by the second speaker diarization and used to refine the first speaker diarization (e.g., the start/end times assigned to different speakers) by ensuring consistent correspondence between the active voice and the face seen in the video. In some embodiments, the first and second speaker diarizations are combined using the Hungarian algorithm to find the optimal assignment of speaker identity from one diarization to the other. Embodiments that rely on this hybrid diarization technique avoid or reduce the conventional problem with over-segmentation by leveraging a video signal in conjunction with the audio signal, while retaining the accuracy benefits of the audio-only speaker diarization. As such, in some embodiments, faces are linked to voices, so that instead of “speaker 1”, “speaker 2”, etc., the transcript interface that displays the diarized transcript can show the faces of each speaker.”; Paragraph [0078] discloses: In some embodiments, video segmentation component 180 identifies candidate boundaries for video segments based on sentences boundaries and word boundaries in transcript 195. In FIG. 1A, video segmentation component 180 includes sentence segmentation component 182 that identifies sentence segments from transcript 195. In an example embodiment, video segmentation component 180 includes gap closing component 186 that retimes boundaries of the sentence segments based on voice or audio activity (e.g., closing non-speech silence gaps between sentences, expanding sentence boundaries to a location within a threshold duration where voice or audio activity is a minimum). Word segmentation component 182 segments the sentence segments into word segments based on transcript 195, and in some embodiments, gap closing component 186 retimes boundaries of the word segments based on voice or audio activity. The resulting boundaries can be thought of as audio cuts in embodiments in which they are derived at least in part using the audio track (e.g., the transcript is generated from the audio track, so the sentence and/or word boundaries are detected from the audio track).”; Paragraph [0099], discloses: “Additionally or alternatively, if there is a conflict between assigned speaker identifies between the two diarization hypotheses for a particular temporal segment, and the predicted active speaker score for the assigned speaker in the second speaker diarization is greater than a designated threshold, face-aware speaker diarization component 166 refines the identity assigned in the first speaker diarization to reflect the identity assigned by the second speaker diarization.”). Therefore, it would obvious for one of ordinary skill in the art to use the known techniques of using a global optimization model such as the Hungarian algorithm seen in Caba to select a number of breakpoints that meet a threshold confidence score, which is described in Claim 14.
Claim 15 recites a computer-readable storage medium storing a program with instructions corresponding to the steps recited in Claim 1. Therefore, the recited programming instructions of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to use the Caba reference, presented in rejection of Claim 1, apply to this claim. Finally, the Caba references discloses a computer readable storage medium storing instructions executed by at least one processor (for example, see Caba, Paragraph [0156]).
Regarding Claim 18, Caba discloses “The non-transitory computer-readable medium of claim 15, further comprising instructions that, when executed by the at least one processor, cause the computer system to:” (Caba, Paragraphs [0159], [0050], and [0156], please see the above described analysis for Claim 15); “combine one or more separated transcript sections within the segmented video transcript that relate to a common topic” (Caba, Paragraph [0052], discloses: “a face-aware speaker diarization technique initially applies an audio-only speaker diarization technique that considers voice identities detected from the audio track of the video to compute a first speaker diarization (e.g., identifying speakers in the audio track as “speaker 1”, “speaker 2”, etc.). An audio-visual speaker diarization technique that considers face and voice identities detected from the audio and video tracks, respectively, is applied to compute a second speaker diarization. As such, the faces in the video that correspond to the active voice at each moment in the video are identified by the second speaker diarization and used to refine the first speaker diarization (the start/end times assigned to different speakers) by ensuring consistent correspondence between the active voice and the face seen in the video. In some embodiments, the first and second speaker diarizations are combined using the Hungarian algorithm to find the optimal assignment of speaker identity from one diarization to the other. Since audio-only approaches tend to over segment (detect more speakers than there actually are in the audio, assigning portions spoken by the same person to two different speakers), leveraging the correspondence between voices and faces in the video reduces this over segmentation, producing a more accurate diarization.”); and “generate, for display on a client device, a video break notification to suggest rearranging frames of the digital video to coincide with the one or more separated transcript sections combined based on the common topic” (Caba, Paragraph [0142] and Figure 10, discloses “In FIG. 10, transcript interface 1010 presents a diarized transcript segmented into paragraphs (e.g., based on change in speaker, change in topic). Depending on the embodiment, transcript interface 1010 presents each paragraph of transcript text 1015 (e.g., paragraph 1030a, paragraph 1030b) with a representation of a visualization of the person speaking that paragraph (e.g., a representative speaker thumbnail for that speaker, such as speaker thumbnail 1020), and/or a visualization of one or more video thumbnails of the video segment corresponding to that paragraph (e.g., video thumbnail 1025). In some embodiments, transcript interface 1010 accepts input selecting some transcript text 1015 (e.g., clicking or tapping and dragging along the transcript), snaps the selected transcript text to word and/or sentence boundaries, and/or snaps a selection of a corresponding video segment to corresponding boundaries. As such, transcript interface 1010 uses a text-based selection to define a corresponding video segment. In some embodiments, transcript interface 1010 accepts input identifying a text-based command (e.g., cut, copy, paste), and in response, executes a corresponding video editing operation on the video segment, as described in more detail below with respect to FIG. 11. In some embodiments, transcript interface 1010 annotates transcript text with an indication of corresponding portions of the video where various features were detected, displays a visual representation of detected non-speech audio or pauses (e.g., as sound bars), and/or displays video thumbnails corresponding to each line of transcript text in a timeline view below each line of text in the transcript (e.g., as a thumbnail bar), as described in greater detail below with respect to FIGS. 11-13.”).
PNG
media_image2.png
626
963
media_image2.png
Greyscale
Accordingly, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to use the techniques for combining similar transcript sections together and rearrange video breaks based on the ones that share a common topic seen in Caba to improve the claimed video segmentation system. For the speakers that speak or are having a conversation within the video in the Caba reference, the transcript is segmented accordingly to map both diarizations at the same time, which closely relates to grouping the transcript sections based on a particular topic. Although the Caba reference does not directly suggest rearranging the frames, the transcript interface that it provides is intuitive enough to notify the user of the current segments that the system has segmented or recommended to segment in order to rearrange different frames of the video using the variety of editing options (cut, copy, paste) available on the interface to correlate with a common topic that is described within the video and the transcript. Thus, it would be obvious for one of ordinary skill in the art to use the techniques and interface seen in Caba to yield predictable results and achieve the same computer-readable storage medium described in Claim 18.
Claims 4 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Caba in view of Hoashi et al. (US 2006/0092327), and further in view of Chatoo et al. (US 2022/0101013).
Regarding Claim 4, Caba discloses “The computer-implemented method of claim 1, wherein extracting the set of video features comprises extracting features indicating one or more of:” (Caba, Paragraphs [0159] & [0050], please refer to the above-described analysis for Claim 1); (Hoashi, Paragraph [0031]). Therefore, it would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the video segmentation method seen in Caba with the technique of indicating the visual makeup frame seen in Hoashi to achieve above-described limitations recited in Claim 4.
The combination of Caba and Hoashi does not explicitly disclose “or a threshold change in composition of the video content between frames of the digital video”. However, in an analogous field of endeavor, Chatoo discloses “The color histogram test is configured to detect a threshold difference in color content between adjacent shots corresponding to the shot boundary, based on respective color histograms for the adjacent shots. For example, a scene/segment change may occur when the on-camera action shifts from one physical location to another. In another example, a shift from live-action camera footage to a graphic:/titles shot may indicate a scene/segment change. In each of these examples, the typical color content of the frames in the previous and succeeding shots may be significantly different (e.g., it is noted that the pHash algorithm used for shot detection may discard color information). The collection management system 204 may be configured to implement or otherwise access algorithm(s) to detect adjacent shots with a threshold difference in color content. For example, the collection management system 204 may (1) build a single color histogram for all the frames of each shot, (2) implement a distance metric on these histograms (e.g., earth mover's distance), and (3) use thresholding to identify pairs of shots with especially distant histograms. In this example, each bin in the histogram may describe the number of red, green or blue pixels of a certain range of values (e.g., 0-32, 33-64, . . . , 223-255).” (Chatoo, Paragraph [0159]). Therefore, it would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the video segmentation method described in the combination of Caba and Hoashi with the threshold change technique found in Chatoo to achieve the same method found in Claim 4. By using either extracting the visual makeup in the frame as seen in the combination of Caba and Hoashi or analyzing the threshold change in between frames of the video as described in Chatoo, one of ordinary skill in the art is able to effectively determine which features are needed to determine specific sections in the transcript to make it more accurate to the events of the video. Therefore, it would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the video segmentation method seen in the combination of Caba and Hoashi with the threshold change technique see in Chatoo to obtain the method of Claim 4.
Regarding Claim 9, Caba discloses “The system of claim 8, further comprising instructions that, when executed by the at least one processor, cause the system to extract the set of video features by extracting features indicating” (Caba, Paragraphs [0183], [0156], [0050], [0142], and [0147] as well as Figures 10 and 11; please refer to the above-described analysis for Claim 8); (Hoashi, Paragraph [0031]). Therefore, it would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the video segmentation system seen in Caba with the technique of indicating the visual makeup frame seen in Hoashi to achieve above-described limitations recited in Claim 9.
The combination of Caba and Hoashi does not explicitly disclose “based on performing a color analysis of pixels in the frame”. However, in an analogous field of endeavor, Chatoo discloses “The color histogram test is configured to detect a threshold difference in color content between adjacent shots corresponding to the shot boundary, based on respective color histograms for the adjacent shots. For example, a scene/segment change may occur when the on-camera action shifts from one physical location to another. In another example, a shift from live-action camera footage to a graphic:/titles shot may indicate a scene/segment change. In each of these examples, the typical color content of the frames in the previous and succeeding shots may be significantly different (e.g., it is noted that the pHash algorithm used for shot detection may discard color information). The collection management system 204 may be configured to implement or otherwise access algorithm(s) to detect adjacent shots with a threshold difference in color content. For example, the collection management system 204 may (1) build a single color histogram for all the frames of each shot, (2) implement a distance metric on these histograms (e.g., earth mover's distance), and (3) use thresholding to identify pairs of shots with especially distant histograms. In this example, each bin in the histogram may describe the number of red, green or blue pixels of a certain range of values (e.g., 0-32, 33-64, . . . , 223-255)” (Chatoo, Paragraph [0159]). Therefore, it would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the video segmentation system described in the combination of Caba and Hoashi with the color analysis text of pixels of each frame found in Chatoo to achieve the same system found in Claim 9. By determining the visual makeup in the frame as seen in the combination of Caba and Hoashi through the color analysis of pixels seen in Chatoo, one of ordinary skill in the art can get a comprehensive overview of the frame to allow the system to make an effective decision on where a video break would happen for the segmented video transcript. Therefore, it would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the video segmentation system seen in the combination of Caba and Hoashi with the color analysis test seen in Chatoo to obtain the system of Claim 9.
Claims 5 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Caba in view of Zhang et al. (US 2023/0142444).
Regarding Claim 5, Caba discloses “The computer-implemented method of claim 1 , wherein determining the set of break points comprises” (Caba, Paragraphs [0159] & [0050], please refer to the above-described analysis for Claim 1); Caba does not explicitly disclose “utilizing a heuristic model to generate confidence scores for potential break points with the digital video by processing the set of audio features and the set of video features”. However, in an analogous field of endeavor, Zhang discloses how, in their invention for determining ad breakpoints within media playback, that “one innovative aspect of the subject matter described in this specification can be embodied in methods including the operations of determining, a candidate set of breakpoints within a media item; generating, using a machine learning model that includes a plurality of parameters, a score for each particular candidate breakpoint in the set of candidate breakpoints within the media item based on presentation features of the media item at the particular candidate breakpoint” (Zhang, Paragraph [0005]). Accordingly, it would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the video segmentation method seen in Caba with the confidence score generation found in Zhang to achieve determining the correct location for specific breakpoints in the video. By doing this, one of ordinary skill in the art minimizes errors that may exist when the method determines the breakpoints in different styles of videos. Thus, it would be obvious for one of ordinary skill in the art to combine the Caba and Zhang references to achieve the method described in Claim 1.
Regarding Claim 19, Caba discloses “The non-transitory computer-readable medium of claim 15, further comprising instructions that, when executed by the at least one processor, cause the computer system to” (Caba, Paragraphs [0159], [0050], and [0156], please see the above described analysis for Claim 15); . Caba does not explicitly disclose “generate the set of break points by using a break point prediction model to predict timestamp locations for the digital video based on the set of audio features and the set of video features”. However, in an analogous field of endeavor, Zhang discloses the following in Paragraphs [0040] and [0041]:
PNG
media_image4.png
522
292
media_image4.png
Greyscale
Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the video segmentation computer-readable storage medium found in Caba with the technique of predict timestamp locations seen in Zhang to achieve an accurate placement of breakpoints within the video for the segmented video transcript. By accurately predicting the timestamp locations using Zhang’s technique with breakpoint prediction model disclosed in Caba, one of ordinary skill in the art can sync the corresponding timestamps together within the video break and transcript. Thus, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the video segmentation computer-readable storage medium seen in Caba with the timestamp estimation technique seen in Zhang to achieve the same computer-readable storage medium seen in Claim 19.
Claims 6 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Caba in view of Zhang, and further in view of Chatoo.
Regarding Claim 6, the combination of Caba and Zhang discloses “The computer-implemented method of claim 5” (Caba, Paragraphs [0159] & [0050]; Zhang, Paragraph [0005], please see above-described analysis for Claim 5). (Chatoo, Paragraph [0178]). Chatoo also further discloses how “The set of breakpoint tests may include a video fade-out test configured to detect a video fade-out between adjacent shots corresponding to the shot boundary, and to return a score indicating that the shot boundary corresponds to the breakpoint based on the detecting. The set of breakpoint tests may include an audio fade-out test configured to detect an audio fade-out between adjacent shots corresponding to the shot boundary, and to return a score indicating that the shot boundary corresponds to the breakpoint based on the detecting.” (Chatoo, Paragraph [0179]). Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the video segmentation method seen in the combination of Caba and Zhang with the technique for finding the first confidence score (audio), second confidence score (video), and combined confidence scores (both audio and video) for a particular location seen in Chatoo to achieve the same method described in Claim 6.
Regarding Claim 7, the combination of Caba, Zhang, and Chatoo discloses “The computer-implemented method of claim 6, wherein combining the first confidence score and the second confidence score comprises:” (Caba, Paragraphs [0159] & [0050]; Zhang, Paragraph [0005]; Chatoo, Paragraphs [0178] and [0179], please see above-described analysis for Claim 6); “comparing a first timestamp for an audio-based potential break point determined from the set of audio features with a second timestamp for a video-based potential break point determined from the set of video features” (Caba, Paragraph [0003], discloses:” In an example embodiment, an audio track from a video is transcribed, generating a transcript that identifies sentences, words, and timestamps representing when in the video each word is”; Paragraph [0135] discloses, “As such, a representation of the resulting (e.g., retimed) sentence and/or word boundaries (e.g., video/audio timestamps) are stored and used to snap a selection to the closest corresponding boundaries. Returning to FIG. 1A, in some embodiments, video segmentation component 180 stores a representation of video segmentation 196 defined by the boundaries of the word and/or sentence segments using one or more data structures. In an example implementation, video segments of a video segmentation(s) 196 are identified by values that represent, or references to, timeline locations (e.g., boundary locations, IDs, etc.), segment durations, separations between boundaries (e.g., snap points), and/or other representations.”); and “determining, based on comparing the first timestamp and the second timestamp, that the first timestamp and the second timestamp are combinable into a single break point for the digital video.” (Caba, Paragraph [0051], discloses: “In some embodiments, to facilitate selecting and performing operations on video segments corresponding to selected transcript text (text-based editing of audio and video assets), candidate boundaries for video segments are identified based on detected sentences and words in a transcript. In an example embodiment, an audio track from a video is transcribed, generating a transcript that identifies sentences, words, and timestamps representing when in the video each word is spoken. The transcript is used to identify boundaries for sentence segments, and if there are any non-speech segments (e.g., longer than a designated duration) between sentence segments, the boundaries for the sentence segments are retimed based on voice or audio activity. Each sentence segment is divided into word segments, and if there are any non-speech segments (e.g., longer than a designated duration) between word segments, the boundaries for the word segments are retimed based on voice or audio activity. As such, a transcript interface presents the transcript and accepts an input selecting individual sentences or words from the transcript (e.g., by clicking or tapping and dragging across the transcript), and the identified boundaries corresponding to the selected transcript text are used as boundaries for a selected video segment. In some embodiments, the transcript interface accepts commands that are traditionally thought of as text-based operations (e.g., instructions to cut, copy, paste, or delete selected transcript text), and in response, performs corresponding video editing operations using the selected video segment.”). It is important to mention, as stated before, that boundaries are analogous to breakpoints when the combination of Caba, Zhang and Chatoo describe the limitation of their claimed invention. Therefore, it would be obvious to one of ordinary skill in the art to use the known technique of combining both audio and video timestamps into one single breakpoint in the video seen in the combination of Caba, Zhang, and Chatoo to improve the video segmentation method seen in Claim 7.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Caba in view of Chatoo.
Regarding Claims 16, Caba discloses “The non-transitory computer-readable medium of claim 15, further comprising instructions that, when executed by the at least one processor, cause the computer system to extract the set of video features by” (Caba, Paragraphs [0159], [0050], and [0156], please see the above-described analysis for Claim 15); 208 provides for presenting augmented reality content in association with an image or a video captured by a camera of the client device 102. The augmentation system 208 may implement or otherwise access augmented reality content items (e.g., corresponding to applying Lenses or augmented reality experiences) for providing real-time special effect(s) and/or sound(s) that may be added to the image or video. To facilitate the presentation of augmented reality content, the augmentation system 208 may implement or otherwise access object recognition algorithms (e.g., including machine learning algorithms) configured to scan an image or video, and to detect/track the movement of objects within the image or video” (Chatoo, Paragraph [0044]). Therefore, it would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the video segmentation computer-readable medium seen in Caba with the technique for identifying a change in an object seen in Chatoo to improve the claimed video segmentation computer-readable medium. The method described in Chatoo of detecting and tracking the movement of objects indicate the change in the objects position, so having this implemented in the claimed invention allows one of ordinary skill in the art to effectively translate those changes in the video breaks that the computer-readable medium generates. Thus, it would be obvious for one of ordinary skill in the art to combine the Caba and Chatoo references to achieve the same claim found in Claim 16.
Claims 10 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Caba in view of Hoashi.
Regarding Claim 10, Caba discloses “The system of claim 8, further comprising instructions that, when executed by the at least one processor, cause the system to extract the set of video features by:” (Caba, Paragraphs [0183], [0156], [0050], [0142], and [0147] as well as Figures 10 and 11; please refer to the above-described analysis for Claim 8) encodes video content of a second frame within the digital video; and comparing the first video frame embedding and the second video frame embedding”. However, in an analogous field of endeavor, Hoashi discloses “the second feature of this invention is that the story segmentation method for video, wherein the training process includes a first shot segmentation process for segmenting the training data per shot, a first section extraction process for extracting a section from the training data, a first feature extraction process for extracting features from each shot obtained by the first shot segmentation process, a training process for producing the story segmentation point recognizing device which conducts story segmentation for the entire video content based on the features of all shots extracted in the first feature extraction process, and a training process for producing the story segmentation point recognizing device for the specific sections based on the feature obtained from shots within specific sections in the first feature extraction process, and the evaluation process includes a second shot segmentation process for segmenting the input data per shot, a second section extraction process for extracting a section of the input data, a second feature extraction process for extracting the feature of each shot obtained by the second shot segmentation process, an entire story segmentation process for recognizing the entire story segmentation points using the entire feature of each shot obtained in the second feature extraction process and the story segmentation point recognizing device for entire video content, and a specific sections story segmentation process for recognizing the story segmentation points for specific sections using the feature of each shot out of the feature of each shot obtained in the second feature extraction process and the story segmentation point recognizing device for specific sections” (Hoashi, Paragraph [0016]). Therefore, it would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the video segmentation system seen in Caba with the technique of acquiring the first and second video frame embeddings seen in Hoashi to achieve the same video segmentation system described in Claim 10.
Regarding Claim 13, Caba discloses “The system of claim 8, further comprising instructions that, when executed by the at least one processor, cause the system to generate the segmented video transcript by” (Caba, Paragraphs [0183], [0156], [0050], [0142], and [0147] as well as Figures 10 and 11; please refer to the above-described analysis for Claim 8) (Hoashi, Paragraph [0024]). Hoashi further discloses “The section extraction process 12 extracts specific sections from the training data. The sections are the portions segmented as sections in video content. For example in the case of a news section, an commentary section, a sports section, an economy section, a special section, a weather section, or the like are present.” (Hoashi, Paragraph [0027]). It should be noted that because the story segmentation device is making decisions as to how to segment the video based on its story, it can indeed be considered a heuristic model. Therefore, it would be obvious for one of ordinary skill in the art to combine the video segmentation system seen in Caba with the heuristic model in the form of the story segmentation device seen in Hoashi to achieve the same method described in Claim 13.
Claims 17 is rejected under 35 U.S.C. 103 as being unpatentable over Caba in view of Chatoo, and further in view of Tan et al. (US 2015/0232764 w/ EFD of January 12, 2024).
Regarding Claim 17, the combination of Caba and Chatoo disclose “The non-transitory computer-readable medium of claim 16, wherein extracting the features indicating the change in the object comprises” (Caba, Paragraphs [0159], [0050], and [0156]; Chatoo, Paragraph [0044], please refer to the above-described analysis for Claim 16) . The combination of Caba and Chatoo does not explicitly disclose “extracting features indicating one or more of presence of a new object within a frame of the digital video or absence of a previously depicted object within the digital video”. However, in an analogous field of endeavor, Tan discloses “Entity analysis module 112 can include one or more image processing models that identify any entities in video data, which can include a physical object, person, organism, logo, and the like. An object recognition model may be employed to identify any objects in the video data, such as a vehicle, tool, electronic device, apparel, or any other physical object. Any organisms can be identified, including humans, animals, plants, fungi, etc. Similarly, entity analysis module 112 may identify any logo, trademark, symbol, or other identifier of a brand, organization, concept, and the like. In some embodiments, entity analysis module 112 may identify actions that are depicted in video data using motion detection or other techniques, including interactions between any of the entities identified in the video data. Entity analysis module 112 may identify particular locations or landmarks based on the presence of content in the video data, thus enabling the identities of these locations or landmarks to be used as candidate words for matching to utterances” (Tan, Paragraph [0023]). Accordingly, it would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the technique for identifying changes in an object in video segmentation computer-readable medium seen in the combination of Caba and Chatoo with the entity analysis module for detecting objects seen in Tan to improve the current video segmentation computer-readable medium. By using the entity analysis module seen in Tan, one of ordinary skill in the art can effectively determine when certain speakers or objects have left the frame to reflect those changes within certain video breaks or specific sections of the video transcript. Therefore, it would be obvious for one of ordinary skill in the art to combine the object change technique in the combination of Caba and Chatoo with the entity analysis module seen in Tan to achieve the same computer-readable medium found in Claim 17.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Caba in view of Nihei et al. (US 2024/0373073 w/ EFD of May 4, 2023).
Regarding Claim 20, Caba discloses “The non-transitory computer-readable medium of claim 15, further comprising instructions that, when executed by the at least one processor, cause the computer system to extract the set of video features by” (Caba, Paragraphs [0159], [0050], and [0156], please see the above-described analysis for Claim 15) . Caba doesn’t explicitly disclose “utilizing a filtering technique to extract features indicating a transition in the video content”. However, in an analogous field of endeavor, Nihei discloses “regarding video transition ML (machine learning) model 232, it is noted that ideal ad-breaks typically occur around direct transitions to black (hard transitions) or fade-to-black transitions (soft transitions). The first and last frames around hard transition segments are easier to detect. Soft transitions can pose a problem because their fade/dissolve sequence can make it difficult to define the beginning and end of the fade frame sequence. Video transition ML model 232 is configured to check for soft and hard transitions as a supplement to black frame detection ML model 224.” (Nihei, Paragraph [0042]). Nihei also further discloses how the “audio transition ML model 238 is configured to ensure that expected audio breaks exhibit enough discontinuity to signal a natural transition point within audio component 225. Audio transition ML model 238 in the form of an NN, for example, may make comparisons between decomposed audio signals (e.g., in the form of spectrograms) taken before and after candidate ad-insertion points and evaluate the two signals for discontinuity. Discontinuity in the audio signal before and after the candidate ad-insertion point indicates that neither speech nor music are interrupted by the candidate ad-insertion point. Audio transition ML model 238 may also check for characteristic patterns in sound levels that indicate a purposeful decrease and subsequent increase in volume around the candidate ad insertion point” (Nihei, Paragraph [0043]). Accordingly, it would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the video segmentation computer-readable medium seen in Caba with the technique of filtering the video features to determine a transition seen in Nihei to achieve an improved video segmentation process for the segmented video transcript. By filtering through specific methods, such as the audio and video machine learning models seen in Nihei, one of ordinary skill in the art can allow the medium to assess the video frames to indicate transitions within the video that could indicate a new video segment and section within the video transcript. Therefore, it would be obvious to combine both the Caba and Nihei references to achieve the computer-readable medium of Claim 20.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Biswas et al. (US 2025/0069600) teaches of a method for obtaining an initial transcription for input natural speech, performing audio segmentation and transcript alignment, and generate a final transcription based off of the final audio segments and transcription portions
Salamon et al. (US 2024/0127820) teaches of systems, methods, and computer storage media for music-aware speaker diarization.
Wang (CN 114187556 A) teaches of a high-definition video intelligent segmentation method based on picture characteristics.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SORIE I KOROMA JR whose telephone number is (571)272-9259. The examiner can normally be reached Monday - Friday 8AM-6:00PM; Alternate Fridays Off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached at 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SORIE I KOROMA JR/Examiner, Art Unit 2662
/Siamak Harandi/Primary Examiner, Art Unit 2662
Prosecution Insights