DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 3/9/2026 has been entered.
Response to Amendment
The amendments, filed 3/9/2026, have been entered and made of record. Claims 1, 5-9, 15-18, and 20 have been amended. Claims 1-3, 5-20 are pending.
Response to Arguments
Applicant’s arguments in the Remarks filed on 3/9/2026 have been considered but are moot in view of the new ground(s) of rejection.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Mazaheri in view of Zavesky
Claim 1-3, 5, 9-16 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Mazaheri et ail.(USPubN 2024/0357191; hereinafter Mazaheri) in view of Zavesky et al.(USPubN 2019/0045194; hereinafter Zavesky).
As per claim 1, Mazaheri teaches a system comprising:
one or more memories; and at least one processor coupled to at least one of the one or more memories and configured to perform operations comprising(“the computing system 900 may include one or more computer processor(s) 902, associated memory 904 (e.g., random access memory (RAM), cache memory, flash memory, etc.), one or more storage device(s) 906 (e.g., a hard disk, an optical drive such as a compact disk (CD) drive or digital versatile disk (DVD) drive, a flash memory stick, etc.), a bus 916, and numerous other elements and functionalities.” in Para.[0153]):
segmenting media content into a sequence of units by detecting unit boundaries in the media content(“The scene break detection service 170 may be configured to perform shot detection in order to identify continuous segments of smaller visually similar frames, or may utilize each shot as a separate frame, in accordance with various embodiments of the invention” in Para.[0025]);
applying one or more feature encoders to generate, in an embedding space, a multimedia representation of features of each unit in the sequence of units across different media modalities for the media content(“he scene break detection service 170 includes functionality to identify a set of candidate ad break timestamps corresponding to instances of a set of the audio break timestamps and a set of video break timestamps within a predefined proximity. In one embodiment, the set of audio break timestamps are obtained from the audio analysis engine 174 and the set of video break timestamps are obtained from the video analysis engine 172” in Para.[0047], “video analysis is optionally performed to identify a set of shots in the media item (STEP 204). The set of shots can consist of one or more frames of the video, for example, grouped on the basis of some criteria for visual similarity” in Para.[0120], “audio analysis is performed. This can include audio break detection on an audio component of the media item to identify one or more audio scenes (STEP 206). Audio analysis can include speech detection, music or background noise detection, and any other procedure for designating one or more segments of the audio file as audio “scenes” that should be treated as uninterruptible or less suited for interruption during playback of the media item” in Para.[0121], The audio analysis engine and the video analysis engine can be interpreted as one or more feature encoder because those can identify a multimedia representation of features such as a set of shots and one or more audio scenes of candidate scene breaks in segment of the media. The video component and the audio component can be interpreted as the different media modalities); and
wherein the different media modalities include at least two of a visual modality, an audio modality, and a timed text modality, and wherein the one or more feature encoders are configured to encode features of each unit in the at least two of the visual modality, the audio modality, and the timed text modality into the embedding space to form the multimedia representation(“computer vision analysis is performed on a video component of the media item. Prior to computer vision analysis, any number of the candidate shots from STEP 204 may be eliminated according to one or more business criteria or heuristics. The remaining shots may be utilized as inputs to the computer vision analysis stage for scoring and selection. The computer vision analysis then involves scoring of each candidate shot by taking N shots before and N shots after a starting timestamp of the candidate shot and performing a visual comparison of the two sets of shots. This multi-shot analysis generates an improved and more contextual score by optimizing for the surround shots rather than a simple shot-by-shot analysis. The output of the computer vision analysis is a score generated for each of the remaining shots” in Para.[0122], “computer vision analysis is performed on the media item. The output of the computer vision analysis is a scoring of each of the remaining shots/timestamps. In STEP 350, a best subset of the score shots is selected on the basis of the calculated scores and/or any number of other business criteria or application-specific heuristics” in Para.[0126], Fig. 2 and 3); and
identifying, through a scoring, whether a unit boundary of the unit boundaries is a scene boundary based on multimedia representations of units in the embedding space in at least a subset of the sequence of units(“he scene break detection service 170 includes functionality to identify a set of candidate ad break timestamps corresponding to instances of a set of the audio break timestamps and a set of video break timestamps within a predefined proximity. In one embodiment, the set of audio break timestamps are obtained from the audio analysis engine 174 and the set of video break timestamps are obtained from the video analysis engine 172” in Para.[0047], “computer vision analysis is performed by executing a computer vision model to generate a scene change score for the candidate scene break using the feature vector as an input. Thus, the computer vision analysis generates a scoring indicating likelihood of a scene change occurring at the given shot/location in the media item” in Para.[0145]).
Mazaheri is silent about identifying, through a sequence classifier, whether a unit boundary of the unit boundaries is a scene boundary based on multimedia representations of units.
Zavesky teaches identifying, through a sequence classifier, whether a unit boundary of the unit boundaries is a scene boundary based on multimedia representations of units(“the process 200 may start with a video program 201 comprising a number of scenes 291-295 (FIG. 2 illustrates a high-level representation of the video program 201 in a timeline form). At stage 210, the scene boundaries of scenes 291-295 may be detected using a shot detection program and/or a scene detection program. At stage 220, various features that may be used for one or more theme classifiers may be extracted from the scenes 291-295, such as low-level invariant image data including colors, shapes, color moments, color histograms, edge distribution histograms, etc., words and phrases identified in an audio portion, captioned text, and so forth” in Para.[0040]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings Mazaheri with the above teachings of Zavesky in order to improve the effect of video processing.
As per claim 2, Mazaheri and Zavesky teach all of limitation of claim 1.
Mazaheri teaches wherein the sequence of units comprises a sequence of shots in the media content(“a scene can refer to a grouping of shots based on something more than a simple frame-by-frame analysis. In one embodiment, a scene refers to a segment of the media file that depicts a division of an act presenting continuous action in one place, a single situation in a narrative depicted by the media item, or a continuous segment of the media item that occurs in a single stage setting.” in Para.[0038]).
As per claim 3, Mazaheri and Zavesky teach all of limitation of claim 1.
Mazaheri teaches wherein the sequence of units comprises frames in the media content(“a scene can refer to a grouping of shots based on something more than a simple frame-by-frame analysis. In one embodiment, a scene refers to a segment of the media file that depicts a division of an act presenting continuous action in one place, a single situation in a narrative depicted by the media item, or a continuous segment of the media item that occurs in a single stage setting.” in Para.[0038]).
As per claim 5, Mazaheri and Zavesky teach all of limitation of claim 1.
Mazaheri teaches wherein the one or more feature encoders are further configured to: convert each unit in the sequence of units into one or more corresponding keyframes representing the visual modality; and encode the one or more corresponding keyframes of each unit into the embedding space to form the multimedia representation of the features of each unit in the visual modality(“the media service 108 includes functionality to generate a media clip of a media item. Similar to media previews, media clips can be generated based on a desired duration or any number of other criteria.” in Para.[0111], “The scene break detection service 170 can be configured to generate a set of feature vectors including at least one visual attribute of the candidate segment, a set of actor attributes of the media item, and a set of genre attributes of the media item. Thus, labeled datasets can be used to generate the inference and to optimize for a variety of correlations that may exist between various criteria and the likelihood of a media item to “go viral” on a social media platform, for example. The scene break detection service 170 can then select a candidate scene or set of scenes having a highest virality score for generation of the media clip” in Para.[0113]).
As per claim 9, Mazaheri and Zavesky teach all of limitation of claim 1.
Mazaheri teaches wherein the one or more feature encoders are trained to encode data associated with the visual modality, the audio modality, the timed text modality, or a combination thereof through contrastive learning(“the scene break detection service 170 includes functionality to deploy and execute a machine learning model for executing computer vision analysis. The machine learning model is a supervised model generated and trained using a labeled dataset of examples. The dataset can be obtained from publicly available media sources comprising a media file and tagged advertisement timestamps. In one example, the scene break detection service 170 includes functionality to obtain the training dataset from cable television, other media streaming platforms, and a variety of available sources of media content comprising advertisements. These media ads can then be labeled by one or more human administrators and provided to a machine learning training module (not shown) for initial training and/or online training of the model” in Para.[0051], [0052], [0053], [0111]-[0118]).
As per claim 10, Mazaheri and Zavesky teach all of limitation of claim 1.
Mazaheri is silent about wherein the sequence classifier is configured to identify the unit boundary as the scene boundary based on similarity between the multimedia representations of the units in the embedding space in the at least a subset of the sequence of units.
Zavesky teaches wherein the sequence classifier is configured to identify the unit boundary as the scene boundary based on similarity between the multimedia representations of the units in the embedding space in the at least a subset of the sequence of units(“derive or obtain theme models (e.g., classifiers) for a number of themes, analyze incoming video programs to identify themes in the video programs, and tag the video programs with the themes that are identified. Notably, classifiers can be trained from any video or image content to recognize various themes, which may include objects like “car,” scenes like “outdoor,” and actions or events like “baseball.” Similarly, shot and scene detection algorithms may locate and tag shot and/or scene boundaries in a video program.” in Para.[0010]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings Mazaheri with the above teachings of Zavesky in order to improve the effect of video processing.
As per claim 11, Mazaheri and Zavesky teach all of limitation of claim 10.
Mazaheri is silent about wherein the sequence classifier is further configured to implement one or more rules related to classifying scene boundaries in identifying the unit boundary as the scene boundary based on the similarity between the multimedia representations of the units in the embedding space in the at least a subset of the sequence of units.
Zavesky teaches wherein the sequence classifier is further configured to implement one or more rules related to classifying scene boundaries in identifying the unit boundary as the scene boundary based on the similarity between the multimedia representations of the units in the embedding space in the at least a subset of the sequence of units (“derive or obtain theme models (e.g., classifiers) for a number of themes, analyze incoming video programs to identify themes in the video programs, and tag the video programs with the themes that are identified. Notably, classifiers can be trained from any video or image content to recognize various themes, which may include objects like “car,” scenes like “outdoor,” and actions or events like “baseball.” Similarly, shot and scene detection algorithms may locate and tag shot and/or scene boundaries in a video program.” in Para.[0010], “Theme identification classifiers may include support vector machine (SVM) based or non-SVM based classifiers, such as neural network based classifiers. The classifiers may be trained upon and utilize various data points to recognize themes in scenes. For instance, classifiers may use low-level invariant image data, such as colors, shapes, color moments, color histograms, edge distribution histograms, etc., may utilize speech recognition pre-processing to obtain an audio transcript and to rely upon various keywords or phrases as data points, may utilize text recognition pre-processing to identify keywords or phrases in captioned text as data points, may utilize image salience to determine whether detected objects are “primary” objects of a scene or are less important or background objects, and so forth. In one example, classifiers may be initially trained on a labeled training data set, such as TRECVID training library, or the like” in Para.[0014]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings Mazaheri with the above teachings of Zavesky in order to improve the effect of video processing.
As per claim 12, Mazaheri and Zavesky teach all of limitation of claim 11.
Mazaheri is silent about wherein the one or more rules are selected as part of a subset of a plurality of rules that can be applied in classifying scene boundaries from identified unit boundaries.
Zavesky teaches wherein the one or more rules are selected as part of a subset of a plurality of rules that can be applied in classifying scene boundaries from identified unit boundaries(Para.[0014], “the theme that is identified in a scene may comprise a primary theme identified from a number of themes in the scene. For instance, the primary theme may be a theme with a maximal likelihood score of a classifier. For example, a classifier can be a support vector machine (SVM) based or non-SVM based classifier, such as a neural network based classifier. In various examples, the present disclosure may employ a number of single class classifiers, or a multi-class classifier. In the example of SVM based classifiers, the primary theme may be the one having a furthest distance from a separation hyperplane for the respective classifier for the class, and so forth. In another example, the encoding strategy may be selected based upon the plurality of themes that is identified in a scene. For instance, different encoding strategies may be associated with the different themes. However, certain encoding strategies may include parameters which are intended to result in a greater storage utilization (e.g., less compression), and hence greater video quality. In such an example, the encoding strategy may be selected as a composite of parameters from the encoding strategies associated with the respective themes that would result in the greatest storage utilization” in Para.[0015]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings Mazaheri with the above teachings of Zavesky in order to improve the effect of video processing.
As per claim 13, Mazaheri and Zavesky teach all of limitation of claim 1.
Mazaheri is silent about wherein the sequence classifier is trained based on labeled data of different media content and the labeled data is indicative of breaks in an audio modality of the different media content, breaks in a visual modality of the different media content, breaks in a timed text modality of the different media content, scene breaks in the different media content, or a combination thereof.
Zavesky teaches wherein the sequence classifier is trained based on labeled data of different media content and the labeled data is indicative of breaks in an audio modality of the different media content, breaks in a visual modality of the different media content, breaks in a timed text modality of the different media content, scene breaks in the different media content, or a combination thereof(Para.[0014], [0015], “the process 200 may start with a video program 201 comprising a number of scenes 291-295 (FIG. 2 illustrates a high-level representation of the video program 201 in a timeline form). At stage 210, the scene boundaries of scenes 291-295 may be detected using a shot detection program and/or a scene detection program. At stage 220, various features that may be used for one or more theme classifiers may be extracted from the scenes 291-295, such as low-level invariant image data including colors, shapes, color moments, color histograms, edge distribution histograms, etc., words and phrases identified in an audio portion, captioned text, and so forth” in Para.[0040]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings Mazaheri with the above teachings of Zavesky in order to improve the effect of video processing.
As per claim 14, Mazaheri and Zavesky teach all of limitation of claim 1.
Mazaheri is silent about wherein the operations further comprise applying the sequence classifier to identify one or more cue points in the sequence of units, the one or more cue points including a start of a title sequence, an end of the title sequence, a start of closing credits, an end of the closing credits, or a combination thereof.
Zavesky teaches wherein the operations further comprise applying the sequence classifier to identify one or more cue points in the sequence of units, the one or more cue points including a start of a title sequence, an end of the title sequence, a start of closing credits, an end of the closing credits, or a combination thereof(Para.[0014], [0015], [0040], “a shot detection program may utilize color histogram differences or a change in color distribution, edge change ratios, standard deviation of pixel intensities, contrast, average brightness, and the like to identify hard cuts, fades, dissolves, etc., which may indicate the end of a shot and the beginning of another shot” in Para.[0017]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings Mazaheri with the above teachings of Zavesky in order to improve the effect of video processing.
As per claim 15, the limitations in the claim 15 has been discussed in the rejection claim 1 and rejected under the same rationale.
As per claim 16, the limitations in the claim 16 has been discussed in the rejection claim 5 and rejected under the same rationale.
As per claim 19, the limitations in the claim 19 has been discussed in the rejection claim 10 and rejected under the same rationale.
As per claim 20, a non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising(Para.[0008]) and the other limitations in the claim 20 has been discussed in the rejection claim 1 and rejected under the same rationale.
Mazaheri in view of Zavesky and Mavlankar
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Mazaheri et ail.(USPubN 2024/0357191; hereinafter Mazaheri) in view of Zavesky et al.(USPubN 2019/0045194; hereinafter Zavesky) further in view of Mavlankar et al.(USPubN 2020/0029131; hereinafter Mavlankar).
As per claim 6, Mazaheri and Zavesky teach all of limitation of claim 1.
Mazaheri and Zavesky are silent about wherein the one or more feature encoders are further configured to: access one or more frames in each unit of the sequence of units that are displayed during a fast-forward operation, a rewind operation, a pause operation, or a combination thereof in reproducing the media content; and encode the one or more frames in each unit into the embedding space to form the multimedia representation of the features of each unit in the visual modality.
Mavlankar teaches wherein the one or more feature encoders are further configured to: access one or more frames in each unit of the sequence of units that are displayed during a fast-forward operation, a rewind operation, a pause operation, or a combination thereof in reproducing the media content; and encode the one or more frames in each unit into the embedding space to form the multimedia representation of the features of each unit in the visual modality (Para.[0019]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings Mazaheri and Zavesky with the above teachings of Mavlankar in order to improve the effect of video processing.
Mazaheri in view of Zavesky and Krishnamurthy
Claims 7 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Mazaheri et ail.(USPubN 2024/0357191; hereinafter Mazaheri) in view of Zavesky et al.(USPubN 2019/0045194; hereinafter Zavesky) further in view of Krishnamurthy(USPubN 2021/0321172)
As per claim 7, Mazaheri and Zavesky teach all of limitation of claim 1.
Mazaheri and Zavesky are silent about wherein the one or more feature encoders are further configured to: convert an audio signal in each unit in the sequence of units into one or more spectrograms representing the audio modality; and encode the one or more spectrograms in each unit into the embedding space to form the multimedia representation of the features of each unit in the audio modality.
Krishnamurthy teaches wherein the one or more feature encoders are further configured to: convert an audio signal in each unit in the sequence of units into one or more spectrograms representing the audio modality; and encode the one or more spectrograms in each unit into the embedding space to form the multimedia representation of the features of each unit in the audio modality (Para.[0046]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings Mazaheri and Zavesky with the above teachings of Krishnamurthy in order to improve the effect of video processing.
As per claim 17, the limitations in the claim 17 has been discussed in the rejection claim 7 and rejected under the same rationale.
Mazaheri in view of Zavesky and Li
Claims 8 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Mazaheri et ail.(USPubN 2024/0357191; hereinafter Mazaheri) in view of Zavesky et al.(USPubN 2019/0045194; hereinafter Zavesky) further in view of Li et al.(USPubN 2024/0064367; hereinafter Li).
As per claim 8, Mazaheri and Zavesky teach all of limitation of claim 1.
Mazaheri and Zavesky are silent about wherein the one or more feature encoders are further configured to: access data associated with display of timed text of the media content in the timed text modality for each unit in the sequence of units; and encode the data associated with display of timed text of the media content for each unit into the embedding space to form the multimedia representation of the features of each unit in the timed text modality.
Li teaches wherein the one or more feature encoders are further configured to: access data associated with display of timed text of the media content in the timed text modality for each unit in the sequence of units; and encode the data associated with display of timed text of the media content for each unit into the embedding space to form the multimedia representation of the features of each unit in the timed text modality (Para.[0050]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings Mazaheri and Zavesky with the above teachings of Li in order to improve the effect of video processing.
As per claim 18, the limitations in the claim 18 has been discussed in the rejection claim 8 and rejected under the same rationale.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SUNGHYOUN PARK whose telephone number is (571)270-1333. The examiner can normally be reached M - Thur 6:00 am - 4 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, THAI Q TRAN can be reached at (571)272-7382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SUNGHYOUN PARK/Examiner, Art Unit 2484
Prosecution Insights