DETAILED ACTION
Response to Arguments
Applicant’s argument: Applicant notes that the Office Action is incomplete. The bottom of page 15 is the end of a paragraph. Page 16 picks up in the middle of a paragraph with a sentence fragment. There is no indication of how much of the Office Action is missing between the bottom of page 15 and the top of page 16. Something is missing between the bottom of page 15 and the top of page 16. To the extent that this application is not allowed in the next Office Action, Applicant requests that it be given an opportunity to respond to a complete office action before "final" status is imposed. Any new complete Office Action will necessarily be a new ground of rejection not necessitated by any amendment.
Examiner’s response: The only words missing at the top of page 16 were, “However Laksono discloses.” Whereby the secondary reference of Laksono is first introduced to teach limitations that were not taught by Xu. Nothing of substance was missing from the rejection that would require this office action be a second action Non-Final Rejection.
Applicant’s argument: The Office Action states that "Xu discloses the above limitations, whereby the aggregated metadata corresponds to a specific scene." Applicant respectfully disagrees. Applicant acknowledges that Xu, Figure 11, and the description thereof show a video data stream broken into a series of frames. The frames are grouped by shots, and shot clustering is used to develop a scene. Xu does not disclose aggregated metadata corresponding to a specific scene. The Office Action does not identify any disclosure in Xu showing metadata aggregated to correspond to a specific scene. A review of Xu does not reveal any such disclosure.
The stated premise of the rejection is that Xu shows aggregated metadata corresponding to a specific scene. That premise is not correct, and as such, the combination of references does not establish that claim 1 is obvious.
Examiner’s response: As disclosed in the rejection of claim 1, Xu does in fact disclose, aggregated metadata that corresponds to a specific scene. This is disclosed in Xu ¶203-204, whereby a specific “target type clip” has its metadata fused and then the fused feature vector is input to a classification model that outputs a preset category, which is considered to be aggregated metadata. This is also shown in Xu Fig. 12, whereby the input is a “highlight clip”, and the output from the classification model is “first label”, which is considered to be aggregated metadata.
Applicant’s argument: Furthermore, there is no reason to modify Xu by reference to Laksono. Laksono is a system for chaptering an individual video content stream, and Xu is for processing multiple content streams and selecting clips from multiple videos that correspond. Xu does not include anything that suggests indexing based on scene content. Thus, one of ordinary skill in the art would not be led to the instant claims by the combined teachings of Xu and Laksono.
Examiner’s response: While Xu does not disclose indexing by a scene, there is a motivation to combine Xu with Laksono since it is advantageous to have scenes that are indexed by metadata. The motivation for Indexing video scenes with metadata is because it makes video content highly discoverable, searchable, and manageable, enabling rapid retrieval, improved search engine optimization. Users can find specific moments or objects in vast libraires, rather than scanning hours of video. Search engine can use metadata such as titles, tags, and description, to index and rank videos.
Furthermore, the indexing of Laksono is not specific to only one video but can be used to find a plurality of videos, as disclosed in ¶50, “In addition to use by custom chapter generator 130, the custom chapter data 132 can include such index data 115 and be used in video storage and retrieval, and particularly to find videos of interest (e.g. relating to sports or cooking), locate videos containing certain scenes (e.g. a man and a woman on a beach).”
CLAIM INTERPRETATION
The 35 USC 112(f) interpretation of claims 1 and 4 has been withdrawn. As result the 35 USC 112(a) and 35 USC 112(b) rejection have also been withdrawn.
Double Patenting
The non-statutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A non-statutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on non-statutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP §§ 706.02(l)(1) - 706.02(l)(3) for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claim 1 is rejected on the ground of non-statutory double patenting as being unpatentable over claim 1 of Application No. 18/581,329. Although the claims at issue are not identical, they are not patentably distinct from each other because the present application is anticipated by Application No. 18/581,329.
Table 1 illustrates the conflicting claim pairs:
Present Application
1
App. No. 18/581,329
1
Table 2 illustrates a mapping between the limitations claim 1 of the present application and claim 1 of US App 18/581,329.
Claim 1 of present App.
Claim 1 of App. No. 18/581,329
A multimodal metadata extraction system comprising:
A system for contextual matching video content based on multimodal metadata extraction generated by processing one or more scenes to extract metadata corresponding to multiple extraction modes, and an embedding model for each extraction mode wherein an aggregated embedding model responsive to said metadata embeddings for each mode formulates an aggregated embedding comprising:
a scene detector having a video content input and an output representing scene boundaries; a metadata extractor responsive to content of a scene as identified by said output representing scene boundaries and having an extracted metadata output corresponding to several extraction modes; a metadata embedding for each extraction mode; and an embedding aggregator responsive to said metadata embedding and having an aggregated embedding output indexing each scene of said content.
a scene detector having a video content input and an output representing scene boundaries; a metadata extractor responsive to content of a scene as identified by said scene boundaries to extract metadata corresponding to several extraction modes; a metadata embedding for each extraction mode; an embedding aggregator responsive to said metadata embedding is to formulate an aggregated embedding for each scene indexing said content;
an embedding extractor responsive to a text input with an embedding model coordinated with said embedding model for one or more of said embedding modes; wherein said embeddings are in the form of a vector, and a vector comparison processor for determining the distance between the query vector and a vector representing said aggregated embedding and indicating the result.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3 and 5-6 are rejected under 35 U.S.C. 103 as being unpatentable over Xu et al. (US Pub. No. 2024/0031655 A1) in view of Laksono (US Pub. No. 2016/0148650 A1).
Regarding claim 1, Xu discloses, a multimodal metadata extraction system comprising: a scene detector having a video content input and an output representing scene boundaries; (See Xu ¶186, “1. Shot splitting. A current video is read, a shot boundary detection method is used to detect a shot boundary in the video, and then the video is split into a plurality of independent shots from the shot boundary, that is, a shot set corresponding to the current video is obtained. The current video is any video in the plurality of videos.”
Further see Xu ¶189, “2. Shot clustering. A cluster algorithm is used to perform shot clustering on a plurality of shots in the shot set corresponding to the current video, to obtain video clips, that is, obtain a video clip set corresponding to the current video.”
Further see Fig. 11, which shows the shot boundaries between the scenes.)
a metadata extractor responsive to content of a scene as identified by said output representing scene boundaries and having an extracted metadata output corresponding to several extraction modes; (See Xu ¶194, “FIG. 12 is a diagram of an architecture of a multi-modal video analytics system according to an embodiment of this disclosure. The system includes an information extraction module, a feature extraction module, a feature refinement module, a feature fusion module, and a classification model. The feature extraction module includes an audio feature extraction model, an image feature extraction model, and a document feature extraction model.”
Further see Xu ¶196, “Further, as shown in FIG. 12, audio extracted from the current target type clip is first input into an audio feature extraction model, to obtain a plurality of first audio feature vectors.”
Further see Xu ¶197, “As shown in FIG. 12, an image (including a plurality of image frames) extracted from the current target type clip is first input into the image feature extraction model to obtain a plurality of first image feature vectors.”)
Further see Xu ¶198, “As shown in FIG. 12, first, an OCR technology is used to extract a subtitle document from the current target type clip, to obtain the OCR subtitle document.”
Whereby all of these feature vectors are considered metadata, since metadata can be defined as data describing data.)
a metadata embedding for each extraction mode; (See Xu ¶194, “The feature refinement module includes an audio feature refinement model, an image feature refinement model, and a document feature refinement model.”
Further see Xu ¶196, “The plurality of extracted first audio feature vectors are input into an audio feature refinement model, to obtain a second audio feature vector.”
Further see Xu ¶197, “The plurality of extracted first image feature vectors are input into the image feature refinement model, to obtain a second image feature vector.”
Further see Xu ¶198, “Then, the extracted OCR subtitle document is input into the document feature extraction model to obtain a first document feature vector.”)
and an embedding aggregator responsive to said metadata embedding and having an aggregated embedding output for each scene of said content. (See Xu ¶203, “(2) As shown in FIG. 12, the second audio feature vector, the second image feature vector, and the second document feature vector of the current target type clip are input into the feature fusion module to perform feature fusion, to obtain a fused feature vector of the current target type clip.”
Further see Xu ¶204, “(3) As shown in FIG. 12, the fused feature vector of the current target type clip is input into the classification model, to obtain a probability that the current target type clip belongs to each preset category, and then the current target type clip is classified into a preset category with a highest probability, to obtain the first label of the current target type clip.”)
Xu discloses the above limitations, whereby the aggregated metadata corresponds to a specific scene, but he fails to disclose embedding for each scene indexing said content.
However, Laksono discloses, embedding output indexing each scene of said content. (See Laksono, ¶30, “In a further example, the index data 115 includes a database of metadata items that can grow or shrink dynamically. The database can store unique identifiers that correspond to particular metadata that identify content of the video in a time synchronized fashion. These metadata items can be stored at either a certain event (start of a scene change, shot transition or start of a new Group of Pictures encoding) or a certain time interval (e.g.: every 1 second) or it can be done at every picture.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the indexing of metadata for each scene as suggested by Laksono to Xu’s extraction of aggregated metadata for each scene. This can be done using known engineering techniques, with a reasonable expectation of success. The motivation for Indexing video scenes with metadata makes video content highly discoverable, searchable, and manageable, enabling rapid retrieval, improved search engine optimization. Users can find specific moments or objects in vast libraires, rather than scanning hours of video. Search engine can use metadata such as titles, tags, and description, to index and rank videos.
Regarding claim 2, Xu and Laksono disclose, the multimodal metadata extraction system according to claim 1 wherein said output representing scene boundaries is a set of video clips in each scene. (See Xu ¶189, “2. Shot clustering. A cluster algorithm is used to perform shot clustering on a plurality of shots in the shot set corresponding to the current video, to obtain video clips, that is, obtain a video clip set corresponding to the current video.”)
Regarding claim 3, Xu and Laksono disclose, the multimodal metadata extraction system according to claim 1 wherein said output representing scene boundaries is an index to said video content corresponding to said identified scenes. (See Laksono, ¶30, “In a further example, the index data 115 includes a database of metadata items that can grow or shrink dynamically. The database can store unique identifiers that correspond to particular metadata that identify content of the video in a time synchronized fashion. These metadata items can be stored at either a certain event (start of a scene change, shot transition or start of a new Group of Pictures encoding) or a certain time interval (e.g.: every 1 second) or it can be done at every picture.”)
Regarding claim 5, Xu and Laksono disclose, the multimodal metadata extraction system according to claim 1 further comprising an embedding database connected to said embedding aggregator for storing said aggregated embedding (See Laksono, ¶30, “In a further example, the index data 115 includes a database of metadata items that can grow or shrink dynamically. The database can store unique identifiers that correspond to particular metadata that identify content of the video in a time synchronized fashion. These metadata items can be stored at either a certain event (start of a scene change, shot transition or start of a new Group of Pictures encoding) or a certain time interval (e.g.: every 1 second) or it can be done at every picture.”)
for use as a search index for scenes of said content. (See Laksono ¶51, “In this fashion, object/features in each shot can be correlated to the shots that contain these objects and features that can be used for indexing and search of indexed video for key objects/features and the shots that contain these objects/features.”)
Regarding claim 6, Xu and Laksono disclose, the multimodal metadata extraction system according to claim 1 wherein said extraction modes include at least one of Audio (speech recognition, music recognition); (See Xu ¶196, “Further, as shown in FIG. 12, audio extracted from the current target type clip is first input into an audio feature extraction model, to obtain a plurality of first audio feature vectors.”)
Image recognition (feature recognition with temporal understanding); (See Xu ¶197, “As shown in FIG. 12, an image (including a plurality of image frames) extracted from the current target type clip is first input into the image feature extraction model to obtain a plurality of first image feature vectors.”)
Text (caption, scene summarization, text recognition; and scene interpretation (sentiment, profanity, acting level). (See Xu ¶198, “As shown in FIG. 12, first, an OCR technology is used to extract a subtitle document from the current target type clip, to obtain the OCR subtitle document.”)
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Xu et al. (US Pub. No. 2024/0031655 A1) in view of Laksono (US Pub. No. 2016/0148650 A1) in view of Jain et al. (US Pub. No. 2021/0326599 A1) and in further view of Tu et al. (US Pub. No. 2011/0225136 A1).
Regarding claim 4, Xu and Laksono he multimodal metadata extraction system according to claim 1, where shots are detected and then clustered to form scenes, but they fail to disclose wherein said scene detector further comprising: a frame analyzer for identifying consecutive frames having similar characteristics, a boundary detector identifying boundaries of consecutive frames having such similar characteristics responsive to said frame analyzer; an embedding system formulating a composite distance matrix capturing distance between shot embedding; and a temporal clustering system connected to said composite distance matrix and having an output identifying scene boundaries of said content.
However, Jain discloses, wherein said scene detector further comprising: a boundary table responsive to said frame analyzer storing boundaries of consecutive frames having such similar characteristics (See Jain ¶38, “The scene detection engine performs 102 video shot detection by detecting boundaries of multiple shots in the video content. That is, for each of the shots in the video content, the scene detection engine detects an end point of a shot and a starting point of a consecutive shot. The scene detection engine, therefore, establishes when a shot is ending and when a consecutive shot is starting.”
Further see Jain ¶50, “The media shot detector 1212a stores the media content in a media database 1307. The media shot detector 1212a detects boundaries of multiple shots in the video content.” Whereby a database inherently stores data in tables, and in this case the data is boundary points of shots.)
an embedding responsive to said content and said boundaries stored in said boundary table and having a composite distance matrix output representing distance between shot embeddings; (See Jain ¶38, “The scene detection engine performs 102 video shot detection by detecting boundaries of multiple shots in the video content. That is, for each of the shots in the video content, the scene detection engine detects an end point of a shot and a starting point of a consecutive shot. The scene detection engine, therefore, establishes when a shot is ending and when a consecutive shot is starting. The scene detection engine extracts 103 a middle frame of each of the shots. The scene detection engine then extracts 104 col or histograms for the middle frames as disclosed in the detailed description of FIG. 5. The scene detection engine generates 105 an image similarity matrix as illustrated in FIG. 6, by extracting color features from the middle frames of the shots.”
Further see Jain ¶39, “The scene detection engine extracts 107 audio features from the audio content of each of the shots as disclosed in the detailed description of FIG. 7 and generates 108 an audio similarity matrix as illustrated in FIG. 8. … The scene detection engine then generates 109 a resultant similarity matrix from the image similarity matrix and the audio similarity matrix as illustrated in FIG. 9. The resultant similarity matrix is a merged similarity matrix, that is, a combination of the image similarity matrix and the audio similarity matrix.”)
and a temporal clustering system connected to said composite distance matrix and having an output identifying scene boundaries of said content. (See Jain ¶40, “The scene detection engine performs automatic clustering of video content using the affinity propagation clustering algorithm on two dimensions, that is, visual features and audio features. On execution of the clustering algorithm, the scene detection engine generates an ordered sequence of shots that define a boundary of each of the scenes of the video content, thereby automatically detecting and marking logical scenes in the video content.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the scene detection based on clustering of video shots according to a similarity matrix as suggested by Jain to Xu and Laksono’s scene detection. This can be done using known engineering techniques, with a reasonable expectation of success. The motivation for doing so is improved robustness. Using similarity matrices helps in quantifying image and audio relationships, allowing for robust scene detection even with varying shot lengths or complex features.
Xu, Laksono, and Jain disclose the above limitations, but they fail to disclose, a frame analyzer for identifying consecutive frames having similar characteristics, a boundary table responsive to said frame analyzer storing boundaries of consecutive frames having such similar characteristics responsive to said frame analyzer.
However, Tu discloses, a frame analyzer for identifying consecutive frames having similar characteristics, a boundary detector identifying boundaries of consecutive frames having such similar characteristics responsive to said frame analyzer; (See Tu ¶67 Each section of the video file obtained by segmenting the video file according to the scene change point is referred to as a "shot". Whether a current frame is served as a shot boundary is determined according to, for example, the distance of HSV histogram between frames. Whether the current frame is served as a shot boundary is determined according to the obtained HSV histogram feature (a shot detection algorithm).”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the shot detection using a distance of a histogram feature between frames as suggested by Tu to Xu, Laksono, and Jain’s shot detection. This can be done using known engineering techniques, with a reasonable expectation of success. The motivation for doing so is computational efficiency, robustness to motion or camera shake, and detecting abrupt global scenes.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID PERLMAN whose telephone number is (571) 270-1417.
The examiner can normally be reached on Monday - Friday; 10:00am -6:30pm.
Examiner interviews are available via telephone and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is
(571) 273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at (866) 217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call (800) 786-9199 (IN USA OR CANADA) or (571) 272-1000.
/DAVID PERLMAN/Primary Examiner, Art Unit 2673
Prosecution Insights