Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Amendments
This action is in response to amendments filed October 14th, 2025, in which Claims 1, 3, 9, 10, & 16-18 have been amended. Claims 2 & 8 have been cancelled. No new claims have been added. The amendments have been entered, and Claims 1, 3-7, & 9-20 are currently pending.
Response to Arguments
Regarding the applicant’s traversal of the 35 U.S.C. 101 rejections of the previous office action, the applicant’s arguments filed October 14th, 2025 have been fully considered, and are unpersuasive.
Applicant asserts that “training the machine learning model using at least the labeled portion of the content asset” as recited by amended claim 1, does not recite a mental process under Step 2A Prong One, and as such, recites an “additional element” that integrates the mental processes into a practical application by improving the functioning of a multimedia detector that is implemented using the trained machine learning model, further citing [0012-0013] of the specification as support.
The examiner notes that it is agreed that this new limitation does not recite a mental process, and is thus an “additional element.” However, the examiner notes that this limitation merely constitutes as mere instructions to apply the judicial exception, which is not indicative of significantly more. These claims are using machine learning at a high level, and as we know from the court in Recentive Analytics, Inc. v. Fox Corp., No. 2023-2437, slip op. at 18 (Fed. Cir. April 18, 2025), “using a machine learning technique.. necessarily includes an interactive training step. Iterative training using selected training materials and dynamic adjustments based on real-time changes are incident to the very nature of machine learning.” Therefore, the rejection is maintained.
Further, amended independent claims 9 & 16 also have similar amended limitations as claim 1, and the rejections for these claims on the basis of 35 U.S.C. 101 have also been maintained under the same rationale. All other claims are dependent upon one of these claims and their rejections under 35 U.S.C. 101 have also been subsequently maintained.
Regarding the applicant’s traversal of the 35 U.S.C. 103 rejections of the previous office action, the applicant’s arguments filed October 14th, 2025 have been fully considered, and are unpersuasive.
The applicant asserts that in view of the specification at [0031] & [0048-0049], claim 1’s limitation “determining, based on metadata associated with a content asset, a relevance score indicative of a likelihood of the content asset comprising at least one instance of an object” is not taught by MCCLOSKEY, further citing the specification at [0031] and [0048-0049] to further discuss perceived differences between the claimed limitation and the disclosure of MCCLOSKEY, asserting that partitioning of the video corpus into subsets does not correlate to the determination of a relevance score in the manner claimed.
The examiner respectfully asserts that this is still taught by the previously cited section of MCCLOSKY ([Abstract] “In the following sections, we develop a fully automated algorithm to partition the video corpus according to video metadata, and to learn local weightings which are applied to base classifier scores (used for relevance/likelihood) in order to improve detection performance.”) This paper discusses using metadata to partition video datasets and learn local weightings for classifier scores, which directly relates to determining a relevance score for content assets based on metadata. The method incorporates metadata (e.g., video tags, timestamps) to weight the likelihood of detecting specific events or objects.
Applicant further asserts that [0031] of the specification describes metadata of a content asset labelled as “sitcom” with the object of interest to be detected as “a gunshot sound”, the content asset is found to not be likely to have the object of interest and would thus, receive a low relevance score, in order to further explain the first limitation of claim 1, further asserting that MCCLOSKY only describes “automatically partition[ing] [a] video corpus into relevant subsets”, “video quality metadata extracted from the header information . . . allow[s] us to determine salient divisions of the data - e.g., separating the video corpus into highly- and lightly-compressed subsets”, and using the partitioned video corpus to “learn local weightings which are applied to base classifier scores in order to improve detection performance” , further asserting that the use of video quality metadata to partition a video corpus into subsets does not teach or suggest “determining, based on metadata associated with a content asset, a relevance score indicative of a likelihood of the content asset comprising at least one instance of an object” as claimed.
Examiner respectfully submits that partitioning the video corpus based on metadata (such as a “sitcom” tag) to learn local weightings applied to classifier scores (such as what might be used to detect “a gunshot sound”) is equivalent to the determination of a relevance score for specific areas of the content asset that reflects the likelihood of the content asset having at least one instance of the said object ([Abstract] “In the following sections, we develop a fully automated algorithm to partition the video corpus according to video metadata, and to learn local weightings which are applied to base classifier scores (used for relevance/likelihood) in order to improve detection performance.”) This paper discusses using metadata to partition video datasets and learn local weightings for classifier scores, which directly relates to determining a relevance score for content assets based on metadata. The method incorporates metadata (e.g., video tags, timestamps) to weight the likelihood of detecting specific events or objects.
Applicant further asserts that learning local weightings for base classifier scores teaches determining weights to be applied to the scores generated by each base classifier, but that MCCLOSKY does not disclose using the video quality metadata to determine the base classifier scores themselves. Examiner respectfully submits that learning local weightings for the classifier scores from the metadata, does result in the classifier scores being determined based on the metadata. ([Abstract] “In the following sections, we develop a fully automated algorithm to partition the video corpus according to video metadata, and to learn local weightings which are applied to base classifier scores (used for relevance/likelihood) in order to improve detection performance.”) This paper discusses using metadata to partition video datasets and learn local weightings for classifier scores, which directly relates to determining a relevance score for content assets based on metadata. The method incorporates metadata (e.g., video tags, timestamps) to weight the likelihood of detecting specific events or objects.
Further, applicant asserts that MCCLOSKEY teaches using metadata to learn local weightings after the classifiers were already generated, rather than base classifier scores being generated based on the metadata.
The examiner respectfully submits that the claim does not currently state or specify whether base classifiers are generated from metadata or if local weightings are learned after the classifiers are already generated. The limitation, as claimed, currently states that a “relevance score” of some kind is “determined” based on metadata, whether this “relevance value” was already generated at some other base value or not. The examiner would like to clarify that if these distinctions are what is trying to be patented, then they must be made clear within the claims themselves.
Further, applicant asserts that “labeling the portion of the content asset detected by the machine learning model as a false identification of the object” is not taught by MCCLOSKEY, further asserting that MCCLOSKEY does not teach that the machine learning model itself is negatively labelling any training clips.
The examiner respectfully asserts, as pointed out in the final paragraph of page 2 of MCCLOSKEY: the algorithm of MCCLOSKEY is a “a fully automated algorithm to partition the video corpus according to video metadata, and to learn local weightings which are applied to base classifier scores in order to improve detection performance.” Since the process of MCCLOSKEY is “fully automated” by the machine learning model, and “Section IV: Metadata-Weighted Fusion Algorithm, A. Partitioning” explicitly describes negatively labelling training clips based on “degenerate partitions” which contain “negative examples” which does, in fact, teach that the machine learning model is determining that the portion of the content asset is not an instance of the object, as taught by MCCLOSKY ([Section IV. A. Partitioning, paragraph 1] “In order to avoid degenerate partitions (e.g., a threshold above which all training clips are negatively labeled), we generate Nᵗ=20 candidate thresholds linearly spaced between the 25th and 75th percentiles of the metadata values of positive clips.”) The paper emphasizes thresholding on metadata-weighted classifier scores (relevance scores), negatively labelling unlikely events or objects, which aligns with the described limitation. Negatively labelling those events/objects is equivalent to determining that they are not instances of the object. Therefore, the rejections under 35 U.S.C. 103 are maintained.
Claim Rejections - 35 USC § 101
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is
directed to an abstract idea (mental process) without significantly more.
Regarding claim 1, in Step 1 of the 101-analysis set forth in MPEP 2106, the claim
recites a method for analyzing metadata. A method is one of the four statutory categories of invention.
In Step 2a Prong 1 of the 101-analysis set forth in the MPEP 2106, the examiner has
determined that the following limitations recite a process that, under the broadest reasonable
interpretation, covers a mental process but for recitation of generic computer components:
“determining, based on metadata associated with a content asset, a relevance score indicative of a likelihood of the content asset comprising at least one instance of an object” (A person can mentally determine the likelihood of something by evaluating tags (metadata) and making a judgement based on that of whether it might have some sort of content in it (MPEP 2106).)
“based on the relevance score, labeling the portion of the content asset detected by the machine learning model as a false identification” (A person can mentally determine whether something is said type of content by evaluating it compared to their own understanding (a threshold value) and making a judgement of whether it is that type of content (MPEP 2106).)
If claim limitations, under their broadest reasonable interpretation, covers performance of
the limitations as a mental process but for the recitation of generic computer components, then it
falls within the mental process grouping of abstract ideas. Accordingly, the claim “recites” an
abstract idea.
In Step 2a Prong 2 of the 101-analysis set forth in MPEP 2106, the examiner has
determined that the following additional elements do not integrate this judicial exception into a
practical application:
“receiving, based on input of at least a portion of the content asset to a machine learning model trained to detect instances of the object, data indicating that the machine learning model detected the portion of the content asset comprising an instance of the object” (Adding insignificant extra-solution activity (mere data gathering) to the judicial exception (MPEP 2106.05(g)).
“training the machine learning model using at least the labeled portion of the content asset” (Mere instructions to apply the judicial exception (MPEP 2106.05(f)).)
Since the claim as a whole, looking at the additional elements individually and in
combination, does not contain any other additional elements that are indicative of
integration into a practical application, the claim is “directed” to an abstract idea.
In Step 2b of the 101-analysis set forth in the 2019 PEG, the examiner has determined
that the claim does not include additional elements that are sufficient to amount to significantly
more than the judicial exception.
As discussed above, additional element (iii) recites insignificant extra solution activity. Further, element (iii) recites a step that stores and retrieves information in memory, which has been determined by the courts to recite a well understood, routine and conventional activity which is not indicative of significantly more (Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015)).). Further, element (iv) recites mere instructions to apply the judicial exception, which is not indicative of significantly more. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 3, it is dependent upon claim 2, and thereby incorporates the limitations of, and corresponding analysis applied to claim 2. Further, claim 3 recites “wherein the data comprises an output of the machine learning model trained to detect instances of the object” (In step 2A prong 2, using data that is output from a machine learning model represents mere application of the judicial exception (machine learning model), which is not indicative of integration into a practical application. In step 2B, mere application of the judicial exception is not indicative of significantly more (MPEP 2106.05(g)).
Further, claim 3 recites “and wherein training the machine learning model using at least the labeled portion of the content asset comprises using the labeled portion of the content asset as negative training data for the machine learning model” (In step 2A prong 2, a machine learning model being trained on new data from false detections represents mere application of the judicial exception (machine learning model), which is not indicative of integration into a practical application (MPEP 2106.05(g). In step 2B, mere application of the judicial exception is not indicative of significantly more).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 4, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 4 recites “wherein the content asset comprises at least one of a video program, an audio program, a movie, a television show, or a video-on-demand asset” (In step 2a, prong 2, this recites generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h).) In step 2B, generally linking the use of the judicial exception to a particular technological environment or field of use is not indicative of significantly more.)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 5, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 5 recites “wherein the object comprises at least one of a sound, an audio event, a tangible item, or a concept.” (In step 2a, prong 2, this recites generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h).) In step 2B, generally linking the use of the judicial exception to a particular technological environment or field of use is not indicative of significantly more.)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 6, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 6 recites “wherein the metadata associated with the content asset comprises a plurality of labels associated with the content asset” (In step 2a, prong 2, this recites generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h).) In step 2B, generally linking the use of the judicial exception to a particular technological environment or field of use is not indicative of significantly more.)
Further, claim 6 recites “wherein each of the plurality of labels is associated with a relevance value indicative of a correlation between the label and the object.” (In step 2a, prong 2, this recites generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h).) In step 2B, generally linking the use of the judicial exception to a particular technological environment or field of use is not indicative of significantly more.)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 7, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 7 recites “wherein the relevance score is equal to the relevance value associated with the label, of the plurality of labels, having the highest correlation with the object” (In step 2a, prong 2, this recites generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h).) In step 2B, generally linking the use of the judicial exception to a particular technological environment or field of use is not indicative of significantly more.)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 9, in Step 1 of the 101-analysis set forth in MPEP 2106, the claim
recites a method. A method is one of the four statutory categories of invention.
In Step 2a Prong 1 of the 101-analysis set forth in the MPEP 2106, the examiner has
determined that the following limitations recite a process that, under the broadest reasonable
interpretation, covers a mental process but for recitation of generic computer components:
“determining, based on metadata associated with a first content asset, a first relevance score indicative of a likelihood of the first content asset comprising at least one instance of an object” (A person can mentally determine the likelihood of something by evaluating tags (metadata) and making a judgement based on that of whether it might have some sort of content in it (MPEP 2106).)
“based on the first relevance score, labeling the portion of the first content asset identified by the machine learning model as a false identification of the object.” (A person can mentally determine whether something is said type of content by evaluating it compared to their own understanding and making a judgement of whether it is that type of content (MPEP 2106).)
If claim limitations, under their broadest reasonable interpretation, covers performance of
the limitations as a mental process but for the recitation of generic computer components, then it
falls within the mental process grouping of abstract ideas. Accordingly, the claim “recites” an
abstract idea.
In Step 2a Prong 2 of the 101-analysis set forth in MPEP 2106, the examiner has
determined that the following additional elements do not integrate this judicial exception into a
practical application:
“receiving, based on input of at least a portion of the content asset to a machine learning model trained to detect instances of the object, output of the machine learning model identifying the portion of the first content asset as an instance of the object” (Adding insignificant extra-solution activity (mere data gathering) to the judicial exception (MPEP 2106.05(g)).
“training the machine learning model using at least the labeled portion of the content asset” (Mere instructions to apply the judicial exception (MPEP 2106.05(f)).)
Since the claim as a whole, looking at the additional elements individually and in
combination, does not contain any other additional elements that are indicative of
integration into a practical application, the claim is “directed” to an abstract idea.
In Step 2b of the 101-analysis set forth in the 2019 PEG, the examiner has determined
that the claim does not include additional elements that are sufficient to amount to significantly
more than the judicial exception.
As discussed above, additional element (iii) recites insignificant extra solution activity. Further, element (iii) recites a step that stores and retrieves information in memory, which has been determined by the courts to recite a well understood, routine and conventional activity which is not indicative of significantly more (Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015)).). Further, element (iv) recites mere instructions to apply the judicial exception, which is not indicative of significantly more. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claims 10-11 & 12-14, they are dependent upon claim 9 and thereby incorporate the limitations of, and corresponding analysis applied to claim 9. Further, they comprise similar limitations to claims 3, 9, & 5-7, respectively, and are rejected under the same rationale.
Regarding claim 15, it is dependent upon claim 13, and thereby incorporates the limitations of, and corresponding analysis applied to claim 13. Further, claim 15 recites “wherein the plurality of labels associated with the first content asset comprise at least one of a theme, a subject, a genre, a setting, a character, a tone, a rating, or a time period associated with the first content asset” (In step 2a, prong 2, this recites generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h).) In step 2B, generally linking the use of the judicial exception to a particular technological environment or field of use is not indicative of significantly more.)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 16, in Step 1 of the 101-analysis set forth in MPEP 2106, the claim
recites a method. A method is one of the four statutory categories of invention.
In Step 2a Prong 1 of the 101-analysis set forth in the MPEP 2106, the examiner has
determined that the following limitations recite a process that, under the broadest reasonable
interpretation, covers a mental process but for recitation of generic computer components:
“determining, for each of a plurality of metadata labels associated with a content asset, a relevance value indicative of a correlation between the metadata label and an object” (A person can mentally determine the likelihood of something by evaluating tags (metadata) and making a judgement based on that of whether it might have some sort of content in it (MPEP 2106).)
“determining, based on the plurality of relevance values, a relevance score indicative of a likelihood of the content asset comprising at least one instance of an object.” (A person can mentally evaluate the plurality of relevance values and make a judgement to determine a relevance score that indicates likelihood of a certain content asset to comprise at least one instance of a specific object (MPEP 2106).)
“based on the relevance score, labeling the at least one instance of the object as a false identification of the object” (A person can make a judgement to mentally label something they thought might be something as a false identification by evaluating what they’ve identified based on their understanding of it. (MPEP 2106).)
If claim limitations, under their broadest reasonable interpretation, covers performance of
the limitations as a mental process but for the recitation of generic computer components, then it
falls within the mental process grouping of abstract ideas. Accordingly, the claim “recites” an
abstract idea.
In Step 2a Prong 2 of the 101-analysis set forth in MPEP 2106, the examiner has
determined that the following additional elements do not integrate this judicial exception into a
practical application:
“detecting, based on input of at least a portion of the content asset to a machine learning model trained to detect instances of the object, at least one instance of the object in the content asset” (Mere instructions to apply the judicial exception (MPEP 2106.05(g)).
“training the machine learning model using at least the labeled portion of the content asset” (Mere instructions to apply the judicial exception (MPEP 2106.05(f)).)
Since the claim as a whole, looking at the additional elements individually and in
combination, does not contain any other additional elements that are indicative of
integration into a practical application, the claim is “directed” to an abstract idea.
In Step 2b of the 101-analysis set forth in the 2019 PEG, the examiner has determined
that the claim does not include additional elements that are sufficient to amount to significantly
more than the judicial exception.
As discussed above, additional elements (iv) & (v) recite mere instructions to apply the judicial exception, which is not indicative of significantly more. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 17, it is dependent upon claim 16 and thereby incorporates the limitations of, and corresponding analysis applied to claim 16. Further, it comprises similar additional limitations to claim 3, and is rejected under the same rationale.
Regarding claim 18, it is dependent upon claim 16, and thereby incorporates the limitations of, and corresponding analysis applied to claim 16. Further, claim 18 recites “sending the labeled at least one instance of the object to a computing device for storage, by the computing device” (In step 2A, prong 2, this recites insignificant extra solution activity of mere data storage, which is not indicative of integration into a practical application (MPEP 2106.05(h).). In step 2B, this recites storing and retrieving information in memory, which is a well-understood, routine and conventional activity, which is not indicative of significantly more (Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015)).).)
Further, claim 18 recites “in a database comprising training data for the machine learning model” (In step 2a, prong 2, this recites generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h).) In step 2B, generally linking the use of the judicial exception to a particular technological environment or field of use is not indicative of significantly more.)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 19, it is dependent upon claim 16, and thereby incorporates the limitations of, and corresponding analysis applied to claim 16. Further, claim 19 recites two more mental processes.
“determining, based on the plurality of relevance values, the relevance score indicative of the likelihood of the content asset comprising at least one instance of an object comprises: identifying a relevance value from the plurality of relevance values having the greatest value” (A person can mentally identify the highest of a group of values by evaluating the values and making a judgement for which one is the highest (MPEP 2106).)
“and assigning the relevance score a value equal to the relevance value” (A person can mentally evaluate multiple values and make a judgement to use whichever value is the greatest (MPEP 2106).)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 20, it is dependent upon claim 16, and thereby incorporates the limitations of, and corresponding analysis applied to claim 16. Further, claim 20 recites “wherein detecting the at least one instance of the object in the content asset comprises using the machine learning model to detect the at least one instance of the object in the content asset.” (In step 2A prong 2, using a machine learning model is mere application of the judicial exception, which is not indicative of integration into a practical application (MPEP 2106.05(f).) In step 2B, mere application of the judicial exception is not indicative of significantly more.)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-7, 9-14, 16-17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over McCloskey, Scott et al. "Metadata-Weighted Score Fusion for Multimedia Event Detection" Available on 19 May 2014 (hereafter, MCCLOSKY), and further in view of Jin, Sou Young, et al. “Unsupervised Hard Example Mining from Videos for Improved Object Detection” Available in 2018 (hereafter, JIN)
Regarding claim 1, MCCLOSKY teaches “A method comprising: determining, based on metadata associated with a content asset, a relevance score indicative of a likelihood of the content asset comprising at least one instance of an object”:
([Abstract] “In the following sections, we develop a fully automated algorithm to partition the video corpus according to video metadata, and to learn local weightings which are applied to base classifier scores (used for relevance/likelihood) in order to improve detection performance.”) This paper discusses using metadata to partition video datasets and learn local weightings for classifier scores, which directly relates to determining a relevance score for content assets based on metadata. The method incorporates metadata (e.g., video tags, timestamps) to weight the likelihood of detecting specific events or objects.
Further, MCCLOSKY teaches “…based on the relevance score labeling the portion of the content asset detected by the machine learning model as a false identification of the object”
([Section IV. A. Partitioning, paragraph 1] “In order to avoid degenerate partitions (e.g., a threshold above which all training clips are negatively labeled), we generate Nᵗ=20 candidate thresholds linearly spaced between the 25th and 75th percentiles of the metadata values of positive clips.”) The paper emphasizes thresholding on metadata-weighted classifier scores (relevance scores), negatively labelling unlikely events or objects, which aligns with the described limitation. Negatively labelling those events/objects is equivalent to determining that they are not instances of the object.
Further, MCCLOSKEY teaches “training the machine learning model using at least the labeled portion of the content asset”:
([Page 4, Section IV. Metadata-Weighted Fusion Algorithm] This section describes the training algorithm used to train the ML model including partitioning and labelling negative samples, which are also used to train the model “Consistent with other detection problems, we assume that the number of negative training examples far outweighs the number of positive examples, so no explicit mechanism is used to avoid degenerate partitions of the negative examples”.)
MCCLOSKY fails to explicitly teach “receiving, based on input of at least a portion of the content asset to a machine learning model trained to detect instances of the object, data indicating that the machine learning model detected the portion of the content asset comprising an instance of the object;” However, in analogous art that uses unsupervised mined hard examples in multimedia object detection, JIN, does teach this:
([Figure 1] As can be seen in the figure and its description, the detections were made by the model and identified to be either true positives (yellow boxes) or false positives (red boxes).)
It would be obvious for one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to combine the base reference, MCCLOSKY (a multimedia detector that detects objects based on identified areas and metadata, which also uses thresholds to identify false detections) with the teachings of JIN (a multimedia detector that receives data from a machine learning model of its detections) because both references address similar objectives of improving performance and accuracy of multimedia detectors.
One of ordinary skill in the art would be motivated to do so because a multimedia detector that detects objects based on identified areas and metadata will be more useful if it can analyze the data properly to determine which detections are accurate and which are false.
Regarding claim 3, MCCLOSKY and JIN teach the limitations of claim 2. Further, MCCLOSKY teaches “wherein the data comprises an output of the machine learning model trained to detect instances of the object”:
([Abstract] “We address the problem of multimedia event detection from videos captured 'in the wild,' in particular the fusion of cues from multiple aspects of the video's content: detected objects, observed motion, audio signatures, etc. We employ score fusion, also known as late fusion, and propose a method that learns local weightings of the various base classifier scores which respect the performance differences arising from the video quality.”)
MCCLOSKY fails to explicitly teach “and wherein training the machine learning model using at least the labeled portion of the content asset comprises using the labeled portion of the content asset as negative training data for the machine learning model.” However, analogous art, JIN, does teach this:
([page 2, Figure 1 description] “For the true positives, the same object is detected in all three frames whereas for the false positives, the detection is isolated– it occurs neither in the previous nor the subsequent frame. These detections that are “isolated in time” frequently turn out to be false positives, and hence provide important sources of hard negative training data for detectors.”)
It would be obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to combine the base reference of MCCLOSKY (a multimedia detector that detects objects based on identified areas and metadata) with the teachings of JIN (a multimedia detector that retains negative training data to use for further training) because both references address similar objectives of improving performance and accuracy of multimedia detectors.
One of ordinary skill in the art would be motivated to do so because JIN provides evidence that negative training samples improve accuracy and that the ability to identify them and use them will increase the accuracy of the model shown in MCCLOSKY.
Regarding claim 4, MCCLOSKY and JIN teach the limitations of claim 1. Further, MCCLOSKY teaches “wherein the content asset comprises at least one of a video program, an audio program, a movie, a television show, or a video-on-demand asset”:
([Section V. A. TRECVid Events, paragraph 1] “The Multimedia Event Detection (MED) task was added to the annual TRECVid1 evaluation in 2011 to assess the performance of event detection techniques on open source video clips. The evaluation provides training and testing video clips for several semantically rich events. Our results are presented on the MEDTest data, and ten events: E06 — Birthday party, E07 — Changing a vehicle tire, E08 — Flash mob gathering, E09 — Getting a vehicle unstuck, E10 — Grooming an animal, E11 — Making a sandwich, E12 — Parade, E13 — Parkour, E14 — Repairing an appliance, E15 — Working on a sewing project.”) In the cited examples, the invention was used on various types of video content.
Regarding claim 5, MCCLOSKY and JIN teach the limitations of claim 1. Further, MCCLOSKY teaches “wherein the object comprises at least one of a sound, an audio event, a tangible item, or a concept.”:
([Section V. C. Base Classifiers and Scores, paragraph 1] “Our experiments were performed with M=5 base classifiers, each of which estimates event probability based on a different multimedia feature. C1 (audio) Low-level audio information is captured using Mel-Frequency Cepstral Coefficients (MFCCs), computed using the HTK Speech Recognition Toolkit2, and a Support Vector Machine (SVM) with Histogram Intersection Kernel (HIK) is trained using a bag of words quantization of the MFCC features. C2 (visual — motion) A Bag of Words (BOW) Histogram of Optical Flow (HOF) feature extracted on Dense Trajectories [14], classified with a bagged HIK SVM. Note: this is the motion feature whose performance is shown in Fig. 3. C3 (visual — objects) Max frame-level Object Bank [15] scores maximized across video, classified with a bagged HIK SVM. C4 (visual — texture) Spatial pyramid feature on a Histogram of Oriented Gradient (HOG 2D) [16], classified with an HIK SVM. Note: this is the texture feature whose performance is shown in Fig. 3. C5 (visual — color) Color SIFT BoW feature [17] with 4096 codewords and spatial pyramid (1 global and 3 vertical layers), classified with an NGD kernel SVM.”) In this citation, it describes the types of objects used in their example to be detected which include sounds and audio events (audio), tangible items (visual - objects) in addition to the “concepts” cited in the claim 4 rejection above (e.g., parkour or a birthday party.)
Regarding claim 6, MCCLOSKY and JIN teach the limitations of claim 1. Further, MCCLOSKY teaches “wherein the metadata associated with the content asset comprises a plurality of labels associated with the content asset”:
([Section IV. D. Using Models in Test, paragraph 1] “For each metadata feature (for each one of them), we compute its estimated label (for each, meaning there is a plurality of labels) as a linear combination of base classifier scores, with weights determined by the metadata (working through the gating function).”)
and “wherein each of the plurality of labels is associated with a relevance value indicative of a correlation between the label and the object”:
([Section IV. B. Learning Weights, paragraph 1] “Let S=(s1,…,sM,1n) be an N-by-M+1 likelihood matrix with entry s(i,j) the score from the jth base classifier on the ith clip, and 1Nc a Nc-by-1 vector appended for adjusting the global offset. L∈{0,1}NC is the binary vector of training labels, and Λ a diagonal matrix with the ith entry indicating the gate response on the metadata of the ith video clip.”) This describes the algorithm used to assign weights to the labels to help it determine the relevance before being compared to a threshold.
Regarding claim 7, MCCLOSKY and JIN teach the limitations of claim 1. Further, MCCLOSKY teaches “wherein the relevance score is equal to the relevance value associated with the label, of the plurality of labels, having the highest correlation with the object.”:
([Section IV. Metadata-Weighted Fusion Algorithm, paragraph 1] “The high-level steps of the training algorithm are given in Algorithm 1. Inputs from the training set of Nc video clips consist of the associated ground truth labels L, scores S from Nb base classifiers, and metadata values M over Nm different quantities. The final trained model consists of Nm pairs of weights Wi, and performance metrics pi associated with each. Details of these high-level steps are given in the following sub-sections, and the use of the model in testing is described in the final sub-section.”) This algorithm explicitly uses metadata and assigns weightings to classifier scores, which correlates to determining a relevance score based on the labels. Additionally, the algorithm identifies the label or metadata with the strongest correlation to the object with successful detection, by assigning weightings to the scores, ensuring the final score is weighted by the label most strongly correlated with the object.
Regarding claim 9, MCCLOSKY teaches “A method comprising: determining, based on metadata associated with a first content asset, a first relevance score indicative of a likelihood of the first content asset comprising at least one instance of an object”:
([Abstract] “In the following sections, we develop a fully automated algorithm to partition the video corpus according to video metadata, and to learn local weightings which are applied to base classifier scores (used for relevance/likelihood) in order to improve detection performance.”) This paper discusses using metadata to partition video datasets and learn local weightings for classifier scores, which directly relates to determining a relevance score for content assets based on metadata. The method incorporates metadata (e.g., video tags, timestamps) to weight the likelihood of detecting specific events or objects.
Further, MCCLOSKY teaches “…based on the first relevance score labeling the portion of the first content asset identified by the machine learning model as a false identification of the object.”:
([Section IV. A. Partitioning, paragraph 1] “In order to avoid degenerate partitions (e.g., a threshold above which all training clips are negatively labeled), we generate Nᵗ=20 candidate thresholds linearly spaced between the 25th and 75th percentiles of the metadata values of positive clips.”) The paper emphasizes thresholding on metadata-weighted classifier scores (relevance scores), negatively labelling unlikely events or objects, which aligns with the described limitation. Negatively labelling those events/objects is equivalent to determining that they are not instances of the object.
Further, MCCLOSKEY teaches “training the machine learning model using at least the labeled portion of the first content asset”:
([Page 4, Section IV. Metadata-Weighted Fusion Algorithm] This section describes the training algorithm used to train the ML model including partitioning and labelling negative samples, which are also used to train the model “Consistent with other detection problems, we assume that the number of negative training examples far outweighs the number of positive examples, so no explicit mechanism is used to avoid degenerate partitions of the negative examples”.)
MCCLOSKY fails to explicitly teach “receiving, based on input of at least a portion of the content asset to a machine learning model trained to detect instances of the object, output of the machine learning model identifying the portion of the first content asset as an instance of the object;” However, in analogous art that uses unsupervised mined hard examples in multimedia object detection, JIN, does teach this:
([Figure 1] As can be seen in the figure and its description, the detections were made by the model and identified to be either true positives (yellow boxes) or false positives (red boxes).)
Regarding claims 10, MCCLOSKY in view of JIN teaches the limitations of claim 9. Further, claim 10 comprises similar additional limitations to claim 3, and is rejected under the same rationale.
Regarding claim 11, MCCLOSKY and JIN teaches the limitations of claim 9. MCCLOSKY teaches “determining, based on metadata associated with a … content asset, a … relevance score indicative of a likelihood of the … content asset comprising at least one instance of the object”:
([Abstract] “In the following sections, we develop a fully automated algorithm to partition the video corpus according to video metadata, and to learn local weightings which are applied to base classifier scores (used for relevance/likelihood) in order to improve detection performance.”) This paper discusses using metadata to partition video datasets and learn local weightings for classifier scores, which directly relates to determining a relevance score for content assets based on metadata. The method incorporates metadata (e.g., video tags, timestamps) to weight the likelihood of detecting specific events or objects.
MCCLOSKY fails to explicitly teach “…based on the … relevance score, determining that the portion of the … content asset identified by the machine learning model is not a false identification of the object.”:
However, JIN does teach this. ([Section IV. A. Partitioning, paragraph 1] “In order to avoid degenerate partitions (e.g., a threshold above which all training clips are negatively labeled), we generate Nᵗ=20 candidate thresholds linearly spaced between the 25th and 75th percentiles of the metadata values of positive clips.”) The paper emphasizes thresholding on metadata-weighted classifier scores (relevance scores), negatively labelling unlikely events or objects, which aligns with the described limitation. Negatively labelling those events/objects is equivalent to determining that they are not instances of the object. Since only those detections are negatively labelled, others would not be, and are thus not identified as false identifications.
MCCLOSKY fails to explicitly teach “receiving output of a machine learning model identifying a portion of the … content asset as an instance of the object;” However, in analogous art that uses unsupervised mined hard examples in multimedia object detection, JIN, does teach this:
([Figure 1] As can be seen in the figure and its description, the detections were made by the model and identified to be either true positives (yellow boxes) or false positives (red boxes).)
MCCLOSKY fails to explicitly teach performance of the same features on a “second content asset”. However, JIN, does teach this:
([page 7, 4. Experiments.] “We evaluate our method on face and pedestrian detection and perform ablation studies analyzing the effect of the hard examples. For pedestrians, we show results on the Caltech dataset [12], while for face detection, we show results on the WIDER Face [61] dataset. The Caltech Pedestrian Dataset [12] consists of videos taken from a vehicle driving through urban traffic, with about 350k annotated bounding-boxes from 250k video frames. The WIDER dataset consists of 32,203 images having 393,703 labeled faces in challenging situations of scale, pose and occlusion. The evaluation set of WIDER is divided into easy, medium, and hard sets according to the detection scores of object proposals from EdgeBox [67]. From easy to hard, the faces get smaller and more crowded.” Using the easy, medium, and hard sets, shows using the invention on multiple (a first, second, and third) collection of content assets, each with more than two content assets.)
It would be obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to combine the base reference of MCCLOSKY (a multimedia detector that detects objects based on identified areas and metadata) with the teachings of JIN (a multimedia detector that performs its functions on more than a single content asset) because both references address similar objectives of improving performance and accuracy of multimedia detectors.
One of the ordinary skill in the art would be motivated to do so because a multimedia detector that can only read one content asset will not be as effective or as efficient at labelling content as a multimedia detector that can perform its functions on more than one content asset.
Regarding claims 12-14, MCCLOSKY in view of JIN teaches the limitations of claim 9. Further, they comprise similar limitations to claims 5-7 respectively, and are rejected under the same rationale.
Regarding claim 16, MCCLOSKY teaches “A method comprising: determining, for each of a plurality of metadata labels associated with a content asset, a relevance value indicative of a correlation between the metadata label and an object; determining, based on the plurality of relevance values, a relevance score indicative of a likelihood of the content asset comprising at least one instance of an object”:
([Abstract] “In the following sections, we develop a fully automated algorithm to partition the video corpus according to video metadata, and to learn local weightings which are applied to base classifier scores (used for relevance/likelihood) in order to improve detection performance.”) This paper discusses using metadata to partition video datasets and learn local weightings for classifier scores, which directly relates to determining a relevance score for content assets based on metadata. The method incorporates metadata (e.g., video tags, timestamps) to weight the likelihood of detecting specific events or objects.
Further, MCCLOSKY teaches “based on the relevance score, labeling the at least one instance of the object as a false identification of the object.” However, JIN does teach this. ([Section IV. A. Partitioning, paragraph 1] “In order to avoid degenerate partitions (e.g., a threshold above which all training clips are negatively labeled), we generate Nᵗ=20 candidate thresholds linearly spaced between the 25th and 75th percentiles of the metadata values of positive clips.”):
The paper emphasizes thresholding on metadata-weighted classifier scores (relevance scores), negatively labelling unlikely events or objects, which aligns with the described limitation. Negatively labelling those events/objects is equivalent to determining that they are not instances of the object.
Further, MCCLOSKEY teaches “training the machine learning model using at least the labeled at least one instance of the object”:
([Page 4, Section IV. Metadata-Weighted Fusion Algorithm] This section describes the training algorithm used to train the ML model including partitioning and labelling negative samples, which are also used to train the model “Consistent with other detection problems, we assume that the number of negative training examples far outweighs the number of positive examples, so no explicit mechanism is used to avoid degenerate partitions of the negative examples”.)
MCCLOSKY fails to explicitly teach “detecting, based on input of at least a portion of the content asset to a machine learning model trained to detect instances of the object, at least one instance of the object in the content asset;” However, in analogous art that uses unsupervised mined hard examples in multimedia object detection, JIN, does teach this:
([Figure 1] As can be seen in the figure and its description, the detections were made by the model and identified to be either true positives (yellow boxes) or false positives (red boxes).)
Regarding claim 17, it comprises similar limitations to claim 3, and is rejected under the same rationale.
Regarding claim 20, MCCLOSKY and JIN teach the limitations of claim 16. Further, MCCLOSKY teaches “wherein detecting the at least one instance of the object in the content asset comprises using the machine learning model to detect the at least one instance of the object in the content asset.”:
([Section I. Introduction, paragraph 4] “We address score fusion of a small number of base classifiers (we use 5 in our experiments), and automatically learn local models that account for variations in classifier performance within the video corpus. Rather than weighting a classifier's output score based on its global performance over the training data, we learn local models which account for local variations in base classifier performance.” Classifiers representing machine learning models are used to detect and classify objects in content
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over MCCLOSKY in view of JIN as applied to claims above, and further in view of Movie Labs. “Common Metadata ‘md’ namespace” Available at https://movielabs.com/md/md/v2.8/Common_Metadata_v2.8.pdf on December 14, 2019 (hereafter, MOVIELABS)
Regarding claim 15, MCCLOSKY in view of JIN teaches the limitations of claim 13. MCCLOSKY in view of JIN fails to teach “wherein the plurality of labels associated with the first content asset comprise at least one of a theme, a subject, a genre, a setting, a character, a tone, a rating, or a time period associated with the first content asset”
However, analogous art of a website documenting common types of metadata in multimedia content, MOVIELABS, does teach this:
([1.1 Overview of Common Metadata] “Common Metadata includes elements that cover typical definitions of media, particularly movies and television. Common Metadata has two parts: Basic Metadata and Digital Asset Metadata. Basic Metadata includes descriptions such as title and artists. It describes information about the work independent of encoding.”) Basic Metadata is a “plurality of labels associated with a content asset.” and ([Table of Contents, page iii, section 4.3] The section lists various types of basic metadata associated with multimedia content assets such as Character, Period, Place (setting), etc.)
It would be obvious to one of ordinary skill in the art, prior to the date of the claimed invention, to combine the base reference of MCCLOSKY in view of JIN (A multimedia detector that detects various types of content based on metadata) with the teachings of MOVIELABS (A document describing the common types of multimedia metadata) because MOVIELABS teaches the most common types of the data types that the other teaches using.
One of ordinary skill in the art would be motivated to do so because using the most common types of the metadata associated with the content being used will likely result in more accurate detections.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over MCCLOSKY in view of JIN as applied to claims above, and further in view of Russom, Philip. “Data Requirements for Machine Learning” Available at https://tdwi.org/articles/2018/09/14/adv-all-data-requirements-for-machine-learning.aspx#:~:text=Infrastructure%20for%20training%20data%20for,Spark%2C%20and%20cloud%20storage). on September 14, 2018 (hereafter, RUSSOM)
Regarding claim 18, MCCLOSKY in view of JIN teaches the limitations of claim 16. MCCLOSKY in view of JIN fails to explicitly teach “sending the labeled instance of the object to a computing device for storage, by the computing device, in a database comprising training data for a machine learning model.” However, analogous art about the typical requirements of machine learning models, RUSSOM, does teach this:
([#2: Large, diverse infrastructure for data management] “Infrastructure for training data for machine learning typically involves multiple data platforms, tools, and processing engines, ranging from traditional (relational and columnar databases) to modern (Hadoop, Spark, and cloud storage). Multiple technologies are required to cope with training data's extreme size, multiple data structures, and (in some cases) multiple latencies. Tools for machine learning are obviously important, but data management infrastructure is just as important.”)
It would be obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to combine the teachings of MCCLOSKY in view of JIN(The multimedia detector that labels false positives) and RUSSOM (A page detailing the typical requirements of data storage for training data) because it would be an inherent requirement in order to perform the operations of MCCLOSKY in view of JIN.
One of ordinary skill in the art would be motivated to do so because it is standard practice for all forms of machine learning models that their training data is stored in some type of memory (computing device) in order to be accessed by the model. Further, storing machine learning model training data in memory facilitates retrieval to redetermine it on demand.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over MCCLOSKY in view of JIN as applied to claims above, and further in view of Cheng, Hui, et al. “Multimedia Event Detection and Recounting” Available at https://www-nlpir.nist.gov/projects/tvpubs/tv12.papers/aurora.pdf in 2012 (hereafter, CHENG)
Regarding claim 19, MCCLOSKY and JIN teach the limitations of claim 16. MCCLOSKY in view of JIN fails to teach “wherein determining, based on the plurality of relevance values, the relevance score indicative of the likelihood of the content asset comprising at least one instance of an object comprises: identifying a relevance value from the plurality of relevance values having the greatest value; and assigning the relevance score a value equal to the relevance value.” However, analogous art of a paper about multimedia event detection, CHENG, does teach this:
([2.2.2 Concept Based Event Modeling, paragraph 3] “Given the raw detection scores of concepts over the full video, the event depicted in the clip can be represented using a number of features derived from Cij. One option is to select the maximum detection score over all sliding windows as the detection confidence of concept detector ꝕi. As a result, we are able to obtain a K-dimensional vector Cmax to represent a video. Meanwhile, we have embedded a video into the concept space defined above. What is more, based on the K-dimensional semantic space, we also explore the following four representations: MAX pooling: for each concept detector, only the maximum detecting score over all sliding windows is pooled to show the probability of concept given a video.”)
It would have been obvious to one of ordinary skill in the art at the time of the invention to combine the base reference of MCCLOSKY in view of JIN (A multimedia detector that detects various types of content) with the teachings of CHENG (the concept of max-pooling applied to multimedia detection) because both references address similar objectives of detecting various types of content within multimedia. Combining these teachings would be a logical step in achieving a multimedia detector that uses the highest relevance score of a plurality to indicate the level of relevance toward a specific type of object.
One of ordinary skill in the art would be motivated to combine these references because the combination aligns with well-established design principles in multimedia detection, such as accuracy. Further, CHENG’s disclosure of using the highest relevance score would provide a predictable solution to enhance the configuration disclosed in MCCLOSKY in view of JIN, yielding the claimed arrangement, where the overall relevance score of detections is equal to the highest score among a plurality, which will result in a true positive detection not being discounted due to a number of false detections around it decreasing the “average” of the relevance scores.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW LEE LEWIS whose telephone number is (571)272-1906. The examiner can normally be reached Monday: 12:00PM - 4:00PM and Tuesday - Friday: 12:00PM - 9PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tamara Kyle can be reached at (571)272-4241. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Matthew Lee Lewis/Examiner, Art Unit 2144
/TAMARA T KYLE/Supervisory Patent Examiner, Art Unit 2144