DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in
37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible
for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has
been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37
CFR 1.114. Applicant's submission filed on 02/26/2026 has been entered.
Examiner Remarks
References are defined in this application as:
Nabi et al (US 11816188 B2), referred to as Nabi in this document,
Foster et al (“Improving Transformation Invariance In Contrastive Representation Learning”), referred to as Foster in this document,
Yi et al (“Human action recognition based on action relevance weighted encoding”), referred to as Yi in this document,
Singh et al (US 11308353 B2), referred to as Singh in this document.
Claim 16 is not rejected under 101 as a result of being seen as indicating how the elements in claim 1 improve a machine learning model, such as by utilizing the contrastive loss to update the weights of the model, thus providing indication on how the application of the contrastive loss results in the claimed compression of feature space noted in claim 1.
Aspects of paragraph 38 or paragraph 11 of the specification are still noted to recite a particular element of the claimed invention, but the claims as currently recited also appear to note elements not obvious according to prior art as described in the allowable subject matter section.
The details around the current claimed limitations are not clear as a result of the numerous 112 rejections and the questions raised as to how the interactions between elements work as a result of the confusion from the 112 rejections. As noted in the 112 rejection section, some of the 112 rejections pose issues that contrast limitations in the claims or what is being referred to by specific elements. As a result many of the 103 rejections were done with an interpretation akin to the previous claim set or simply rejected as the limitation described (as many limitations appear disconnected thus unable to properly determine what connection to other elements needed to be taught), as the current claim set could not be properly determined how the limitations were narrowed or changed as a result of the amendments 112 issues.
Should the 112 rejections be addressed (and no other rejections or objections on the claims are standing), the current application stands to potentially be narrowed to a point requiring further search and consideration to check if allowable subject matter is recited. This is as a result of a potential interpretation of the claims. Such as should the pseudo labellings within claim 1 be tied to using the extracted features rather than just using a disconnected from the rest of the claims classification probability (as figure 2 of the invention indicates the features are used for the labelling but the claims do not have a limitation for such), then the second pseudo labelling recited in the claims could in this hypothetical be based off the updated/weighted extracted features resulting from the comparison/transform similarity to the background (assumed to potentially be the non-informative background). A requirement of a second pseudo labelling based off a first pseudo labelling (as a result of weighted features from the first pseudo labelling) are elements not obvious under the currently recited art. However the current claims are recited with too many 112 rejections to know if the above hypothetical is a possible interpretation, thus the 112 rejections should be fixed to help ensure aspects of the current invention are trying to be directed to non-obvious subject matter.
Response to Arguments
Remarks page 7-9, Applicant contends:
Amended application satisfies 101 requirements.
Response:
In regards to elements added by the amendments, applicant’s arguments with respect to claim(s) 1 have been considered but are moot because the new ground of rejection contain elements that have not been previously examined or does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. While the previous abstract idea limitation was removed, the new limitations have to be examined for abstract ideas. The amendments do provide additional details regarding the invention and the additional details are noted to be appreciated for progressing prosecution of the case. The premise of improving a machine learning model to satisfy the requirement for 101 is noted as being considered in the examiner remarks.
Remarks page 9, Applicant contends:
Amended application overcomes the 103 rejections by providing more details on the steps of the method.
Response:
Applicant’s arguments with respect to claim(s) 1 have been considered but are moot because the new ground of rejection contain elements that have not been previously examined or does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 112
Regarding 112(b):
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-5, 7-11, 14, and 16rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
In regard to Claim 1:
Claim 1 rejected as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 1 recites “pseudo-labelling each video segment as informative background or foreground based on a classification probability for that video segment”. The limitation recited creates a conflict with the other elements of the claims. Claim 1 notes using elements such as contrastive loss which are given further detail in claims 8 and 9. However, aspects related to those limitations and claims are not possible, as there is no element to perform contrastive loss. When claim 1 recites “pseudo-labelling each video segment as informative background or foreground” the claim is noting each segment is either informative background or foreground. There is no non-informative background or anything considered not one of those two labels, as the limitation recited labels only as informative background or foreground.
The issue noted above is noted to be likely unintentional, as claim 7 goes into details involving the labeling and utilizes thresholds to determine what gets labeled. The utilization of some form of threshold like claim 7 or some limit could prevent the pseudo labeling as informative background or foreground from relabeling everything to only those two things. Aspects about ensuring there is non-informative background is discussed later in another 112 rejection.
Claim 1 recites the limitation "a classification probability" in “pseudo-labelling each video segment as informative background or foreground based on a classification probability for that video segment.” There is insufficient antecedent basis for this limitation in the claim. Claim 1 recites two iterations of “a classification probability”. The first is in “pseudo-labelling each video segment as background or foreground based on a classification probability for that video segment” and the second is in “pseudo-labelling each video segment as informative background or foreground based on a classification probability for that video segment”. One of ordinary skill in the art cannot determine whether the classification probabilities mentioned here are intended to be the same classification probability or two separate classification probabilities. Claim 7 notes utilizing a maximum classification probability as a confidence or confidence score for the determinations and claim 1 also notes applying the self-weighting mechanism to make a combination of features weighted. Providing an indication in claim 1 whether any of these elements pertain to the classification probability, such as whether the self-weighting mechanism alters the classification probability or makes a new one that is utilized for the pseudo-labelling of video segments as informative background or foreground. Any amendments to help provide the improved indication should likely update any references in other claims to align, such as updating claim 7 to align with the indications provided in claim 1.
Claim 1 recites the limitation "the weighted feature vector" in “is compressed and separated from feature spaces of the informative background and the foreground based on the weighted feature vector of the video segment”. There is insufficient antecedent basis for this limitation in the claim. There is no previous recitation of “a weighted feature vector”. Weighting of features is noted in “applying the self-weighting mechanism to each video segment such that the feature vector for that video segment represents a combination of features weighted by the self-weighting mechanism” but no indication of whether the result of that limitation or another one is what creates the “weighted feature vector”.
In regard to claim 7:
Claim 7 recites the limitations “pseudo-labelling a video segment as non-informative background when the maximum classification probability for that video segment falls below a threshold” in “and pseudo-labelling video segments having the highest confidence scores as foreground or informative background”. There is insufficient antecedent basis for this limitation in the claim. Claim 7 is reintroducing pseudo labelling of the video segments which is already recited in claim 1 (such as “pseudo-labelling each video segment as informative background or foreground based on a classification probability for that video segment”). One of ordinary skill in the art cannot determine which labeling to utilize, the labels as described in claim 1 or the labels as described in claim 7.
Claim 7 recites the limitation "video features" in “based on the video features extracted from the video segments in the untrimmed video clip”. There is insufficient antecedent basis for this limitation in the claim. Video features are no longer recited in claim 1 after the current amendments, thus one cannot tell what claim 7 is referring to by “video features”.
The term “highest” in claim 7 is a relative term which renders the claim indefinite. The term “highest” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The term “highest” does not convey in what regard would create the highest. Is the highest whatever confidence score noted in claim 7 is the largest out of all of the scores? Are the highest scores what scores satisfied or exceeded a threshold? Without an indication on the limit of “highest” one of ordinary skill could not determine what elements would fail to meet the criteria. Providing details as to what “highest” means or the limits of what would be the “highest” would resolve the relative term issue.
In regard to claim 8:
Claim 8 recites the limitation " non-informative background". There is insufficient antecedent basis for this limitation in the claim. Claim 1 only recites “background or foreground” or “informative background or foreground”. Previous claim 1 mentioned “non-informative background” in regards to contrastive loss, but the claim that appeared to identify or label a video segment as non-informative background is claim 7 (“pseudo-labelling a video segment as non-informative background”).
In regards to dependent claims:
Claims depending upon a claim rejected under 112(b) are rejected for being dependent upon a rejected claim.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-5, 7-11, and 14 are rejected under 35 U.S.C. 101 because the claimed invention is directed towards an abstract idea without significantly more.
In regards to Claim 1:
Step 1: Is the claim directed towards a process, machine, manufacture, or composition of matter?
Yes, it is directed towards a method, so a process.
Step 2A Prong 1: Does the claim recite a law of nature, a natural phenomenon, or an abstract idea?
Yes, the claim does recite a(n) abstract idea.
Claim 1 recites the following abstract ideas:
pseudo-labelling each video segment as background or foreground based on a classification probability for that video segment;
This limitation is directed towards the abstract idea of a mental process, or a concept performed in the human mind, including observation, evaluation, judgement or opinion (see MPEP 2106.04(a)(2) subsection 3). Here it is seen as evaluation and judgement. Labeling based off a value is an aspect performable in the human mind.
pseudo-labelling each video segment as informative background or foreground based on a classification probability for that video segment;
This limitation is directed towards the abstract idea of a mental process, or a concept performed in the human mind, including observation, evaluation, judgement or opinion (see MPEP 2106.04(a)(2) subsection 3). Here it is seen as evaluation and judgement. Labeling based off a value is an aspect performable in the human mind.
Step 2A Prong 2: Does the claim recite additional elements that integrate the exception into a practical application of the exception?
No, the application does not recite any additional elements that would integrate the abstract idea into a practical application.
Claim 1 recites the following additional elements:
initially training the base class model by adjusting weights of the base class model to recognize a plurality of base classes using a training dataset consisting of a plurality of untrimmed video clips containing labelled actions in the plurality of base classes
At a high level of generality, this activity of training a base class model is an “apply it” use (see MPEP 2106.05(f)).
splitting the untrimmed video clips into a plurality of video segments each containing non-informative background, informative background and/or foreground
This is a continuation of the training of the base class model.
extracting a feature vector from each video segment using the base class model;
At a high level of generality, this is an activity of using an element as an “apply it” use (see MPEP 2106.05(f)).
using a transform similarity between each segment and the pseudo-labelled background segments to create a self-weighting mechanism;
At a high level of generality, this is an activity of using an element as an “apply it” use (see MPEP 2106.05(f)).
applying the self-weighting mechanism to each video segment such that the feature vector for that video segment represents a combination of features weighted by the self-weighting mechanism;
At a high level of generality, this is an activity of using an element as an “apply it” use (see MPEP 2106.05(f)).
and applying a self-supervised contrastive loss to the base model, the self- supervised contrastive loss being between non-informative background and informative background + foreground such that a feature space of the non-informative background is compressed and separated from feature spaces of the informative background and the foreground based on the weighted feature vector of the video segment
At a high level of generality, this is an activity of using an element as an “apply it” use (see MPEP 2106.05(f)).
further training the base class model by updating the weights of the base class model to classify and localize one or more novel classes using a training data set comprising a few trimmed video segments of actions comprising the novel classes
At a high level of generality, this activity of training a base class model is an “apply it” use (see MPEP 2106.05(f)).
extracting feature vectors, from the few trimmed video segments containing the novel classes
At a high level of generality, this activity of extracting features as an “apply it” use (see MPEP 2106.05(f)).
performing a nearest neighbor match to the feature vectors extracted from the untrimmed video clips in the base classes
At a high level of generality, this activity of performing a nearest neighbor match as an “apply it” use (see MPEP 2106.05(f)).
Step 2B: Does the claim as a whole amount to significantly more than the judicial exception?
No, the claim as a whole does not amount to significantly more than the judicial exception. All elements of the claim, viewed individually or wholistically, do not provide an inventive concept or otherwise significantly more than the abstract idea itself.
Claim 1 recites the following additional elements:
initially training the base class model by adjusting weights of the base class model to recognize a plurality of base classes using a training dataset consisting of a plurality of untrimmed video clips containing labelled actions in the plurality of base classes
At a high level of generality, this activity of training a base class model is an “apply it” use (see MPEP 2106.05(f)). At said high level of generality, a generic recitation of “train a base class model” using a training dataset or by generically adjusting weights does not incorporate the abstract idea into a practical invention and is seen as a variation of the phrase “apply it”.
splitting the untrimmed video clips into a plurality of video segments each containing non-informative background, informative background and/or foreground
This is a continuation of the training of the base class model. Noting the training data is split into segments or giving a description of what the training data contains does not incorporate the abstract idea into a practical invention is seen as a continuation of the training apply it.
extracting a feature vector from each video segment using the base class model;
At a high level of generality, this is an activity of using an element as an “apply it” use (see MPEP 2106.05(f)). At said high level of generality, a generic recitation of “apply” or equivalent does not incorporate the abstract idea into a practical invention and is seen as a variation of the phrase “apply it”. Extracting features using the base class model is applying the base class model.
using a transform similarity between each segment and the pseudo-labelled background segments to create a self-weighting mechanism;
At a high level of generality, this is an activity of using an element as an “apply it” use (see MPEP 2106.05(f)). At said high level of generality, a generic recitation of “apply” or equivalent does not incorporate the abstract idea into a practical invention and is seen as a variation of the phrase “apply it”. A self-weighting mechanism being made from a transform similarity does not integrate the abstract ideas.
applying the self-weighting mechanism to each video segment such that the feature vector for that video segment represents a combination of features weighted by the self-weighting mechanism;
At a high level of generality, this is an activity of using an element as an “apply it” use (see MPEP 2106.05(f)). At said high level of generality, a generic recitation of “apply” or equivalent does not incorporate the abstract idea into a practical invention and is seen as a variation of the phrase “apply it”. Applying the self-weighting mechanism to have weighted features does not integrate the abstract ideas into a practical application.
and applying a self-supervised contrastive loss to the base model, the self- supervised contrastive loss being between non-informative background and informative background + foreground such that a feature space of the non-informative background is compressed and separated from feature spaces of the informative background and the foreground based on the weighted feature vector of the video segment
At a high level of generality, this is an activity of using an element as an “apply it” use (see MPEP 2106.05(f)). At said high level of generality, a generic recitation of “apply” or equivalent does not incorporate the abstract idea into a practical invention and is seen as a variation of the phrase “apply it”. Generically applying a contrastive loss to claim a result does not integrate the abstract idea into a practical application as how a model is improved is not apparent as a result of the outcome of the application being claimed but not the steps to achieve the outcome.
further training the base class model by updating the weights of the base class model to classify and localize one or more novel classes using a training data set comprising a few trimmed video segments of actions comprising the novel classes
At a high level of generality, this activity of training a base class model is an “apply it” use (see MPEP 2106.05(f)). At said high level of generality, a generic recitation of “train the base class model” using a training dataset or generically adjusting weights does not incorporate the abstract idea into a practical invention and is seen as a variation of the phrase “apply it”.
extracting feature vectors, from the few trimmed video segments containing the novel classes
At a high level of generality, this activity of extracting features as an “apply it” use (see MPEP 2106.05(f)). At said high level of generality, a generic recitation of extracting features does not incorporate the abstract idea into a practical invention and is seen as a variation of the phrase “apply it”.
performing a nearest neighbor match to the feature vectors extracted from the untrimmed video clips in the base classes
At a high level of generality, this activity of performing a nearest neighbor match as an “apply it” use (see MPEP 2106.05(f)). At said high level of generality, a generic recitation of performing a nearest neighbor match does not incorporate the abstract idea into a practical invention and is seen as a variation of the phrase “apply it”.
In regards to Claim 2:
Step 2A Prong 1: Does the claim recite a law of nature, a natural phenomenon, or an abstract idea?
Yes, the claim does recite a(n) abstract idea.
Claim 2 recites the following abstract ideas:
wherein the base class model is able to classify and localize the action depicted in the novel class
This limitation is directed towards the abstract idea of a mental process, or a concept performed in the human mind, including observation, evaluation, judgement or opinion (see MPEP 2106.04(a)(2) subsection 3). Here it is seen as observation and evaluation. The base class model is seen as a computer performing the abstract idea of classification and localizing of an input.
Step 2A Prong 2: Does the claim recite additional elements that integrate the exception into a practical application of the exception?
No, the application does not recite any additional elements that would integrate the abstract idea into a practical application.
Claim 2 recites the following additional elements:
exposing the base class model to untrimmed testing video segments comprising action in the novel class
This limitation is directed towards the insignificant extra solution activity of mere data gathering (see MPEP § 2106.05(g)).
Step 2B: Does the claim as a whole amount to significantly more than the judicial exception?
No, the claim as a whole does not amount to significantly more than the judicial exception. All elements of the claim, viewed individually or wholistically, do not provide an inventive concept or otherwise significantly more than the abstract idea itself.
Claim 2 recites the following additional elements:
exposing the base class model to untrimmed testing video segments comprising action in the novel class
This limitation is directed towards the insignificant extra solution activity of mere data gathering (see MPEP § 2106.05(g)). This is a well understood, routine, conventional activity of transmitting data (see MPEP 2106.05(d) example i in computer functions).
In regards to Claim 3:
Step 2A Prong 2: Does the claim recite additional elements that integrate the exception into a practical application of the exception?
No, the application does not recite any additional elements that would integrate the abstract idea into a practical application.
Claim 3 recites the following additional elements:
video segments containing foreground are video segments containing an action which the base class model is trained to recognize
At a high level of generality, this is an activity of using training as an “apply it” use (see MPEP 2106.05(f)).
Step 2B: Does the claim as a whole amount to significantly more than the judicial exception?
No, the claim as a whole does not amount to significantly more than the judicial exception. All elements of the claim, viewed individually or wholistically, do not provide an inventive concept or otherwise significantly more than the abstract idea itself.
Claim 3 recites the following additional elements:
video segments containing foreground are video segments containing an action which the base class model is trained to recognize
At a high level of generality, this is an activity of using training as an “apply it” use (see MPEP 2106.05(f)). At said high level of generality, a generic recitation of being trained to recognize something does not incorporate the abstract idea into a practical invention and is seen as a variation of the phrase “apply it”.
In regards to Claim 4:
Step 2A Prong 2: Does the claim recite additional elements that integrate the exception into a practical application of the exception?
No, the application does not recite any additional elements that would integrate the abstract idea into a practical application.
Claim 4 recites the following additional elements:
wherein video segments containing informative background are video segments containing informative objects or actions which the base class model is not trained to recognize
At a high level of generality, this is an activity of using training as an “apply it” use (see MPEP 2106.05(f)).
Step 2B: Does the claim as a whole amount to significantly more than the judicial exception?
No, the claim as a whole does not amount to significantly more than the judicial exception. All elements of the claim, viewed individually or wholistically, do not provide an inventive concept or otherwise significantly more than the abstract idea itself.
Claim 4 recites the following additional elements:
wherein video segments containing informative background are video segments containing informative objects or actions which the base class model is not trained to recognize
At a high level of generality, this is an activity of using training as an “apply it” use (see MPEP 2106.05(f)). At said high level of generality, a generic recitation of training to not recognize something does not incorporate the abstract idea into a practical invention and is seen as a variation of the phrase “apply it”.
In regards to Claim 5:
Step 1: Is the claim directed towards a process, machine, manufacture, or composition of matter?
Yes, it is directed towards a method, so a process.
Step 2A Prong 1: Does the claim recite a law of nature, a natural phenomenon, or an abstract idea?
Yes, the claim does recite a(n) abstract idea.
Claim 5 recites the following abstract ideas:
wherein video segments containing non-informative background are video clips not containing informative objects or actions
This limitation is directed towards a continuation of the abstract idea of a mental process, or a concept performed in the human mind, including observation, evaluation, judgement or opinion (see MPEP 2106.04(a)(2) subsection 3). Here it is seen as a continuation of the abstract ideas in claim 1.
In regards to Claim 7:
Step 2A Prong 1: Does the claim recite a law of nature, a natural phenomenon, or an abstract idea?
Yes, the claim does recite a(n) abstract idea.
Claim 7 recites the following abstract ideas:
pseudo-labelling a video segment as non-informative background when the maximum classification probability for that video segment falls below a threshold
This limitation is directed towards the abstract idea of a mental process, or a concept performed in the human mind, including observation, evaluation, judgement or opinion (see MPEP 2106.04(a)(2) subsection 3). Here it is seen as evaluation and judgement.
and measuring a confidence score as the maximum value of each video segment's classification probabilities
This limitation is directed towards the abstract idea of a mental process, or a concept performed in the human mind, including observation, evaluation, judgement or opinion (see MPEP 2106.04(a)(2) subsection 3). Here it is seen as evaluation.
and pseudo-labelling video segments having the highest confidence scores as foreground or informative background
This limitation is directed towards the abstract idea of a mental process, or a concept performed in the human mind, including observation, evaluation, judgement or opinion (see MPEP 2106.04(a)(2) subsection 3). Here it is seen as evaluation.
Step 2A Prong 2: Does the claim recite additional elements that integrate the exception into a practical application of the exception?
No, the application does not recite any additional elements that would integrate the abstract idea into a practical application.
Claim 7 recites the following additional elements:
determining, based on the video features extracted from the video segments in the untrimmed video clip, a maximum classification probability of each video segment, the maximum classification probability indicating a confidence for the given segment belongs to one of the base classes
At a high level of generality, this is an activity of using determining a maximum classification probability as an “apply it” use (see MPEP 2106.05(f)).
Step 2B: Does the claim as a whole amount to significantly more than the judicial exception?
No, the claim as a whole does not amount to significantly more than the judicial exception. All elements of the claim, viewed individually or wholistically, do not provide an inventive concept or otherwise significantly more than the abstract idea itself.
Claim 7 recites the following additional elements:
determining, based on the video features extracted from the video segments in the untrimmed video clip, a maximum classification probability of each video segment, the maximum classification probability indicating a confidence for the given segment belongs to one of the base classes
At a high level of generality, this is an activity of using determining a maximum classification probability as an “apply it” use (see MPEP 2106.05(f)). At said high level of generality, a generic recitation of determining a maximum classification probability does not incorporate the abstract idea into a practical invention and is seen as a variation of the phrase “apply it”. The addition of a description of what maximum classification probability indicates does not integrate the invention into a practical application.
In regards to Claim 8:
Step 2A Prong 1: Does the claim recite a law of nature, a natural phenomenon, or an abstract idea?
Yes, the claim does recite a(n) abstract idea.
Claim 8 recites the following abstract ideas:
Defining, as a negative pair, a feature extracted from non-informative background video segments and a feature extracted from both informative background and foreground segments
This limitation is directed towards a continuation of the abstract idea of a mental process, or a concept performed in the human mind, including observation, evaluation, judgement or opinion (see MPEP 2106.04(a)(2) subsection 3). Here it is seen as a continuation of the abstract ideas in claims 7. The defining of a negative pair does not integrate the invention into a practical application.
In regards to Claim 9:
Step 2A Prong 1: Does the claim recite a law of nature, a natural phenomenon, or an abstract idea?
Yes, the claim does recite a(n) abstract idea.
Claim 9 recites the following abstract ideas:
enlarging a distance in the base class model between features in the negative pair by maximizing a contrastive loss
This limitation is directed towards the abstract idea of a mathematical concept (see MPEP 2106.04(a)(2) subsection 1). Contrastive loss is seen as a mathematical concept.
In regards to Claim 10:
Step 2A Prong 1: Does the claim recite a law of nature, a natural phenomenon, or an abstract idea?
Yes, the claim does recite a(n) abstract idea.
Claim 10 recites the following abstract ideas:
defining, as a positive pair, features extracted from non-informative background video segments
This limitation is directed towards a continuation of the abstract idea of a mental process, or a concept performed in the human mind, including observation, evaluation, judgement or opinion (see MPEP 2106.04(a)(2) subsection 3). Here it is seen as a continuation of the abstract ideas in claims 9. The defining of a positive pair does not integrate the invention into a practical application.
In regards to Claim 11:
Step 2A Prong 1: Does the claim recite a law of nature, a natural phenomenon, or an abstract idea?
Yes, the claim does recite a(n) abstract idea.
Claim 11 recites the following abstract ideas:
reducing a distance in the base class model between features in the positive pair by minimizing the contrastive loss
This limitation is directed towards the abstract idea of a mathematical concept (see MPEP 2106.04(a)(2) subsection 1). Contrastive loss is seen as a mathematical concept.
In regards to Claim 14:
Step 1: Is the claim directed towards a process, machine, manufacture, or composition of matter?
No, the claim is directed towards software per se.
Claim 14 recites the following software per se:
a processor; software, executing on the processor, the software performing the steps of the method of claim 1
The specification does not define a processor as hardware. As a result, this claim can be seen as only being a system of software.
Step 2A Prong 2: Does the claim recite additional elements that integrate the exception into a practical application of the exception?
No, the application does not recite any additional elements that would integrate the abstract idea into a practical application.
Claim 14 recites the following additional elements:
a processor; software, executing on the processor, the software performing the steps of the method of claim 1
At a high level of generality, this is an activity of using a processor and software as an “apply it” use (see MPEP 2106.05(f)).
Step 2B: Does the claim as a whole amount to significantly more than the judicial exception?
No, the claim as a whole does not amount to significantly more than the judicial exception. All elements of the claim, viewed individually or wholistically, do not provide an inventive concept or otherwise significantly more than the abstract idea itself.
Claim 14 recites the following additional elements:
a processor; software, executing on the processor, the software performing the steps of the method of claim 1
At a high level of generality, this is an activity of using a processor and software as an “apply it” use (see MPEP 2106.05(f)). At said high level of generality, a processor and software appears to be an implementation of the abstract idea on a computer, so merely using a computer as a tool to perform the abstract idea.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Nabi et al (US 11816188 B2), referred to as Nabi in this document
Foster et al (“Improving Transformation Invariance In Contrastive Representation Learning”), referred to as Foster in this document,
Yi et al (“Human action recognition based on action relevance weighted encoding”), referred to as Yi in this document,
Singh et al (US 11308353 B2), referred to as Singh in this document.
Claims 1-5, 7-11, and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nabi, in further view of Foster, and in further view of Yi.
In regards to Claim 1:
Nabi teaches:
initially training the base class model by adjusting weights of the base class model to recognize a plurality of base classes using a training dataset consisting of a plurality of untrimmed video clips containing labelled actions in the plurality of base classes, the initial training comprising:
[Nabi Column1 Line 33]: “Systems, methods, and articles of manufacture, including computer program products, are provided for weakly supervised, one-shot image segmentation. In some example embodiments, there is provided a system that includes at least one processor and at least one memory. The at least one memory may include program code that provides operations when executed by the at least one processor. The operations may include: training, in a supervised manner, a machine learning model to learn a plurality of base class prototypes corresponding to a plurality of base objects [initially training the base class model by adjusting weights of the base class model to recognize a plurality of base classes using a training dataset consisting of a plurality of untrimmed video clips containing labelled actions in the plurality of base classes, the initial training comprising where the labelled aspect is supported by the training for the base class prototypes noted to be supervised], each of the plurality of base class prototypes corresponding to a segmentation of a class of one or more similar base objects; training, based at least on a support image depicting a novel object, the machine learning model to learn a novel class prototype corresponding to the novel object, the machine learning model being trained to learn the novel class prototype based at least on one of the plurality of base class prototypes identified as being similar to the support image, and the novel object being a different object than the plurality of base objects; and applying the trained machine learning model to segment a query image.”
and further training the base class model by adjusting weights of the base class model to classify and localize one or more novel classes using a training data set comprising a few trimmed video segments of actions comprising the novel classes,
the further training comprising: extracting feature vectors, from the few trimmed video segments containing the novel classes
[Nabi Column1 Line 33]: “Systems, methods, and articles of manufacture, including computer program products, are provided for weakly supervised, one-shot image segmentation. In some example embodiments, there is provided a system that includes at least one processor and at least one memory. The at least one memory may include program code that provides operations when executed by the at least one processor. The operations may include: training, in a supervised manner, a machine learning model to learn a plurality of base class prototypes corresponding to a plurality of base objects, each of the plurality of base class prototypes corresponding to a segmentation of a class of one or more similar base objects; training, based at least on a support image depicting a novel object [and further training the base class model by adjusting weights of the base class model to classify and localize one or more novel classes using a training data set comprising a few trimmed video segments of actions comprising the novel classes][the further training comprising: extracting feature vectors, from the few trimmed video segments containing the novel classes where the input trimmed video segments are taught by Yi teaching the training data being video segments], the machine learning model to learn a novel class prototype corresponding to the novel object, the machine learning model being trained to learn the novel class prototype based at least on one of the plurality of base class prototypes identified as being similar to the support image, and the novel object being a different object than the plurality of base objects; and applying the trained machine learning model to segment a query image.”
and performing a nearest neighbor match to the feature vectors extracted from the untrimmed video clips in the base classes
[Nabi Column 1 Line 56]: “In some variations, one or more features disclosed herein including the following features can optionally be included in any feasible combination. A nearest neighbor search [and performing a nearest neighbor match to the feature vectors extracted from the untrimmed video clips in the base classes] may be performed in order to identify the one of the plurality of base class prototypes similar to the support image”
As a note to help prevent confusion later, the aspect of informative background is seen as classes that are not the base classes/currently trained to recognize classes being detected by the model (thus are novel classes). This interpretation is supported in Nabi by [Nabi Column1 Line 33]: “the machine learning model being trained to learn the novel class prototype based at least on one of the plurality of base class prototypes identified as being similar to the support image, and the novel object being a different object than the plurality of base objects”. The support for the interpretation from the current disclosure comes from [Current Specification 0008]: “(2) overlapped base class background and novel class foreground: BG segments in base classes could be similar to FG segments in novel classes with similar appearances and motions. That is, unrecognized action (i.e., action not falling into one of the base classes) may be the action depicted in a novel class.” and [current claim 4]: “wherein video segments containing informative background are video segments containing informative objects or actions which the base class model is not trained to recognize”.
Nabi teaches the ideas of foreground and background (equivalent to non-informative background) in [Nabi Column 7 Line 19]: “When the machine learning model 115 is trained in a fully supervised manner, the machine learning model 115 may encode a support image I.sub.support from the support set D.sub.support to produce a feature map F.sub.s∈customcharacter.sup.h.sup.d.sup.×w.sup.s.sup.×d. The binarized segmentation masks for foreground-background (FG-BG) {M.sub.S.sub.bg, M.sub.S.sub.ƒg} of the support image I.sub.support may be used to encode the class information in the support feature map to produce a foreground feature map F.sub.S.sub.ƒg and a background feature map F.sub.S.sub.bg. These feature maps may be pooled spatially to form a pooled feature map F.sub.pooled∈custom character.sup.1×1×d for each of the foreground-background (FG-BG) segmentation masks.” Informative background is grouped with foreground in the current interpretation, as both contain elements to be classified, thus not elements to be ignored like the background.
Nabi does not explicitly teach:
using a training dataset consisting of a plurality of untrimmed video clips
splitting the untrimmed video clips in the training dataset into a plurality of video segments each containing non-informative background, informative background and/or foreground;
extracting a feature vector from each video segment using the base class model;
pseudo-labelling each video segment as background or foreground based on a classification probability for that video segment;
using a transform similarity between each segment and the pseudo-labelled background segments to create a self-weighting mechanism
applying the self-weighting mechanism to each video segment such that the feature vector for that video segment represents a combination of features weighted by the self-weighting mechanism
pseudo-labelling each video segment as informative background or foreground based on a classification probability for that video segment;
and applying a self-supervised contrastive loss to the base model, the self- supervised contrastive loss being between non-informative background and informative background + foreground such that a feature space of the non-informative background is compressed and separated from feature spaces of the informative background and the foreground based on the weighted feature vector of the video segment
and localize one or more novel classes using a training data set comprising a few trimmed video segments of actions
Foster teaches:
and applying a self-supervised contrastive loss to the base model, the self- supervised contrastive loss being between non-informative background and informative background + foreground such that a feature space of the non-informative background is compressed and separated from feature spaces of the informative background and the foreground based on the weighted feature vector of the video segment
[Foster et al 2. Probabilistic Formulation of Contrastive Learning page 3]: “If the encoder is to discard irrelevant information, we would expect different encodings of x formed with different transformations to be close in representation space. Altering the transformation should not lead to big changes in the representations of the same input. In other words, the distribution p(z|x) should place most probability mass in a small region. However, this does not provide a sufficient training signal for the encoder fθ as it fails to penalize trivial solutions in which all inputs are mapped to the same point. To preserve meaningful information about the input x whilst discarding purely cosmetic features, we should require p(z|x) to be focused on a single point whilst simultaneously requiring the representations of different inputs not to be close. That is, the marginal
PNG
media_image1.png
43
293
media_image1.png
Greyscale
should distribute probability mass over representation space. This intuition is directly reflected in contrastive learning. Most state-of-the-art contrastive learning methods utilize the InfoNCE objective (van den Oord et al., 2018), or close variants of it (Chen et al., 2020a). InfoNCE uses a batch x1, ..., xK of inputs, from which we form pairs of representations (z1, z’1), ...,(zK, z’K) by applying two random transformations to each input followed by the encoder fθ. In probabilistic language
PNG
media_image2.png
355
1302
media_image2.png
Greyscale
Written in this way, we see that the loss will be minimized when sφ(zi , z’i ) is large, but sφ(zi , z’j ) is small for i != j. In other words, InfoNCE makes the two samples zi , z’i of p(z|x = xi) similar [and applying a self-supervised contrastive loss to the base model, the self- supervised contrastive loss being between non-informative background and informative background + foreground such that a feature space of the non-informative background is compressed and separated from feature spaces of the informative background and the foreground based on the weighted feature vector of the video segment], whilst making samples zi, z’j of p(z) dissimilar. This can also be understood through the lens of mutual information, for more details see Appendix A.”
As noted in the quote, part of the reason for such a loss equation being used is to have “representations of different inputs not to be close”. This means that making a pair similar, or two samples, is bringing the two samples closer together or compressing. This idea is supported by phrasing in [Current Application 0011]: “For NBG, the model should compress its feature space and pull it away from FG…”
The informative background, foreground, and non-informative background portions are taught by Nabi, wherein Foster is used here to add contrastive learning.
using a transform similarity between each segment and the pseudo-labelled background segments to create a self-weighting mechanism
applying the self-weighting mechanism to each video segment such that the feature vector for that video segment represents a combination of features weighted by the self-weighting mechanism
[Foster Introduction page 1]: “The objective functions used for contrastive learning encourage representations to remain similar under transformation, whilst simultaneously requiring different inputs to be well spread out in representation space (Wang & Isola, 2020)… However, representations are compared using a similarity function [using a transform similarity between each segment and the pseudo-labelled background segments to create a self-weighting mechanism][applying the self-weighting mechanism to each video segment such that the feature vector for that video segment represents a combination of features weighted by the self-weighting mechanism as using the transform similarity to make a self-weighting mechanism means that when this mechanism is applied the application of the mechanism is apparent. The aspects related to video are taught as indicated by the Yi in claim 1] that can be maximized even for representations that are far apart, meaning that the invariance learned is relatively weak. Unfortunately, directly changing the similarity measure hampers the algorithm. We therefore investigate methods to improve contrastive representations by explicitly encouraging stronger invariance to the set of transformations, without changing the similarity function. We look to extract more information about how representations are changing with respect to transformation, and use this to direct the encoder towards greater invariance”
One of ordinary skill in the art, prior to the effective filing date, would have been motivated to combine Nabi and Foster. Nabi and Foster are of the same field of endeavor of machine learning. One of ordinary skill in the art would have been motivated to combine Nabi and Foster in order to compress data or focus the model on more important information, such as by ignoring purely cosmetic details ([Foster et al 2. Probabilistic Formulation of Contrastive Learning page 3]: “In other words, the distribution p(z|x) should place most probability mass in a small region. However, this does not provide a sufficient training signal for the encoder fθ as it fails to penalize trivial solutions in which all inputs are mapped to the same point. To preserve meaningful information about the input x whilst discarding purely cosmetic features, we should require p(z|x) to be focused on a single point whilst simultaneously requiring the representations of different inputs not to be close.”).
Yi teaches:
using a training dataset consisting of a plurality of untrimmed video clips
[Yi 4.1 Datasets and implementation page 5]: “HMDB51 is a challenging video dataset [using a training dataset consisting of a plurality of untrimmed video clips] for action recognition, which pose challenges including large intra-class variations, camera movements, viewpoint variation, etc. It contains 6766 video clips collected from various sources, such as movies and YouTube. The dataset is composed of 51 distinct action categories, each including at least 101 video clips.”
splitting the untrimmed video clips into a plurality of video segments each containing non-informative background, informative background and/or foreground;
[Yi Introduction page 2]: “To characterize the relevance between video frames and target action, another new attribute named frame action relevance is defined based on the displacement vectors of all the trajectories belonging to a video frame. In the process of encoding, it is utilized as weights to describe the different importance of every video frame [splitting the untrimmed video clips into a plurality of video segments each containing non-informative background, informative background and/or foreground] in a video for recognizing actions.”
Nabi is noted for teaching elements of foreground, non-informative background, or informative background [Nabi Column 7 Line 19]. Yi supports the idea that the video clips and video segments contain some of these elements in [Yi Introduction page 2]: “A new method to calculate saliency map based on optical flow is proposed to highlight regions of foreground [foreground] motion. And a new attribute named trajectory action relevance is defined, which is calculated based on saliency map, and during encoding, it is used to set weight on each trajectory by the respective relevance between every trajectory and target action”
The idea of separating the video into frames for the segments is supported by the current disclosure in [Current Specification 0025]: “For untrimmed video recognition, to obtain the video feature F(x) given x, each video is split into T overlapped or un-overlapped video segments, where each segment contains t consecutive frames.”
Details supporting the relevance of Yi of focusing on recognized actions in video is also shown in [Yi Introduction page 2]: “To characterize the relevance between video frames and target action, another new attribute named frame action relevance is defined based on the displacement vectors of all the trajectories belonging to a video frame. In the process of encoding, it is utilized as weights to describe the different importance of every video frame in a video for recognizing actions.”
As Yi notes the purpose of the weighting through saliency maps helps find foreground motion ([Yi Introduction page 2]: “A new method to calculate saliency map based on optical flow is proposed to highlight regions of foreground motion.”), the weighting done by Yi is considered an equivalent to being based on the non-informative background present in a segment, as the weighting is finding the extend of foreground, which as a value would be based on the extend of non-informative background.
extracting a feature vector from each video segment using the base class model;
[Yi Introduction page 2]: “Furthermore, a novel approach is proposed to calculate saliency map based on optical flow, which can highlight important action regions [extracting a feature vector from each video segment using the base class model] around actors. To determine the contribution of each trajectory to the recognition accuracy, trajectory action relevance is defined and used as weights during encoding process.”
pseudo-labelling each video segment as background or foreground based on a classification probability for that video segment;
[Yi Introduction page 2]: “A new method to calculate saliency map based on optical flow is proposed to highlight regions of foreground motion [pseudo-labelling each video segment as background or foreground based on a classification probability for that video segment as the weighting from the saliency map is noted to highlight foreground motion, thus elements with the highest weighting would be considered based on some classification probability]. And a new attribute named trajectory action relevance is defined, which is calculated based on saliency map, and during encoding, it is used to set weight on each trajectory by the respective relevance between every trajectory and target action”
pseudo-labelling each video segment as informative background or foreground based on a classification probability for that video segment;
[Yi Introduction page 2]: “A new method to calculate saliency map based on optical flow is proposed to highlight regions of foreground motion [pseudo-labelling each video segment as informative background or foreground based on a classification probability for that video segment where the premise of the labelling the segments is the same as in claim 7 (here the classification is being used to indicate as foreground thus satisfying the claim at least in that aspect where the other aspects related to informative background are interpreted like claim 7 as a result of the 112 issues with claim 1)]. And a new attribute named trajectory action relevance is defined, which is calculated based on saliency map, and during encoding, it is used to set weight on each trajectory by the respective relevance between every trajectory and target action”
and localize one or more novel classes using a training data set comprising a few trimmed video segments of actions
[Yi Introduction page 2]: “A new method to calculate saliency map based on optical flow is proposed to highlight regions of foreground motion [and localize one or more novel classes]. And a new attribute named trajectory action relevance is defined, which is calculated based on saliency map, and during encoding, it is used to set weight on each trajectory by the respective relevance between every trajectory and target action”
[Yi 4.1 Datasets and implementation page 5]: “HMDB51 is a challenging video dataset [using a training data set comprising a few trimmed video segments of actions] for action recognition, which pose challenges including large intra-class variations, camera movements, viewpoint variation, etc. It contains 6766 video clips collected from various sources, such as movies and YouTube. The dataset is composed of 51 distinct action categories, each including at least 101 video clips.”
One of ordinary skill in the art, prior to the effective filing date, would have been motivated to combine Nabi and Yi. Nabi and Yi are in the same field of endeavor of machine learning. One of ordinary skill in the art would have been motivated to combine Nabi and Yi in order to utilize video data in action recognition. Video is considered a popular format for the use in relation to things like surveillance and computer interaction ([Yi Introduction page 1]: “Action recognition in videos has received intensive attention owing to its broad range of applications such as content-based video retrieval, intelligent video surveillance and human–computer interaction.”). Action recognition is considered important to help prevent elements of the background from distracting from the more important elements in a video ([Yi Introduction page 1]: “However, discriminative human actions may happen sparsely in a few key video frames within a video sequence [10], thus, despite the good performance of the aforementioned methods, there may still exist some trajectories that locate in frames which are irrelevant to the action class, and distracting trajectories that lie in key video frames but belong to background. Identify the relevant trajectories belonging to human action in the key frames is necessary for better classification. To further improve performance, saliency algorithms have been used to prune redundant trajectories and acquire the informative ones.”).
In regards to Claim 2:
The method of claim 1 is taught by Nabi, Foster, and Yi.
Nabi teaches:
in the novel class
wherein the base class model is able to classify and localize the action depicted in the novel class
[Nabi Column1 Line 33]: “Systems, methods, and articles of manufacture, including computer program products, are provided for weakly supervised, one-shot image segmentation. In some example embodiments, there is provided a system that includes at least one processor and at least one memory. The at least one memory may include program code that provides operations when executed by the at least one processor. The operations may include: training, in a supervised manner, a machine learning model to learn a plurality of base class prototypes corresponding to a plurality of base objects, each of the plurality of base class prototypes corresponding to a segmentation of a class of one or more similar base objects; training, based at least on a support image depicting a novel object, the machine learning model to learn a novel class [in the novel class] prototype corresponding to the novel object, the machine learning model being trained to learn the novel class prototype based at least on one of the plurality of base class prototypes [wherein the base class model is able to classify and localize the action depicted in the novel class] identified as being similar to the support image, and the novel object being a different object than the plurality of base objects; and applying the trained machine learning model to segment a query image.”
Yi teaches:
localize
[Yi Introduction page 2]: “A new method to calculate saliency map based on optical flow is proposed to highlight regions of foreground motion [localize]. And a new attribute named trajectory action relevance is defined, which is calculated based on saliency map, and during encoding, it is used to set weight on each trajectory by the respective relevance between every trajectory and target action”
exposing the base class model to untrimmed testing video segments comprising action
[Yi 4.1 Datasets and implementation page 5]: “HMDB51 is a challenging video dataset [exposing the base class model to untrimmed testing video segments comprising action] for action recognition, which pose challenges including large intra-class variations, camera movements, viewpoint variation, etc. It contains 6766 video clips collected from various sources, such as movies and YouTube. The dataset is composed of 51 distinct action categories, each including at least 101 video clips.”
The motivation to combine with Yi is the same as the motivation to combine with Yi in claim 1.
In regards to Claim 3:
The method of claim 1 is taught by Nabi, Foster, and Yi.
Nabi teaches:
which the base class model is trained to recognize
[Nabi Column1 Line 33]: “Systems, methods, and articles of manufacture, including computer program products, are provided for weakly supervised, one-shot image segmentation. In some example embodiments, there is provided a system that includes at least one processor and at least one memory. The at least one memory may include program code that provides operations when executed by the at least one processor. The operations may include: training, in a supervised manner, a machine learning model to learn a plurality of base class prototypes [which the base class model is trained to recognize] corresponding to a plurality of base objects, each of the plurality of base class prototypes corresponding to a segmentation of a class of one or more similar base objects; training, based at least on a support image depicting a novel object, the machine learning model to learn a novel class prototype corresponding to the novel object, the machine learning model being trained to learn the novel class prototype based at least on one of the plurality of base class prototypes identified as being similar to the support image, and the novel object being a different object than the plurality of base objects; and applying the trained machine learning model to segment a query image.”
Yi teaches the video aspect containing foreground and action, but the combination is considered supported and obvious, on top of the motivation to combine with Yi, for Nabi notes foreground and background aspects as explained in claim 1 mapping.
Yi teaches:
wherein video segments containing foreground are video segments containing an action
[Yi Introduction page 2]: “A new method to calculate saliency map based on optical flow is proposed to highlight regions of foreground motion [wherein video segments containing foreground are video segments containing an action]. And a new attribute named trajectory action relevance is defined, which is calculated based on saliency map, and during encoding, it is used to set weight on each trajectory by the respective relevance between every trajectory and target action”
The motivation to combine with Yi is the same as the motivation to combine with Yi in claim 1.
Regarding Claim 4:
The method of claim 1 is taught by Nabi, Foster, and Yi.
Nabi teaches:
wherein video segments containing informative background are video segments containing informative objects or actions which the base class model is not trained to recognize
[Nabi Column1 Line 33]: “Systems, methods, and articles of manufacture, including computer program products, are provided for weakly supervised, one-shot image segmentation. In some example embodiments, there is provided a system that includes at least one processor and at least one memory. The at least one memory may include program code that provides operations when executed by the at least one processor. The operations may include: training, in a supervised manner, a machine learning model to learn a plurality of base class prototypes corresponding to a plurality of base objects, each of the plurality of base class prototypes corresponding to a segmentation of a class of one or more similar base objects; training, based at least on a support image depicting a novel object [wherein video segments containing informative background are video segments containing informative objects or actions], the machine learning model to learn a novel class prototype corresponding to the novel object, the machine learning model being trained to learn the novel class prototype based at least on one of the plurality of base class prototypes identified as being similar to the support image, and the novel object being a different object [which the base class model is not trained to recognize] than the plurality of base objects; and applying the trained machine learning model to segment a query image.”
As explained in some detail in claim 1 mapping, the informative background is seen as the novel classes, as those are classes the base class model can perceive as actions, but are not actions recognized by the model, for the model was only originally trained to recognize the base classes. As part of few shot learning, the novel classes are then learned utilizing a few pieces of training data for the novel class.
Yi teaches the aspects of the input being video segments.
Regarding Claim 5:
The method of claim 1 is taught by Nabi, Foster, and Yi.
Yi teaches:
wherein video segments containing non-informative background are video clips not containing informative objects or actions
[Yi Introduction page 1]: “However, discriminative human actions may happen sparsely in a few key video frames within a video sequence [10], thus, despite the good performance of the aforementioned methods, there may still exist some trajectories that locate in frames which are irrelevant to the action class, and distracting trajectories that lie in key video frames but belong to background [wherein video segments containing non-informative background are video clips not containing informative objects or actions]. Identify the relevant trajectories belonging to human action in the key frames is necessary for better classification. To further improve performance, saliency algorithms have been used to prune redundant trajectories and acquire the informative ones.”
The motivation to combine with Yi is the same as the motivation to combine with Yi in claim 1.
Regarding Claim 7:
The method of claim 1 is taught by Nabi, Foster, and Yi.
Aspects of claim 7 are interpreted akin to the previous claim set as the 112 issues with the current claims prevent a proper interpretation of claim 7 (as some elements of the amended claim 1 conflict with the recited elements in claim 7 as indicated by the 112 rejection section).
Nabi teaches:
belongs to one of the base classes
[Nabi Column1 Line 33]: “Systems, methods, and articles of manufacture, including computer program products, are provided for weakly supervised, one-shot image segmentation. In some example embodiments, there is provided a system that includes at least one processor and at least one memory. The at least one memory may include program code that provides operations when executed by the at least one processor. The operations may include: training, in a supervised manner, a machine learning model to learn a plurality of base class prototypes corresponding to a plurality of base objects, each of the plurality of base class prototypes corresponding to a segmentation of a class of one or more similar base objects [belongs to one of the base classes]; training, based at least on a support image depicting a novel object, the machine learning model to learn a novel class prototype corresponding to the novel object, the machine learning model being trained to learn the novel class prototype based at least on one of the plurality of base class prototypes identified as being similar to the support image, and the novel object being a different object than the plurality of base objects; and applying the trained machine learning model to segment a query image.”
Yi teaches:
determining, based on the video features extracted from the video segments in the untrimmed video clip, a maximum classification probability of each video segment, the maximum classification probability indicating a confidence for the given segment belongs to one of the base classes
and measuring a confidence score as the maximum value of each video segment's classification probabilities,
[Yi 2 Related Work page 2]: “As for the saliency-based algorithms, they focus on finding a suitable salient mask, which is a critical step in feature points selection procedure. Aye et al. [35] presented a saliency navigated approach to localize action region with saliency detection, and a feature descriptor for extracting changing gradient orientation features from salient action region was introduced…Similarly, Feichtenhofer et al. [38] introduced a definition of spatiotemporal saliency weights [determining, based on the video features extracted from the video segments in the untrimmed video clip, a maximum classification probability of each video segment where the maximum classification probability is seen as an aspect of the saliency weights as the use (determining if foreground or background) of the maximum classification probability and the saliency weights is the same where the specifics are in the combination with Nabi (see motivation to combine below)][and measuring a confidence score as the maximum value of each video segment's classification probabilities as the claim limitation notes the confidence score is a value equivalent to the maximum classification probability thus teaching the maximum classification probability teaches the confidence score] to encode and pool primitive feature measurements to capture actions in videos. Moreover, the contribution of local features was enhanced through weighting with their respective saliency.”
and pseudo-labelling video segments having the highest confidence scores as foreground or informative background
[Yi Introduction page 2]: “A new method to calculate saliency map based on optical flow is proposed to highlight regions of foreground motion [and pseudo-labelling video segments having the highest confidence scores as foreground or informative background as the weighting from the saliency map is noted to highlight foreground motion, thus elements with the highest weighting would be above the implied threshold to be foreground. The implied threshold comes from the regions noted as being foreground or background thus implying a threshold exists separating what makes a region foreground or background]. And a new attribute named trajectory action relevance is defined, which is calculated based on saliency map, and during encoding, it is used to set weight on each trajectory by the respective relevance between every trajectory and target action”
pseudo-labelling a video segment as non-informative background when the maximum classification probability for that video segment falls below a threshold;
[Yi Introduction page 1]: “However, discriminative human actions may happen sparsely in a few key video frames within a video sequence [10], thus, despite the good performance of the aforementioned methods, there may still exist some trajectories that locate in frames which are irrelevant to the action class, and distracting trajectories that lie in key video frames but belong to background [pseudo-labelling a video segment as non-informative background when the maximum classification probability for that video segment falls below a threshold where the background supports the idea there is a threshold as the quote later notes pruning which would also imply a threshold for what is pruned for being irrelevant/background]. Identify the relevant trajectories belonging to human action in the key frames is necessary for better classification. To further improve performance, saliency algorithms have been used to prune redundant trajectories and acquire the informative ones.”
The motivation to combine with Yi is the same motivation to combine with Yi in claim 1. Further support for the idea of combination involving “the maximum classification probability indicating a confidence of belonging to one of the base class objects” is noted to be that Nabi teaches the elements related to classification for base classes as shown earlier, thus the note of the weights utilized in Yi for classification would be of relation to base classes when utilized in combination with Nabi. The motivation for utilizing the aspects of Yi for combination with Nabi is indicated in the motivation for the combination in claim 1, meaning the combination to adapt to video and focus on relevant aspects of a video would motivate to have such a combination.
Regarding Claim 8:
The method of claim 1 is taught by Nabi, Foster, and Yi.
Foster teaches:
defining, as a negative pair, a feature extracted from non-informative background video segments and a feature extracted from both informative background and foreground segments
[Foster et al 2. Probabilistic Formulation of Contrastive Learning page 3]: “If the encoder is to discard irrelevant information, we would expect different encodings of x formed with different transformations to be close in representation space. Altering the transformation should not lead to big changes in the representations of the same input. In other words, the distribution p(z|x) should place most probability mass in a small region. However, this does not provide a sufficient training signal for the encoder fθ as it fails to penalize trivial solutions in which all inputs are mapped to the same point. To preserve meaningful information about the input x whilst discarding purely cosmetic features, we should require p(z|x) to be focused on a single point whilst simultaneously requiring the representations of different inputs not to be close. That is, the marginal
PNG
media_image1.png
43
293
media_image1.png
Greyscale
should distribute probability mass over representation space. This intuition is directly reflected in contrastive learning. Most state-of-the-art contrastive learning methods utilize the InfoNCE objective (van den Oord et al., 2018), or close variants of it (Chen et al., 2020a). InfoNCE uses a batch x1, ..., xK of inputs, from which we form pairs of representations (z1, z’1), ...,(zK, z’K) by applying two random transformations to each input followed by the encoder fθ. In probabilistic language
PNG
media_image2.png
355
1302
media_image2.png
Greyscale
Written in this way, we see that the loss will be minimized when sφ(zi , z’i ) is large, but sφ(zi , z’j ) is small for i != j. In other words, InfoNCE makes the two samples zi , z’i of p(z|x = xi) similar, whilst making samples zi, z’j of p(z) dissimilar [defining, as a negative pair, a feature extracted from non-informative background video segments and a feature extracted from both informative background and foreground segments]. This can also be understood through the lens of mutual information, for more details see Appendix A.”
The motivation to combine with Foster is the same as in claim 1.
Nabi teaches the elements of foreground, non-informative background, and informative background as discussed in the mapping of claim 1. Aspects of labelling as foreground or background based on the weights is taught in claim 7 by Yi. The aspect of a negative pair is taught below by Foster as Foster teaches contrastive learning. The combination of a negative pair for the non-informative background and an element from the foreground/informative background is sensible as Foster notes [Foster et al 2. Probabilistic Formulation of Contrastive Learning page 3]: “If the encoder is to discard irrelevant information, we would expect different encodings of x formed with different transformations to be close in representation space.”. This quote supports the combination of Nabi with Yi, as Yi notes the background as distracting elements to be pruned ([Yi Introduction page 1]: “However, discriminative human actions may happen sparsely in a few key video frames within a video sequence [10], thus, despite the good performance of the aforementioned methods, there may still exist some trajectories that locate in frames which are irrelevant to the action class, and distracting trajectories that lie in key video frames but belong to background. Identify the relevant trajectories belonging to human action in the key frames is necessary for better classification. To further improve performance, saliency algorithms have been used to prune redundant trajectories and acquire the informative ones.”).
Regarding Claim 9:
The method of claim 8 is taught by Nabi, Foster, and Yi.
Foster teaches:
enlarging a distance in the base class model between features in the negative pair by maximizing a contrastive loss
[Foster et al 2. Probabilistic Formulation of Contrastive Learning page 3]: “If the encoder is to discard irrelevant information, we would expect different encodings of x formed with different transformations to be close in representation space. Altering the transformation should not lead to big changes in the representations of the same input. In other words, the distribution p(z|x) should place most probability mass in a small region. However, this does not provide a sufficient training signal for the encoder fθ as it fails to penalize trivial solutions in which all inputs are mapped to the same point. To preserve meaningful information about the input x whilst discarding purely cosmetic features, we should require p(z|x) to be focused on a single point whilst simultaneously requiring the representations of different inputs not to be close. That is, the marginal
PNG
media_image1.png
43
293
media_image1.png
Greyscale
should distribute probability mass over representation space. This intuition is directly reflected in contrastive learning. Most state-of-the-art contrastive learning methods utilize the InfoNCE objective (van den Oord et al., 2018), or close variants of it (Chen et al., 2020a). InfoNCE uses a batch x1, ..., xK of inputs, from which we form pairs of representations (z1, z’1), ...,(zK, z’K) by applying two random transformations to each input followed by the encoder fθ. In probabilistic language
PNG
media_image2.png
355
1302
media_image2.png
Greyscale
Written in this way, we see that the loss will be minimized when sφ(zi , z’i ) is large, but sφ(zi , z’j ) is small for i != j. In other words, InfoNCE makes the two samples zi , z’i of p(z|x = xi) similar, whilst making samples zi, z’j of p(z) dissimilar [enlarging a distance in the base class model between features in the negative pair by maximizing a contrastive loss]. This can also be understood through the lens of mutual information, for more details see Appendix A.”
As noted in the quote, part of the reason for such a loss equation being used is to have “representations of different inputs not to be close”. This means that making a pair, or two samples, dissimilar is moving the two samples farther apart.
Motivation to combine with Foster is the same as in claim 1.
Regarding Claim 10:
The method of claim 9 is taught by Nabi, Foster, and Yi.
Foster teaches:
defining, as a positive pair, features extracted from non-informative background video segments
[Foster et al 2. Probabilistic Formulation of Contrastive Learning page 3]: “If the encoder is to discard irrelevant information, we would expect different encodings of x formed with different transformations to be close in representation space. Altering the transformation should not lead to big changes in the representations of the same input. In other words, the distribution p(z|x) should place most probability mass in a small region. However, this does not provide a sufficient training signal for the encoder fθ as it fails to penalize trivial solutions in which all inputs are mapped to the same point. To preserve meaningful information about the input x whilst discarding purely cosmetic features, we should require p(z|x) to be focused on a single point whilst simultaneously requiring the representations of different inputs not to be close. That is, the marginal
PNG
media_image1.png
43
293
media_image1.png
Greyscale
should distribute probability mass over representation space. This intuition is directly reflected in contrastive learning. Most state-of-the-art contrastive learning methods utilize the InfoNCE objective (van den Oord et al., 2018), or close variants of it (Chen et al., 2020a). InfoNCE uses a batch x1, ..., xK of inputs, from which we form pairs of representations (z1, z’1), ...,(zK, z’K) by applying two random transformations to each input followed by the encoder fθ. In probabilistic language
PNG
media_image2.png
355
1302
media_image2.png
Greyscale
Written in this way, we see that the loss will be minimized when sφ(zi , z’i ) is large, but sφ(zi , z’j ) is small for i != j. In other words, InfoNCE makes the two samples zi , z’i of p(z|x = xi) similar [defining, as a positive pair, features extracted from non-informative background video segments], whilst making samples zi, z’j of p(z) dissimilar. This can also be understood through the lens of mutual information, for more details see Appendix A.”
Support for the positive pair including features from the non-informative background is shown by focusing on a piece of the above quote: “If the encoder is to discard irrelevant information, we would expect different encodings of x formed with different transformations to be close in representation space”. The section is noting that to preserve meaningful information and to discard purely cosmetic features (or non-informative information), same classifications should be grouped closer together.
The motivation to combine with Foster is the same as in claim 1.
Regarding Claim 11:
The method of claim 10 is taught by Nabi, Foster, and Yi.
Foster teaches:
reducing a distance in the base class model between features in the positive pair by minimizing the contrastive loss
[Foster et al 2. Probabilistic Formulation of Contrastive Learning page 3]: “If the encoder is to discard irrelevant information, we would expect different encodings of x formed with different transformations to be close in representation space. Altering the transformation should not lead to big changes in the representations of the same input. In other words, the distribution p(z|x) should place most probability mass in a small region. However, this does not provide a sufficient training signal for the encoder fθ as it fails to penalize trivial solutions in which all inputs are mapped to the same point. To preserve meaningful information about the input x whilst discarding purely cosmetic features, we should require p(z|x) to be focused on a single point whilst simultaneously requiring the representations of different inputs not to be close. That is, the marginal
PNG
media_image1.png
43
293
media_image1.png
Greyscale
should distribute probability mass over representation space. This intuition is directly reflected in contrastive learning. Most state-of-the-art contrastive learning methods utilize the InfoNCE objective (van den Oord et al., 2018), or close variants of it (Chen et al., 2020a). InfoNCE uses a batch x1, ..., xK of inputs, from which we form pairs of representations (z1, z’1), ...,(zK, z’K) by applying two random transformations to each input followed by the encoder fθ. In probabilistic language
PNG
media_image2.png
355
1302
media_image2.png
Greyscale
Written in this way, we see that the loss will be minimized when sφ(zi , z’i ) is large, but sφ(zi , z’j ) is small for i != j. In other words, InfoNCE makes the two samples zi , z’i of p(z|x = xi) similar [reducing a distance in the base class model between features in the positive pair by minimizing the contrastive loss], whilst making samples zi, z’j of p(z) dissimilar. This can also be understood through the lens of mutual information, for more details see Appendix A.”
As noted in the quote, part of the reason for such a loss equation being used is to have “If the encoder is to discard irrelevant information, we would expect different encodings of x formed with different transformations to be close in representation space.” This means that making a pair, or two samples, similar is bringing the two samples closer together or compressing.
Motivation to combine with Foster is the same as in claim 1.
In regards to Claim 14:
Nabi teaches:
A system comprising: a processor; software, executing on the processor, the software performing the steps of claim 1
[Nabi Column1 Line 33]: “Systems, methods, and articles of manufacture, including computer program products, are provided for weakly supervised, one-shot image segmentation. In some example embodiments, there is provided a system that includes at least one processor [A system comprising: a processor;] and at least one memory. The at least one memory may include program code [software, executing on the processor] that provides operations when executed by the at least one processor. The operations may include: training, in a supervised manner, a machine learning model to learn a plurality of base class prototypes corresponding to a plurality of base objects, each of the plurality of base class prototypes corresponding to a segmentation of a class of one or more similar base objects; training, based at least on a support image depicting a novel object, the machine learning model to learn a novel class prototype corresponding to the novel object, the machine learning model being trained to learn the novel class prototype based at least on one of the plurality of base class prototypes identified as being similar to the support image, and the novel object being a different object than the plurality of base objects; and applying the trained machine learning model to segment a query image.”
the software performing the steps of claim 1
This limitation is seen as being taught by the elements mapping claim 1.
Claim 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nabi, in further view of Foster, in further view of Yi, and in further view of Singh.
In regards to claim 16:
The method of claim 1 is taught by Nabi, Foster, and Yi.
Updating the weights of the base class model based on the contrastive loss
[Foster et al 2. Probabilistic Formulation of Contrastive Learning page 3]: “If the encoder is to discard irrelevant information, we would expect different encodings of x formed with different transformations to be close in representation space. Altering the transformation should not lead to big changes in the representations of the same input. In other words, the distribution p(z|x) should place most probability mass in a small region. However, this does not provide a sufficient training signal for the encoder fθ as it fails to penalize trivial solutions in which all inputs are mapped to the same point. To preserve meaningful information about the input x whilst discarding purely cosmetic features, we should require p(z|x) to be focused on a single point whilst simultaneously requiring the representations of different inputs not to be close. That is, the marginal
PNG
media_image1.png
43
293
media_image1.png
Greyscale
should distribute probability mass over representation space. This intuition is directly reflected in contrastive learning. Most state-of-the-art contrastive learning methods utilize the InfoNCE objective (van den Oord et al., 2018), or close variants of it (Chen et al., 2020a). InfoNCE uses a batch x1, ..., xK of inputs, from which we form pairs of representations (z1, z’1), ...,(zK, z’K) by applying two random transformations to each input followed by the encoder fθ. In probabilistic language
PNG
media_image2.png
355
1302
media_image2.png
Greyscale
Written in this way, we see that the loss will be minimized [updating the weights of the base class model based on the contrastive loss as the steps of contrastive learning using a contrastive loss are indicated within the quote] when sφ(zi , z’i ) is large, but sφ(zi , z’j ) is small for i != j. In other words, InfoNCE makes the two samples zi , z’i of p(z|x = xi) similar, whilst making samples zi, z’j of p(z) dissimilar. This can also be understood through the lens of mutual information, for more details see Appendix A.”
As noted in the quote, part of the reason for such a loss equation being used is to have “representations of different inputs not to be close”. This means that making a pair similar, or two samples, is bringing the two samples closer together or compressing. This idea is supported by phrasing in [Current Application 0011]: “For NBG, the model should compress its feature space and pull it away from FG…”
The motivation for combining with Foster for contrastive learning is the same as noted for claim 1.
Nabi does not explicitly teach:
Iterating the steps of the initial training until the contrastive loss falls below a threshold
Singh teaches:
Iterating the steps of the initial training until the contrastive loss falls below a threshold
[Singh Column 13 Line 10]: “In some embodiments, the digital image classification system 102 performs many iterations or epochs of training over the labeled digital images 302 until the base neural network 304 classifies the labeled digital images with at least a threshold accuracy (or until the self-supervised loss and/or the classification loss are below a threshold loss [Iterating the steps of the initial training until the contrastive loss falls below a threshold where the contrastive loss is indicated/taught by Foster as described earlier]). Thus, the digital image classification system 102 trains the base neural network 304 to accurately classify the labeled digital images 302 into the corresponding base classes.”
One of ordinary skill in the art, prior to the effective filing date, would have been motivated to combine Nabi and Singh in order to run until a loss below a threshold. Nabi and Singh are in the same field of endeavor of machine learning. One of ordinary skill in the art would have combined Nabi and Singh in order to ensure a level of accuracy of the created model ([Singh Column 13 Line 10]: “In some embodiments, the digital image classification system 102 performs many iterations or epochs of training over the labeled digital images 302 until the base neural network 304 classifies the labeled digital images with at least a threshold accuracy (or until the self-supervised loss and/or the classification loss are below a threshold loss). Thus, the digital image classification system 102 trains the base neural network 304 to accurately classify the labeled digital images 302 into the corresponding base classes.”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Wang et al (“Plant Leaves Classification: A Few-Shot Learning Method Based on Siamese Network”) is considered relevant art as the reference notes the use of few shot learning, contrastive learning, and nearest neighbor classification, which are all elements of the current application.
Burghouts (US 20150286874 A1) is considered relevant art as the reference teaches elements related to action recognition in video, thus teaches aspects related to weighting elements of a video to identify relevant parts similar to the current application.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHRISTOPHER D DEVORE whose telephone number is (703)756-1234. The examiner can normally be reached Monday-Friday 7:30 am - 5 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael J Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.D.D./Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129
