Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/27/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Allowable Subject Matter
Claims 2, 3, 5-7, 9, and 14-16 objected to as being dependent upon a rejected base claim, but would be allowable over the prior art if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 22 rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because the claim recites "A system comprising: one or more computers; and one or more storage devices" but does not explicitly recite the non-transitory nature of the storage devices and therefore includes signals per se.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-18 and 23 rejected under 35 U.S.C. 101 because they are directed to an abstract idea without significantly more.
Step 1 analysis:
Independent Claim 1 recites, in part, a computer-implemented method, therefore falling into the statutory category of process. Independent Claim 23 recites, in part, one or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a neural network, therefore falling into the statutory category of machine.
Regarding Claim 1:
Step 2A: Prong 1 analysis:
Claim 1 recites in part:
“combining the object segmentation and the first representation to generate a first object representation for each of the objects in the first transformed view”. As drafted and under its broadest reasonable interpretation, this limitation covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper. For example, this limitation encompasses combining data.
“combining the object segmentation and the second representation to generate a second object representation for each of the objects in the second transformed view”. As drafted and under its broadest reasonable interpretation, this limitation covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper. For example, this limitation encompasses combining data.
“updating parameters of the first feature representation neural network by comparing a predicted object representation dependent upon the first object representation, and the second object representation”. As drafted and under its broadest reasonable interpretation, this limitation covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper. For example, this limitation encompasses updating data.
“updating parameters of the second feature representation neural network based on the parameters of the first feature representation neural network”. As drafted and under its broadest reasonable interpretation, this limitation covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper. For example, this limitation encompasses updating data.
“updating parameters of the object discovery neural network based on the parameters of the first feature representation neural network”. As drafted and under its broadest reasonable interpretation, this limitation covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper. For example, this limitation encompasses updating data.
Accordingly, at Step 2A: Prong 1, the claim is directed to an abstract idea.
Step 2A: Prong 2 analysis:
The judicial exception is not integrated into practical application. In particular, the claim recites the additional elements of:
“processing the training data item using an object discovery neural network to generate an object segmentation of the training data item, wherein the object segmentation defines a segmentation of the training data item into a plurality of different objects represented by the training data item”. This additional element is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (object discovery neural network) (See MPEP 2106.05(f)).
“processing a first transformed view of the training data item with a first feature representation neural network to generate a first representation of the first transformed view”. This additional element is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (neural network) (See MPEP 2106.05(f)).
“processing a second transformed view of the training data item with a second feature representation neural network to generate a second representation of the second transformed view”. This additional element is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (neural network) (See MPEP 2106.05(f)).
Accordingly at Step 2A: Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B analysis:
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
As discussed above, the additional element(s) of “processing the training data item using an object discovery neural network to generate an object segmentation of the training data item, wherein the object segmentation defines a segmentation of the training data item into a plurality of different objects represented by the training data item”, “processing a first transformed view of the training data item with a first feature representation neural network to generate a first representation of the first transformed view”, and “processing a second transformed view of the training data item with a second feature representation neural network to generate a second representation of the second transformed view” is/are recited at a high-level of generality such that it/they amount(s) to no more than mere instructions to apply the exception using a generic computer component (neural networks) (See MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional elements individually or in combination do not amount to significantly more than the judicial exception.
Regarding Claim 2:
Step 2A: Prong 1 analysis:
Claim 2 recites in part:
“determining a value of a contrastive objective function, wherein the contrastive objective function has a first term dependent upon a first measure of similarity between the predicted object representation for one of the objects in the first transformed view and the second object representation for the same object in the second transformed view, and a second term dependent upon a second measure of similarity between the predicted object representation for one of the objects in the first transformed view and the second object representation for a different one of the objects in the second transformed view of the training data item or of another training data item”. As drafted and under its broadest reasonable interpretation, this limitation covers a mathematical calculation/equation.
“wherein updating parameters of the first feature representation neural network comprises updating the parameters to maximize the first term and to minimize the second term”. As drafted and under its broadest reasonable interpretation, this limitation covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper. For example, this limitation encompasses updating data.
Accordingly, at Step 2A: Prong 1, the claim is directed to an abstract idea.
Step 2A: Prong 2 analysis:
The claim does not recite any additional elements that integrate the judicial exception into a practical application.
Step 2B analysis:
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
Regarding Claim 3:
Step 2A: Prong 1 analysis:
Claim 3 recites in part:
“determining the first term of the contrastive objective function by computing the first measure of similarity between the predicted object representation for the object in the first transformed view and the second object representation for the same object in the second transformed view”. As drafted and under its broadest reasonable interpretation, this limitation covers a mathematical calculation/equation.
“determining the second term of the contrastive objective function by computing the second measure of similarity between the predicted object representation for the object in the first transformed view and the second object representation for each different object in the second transformed view of the training data item”. As drafted and under its broadest reasonable interpretation, this limitation covers a mathematical calculation/equation.
Accordingly, at Step 2A: Prong 1, the claim is directed to an abstract idea.
Step 2A: Prong 2 analysis:
The claim does not recite any additional elements that integrate the judicial exception into a practical application.
Step 2B analysis:
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
Regarding Claim 4:
Step 2A: Prong 1 analysis:Claim 4 recites in part:
“updating the parameters of the object discovery feature representation neural network to approach or equal values of the parameters of the first feature representation neural network”. As drafted and under its broadest reasonable interpretation, this limitation covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper. For example, this limitation encompasses updating data.
Accordingly, at Step 2A: Prong 1, the claim is directed to an abstract idea.
Step 2A: Prong 2 analysis:
The judicial exception is not integrated into practical application. In particular, the claim recites the additional elements of:
“wherein the object discovery neural network comprises an object discovery feature representation neural network to process the training data item to generate a feature map of the training data item for generating the object segmentation, wherein the object discovery feature representation neural network has the same architecture as the first feature representation neural network”. This limitation merely indicates a field of use or technological environment in which the judicial exception is performed (object discovery) and thus fails to add an inventive concept to the claims. See MPEP 2106.05(h).
“processing the training data item using the object discovery feature representation neural network to generate the feature map of the training data item”. This additional element is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (neural network) (See MPEP 2106.05(f)).
“using the feature map of the training data item to generate the object segmentation of the training data item”. This additional element is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (feature map) (See MPEP 2106.05(f)).
Accordingly at Step 2A: Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B analysis:
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
The additional element(s) of “wherein the object discovery neural network comprises an object discovery feature representation neural network to process the training data item to generate a feature map of the training data item for generating the object segmentation, wherein the object discovery feature representation neural network has the same architecture as the first feature representation neural network” is/are directed to particular field(s) of use (object discovery) (MPEP 2106.05(h)) and therefore do not provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
As discussed above, the additional element(s) of “processing the training data item using the object discovery feature representation neural network to generate the feature map of the training data item” and “using the feature map of the training data item to generate the object segmentation of the training data item” is/are recited at a high-level of generality such that it/they amount(s) to no more than mere instructions to apply the exception using a generic computer component (neural networks and feature maps) (See MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional elements individually or in combination do not amount to significantly more than the judicial exception.
Regarding Claim 5:
Step 2A: Prong 1 analysis:Claim 5 recites in part:
“and wherein processing the training data item using the object discovery neural network to generate the object segmentation of the training data item comprises grouping similar feature vectors to determine a set of groups of feature vectors and corresponding groups of elements of the training data item, one group for each of the different objects, wherein the groups of the elements of the training data item define the object segmentation”. As drafted and under its broadest reasonable interpretation, this limitation covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper. For example, this limitation encompasses grouping similar data elements together.
Accordingly, at Step 2A: Prong 1, the claim is directed to an abstract idea.
Step 2A: Prong 2 analysis:
The judicial exception is not integrated into practical application. In particular, the claim recites the additional elements of:
“wherein the feature map defines, for elements of the training data item, a respective feature vector for the element”. This limitation merely indicates a field of use or technological environment in which the judicial exception is performed (feature maps) and thus fails to add an inventive concept to the claims. See MPEP 2106.05(h).
Accordingly at Step 2A: Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B analysis:
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
The additional element(s) of “wherein the feature map defines, for elements of the training data item, a respective feature vector for the element” is/are directed to particular field(s) of use (feature maps) (MPEP 2106.05(h)) and therefore do not provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Accordingly, at Step 2B, the additional elements individually or in combination do not amount to significantly more than the judicial exception.
Regarding Claim 6:
Step 2A: Prong 2 analysis:The judicial exception is not integrated into practical application. In particular, the claim recites the additional elements of:
“wherein each group of elements of the training data item defines a respective mask for the group of elements, and wherein each mask defines elements for a respective one of the different objects”. This limitation merely indicates a field of use or technological environment in which the judicial exception is performed (data masks) and thus fails to add an inventive concept to the claims. See MPEP 2106.05(h).
Accordingly at Step 2A: Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B analysis:
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
The additional element(s) of “wherein each group of elements of the training data item defines a respective mask for the group of elements, and wherein each mask defines elements for a respective one of the different objects” is/are directed to particular field(s) of use (data masks) (MPEP 2106.05(h)) and therefore do not provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Accordingly, at Step 2B, the additional elements individually or in combination do not amount to significantly more than the judicial exception.
Regarding Claim 7:
Step 2A: Prong 2 analysis:
The judicial exception is not integrated into practical application. In particular, the claim recites the additional elements of:
“wherein the first representation of the first transformed view comprises a first representation feature map”. This limitation merely indicates a field of use or technological environment in which the judicial exception is performed (feature maps) and thus fails to add an inventive concept to the claims. See MPEP 2106.05(h).
“and wherein the second representation of the second transformed view comprises a second representation feature map”. This limitation merely indicates a field of use or technological environment in which the judicial exception is performed (feature maps) and thus fails to add an inventive concept to the claims. See MPEP 2106.05(h).
“generating the first object representation for each of the objects in the first transformed view by applying the respective mask for each of the objects in the first transformed view to the first representation feature map”. This additional element is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (data masks) (See MPEP 2106.05(f)).
“generating the second object representation for each of the objects in the second transformed view by applying the respective mask for each of the objects in the second transformed view to the second representation feature map”. This additional element is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (data masks) (See MPEP 2106.05(f)).
Accordingly at Step 2A: Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B analysis:
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
The additional element(s) of “wherein the first representation of the first transformed view comprises a first representation feature map” and “and wherein the second representation of the second transformed view comprises a second representation feature map” is/are directed to particular field(s) of use (feature maps) (MPEP 2106.05(h)) and therefore do not provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
As discussed above, the additional element(s) of “generating the first object representation for each of the objects in the first transformed view by applying the respective mask for each of the objects in the first transformed view to the first representation feature map” and “generating the second object representation for each of the objects in the second transformed view by applying the respective mask for each of the objects in the second transformed view to the second representation feature map” is/are recited at a high-level of generality such that it/they amount(s) to no more than mere instructions to apply the exception using a generic computer component (neural networks) (See MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional elements individually or in combination do not amount to significantly more than the judicial exception.
Regarding Claim 8:
Step 2A: Prong 1 analysis:
Claim 8 recites in part:
“wherein updating parameters of the object discovery neural network based on the parameters of the first feature representation neural network comprises updating parameters of the object discovery neural network with a moving average of the parameters of the first feature representation neural network”. As drafted and under its broadest reasonable interpretation, this limitation covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper. For example, this limitation encompasses updating data.
Accordingly, at Step 2A: Prong 1, the claim is directed to an abstract idea.
Step 2A: Prong 2 analysis:
The claim does not recite any additional elements that integrate the judicial exception into a practical application.
Step 2B analysis:
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
Regarding Claim 9:
Step 2A: Prong 1 analysis:
Claim 9 recites in part:
“wherein updating parameters of the object discovery neural network based on the parameters of the first feature representation neural network comprises updating parameters of the object discovery neural network with a copy of the parameters of the first feature representation neural network”. As drafted and under its broadest reasonable interpretation, this limitation covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper. For example, this limitation encompasses updating data.
Accordingly, at Step 2A: Prong 1, the claim is directed to an abstract idea.
Step 2A: Prong 2 analysis:
The claim does not recite any additional elements that integrate the judicial exception into a practical application.
Step 2B analysis:
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
Regarding Claim 10:
Step 2A: Prong 1 analysis:
Claim 10 recites in part:
“updating the parameters of the first feature representation neural network by comparing the predicted object representation and the second object representation from the first training data item”. As drafted and under its broadest reasonable interpretation, this limitation covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper. For example, this limitation encompasses updating data.
“updating the parameters of the second feature representation neural network”. As drafted and under its broadest reasonable interpretation, this limitation covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper. For example, this limitation encompasses updating data.
“updating the parameters of the object discovery neural network based on the updated parameters of the first feature representation neural network”. As drafted and under its broadest reasonable interpretation, this limitation covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper. For example, this limitation encompasses updating data.
“updating the parameters of the first feature representation neural network by comparing the predicted object representation and the second object representation from the second training data item”. As drafted and under its broadest reasonable interpretation, this limitation covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper. For example, this limitation encompasses updating data.
“updating the parameters of the second feature representation neural network”. As drafted and under its broadest reasonable interpretation, this limitation covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper. For example, this limitation encompasses updating data.
Accordingly, at Step 2A: Prong 1, the claim is directed to an abstract idea.
Step 2A: Prong 2 analysis:
The judicial exception is not integrated into practical application. In particular, the claim recites the additional elements of:
“processing a first training data item using the object discovery neural network to generate the object segmentation of the first training data item”. This additional element is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (neural network) (See MPEP 2106.05(f)).
“generating the first object representation and the second object representation from the first training data item using the object segmentation of the first training data item”. This additional element is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (object segmentation) (See MPEP 2106.05(f)).
“processing a second training data item using the object discovery neural network, after updating the parameters of the object discovery neural network, to generate the object segmentation of the second training data item”. This additional element is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (neural network) (See MPEP 2106.05(f)).
“generating the first object representation and the second object representation from the second training data item using the object segmentation of the second training data item”. This additional element is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (object segmentation) (See MPEP 2106.05(f)).
Accordingly at Step 2A: Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B analysis:
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
As discussed above, the additional element(s) of “processing a first training data item using the object discovery neural network to generate the object segmentation of the first training data item”, “generating the first object representation and the second object representation from the first training data item using the object segmentation of the first training data item”, “processing a second training data item using the object discovery neural network, after updating the parameters of the object discovery neural network, to generate the object segmentation of the second training data item”, and “generating the first object representation and the second object representation from the second training data item using the object segmentation of the second training data item” is/are recited at a high-level of generality such that it/they amount(s) to no more than mere instructions to apply the exception using a generic computer component (neural networks and object segmentations) (See MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional elements individually or in combination do not amount to significantly more than the judicial exception.
Regarding Claim 11:
Step 2A: Prong 1 analysis:
Claim 11 recites in part:
“wherein updating the parameters of the second feature representation neural network comprises updating the parameters to approach values of the parameters of the first feature representation neural network”. As drafted and under its broadest reasonable interpretation, this limitation covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper. For example, this limitation encompasses updating data.
Accordingly, at Step 2A: Prong 1, the claim is directed to an abstract idea.
Step 2A: Prong 2 analysis:
The judicial exception is not integrated into practical application. In particular, the claim recites the additional elements of:
“wherein the first feature representation neural network and the second feature representation neural network have the same architecture and different parameters”. This limitation merely indicates a field of use or technological environment in which the judicial exception is performed (neural network parameters) and thus fails to add an inventive concept to the claims. See MPEP 2106.05(h).
Accordingly at Step 2A: Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B analysis:
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
The additional element(s) of “wherein the first feature representation neural network and the second feature representation neural network have the same architecture and different parameters” is/are directed to particular field(s) of use (neural network parameters) (MPEP 2106.05(h)) and therefore do not provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Accordingly, at Step 2B, the additional elements individually or in combination do not amount to significantly more than the judicial exception.
Regarding Claim 12:
Step 2A: Prong 1 analysis:
Claim 12 recites in part:
“updating parameters of the prediction neural network by comparing the predicted object representation and the second object representation”. As drafted and under its broadest reasonable interpretation, this limitation covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper. For example, this limitation encompasses updating data.
Accordingly, at Step 2A: Prong 1, the claim is directed to an abstract idea.
Step 2A: Prong 2 analysis:
The judicial exception is not integrated into practical application. In particular, the claim recites the additional elements of:
“processing the first object representation using a prediction neural network to generate the predicted object representation”. This additional element is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (neural network) (See MPEP 2106.05(f)).
Accordingly at Step 2A: Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B analysis:
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
As discussed above, the additional element(s) of “processing the first object representation using a prediction neural network to generate the predicted object representation” is/are recited at a high-level of generality such that it/they amount(s) to no more than mere instructions to apply the exception using a generic computer component (neural network) (See MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional elements individually or in combination do not amount to significantly more than the judicial exception.
Regarding Claim 13:
Step 2A: Prong 2 analysis:
The judicial exception is not integrated into practical application. In particular, the claim recites the additional elements of:
“wherein the predicted object representation is the first object representation”. This limitation merely indicates a field of use or technological environment in which the judicial exception is performed (object representations) and thus fails to add an inventive concept to the claims. See MPEP 2106.05(h).
Accordingly at Step 2A: Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B analysis:
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
The additional element(s) of “wherein the predicted object representation is the first object representation” is/are directed to particular field(s) of use (object representations) (MPEP 2106.05(h)) and therefore do not provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Accordingly, at Step 2B, the additional elements individually or in combination do not amount to significantly more than the judicial exception.
Regarding Claim 14:
Step 2A: Prong 1 analysis:
Claim 14 recites in part:
“combining the object segmentation and the first representation to generate a first intermediate object representation for each of the objects in the first transformed view”. As drafted and under its broadest reasonable interpretation, this limitation covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper. For example, this limitation encompasses combining data.
“combining the object segmentation and the second representation to generate a second intermediate object representation for each of the objects in the second transformed view”. As drafted and under its broadest reasonable interpretation, this limitation covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper. For example, this limitation encompasses combining data.
“updating parameters of the first projection neural network by comparing the predicted object representation and the second object representation”. As drafted and under its broadest reasonable interpretation, this limitation covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper. For example, this limitation encompasses updating data.
“updating parameters of the second projection neural network based on the parameters of the first projection neural network”. As drafted and under its broadest reasonable interpretation, this limitation covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper. For example, this limitation encompasses updating data.
Accordingly, at Step 2A: Prong 1, the claim is directed to an abstract idea.
Step 2A: Prong 2 analysis:
The judicial exception is not integrated into practical application. In particular, the claim recites the additional elements of:
“processing the first intermediate object representation for each of the objects using a first projection neural network to generate the first object representation for each of the objects in the first transformed view”. This additional element is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (neural network) (See MPEP 2106.05(f)).
“processing the second intermediate object representation for each of the objects using a second projection neural network to generate the second object representation for each of the objects in the first transformed view”. This additional element is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (neural network) (See MPEP 2106.05(f)).
Accordingly at Step 2A: Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B analysis:
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
As discussed above, the additional element(s) of “processing the first intermediate object representation for each of the objects using a first projection neural network to generate the first object representation for each of the objects in the first transformed view” and “processing the second intermediate object representation for each of the objects using a second projection neural network to generate the second object representation for each of the objects in the first transformed view” is/are recited at a high-level of generality such that it/they amount(s) to no more than mere instructions to apply the exception using a generic computer component (neural network) (See MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional elements individually or in combination do not amount to significantly more than the judicial exception.
Regarding Claim 15:
Step 2A: Prong 2 analysis:
The judicial exception is not integrated into practical application. In particular, the claim recites the additional elements of:
“generating the first transformed view of the training data item by applying a first data item transformation to the training data item to generate the first transformed view of the training data item”. This additional element is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (data transformations) (See MPEP 2106.05(f)).
“generating the second transformed view of the training data item by applying a second data item transformation to the training data item to generate the second transformed view of the training data item, wherein the second data item transformation is different from the first data item transformation”. This additional element is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (data transformations) (See MPEP 2106.05(f)).
Accordingly at Step 2A: Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B analysis:
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
As discussed above, the additional element(s) of “generating the first transformed view of the training data item by applying a first data item transformation to the training data item to generate the first transformed view of the training data item” and “generating the second transformed view of the training data item by applying a second data item transformation to the training data item to generate the second transformed view of the training data item, wherein the second data item transformation is different from the first data item transformation” is/are recited at a high-level of generality such that it/they amount(s) to no more than mere instructions to apply the exception using a generic computer component (data transformations) (See MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional elements individually or in combination do not amount to significantly more than the judicial exception.
Regarding Claim 16:
Step 2A: Prong 2 analysis:
The judicial exception is not integrated into practical application. In particular, the claim recites the additional elements of:
“wherein first feature representation neural network and the second feature representation neural network are the same neural network such that updating the parameters of the second feature representation neural network is performed by updating the parameters of the first feature representation neural network”. This limitation merely indicates a field of use or technological environment in which the judicial exception is performed (neural networks) and thus fails to add an inventive concept to the claims. See MPEP 2106.05(h).
Accordingly at Step 2A: Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B analysis:
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
The additional element(s) of “wherein first feature representation neural network and the second feature representation neural network are the same neural network such that updating the parameters of the second feature representation neural network is performed by updating the parameters of the first feature representation neural network” is/are directed to particular field(s) of use (neural networks) (MPEP 2106.05(h)) and therefore do not provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Accordingly, at Step 2B, the additional elements individually or in combination do not amount to significantly more than the judicial exception.
Regarding Claim 17:
Step 2A: Prong 2 analysis:
The judicial exception is not integrated into practical application. In particular, the claim recites the additional elements of:
“wherein the training data item comprises an audio data item, an image data item, a multimodal data item, a text data item, or a graph data item”. This limitation merely indicates a field of use or technological environment in which the judicial exception is performed (data types) and thus fails to add an inventive concept to the claims. See MPEP 2106.05(h).
“after the training, using i) the object discovery neural network, or ii) at least part of the first feature representation neural network or iii) at least part of the second feature representation neural network, correspondingly, to process input data comprising audio data, image data, multimodal data, text data, or graph data respectively to perform an audio signal processing task, an image processing task, a multimodal processing task, a text processing task, or a graph processing task”. This additional element is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (neural networks) (See MPEP 2106.05(f)).
Accordingly at Step 2A: Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B analysis:
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
The additional element(s) of “wherein the training data item comprises an audio data item, an image data item, a multimodal data item, a text data item, or a graph data item” is/are directed to particular field(s) of use (data types) (MPEP 2106.05(h)) and therefore do not provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
As discussed above, the additional element(s) of “after the training, using i) the object discovery neural network, or ii) at least part of the first feature representation neural network or iii) at least part of the second feature representation neural network, correspondingly, to process input data comprising audio data, image data, multimodal data, text data, or graph data respectively to perform an audio signal processing task, an image processing task, a multimodal processing task, a text processing task, or a graph processing task” is/are recited at a high-level of generality such that it/they amount(s) to no more than mere instructions to apply the exception using a generic computer component (neural networks) (See MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional elements individually or in combination do not amount to significantly more than the judicial exception.
Regarding Claim 18:
Step 2A: Prong 2 analysis:
The judicial exception is not integrated into practical application. In particular, the claim recites the additional elements of:
“wherein the audio data comprises data representing values of an audio waveform, and wherein the audio signal processing task comprises: processing audio data representing speech to provide output data that detects words or phonemes in the speech or categorizes words, syllables or phonemes in the speech into one or more of a plurality of categories, or processing audio data representing a sound to provide output data that detects presence of a particular sound or event in the sound, or processing audio data representing a sound to provide output data that categorizes a content of the sound into one or more of a plurality of categories”. This limitation merely indicates a field of use or technological environment in which the judicial exception is performed (audio data) and thus fails to add an inventive concept to the claims. See MPEP 2106.05(h).
“or wherein the image data comprises pixels of an image or of video, and wherein the image processing task comprises: processing the image data to provide output data that identifies the location of one or more objects in the image or video, or processing the image data to provide output data that segments pixels of the image or video into regions that represent one or more objects in the image or video signal, or processing the image data to provide output data that categorizes a content of the image or video into one or more of a plurality of categories, or processing the image data to provide output data that predicts depth values for pixels of the image or video; or wherein the image data comprises pixels of a video and the image processing task comprises: processing the image data to provide output data that identifies the location of one or more actions represented in the video, or processing the image data to provide output data that categorizes one or more actions represented in the video into one or more of a plurality of categories”. This limitation merely indicates a field of use or technological environment in which the judicial exception is performed (image data) and thus fails to add an inventive concept to the claims. See MPEP 2106.05(h).
“or wherein the multimodal data comprises audio-visual data comprising a combination of pixels of an image or of video and audio data representing values of an audio waveform, and wherein the multimodal processing task comprises: processing the combination to provide output data that detects presence of a particular event in the combination, or processing the combination to provide output data that categorizes the combination into one or more of a plurality of categories”. This limitation merely indicates a field of use or technological environment in which the judicial exception is performed (multimodal data) and thus fails to add an inventive concept to the claims. See MPEP 2106.05(h).
“or wherein the multimodal data comprises a combination of text data representing text in a natural language and pixels of an image or of video or audio data representing values of an audio waveform, and wherein the multimodal processing task comprises: processing the combination to provide output data that defines whether the image or video or audio waveform is described by the text” This limitation merely indicates a field of use or technological environment in which the judicial exception is performed (multimodal data) and thus fails to add an inventive concept to the claims. See MPEP 2106.05(h).
Accordingly at Step 2A: Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B analysis:
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
The additional element(s) of “wherein the audio data comprises data representing values of an audio waveform, and wherein the audio signal processing task comprises: processing audio data representing speech to provide output data that detects words or phonemes in the speech or categorizes words, syllables or phonemes in the speech into one or more of a plurality of categories, or processing audio data representing a sound to provide output data that detects presence of a particular sound or event in the sound, or processing audio data representing a sound to provide output data that categorizes a content of the sound into one or more of a plurality of categories”, “or wherein the image data comprises pixels of an image or of video, and wherein the image processing task comprises: processing the image data to provide output data that identifies the location of one or more objects in the image or video, or processing the image data to provide output data that segments pixels of the image or video into regions that represent one or more objects in the image or video signal, or processing the image data to provide output data that categorizes a content of the image or video into one or more of a plurality of categories, or processing the image data to provide output data that predicts depth values for pixels of the image or video; or wherein the image data comprises pixels of a video and the image processing task comprises: processing the image data to provide output data that identifies the location of one or more actions represented in the video, or processing the image data to provide output data that categorizes one or more actions represented in the video into one or more of a plurality of categories”, “or wherein the multimodal data comprises audio-visual data comprising a combination of pixels of an image or of video and audio data representing values of an audio waveform, and wherein the multimodal processing task comprises: processing the combination to provide output data that detects presence of a particular event in the combination, or processing the combination to provide output data that categorizes the combination into one or more of a plurality of categories”, and “or wherein the multimodal data comprises a combination of text data representing text in a natural language and pixels of an image or of video or audio data representing values of an audio waveform, and wherein the multimodal processing task comprises: processing the combination to provide output data that defines whether the image or video or audio waveform is described by the text” is/are directed to particular field(s) of use (image, audio, and multimodal data) (MPEP 2106.05(h)) and therefore do not provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Accordingly, at Step 2B, the additional elements individually or in combination do not amount to significantly more than the judicial exception.
Regarding Claim 23:
Due to claim language similar to that of Claim 1, Claim 23 is rejected for the same reasons as presented above in the rejection of Claim 1, with the exception of the limitation(s) covered below.
Step 2A: Prong 2 analysis:
The judicial exception is not integrated into practical application. In particular, the claim recites the additional elements of:
“One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a neural network”. This additional element is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (storage and processor) (See MPEP 2106.05(f)).
Accordingly at Step 2A: Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B analysis:
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
As discussed above, the additional element(s) of “One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a neural network” is/are recited at a high-level of generality such that it/they amount(s) to no more than mere instructions to apply the exception using a generic computer component (storage and processor) (See MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional elements individually or in combination do not amount to significantly more than the judicial exception.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 4, 8, 10-13, 17, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wei et al (Wei, F., Gao, Y., Wu, Z., Hu, H., & Lin, S. (2021). Aligning Pretraining for Detection via Object-Level Contrastive Learning. arXiv [Cs.CV]. Retrieved from http://arxiv.org/abs/2106.02637, hereinafter Wei), in view of Villar-Corrales et al (Villar-Corrales, A., & Behnke, S. (2022). Unsupervised Image Decomposition with Phase-Correlation Networks. arXiv [Cs.CV]. Retrieved from http://arxiv.org/abs/2110.03473, hereinafter Corrales), and in view of Grill et al (Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., … Valko, M. (2020). Bootstrap your own latent: A new approach to self-supervised Learning. arXiv [Cs.LG]. Retrieved from http://arxiv.org/abs/2006.07733, hereinafter Grill).
Regarding Claim 1:
Wei teaches
A computer-implemented method of training a neural network, the method comprising, for a plurality of training data items: processing the training data item using an object discovery neural network to generate an object segmentation of the training data item, wherein the object segmentation defines a segmentation of the training data item into a plurality of different objects represented by the training data item (Wei [Page 3, Section 3.2, Object Proposal Generation, par. 1]: "Inspired by R-CNN [34] and Fast R-CNN [35], we use selective search [14], an unsupervised object proposal generation algorithm which takes into account color similarity, texture similarity, size of region and fit between regions, to generate a set of object proposals for each of the raw images. We represent each object proposal as a bounding box b = {x,y,w,h}, where (x,y) denotes the coordinates of the bounding box center, and w and h are the corresponding width and height, respectively");
processing a first transformed view of the training data item with a first feature representation neural network to generate a first representation of the first transformed view (Wei [Page 4, section 3.3, Aligning Pretraining Architecture to Object Detection, par. 1]: "Following Mask R-CNN, we use a backbone with FPN as the image-level feature extractor fI. We denote the output of FPN as {P2,P3,P4,P5} with a stride of {4,8,16,32}. Here, we do not use P6 due to its low resolution. With the bounding box representation b, RoIAlign [17] is applied to extract the foreground feature from the corresponding scale level. For further architectural alignment, we additionally introduce an R-CNN head fH into pretraining. The object-level feature representation h of bounding box b is extracted from an image view V as: {Eqn. 1}"; [Figure 1 caption]: "We construct three views {V1,V2,V3} where the scales and locations of the same object are different. We adopt a backbone with FPN to encode image-level features and RoIAlign to extract object-level features.");
processing a second transformed view of the training data item with a second feature representation neural network to generate a second representation of the second transformed view (Wei [Page 4, section 3.3, Aligning Pretraining Architecture to Object Detection, par. 1]: "Following Mask R-CNN, we use a backbone with FPN as the image-level feature extractor fI. We denote the output of FPN as {P2,P3,P4,P5} with a stride of {4,8,16,32}. Here, we do not use P6 due to its low resolution. With the bounding box representation b, RoIAlign [17] is applied to extract the foreground feature from the corresponding scale level. For further architectural alignment, we additionally introduce an R-CNN head fH into pretraining. The object-level feature representation h of bounding box b is extracted from an image view V as: {Eqn. 1}"; [Figure 1 caption]: "We construct three views {V1,V2,V3} where the scales and locations of the same object are different. We adopt a backbone with FPN to encode image-level features and RoIAlign to extract object-level features."; (EN): since Wei utilizes different views {V1, V2, V3}, it teaches the processing of a second transformed view);
and updating parameters of the object discovery neural network based on the parameters of the first feature representation neural network (Wei [Figure 1 caption]: "Object proposals are assigned to different pyramid levels according to their image areas. Contrastive learning is performed at the object level to learn translation-invariant and scale-invariant representations. The target network is updated by an exponential moving average of the online network").
Wei does not distinctly disclose
combining the object segmentation and the first representation to generate a first object representation for each of the objects in the first transformed view;
combining the object segmentation and the second representation to generate a second object representation for each of the objects in the second transformed view;
However, Corrales teaches
combining the object segmentation and the first representation to generate a first object representation for each of the objects in the first transformed view (Corrales [Page 5, section 3.3, par. 2]: "The number of possible object combinations grows exponentially with the maximum number of objects and the number of object candidates (TNmax), which quickly makes it infeasible to evaluate all possible combinations. Therefore, similarly to (Monnier et al., 2021), we propose a greedy algorithm that selects in a sequential manner the objects that minimize the reconstruction loss."; [Page 5, section 3.3, par. 3]: "Then, for each subsequent iteration, we greedily select the object that, combined with the previously selected ones, minimizes the reconstruction error. This error is computed using Equation (4), which corresponds to the mean squared error between the input image (I) and a combination of the selected candidates (G(T ))");
combining the object segmentation and the second representation to generate a second object representation for each of the objects in the second transformed view (Corrales [Page 5, section 3.3, par. 2]: "The number of possible object combinations grows exponentially with the maximum number of objects and the number of object candidates (TNmax), which quickly makes it infeasible to evaluate all possible combinations. Therefore, similarly to (Monnier et al., 2021), we propose a greedy algorithm that selects in a sequential manner the objects that minimize the reconstruction loss."; [Page 5, section 3.3, par. 3]: "Then, for each subsequent iteration, we greedily select the object that, combined with the previously selected ones, minimizes the reconstruction error. This error is computed using Equation (4), which corresponds to the mean squared error between the input image (I) and a combination of the selected candidates (G(T ))"; (EN): since the method of Corrales iteratively combines all objects, it is reasonable to assume that the multiple views of Wei would be combined);
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine the concept of object level contrastive learning for object detection of Wei with the techniques for unsupervised image composition of Corrales in order to provide a method for combining multiple object segmentations into transformed views (Corrales [Page 5, section 3.3, par. 2]: "The number of possible object combinations grows exponentially with the maximum number of objects and the number of object candidates (TNmax), which quickly makes it infeasible to evaluate all possible combinations. Therefore, similarly to (Monnier et al., 2021), we propose a greedy algorithm that selects in a sequential manner the objects that minimize the reconstruction loss."; [Page 5, section 3.3, par. 3]: "Then, for each subsequent iteration, we greedily select the object that, combined with the previously selected ones, minimizes the reconstruction error. This error is computed using Equation (4), which corresponds to the mean squared error between the input image (I) and a combination of the selected candidates (G(T ))").
Wei + Corrales does not distinctly disclose
updating parameters of the first feature representation neural network by comparing a predicted object representation dependent upon the first object representation, and the second object representation;
updating parameters of the second feature representation neural network based on the parameters of the first feature representation neural network;
However, Grill teaches
updating parameters of the first feature representation neural network by comparing a predicted object representation dependent upon the first object representation, and the second object representation (Grill [Page 3, section 3, par. 2]: "This experimental finding is the core motivation for BYOL: from a given representation, referred to as target, we can train a new, potentially enhanced representation, referred to as online, by predicting the target representation."; [Figure 2 caption]: "BYOL’s architecture. BYOL minimizes a similarity loss between qθ(zθ) and sg(zξ), where θ are the trained weights, ξ are an exponential moving average of θ and sg means stop-gradient. At the end of training, everything but fθ is discarded, and yθ is used as the image representation."; [Page 16, section A]: it can be seen in algorithm A that both online parameters and target parameters are being updated; [Page 35, section J, J.5]: it can be seen in the pseudocode for the update function that both the online network and target network are being updated.);
updating parameters of the second feature representation neural network based on the parameters of the first feature representation neural network (Grill [Page 16, section A]: it can be seen in algorithm A that both online parameters and target parameters are being updated; [Page 35, section J, J.5]: it can be seen in the pseudocode for the update function that both the online network and target network are being updated.);
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine the concept of object level contrastive learning for object detection of Wei + Corrales with the techniques for self-supervised learning of Grill in order to provide a method for image representation learning that utilizes an online and a target neural network (Grill [Page 3, section 3, par. 2]: "This experimental finding is the core motivation for BYOL: from a given representation, referred to as target, we can train a new, potentially enhanced representation, referred to as online, by predicting the target representation."; [Figure 2 caption]: "BYOL’s architecture. BYOL minimizes a similarity loss between qθ(zθ) and sg(zξ), where θ are the trained weights, ξ are an exponential moving average of θ and sg means stop-gradient. At the end of training, everything but fθ is discarded, and yθ is used as the image representation.").
Regarding Claim 4:
Wei teaches
The method of claim 1, wherein the object discovery neural network comprises an object discovery feature representation neural network to process the training data item to generate a feature map of the training data item for generating the object segmentation, wherein the object discovery feature representation neural network has the same architecture as the first feature representation neural network (Wei [Page 5, section 4.1, Architecture, par. 1]: "RoIAlign [17] is then used to extract RoI features on feature maps {P2,P3,P4,P5} with a stride of {4,8,16,32}. According to the image areas of object proposals, each RoI feature is then transformed to an object-level representation by the head network as in Mask R-CNN. SoCo-R50-C4: on the standard ResNet-50 architecture, we insert the RoI operation on the output of the 4-th residual block. The entire 5-th residual block is treated as the head network to encode object-level features. Both the projection network and prediction network are 2-layer MLPs which consist of a linear layer with output size 4096 followed by batch normalization [37], rectified linear units (ReLU) [38], and a final linear layer with output dimension 256");
the method further comprising: processing the training data item using the object discovery feature representation neural network to generate the feature map of the training data item (Wei [Page 5, section 4.1, Architecture, par. 1]: "To demonstrate the extensibility and flexibility of SoCo, we provide details of SoCo alignment for the detection architectures R50-FPN and R50 C4. SoCo-R50-FPN: ResNet-50 [36] with FPN [15] is used as the image-level feature encoder. RoIAlign [17] is then used to extract RoI features on feature maps {P2,P3,P4,P5} with a stride of {4,8,16,32}.");
using the feature map of the training data item to generate the object segmentation of the training data item (Wei [Page 5, section 4.1, Architecture, par. 1]: "RoIAlign [17] is then used to extract RoI features on feature maps {P2,P3,P4,P5} with a stride of {4,8,16,32}. According to the image areas of object proposals, each RoI feature is then transformed to an object-level representation by the head network as in Mask R-CNN.");
and updating the parameters of the object discovery feature representation neural network to approach or equal values of the parameters of the first feature representation neural network (Wei [Figure 1 caption]: "Object proposals are assigned to different pyramid levels according to their image areas. Contrastive learning is performed at the object level to learn translation-invariant and scale-invariant representations. The target network is updated by an exponential moving average of the online network.").
Regarding Claim 8:
Wei teaches
The method of claim 1 wherein updating parameters of the object discovery neural network based on the parameters of the first feature representation neural network comprises updating parameters of the object discovery neural network with a moving average of the parameters of the first feature representation neural network (Wei [Figure 1 caption]: "Object proposals are assigned to different pyramid levels according to their image areas. Contrastive learning is performed at the object level to learn translation-invariant and scale-invariant representations. The target network is updated by an exponential moving average of the online network.").
Regarding Claim 10:
Wei teaches
The method of claim 1, wherein the method is performed iteratively by: processing a first training data item using the object discovery neural network to generate the object segmentation of the first training data item (Wei [Page 3, Section 3.2, Object Proposal Generation, par. 1]: "Inspired by R-CNN [34] and Fast R-CNN [35], we use selective search [14], an unsupervised object proposal generation algorithm which takes into account color similarity, texture similarity, size of region and fit between regions, to generate a set of object proposals for each of the raw images. We represent each object proposal as a bounding box b = {x,y,w,h}, where (x,y) denotes the coordinates of the bounding box center, and w and h are the corresponding width and height, respectively");
generating the first object representation and the second object representation from the first training data item using the object segmentation of the first training data item (Wei [Page 4, section 3.3, Aligning Pretraining Architecture to Object Detection, par. 1]: "Following Mask R-CNN, we use a backbone with FPN as the image-level feature extractor fI. We denote the output of FPN as {P2,P3,P4,P5} with a stride of {4,8,16,32}. Here, we do not use P6 due to its low resolution. With the bounding box representation b, RoIAlign [17] is applied to extract the foreground feature from the corresponding scale level. For further architectural alignment, we additionally introduce an R-CNN head fH into pretraining. The object-level feature representation h of bounding box b is extracted from an image view V as: {Eqn. 1}"; [Figure 1 caption]: "We construct three views {V1,V2,V3} where the scales and locations of the same object are different. We adopt a backbone with FPN to encode image-level features and RoIAlign to extract object-level features.");
updating the parameters of the object discovery neural network based on the updated parameters of the first feature representation neural network (Wei [Figure 1 caption]: "Object proposals are assigned to different pyramid levels according to their image areas. Contrastive learning is performed at the object level to learn translation-invariant and scale-invariant representations. The target network is updated by an exponential moving average of the online network.");
processing a second training data item using the object discovery neural network, after updating the parameters of the object discovery neural network, to generate the object segmentation of the second training data item (Wei [Page 4, section 3.3, Aligning Pretraining Architecture to Object Detection, par. 1]: "Following Mask R-CNN, we use a backbone with FPN as the image-level feature extractor fI. We denote the output of FPN as {P2,P3,P4,P5} with a stride of {4,8,16,32}. Here, we do not use P6 due to its low resolution. With the bounding box representation b, RoIAlign [17] is applied to extract the foreground feature from the corresponding scale level. For further architectural alignment, we additionally introduce an R-CNN head fH into pretraining. The object-level feature representation h of bounding box b is extracted from an image view V as: {Eqn. 1}"; [Figure 1 caption]: "We construct three views {V1,V2,V3} where the scales and locations of the same object are different. We adopt a backbone with FPN to encode image-level features and RoIAlign to extract object-level features.");
generating the first object representation and the second object representation from the second training data item using the object segmentation of the second training data item (Wei [Page 4, section 3.3, Aligning Pretraining Architecture to Object Detection, par. 1]: "Following Mask R-CNN, we use a backbone with FPN as the image-level feature extractor fI. We denote the output of FPN as {P2,P3,P4,P5} with a stride of {4,8,16,32}. Here, we do not use P6 due to its low resolution. With the bounding box representation b, RoIAlign [17] is applied to extract the foreground feature from the corresponding scale level. For further architectural alignment, we additionally introduce an R-CNN head fH into pretraining. The object-level feature representation h of bounding box b is extracted from an image view V as: {Eqn. 1}"; [Figure 1 caption]: "We construct three views {V1,V2,V3} where the scales and locations of the same object are different. We adopt a backbone with FPN to encode image-level features and RoIAlign to extract object-level features.");
Wei does not distinctly disclose
updating the parameters of the first feature representation neural network by comparing the predicted object representation and the second object representation from the first training data item;
updating the parameters of the second feature representation neural network;
updating the parameters of the first feature representation neural network by comparing the predicted object representation and the second object representation from the second training data item;
updating the parameters of the second feature representation neural network.
However, Grill teaches
updating the parameters of the first feature representation neural network by comparing the predicted object representation and the second object representation from the first training data item (Grill [Page 3, section 3, par. 2]: "This experimental finding is the core motivation for BYOL: from a given representation, referred to as target, we can train a new, potentially enhanced representation, referred to as online, by predicting the target representation."; [Figure 2 caption]: "BYOL’s architecture. BYOL minimizes a similarity loss between qθ(zθ) and sg(zξ), where θ are the trained weights, ξ are an exponential moving average of θ and sg means stop-gradient. At the end of training, everything but fθ is discarded, and yθ is used as the image representation."; [Page 16, section A]: it can be seen in algorithm A that both online parameters and target parameters are being updated; [Page 35, section J, J.5]: it can be seen in the pseudocode for the update function that both the online network and target network are being updated.);
updating the parameters of the second feature representation neural network (Grill [Page 16, section A]: it can be seen in algorithm A that both online parameters and target parameters are being updated; [Page 35, section J, J.5]: it can be seen in the pseudocode for the update function that both the online network and target network are being updated.);
updating the parameters of the first feature representation neural network by comparing the predicted object representation and the second object representation from the second training data item (Grill [Page 3, section 3, par. 2]: "This experimental finding is the core motivation for BYOL: from a given representation, referred to as target, we can train a new, potentially enhanced representation, referred to as online, by predicting the target representation."; [Figure 2 caption]: "BYOL’s architecture. BYOL minimizes a similarity loss between qθ(zθ) and sg(zξ), where θ are the trained weights, ξ are an exponential moving average of θ and sg means stop-gradient. At the end of training, everything but fθ is discarded, and yθ is used as the image representation."; [Page 16, section A]: it can be seen in algorithm A that both online parameters and target parameters are being updated; [Page 35, section J, J.5]: it can be seen in the pseudocode for the update function that both the online network and target network are being updated.);
updating the parameters of the second feature representation neural network (Grill [Page 16, section A]: it can be seen in algorithm A that both online parameters and target parameters are being updated; [Page 35, section J, J.5]: it can be seen in the pseudocode for the update function that both the online network and target network are being updated.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine the concept of object level contrastive learning for object detection of Wei with the techniques for self-supervised learning of Grill in order to provide a method for image representation learning that utilizes an online and a target neural network (Grill [Page 3, section 3, par. 2]: "This experimental finding is the core motivation for BYOL: from a given representation, referred to as target, we can train a new, potentially enhanced representation, referred to as online, by predicting the target representation."; [Figure 2 caption]: "BYOL’s architecture. BYOL minimizes a similarity loss between qθ(zθ) and sg(zξ), where θ are the trained weights, ξ are an exponential moving average of θ and sg means stop-gradient. At the end of training, everything but fθ is discarded, and yθ is used as the image representation.").
Regarding Claim 11:
Wei teaches
The method of claim 1, wherein the first feature representation neural network and the second feature representation neural network have the same architecture and different parameters (Wei [Page 5, section 4.1, Architecture, par. 1]: "RoIAlign [17] is then used to extract RoI features on feature maps {P2,P3,P4,P5} with a stride of {4,8,16,32}. According to the image areas of object proposals, each RoI feature is then transformed to an object-level representation by the head network as in Mask R-CNN. SoCo-R50-C4: on the standard ResNet-50 architecture, we insert the RoI operation on the output of the 4-th residual block. The entire 5-th residual block is treated as the head network to encode object-level features. Both the projection network and prediction network are 2-layer MLPs which consist of a linear layer with output size 4096 followed by batch normalization [37], rectified linear units (ReLU) [38], and a final linear layer with output dimension 256")
Wei does not distinctly disclose
and wherein updating the parameters of the second feature representation neural network comprises updating the parameters to approach values of the parameters of the first feature representation neural network.
However, Grill teaches
and wherein updating the parameters of the second feature representation neural network comprises updating the parameters to approach values of the parameters of the first feature representation neural network (Grill [Page 16, section A]: it can be seen in algorithm A that both online parameters and target parameters are being updated; [Page 35, section J, J.5]: it can be seen in the pseudocode for the update function that both the online network and target network are being updated.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine the concept of object level contrastive learning for object detection of Wei with the techniques for self-supervised learning of Grill in order to provide a method for image representation learning that utilizes an online and a target neural network (Grill [Page 3, section 3, par. 2]: "This experimental finding is the core motivation for BYOL: from a given representation, referred to as target, we can train a new, potentially enhanced representation, referred to as online, by predicting the target representation."; [Figure 2 caption]: "BYOL’s architecture. BYOL minimizes a similarity loss between qθ(zθ) and sg(zξ), where θ are the trained weights, ξ are an exponential moving average of θ and sg means stop-gradient. At the end of training, everything but fθ is discarded, and yθ is used as the image representation.").
Regarding Claim 12:
Wei does not distinctly disclose
The method of claim 1, further comprising: processing the first object representation using a prediction neural network to generate the predicted object representation;
and updating parameters of the prediction neural network by comparing the predicted object representation and the second object representation.
However, Grill teaches
The method of claim 1, further comprising: processing the first object representation using a prediction neural network to generate the predicted object representation (Grill [Page 3, section 3.1, par. 1]: "The online network is defined by a set of weights θ and is comprised of three stages: an encoder fθ, a projector gθ and a predictor qθ, as shown in Figure 2 and Figure 8. The target network has the same architecture as the online network, but uses a different set of weights ξ."; [Page 3, section 3.1, par. 2]: "We then output a prediction qθ(zθ) of zξ and 2-normalize both qθ(zθ) and zξ to qθ(zθ) = ∆ qθ(zθ)/ qθ(zθ) 2 and zξ = ∆ zξ/ zξ 2. Note that this predictor is only applied to the online branch, making the architecture asymmetric between the online and target pipeline.");
and updating parameters of the prediction neural network by comparing the predicted object representation and the second object representation (Grill [Page 4, section 3.1, par. 2]: "Finally we define the following mean squared error between the normalized predictions and target projections, {Eqn. 2}. We symmetrize the loss Lθ,ξ in Eq.2 by separately feeding v to the online network and v to the target network to compute Lθ,ξ.").
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine the concept of object level contrastive learning for object detection of Wei with the techniques for self-supervised learning of Grill in order to provide a method for image representation learning that utilizes an online and a target neural network (Grill [Page 3, section 3, par. 2]: "This experimental finding is the core motivation for BYOL: from a given representation, referred to as target, we can train a new, potentially enhanced representation, referred to as online, by predicting the target representation."; [Figure 2 caption]: "BYOL’s architecture. BYOL minimizes a similarity loss between qθ(zθ) and sg(zξ), where θ are the trained weights, ξ are an exponential moving average of θ and sg means stop-gradient. At the end of training, everything but fθ is discarded, and yθ is used as the image representation.").
Regarding Claim 13: Wei does not distinctly disclose
The method of claim 1, wherein the predicted object representation is the first object representation.
However, Grill teaches
The method of claim 1, wherein the predicted object representation is the first object representation (Grill [Figure 2]: it can be seen in Figure 2 that the predicted object representation is the first object representation).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine the concept of object level contrastive learning for object detection of Wei with the techniques for self-supervised learning of Grill in order to provide a method for image representation learning that utilizes an online and a target neural network (Grill [Page 3, section 3, par. 2]: "This experimental finding is the core motivation for BYOL: from a given representation, referred to as target, we can train a new, potentially enhanced representation, referred to as online, by predicting the target representation."; [Figure 2 caption]: "BYOL’s architecture. BYOL minimizes a similarity loss between qθ(zθ) and sg(zξ), where θ are the trained weights, ξ are an exponential moving average of θ and sg means stop-gradient. At the end of training, everything but fθ is discarded, and yθ is used as the image representation.").
Regarding Claim 17:
Wei teaches
The method of claim 1 wherein the training data item comprises an audio data item, an image data item, a multimodal data item, a text data item, or a graph data item (Wei [Page 1, section 1, par. 2]: "The goal of this work is to develop self-supervised pretraining that is aligned to object detection. In object detection, bounding boxes are widely adopted as the representation for objects. Translation and scale invariance for object detection are reflected by the location and size of the bounding boxes. An obvious representation gap exists between image-level pretraining and the object-level bounding boxes of object detection.");
the method further comprising, after the training, using i) the object discovery neural network, or ii) at least part of the first feature representation neural network or iii) at least part of the second feature representation neural network, correspondingly, to process input data comprising audio data, image data, multimodal data, text data, or graph data respectively to perform an audio signal processing task, an image processing task, a multimodal processing task, a text processing task, or a graph processing task (Wei [Page 3, Section 3.2, Object Proposal Generation, par. 1]: "Inspired by R-CNN [34] and Fast R-CNN [35], we use selective search [14], an unsupervised object proposal generation algorithm which takes into account color similarity, texture similarity, size of region and fit between regions, to generate a set of object proposals for each of the raw images. We represent each object proposal as a bounding box b = {x,y,w,h}, where (x,y) denotes the coordinates of the bounding box center, and w and h are the corresponding width and height, respectively").
Regarding Claim 18:
Wei teaches
The method of claim 17, wherein the audio data comprises data representing values of an audio waveform, and wherein the audio signal processing task comprises: processing audio data representing speech to provide output data that detects words or phonemes in the speech or categorizes words, syllables or phonemes in the speech into one or more of a plurality of categories, or processing audio data representing a sound to provide output data that detects presence of a particular sound or event in the sound, or processing audio data representing a sound to provide output data that categorizes a content of the sound into one or more of a plurality of categories; or wherein the image data comprises pixels of an image or of video, and wherein the image processing task comprises: processing the image data to provide output data that identifies the location of one or more objects in the image or video, or processing the image data to provide output data that segments pixels of the image or video into regions that represent one or more objects in the image or video signal, or processing the image data to provide output data that categorizes a content of the image or video into one or more of a plurality of categories, or processing the image data to provide output data that predicts depth values for pixels of the image or video; or wherein the image data comprises pixels of a video and the image processing task comprises: processing the image data to provide output data that identifies the location of one or more actions represented in the video, or processing the image data to provide output data that categorizes one or more actions represented in the video into one or more of a plurality of categories; or wherein the multimodal data comprises audio-visual data comprising a combination of pixels of an image or of video and audio data representing values of an audio waveform, and wherein the multimodal processing task comprises: processing the combination to provide output data that detects presence of a particular event in the combination, or processing the combination to provide output data that categorizes the combination into one or more of a plurality of categories; or wherein the multimodal data comprises a combination of text data representing text in a natural language and pixels of an image or of video or audio data representing values of an audio waveform, and wherein the multimodal processing task comprises: processing the combination to provide output data that defines whether the image or video or audio waveform is described by the text (Wei [Figure 1 caption]: "Overview of SoCo. SoCo utilizes selective search to generate a set of object proposals foreach raw image. K proposals are randomly selected in each training step. We construct three views {V1,V2,V3} where the scales and locations of the same object are different. We adopt a backbone with FPN to encode image-level features and RoIAlign to extract object-level features. Object proposals are assigned to different pyramid levels according to their image areas. Contrastive learning is performed at the object level to learn translation-invariant and scale-invariant representations. The target network is updated by an exponential moving average of the online network."; (EN): the cited excerpt of Wei applies specifically to the image data section of the claim).
Claim Rejections - 35 USC § 103
Claim(s) 22 and 23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wei + Corrales + Grill as applied to claim 1 above, and further in view of Wang et al (US 12579652 B1, hereinafter Wang).
Regarding Claim 22:
Due to claim language similar to that of Claim 1, Claim 22 is rejected for the same reasons as presented above in the rejection of Claim 1, with the exception of the limitation(s) covered below.
Wei + Corrales + Grill does not distinctly disclose
A system comprising: one or more computers (Wang [Col 12 lines 38-40]: "The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device");
and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations (Wang [Col 12 lines 38-40]: "The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device")
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine the concept of object level contrastive learning for object detection of Wei + Corrales + Grill with the techniques for robotic picking of items based on a reference image of Wang in order to provide a method for image segmentation that utilizes modern computing hardware to perform the task(s) (Wang [Abstract]: " An instance segmentation operation is performed on the image of the picking environment to determine an instance of the first item within the image of the picking environment, based on the reference image and the image of the picking environment. A grasp plan for a robotic arm end-of-arm tool is generated based on segmentation information output from the instance segmentation operation and the item attributes.").
Regarding Claim 23:
Due to claim language similar to that of Claims 1 and 22, Claim 23 is rejected for the same reasons as presented above in the rejection of Claims 1 and 22.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20210383225 A1 – computer programs on one or more computers in one or more locations that is configured to learn a representation of data item such as an image through a self-supervised learning process
Pot, E., Toshev, A., & Kosecka, J. (2018). Self-supervisory Signals for Object Discovery and Detection. arXiv [Cs.CV]. Retrieved from http://arxiv.org/abs/1806.03370 – we propose to leverage the motion of a robotic agent exploring an environment as main means of self-supervision, and show how to discover novel objects and learn effective detection models for them with no or very limited labelling effort
T. Zhou, J. Li, X. Li and L. Shao, "Target-Aware Object Discovery and Association for Unsupervised Video Multi-Object Segmentation," 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 2021, pp. 6981-6990, doi: 10.1109/CVPR46437.2021.00691. – we introduce a novel approach for more accurate and efficient spatio-temporal segmentation
Any inquiry concerning this communication or earlier communications from the examiner should be directed to COREY M SACKALOSKY whose telephone number is (703)756-1590. The examiner can normally be reached M-F 7:30am-3:30pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas can be reached at (571) 272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/COREY SACKALOSKY/Examiner, Art Unit 2128
/OMAR F FERNANDEZ RIVAS/Supervisory Patent Examiner, Art Unit 2128
Prosecution Insights