Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 10-18 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Re claim 10, claim 10 recites “A computing system comprising: one or more processors; and instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising”. The examiner notes that systems are defined by their structure. The examiner note that “instructions” is not a structural element and it is unclear what the structure of “instructions” is. The examiner suggests amending the claims to include a non transitory computer readable medium storing instructions.
Claims 11-18 are unclear because they depend from claim 10.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claim 1, 4-7, 10, 13-16 and 19 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1, of U.S. Patent No. US 11354778 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because recite all the features of the instant claim.
Re claim 1, claim 1 of the patent discloses
A computer-implemented method, comprising (see claim 1 “cause the computing system to perform operations, the operations comprising” note that the operations are a computer implemented method):
obtaining one or more training inputs (see claim 1 “obtaining a training image”);
performing one or more first augmentation operations on at least one of the training inputs to obtain a first augmented input (see claim 1 “performing a plurality of first augmentation operations on the training image to obtain a first augmented image”);
separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the at least one of the training inputs to obtain a second augmented input (see claim 1 “separate from performing the plurality of first augmentation operations, performing a plurality of second augmentation operations on the training image to obtain a second augmented image”);
wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input (see claim 1 “wherein the plurality of first augmentation operations comprise at least a first random crop operation that randomly crops the training image and a first random color distortion operation that randomly modifies color values of the training image to the training image” and “wherein the plurality of second augmentation operations comprise at least a second random crop operation that randomly crops the training image and a second random color distortion operation that randomly modifies color values of the training image”);
computing a contrastive loss value between a first representation based on the first augmented input and a second representation based on the second augmented input (see claim 1 “evaluating a loss function that evaluates a difference between the first projected representation and the second projected representation” and “respectively processing, with the base encoder neural network, the first augmented image and the second augmented image to respectively generate a first intermediate representation for the first augmented image and a second intermediate representation for the second augmented image; respectively processing, with the projection head neural network, the first intermediate representation and the second intermediate representation to respectively obtain a first projected representation for the first augmented image and a second projected representation for the second augmented image” note that a projected representation is for both.),
wherein computing the contrastive loss value comprises: processing, with a base encoder neural network, the first augmented input to generate a first intermediate representation for the first augmented input; (see claim 1 “respectively processing, with the base encoder neural network, the first augmented image and the second augmented image to respectively generate a first intermediate representation for the first augmented image and a second intermediate representation for the second augmented image” note that a first intermediate representation is generated )
and processing, with a projection head neural network, the first intermediate representation to obtain the first representation (see claim 1 “respectively processing, with the projection head neural network, the first intermediate representation and the second intermediate representation to respectively obtain a first projected representation for the first augmented image and a second projected representation for the second augmented image”);
and modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the contrastive loss value. (see claim 1 ”modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the loss function” note that the loss function is between the first and second projected representation of the image and is therefore contrastive.)
Re claim 4 claim 1 of the patent discloses wherein the projection head neural network transforms the first intermediate representation using a nonlinear transformation. (see claim 1 “a projection head neural network configured to process the intermediate representation of the input image to generate a projected representation of the input image, wherein to generate the projected representation, the projection head neural network is configured to perform at least one non-linear transformation”)
Re claim 5 claim 8 discloses wherein the projection head neural network comprises a multilayer perceptron (see claim 8 “wherein the projection head neural network comprises a multi-layer perceptron that comprises one hidden layer and a rectified linear unit non-linear activation function.”).
Re claim 6, claim 6 of the patent discloses wherein the one or more first augmentation operations comprise a blur operation. (see claim 6 “wherein the plurality of first augmentation operations further comprises a first random Gaussian blur operation that randomly applies a Gaussian blur to the training image and wherein the plurality of second augmentation operations further comprises a second random Gaussian blur operation that randomly applies a Gaussian blur to the training image.”)
Re claim 7, claim 7 of the patent discloses wherein the blur operation applies a Gaussian blur based on a randomly sampled value (see claim 6 “wherein the plurality of first augmentation operations further comprises a first random Gaussian blur operation that randomly applies a Gaussian blur to the training image and wherein the plurality of second augmentation operations further comprises a second random Gaussian blur operation that randomly applies a Gaussian blur to the training image”).
Re claim 10 claim 1 of the patent discloses
A computing system comprising: one or more processors; and instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising: ( see claim 1 “A computing system to perform contrastive learning of visual representations, the computing system comprising: one or more processors; and one or more non-transitory computer-readable media that collectively store”)
obtaining one or more training inputs (see claim 1 “obtaining a training image”);
performing one or more first augmentation operations on at least one of the training inputs to obtain a first augmented input (see claim 1 “performing a plurality of first augmentation operations on the training image to obtain a first augmented image”);
separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the at least one of the training inputs to obtain a second augmented input (see claim 1 “separate from performing the plurality of first augmentation operations, performing a plurality of second augmentation operations on the training image to obtain a second augmented image”);
wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input (see claim 1 “wherein the plurality of first augmentation operations comprise at least a first random crop operation that randomly crops the training image and a first random color distortion operation that randomly modifies color values of the training image to the training image” and “wherein the plurality of second augmentation operations comprise at least a second random crop operation that randomly crops the training image and a second random color distortion operation that randomly modifies color values of the training image”);
computing a contrastive loss value between a first representation based on the first augmented input and a second representation based on the second augmented input (see claim 1 “evaluating a loss function that evaluates a difference between the first projected representation and the second projected representation” and “respectively processing, with the base encoder neural network, the first augmented image and the second augmented image to respectively generate a first intermediate representation for the first augmented image and a second intermediate representation for the second augmented image; respectively processing, with the projection head neural network, the first intermediate representation and the second intermediate representation to respectively obtain a first projected representation for the first augmented image and a second projected representation for the second augmented image” note that a projected representation is for both. ),
wherein computing the contrastive loss value comprises: processing, with a base encoder neural network, the first augmented input to generate a first intermediate representation for the first augmented input; (see claim 1 “respectively processing, with the base encoder neural network, the first augmented image and the second augmented image to respectively generate a first intermediate representation for the first augmented image and a second intermediate representation for the second augmented image” note that a first intermediate representation is generated )
and processing, with a projection head neural network, the first intermediate representation to obtain the first representation (see claim 1 “respectively processing, with the projection head neural network, the first intermediate representation and the second intermediate representation to respectively obtain a first projected representation for the first augmented image and a second projected representation for the second augmented image”);
and modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the contrastive loss value. (see claim 1 ”modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the loss function” note that the loss function is between the first and second projected representation of the image and is therefore contrastive.)
Re claim 13 claim 1 of the patent discloses wherein the projection head neural network transforms the first intermediate representation using a nonlinear transformation. (see claim 1 “a projection head neural network configured to process the intermediate representation of the input image to generate a projected representation of the input image, wherein to generate the projected representation, the projection head neural network is configured to perform at least one non-linear transformation”)
Re claim 14 claim 8 discloses wherein the projection head neural network comprises a multilayer perceptron (see claim 8 “wherein the projection head neural network comprises a multi-layer perceptron that comprises one hidden layer and a rectified linear unit non-linear activation function.”).
Re claim 15, claim 6 of the patent discloses wherein the one or more first augmentation operations comprise a blur operation. (see claim 6 “wherein the plurality of first augmentation operations further comprises a first random Gaussian blur operation that randomly applies a Gaussian blur to the training image and wherein the plurality of second augmentation operations further comprises a second random Gaussian blur operation that randomly applies a Gaussian blur to the training image.”)
Re claim 16, claim 7 of the patent discloses wherein the blur operation applies a Gaussian blur based on a randomly sampled value (see claim 6 “wherein the plurality of first augmentation operations further comprises a first random Gaussian blur operation that randomly applies a Gaussian blur to the training image and wherein the plurality of second augmentation operations further comprises a second random Gaussian blur operation that randomly applies a Gaussian blur to the training image”).
Re claim 19, claim 1 of the patent discloses
One or more non-transitory computer-readable media that collectively store a neural network trained by (see claim 1 “and one or more non-transitory computer-readable media that collectively store: a base encoder neural network configured to process an input image to generate an intermediate representation of the input image; a projection head neural network configured to process the intermediate representation of the input image to generate a projected representation of the input image, wherein to generate the projected representation, the projection head neural network is configured to perform at least one non-linear transformation; and instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising” note that a memory stores the neural networks note the steps below comprise training a network):
obtaining one or more training inputs (see claim 1 “obtaining a training image”);
performing one or more first augmentation operations on at least one of the training inputs to obtain a first augmented input (see claim 1 “performing a plurality of first augmentation operations on the training image to obtain a first augmented image”);
separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the at least one of the training inputs to obtain a second augmented input (see claim 1 “separate from performing the plurality of first augmentation operations, performing a plurality of second augmentation operations on the training image to obtain a second augmented image”);
wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input (see claim 1 “wherein the plurality of first augmentation operations comprise at least a first random crop operation that randomly crops the training image and a first random color distortion operation that randomly modifies color values of the training image to the training image” and “wherein the plurality of second augmentation operations comprise at least a second random crop operation that randomly crops the training image and a second random color distortion operation that randomly modifies color values of the training image”);
computing a contrastive loss value between a first representation based on the first augmented input and a second representation based on the second augmented input (see claim 1 “evaluating a loss function that evaluates a difference between the first projected representation and the second projected representation” and “respectively processing, with the base encoder neural network, the first augmented image and the second augmented image to respectively generate a first intermediate representation for the first augmented image and a second intermediate representation for the second augmented image; respectively processing, with the projection head neural network, the first intermediate representation and the second intermediate representation to respectively obtain a first projected representation for the first augmented image and a second projected representation for the second augmented image” note that a projected representation is for both.),
wherein computing the contrastive loss value comprises: processing, with a base encoder neural network, the first augmented input to generate a first intermediate representation for the first augmented input; (see claim 1 “respectively processing, with the base encoder neural network, the first augmented image and the second augmented image to respectively generate a first intermediate representation for the first augmented image and a second intermediate representation for the second augmented image” note that a first intermediate representation is generated )
and processing, with a projection head neural network, the first intermediate representation to obtain the first representation (see claim 1 “respectively processing, with the projection head neural network, the first intermediate representation and the second intermediate representation to respectively obtain a first projected representation for the first augmented image and a second projected representation for the second augmented image”);
and modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the contrastive loss value. (see claim 1 ”modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the loss function” note that the loss function is between the first and second projected representation of the image and is therefore contrastive.)
Claim 2, and 11 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim of U.S. Patent No. US 11354778 B2 in view of Lin US 20190355102 A1.
Re claim 2, claim 1 does not expressly disclose the encoder neural network comprises an attention mechanism. Lin discloses the encoder neural network comprises an attention mechanism. (see paragraph 70 note the encoder contains an attention layer) The motivation to combine is see paragraph 71 “As discussed in the following, the contextual attention pathway of the described system overcomes this limitation—the contextual attention pathway is capable of identifying features for content generation that are from distant spatial locations of the image under consideration”. One of ordinary skill in the art could have easily combine claim 1 with Lin to add an attention layer to the encoders of claim 1. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 and Lin to reach the aforementioned advantage.
Re claim 11, claim 1 does not expressly disclose the encoder neural network comprises an attention mechanism. Lin discloses the encoder neural network comprises an attention mechanism. (see paragraph 70 note the encoder contains an attention layer) The motivation to combine is see paragraph 71 “As discussed in the following, the contextual attention pathway of the described system overcomes this limitation—the contextual attention pathway is capable of identifying features for content generation that are from distant spatial locations of the image under consideration”. One of ordinary skill in the art could have easily combine claim 1 with Lin to add an attention layer to the encoders of Chen. Therefore, it would have been obvious one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 and Lin to reach the aforementioned advantage.
Claims 8, 9, 17 and 18 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 6 of U.S. Patent No. US 11354778 B2 in view of Bachman et al “Learning Representations by Maximizing Mutal Information Across Views” 2019 arXiv:1906.00910
Re claim 8 Claim 6 does not expressly disclose wherein the one or more first augmentation operations comprise a first resize operation that resizes the training input.
Bachman discloses wherein the one or more first augmentation operations comprise a first resize operation that resizes the training input. (See section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations);
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of claim 6. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 6 with Bachman to reach the aforementioned advantage.
Re claim 9 claim 6 further does not disclose wherein the one or more first augmentation operations comprises a first random flip operation that randomly flips the training input.
Bachman discloses wherein the one or more first augmentation operations comprises a first random flip operation that randomly flips the training input.
(See section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations);
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of claim 6. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 6 with Bachman to reach the aforementioned advantage.
Re claim 17 Claim 6 does not expressly disclose wherein the one or more first augmentation operations comprise a first resize operation that resizes the training input.
Bachman discloses wherein the one or more first augmentation operations comprise a first resize operation that resizes the training input. (see section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations);
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of claim 6. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 6 with Bachman to reach the aforementioned advantage.
Re claim 18 claim 6 further does not disclose wherein the one or more first augmentation operations comprises a first random flip operation that randomly flips the training input.
Bachman discloses wherein the one or more first augmentation operations comprises a first random flip operation that randomly flips the training input.
(See section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations);
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of claim 6. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 6 with Bachman to reach the aforementioned advantage.
Claim 20, 21 23 and 24 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 8 and 6 of U.S. Patent No. 11354778 B2 in view of Chen et al “Improved Baselines with Momentum Contrastive Learning” 9 March 2020 arXiv:2003.04297 .
Re claim 20 Claim 20, Claim 1 Further discloses wherein the one or more non-transitory computer-readable media collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations comprising (see claim 1 “one or more non-transitory computer-readable media that collectively store: a base encoder neural network configured to process an input image to generate an intermediate representation of the input image; a projection head neural network configured to process the intermediate representation of the input image to generate a projected representation of the input image, wherein to generate the projected representation, the projection head neural network is configured to perform at least one non-linear transformation; and instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising” )
Claim 1 does not expressly disclose receiving an input; processing the input using the neural network to generate an output representation from the neural network; and generating, based on the output representation, an output value
Chen discloses: receiving an input; processing the input using the neural network to generate an output representation from the neural network; and generating, based on the output representation, an output value (see section 3 setting “Unsupervised learning is conducted on the 1.28MImageNet [3] training set. We follow two common protocols for evaluation. (i) ImageNet linear classification: features are frozen and a supervised linear classifier is trained (ii) Transferring to VOC object detection [4]: a Faster R-CNN detector [14] (C4-backbone) is fine-tuned end-to-end on the VOC 07+12 trainval set1 and evaluated on the VOC 07 testset using the COCO suite of metrics [10]. We use the same hyper-parameters (except when noted) and codebase as MoCo [6]. All results use a standard-size ResNet-50”). Note that after the unsupervised training a test set is applied to the neural network test a classification using a linear classifier). The motivation is to train a classifier (see section 3 setting “Unsupervised learning is conducted on the 1.28MImageNet [3] training set. We follow two common protocols for evaluation. (i) ImageNet linear classification: features are frozen and a supervised linear classifier is trained (ii) Transferring to VOC object detection [4]: a Faster R-CNN detector [14] (C4-backbone) is fine-tuned end-to-end on the VOC 07+12 trainval set1 and evaluated on the VOC 07 testset using the COCO suite of metrics [10]. We use the same hyper-parameters (except when noted) and codebase as MoCo [6]. All results use a standard-size ResNet-50”). Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Claim 1 and Chen to reach the aforementioned advantage.
Re claim 21 Chen further discloses wherein the input comprises image data, and wherein generating the output value comprises: generating, based on the output representation, at least one of: a classification prediction, a detection prediction, a recognition prediction, or a segmentation prediction. (See section 3 setting “Unsupervised learning is conducted on the 1.28MImageNet [3] training set. We follow two common protocols for evaluation. (i) ImageNet linear classification: features are frozen and a supervised linear classifier is trained (ii) Transferring to VOC object detection [4]: a Faster R-CNN detector [14] (C4-backbone) is fine-tuned end-to-end on the VOC 07+12 trainval set1 and evaluated on the VOC 07 testset using the COCO suite of metrics [10]. We use the same hyper-parameters (except when noted) and codebase as MoCo [6]. All results use a standard-size ResNet-50”). Note that after the unsupervised training a test set is applied to the neural network test a classification using a linear classifier).
Re claim 23 claim 8 discloses wherein the projection head neural network comprises a multilayer perceptron (see claim 8 “wherein the projection head neural network comprises a multi-layer perceptron that comprises one hidden layer and a rectified linear unit non-linear activation function.”).
Re claim 24, claim 6 of the patent discloses wherein the one or more first augmentation operations comprise a blur operation. (see claim 6 “wherein the plurality of first augmentation operations further comprises a first random Gaussian blur operation that randomly applies a Gaussian blur to the training image and wherein the plurality of second augmentation operations further comprises a second random Gaussian blur operation that randomly applies a Gaussian blur to the training image.”)
Claim 22 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. 11354778 B2 in view of Chen et al “Improved Baselines with Momentum Contrastive Learning” 9 March 2020 arXiv:2003.04297 in further view of Lin US 20190355102 A1.
Re claim 22, claim 1 does not expressly disclose the encoder neural network comprises an attention mechanism. Lin discloses the encoder neural network comprises an attention mechanism. (see paragraph 70 note the encoder contains an attention layer) The motivation to combine is see paragraph 71 “As discussed in the following, the contextual attention pathway of the described system overcomes this limitation—the contextual attention pathway is capable of identifying features for content generation that are from distant spatial locations of the image under consideration”. One of ordinary skill in the art could have easily combine claim 1 with Lin to add an attention layer to the encoders of claim 1. Therefore, it would have been obvious before the effective filing date of the claimed invention to combine claim 1, Chen and Lin to reach the aforementioned advantage.
Claims 1, 3, 10 and 12 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. US 11386302 B2 in view of Bachman et al “Learning Representations by Maximizing Mutal Information Across Views” 2019 arXiv:1906.00910,
Re claim 1, claim 1 of the patent discloses
A computer-implemented method, comprising: (see claim 1 “A computer-implemented method for performing semi-supervised contrastive learning of visual representations, the method comprising”)
obtaining one or more training inputs; (see claim 1 “obtaining a training image in a set of one or more unlabeled training images”)
performing one or more first augmentation operations on at least one of the training inputs to obtain a first augmented input; separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the at least one of the training inputs to obtain a second augmented input; ( see claim 1 “performing a plurality of first augmentation operations on the training image to obtain a first augmented image; separate from performing the plurality of first augmentation operations, performing a plurality of second augmentation operations on the training image to obtain a second augmented image;”)
computing a contrastive loss value between a first representation based on the first augmented input and a second representation based on the second augmented input, (see claim 1 “evaluating a loss function that evaluates a difference between the first projected representation and the second projected representation” and “respectively processing, with a base encoder neural network, the first augmented image and the second augmented image to respectively generate a first intermediate representation for the first augmented image and a second intermediate representation for the second augmented image; respectively processing, with a projection head neural network comprising a plurality of layers, the first intermediate representation and the second intermediate representation to respectively generate a first projected representation for the first augmented image and a second projected representation for the second augmented image”)
wherein computing the contrastive loss value comprises:
processing, with a base encoder neural network, the first augmented input to generate a first intermediate representation for the first augmented input; (see claim 1 “respectively processing, with a base encoder neural network, the first augmented image and the second augmented image to respectively generate a first intermediate representation for the first augmented image and a second intermediate representation for the second augmented image”)
and processing, with a projection head neural network, the first intermediate representation to obtain the first representation (see claim 1 “respectively processing, with a projection head neural network comprising a plurality of layers, the first intermediate representation and the second intermediate representation to respectively generate a first projected representation for the first augmented image and a second projected representation for the second augmented image”);
and modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the contrastive loss value (see claim 1 “evaluating a loss function that evaluates a difference between the first projected representation and the second projected representation; modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the loss function”).
Claim 1 does not expressly disclose wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input;
Bachman et al discloses wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input; (see section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations );
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of claim 1. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 with Bachman to reach the aforementioned advantage.
Re claim 3 claim 1 discloses generating a classification model from the base encoder neural network and the projection head neural network, the classification model comprising some but not all of the plurality of layers of the projection head neural network (see claim 1 “generating an image classification model from the base encoder neural network and the projection head neural network, the image classification model comprising some but not all of the plurality of layers of the projection head neural network”).
Re claim 10 Claim 1 discloses A computing system comprising: one or more processors; and instructions that, when executed by the one or more processors, cause the computing system to perform operations (see claim 1 “A computer-implemented method for performing semi-supervised contrastive learning of visual representations, the method comprising” note that the computer corresponds to the processor and the method corresponds to the instructions )
obtaining one or more training inputs; (see claim 1 “obtaining a training image in a set of one or more unlabeled training images”)
performing one or more first augmentation operations on at least one of the training inputs to obtain a first augmented input; separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the at least one of the training inputs to obtain a second augmented input; ( see claim 1 “performing a plurality of first augmentation operations on the training image to obtain a first augmented image; separate from performing the plurality of first augmentation operations, performing a plurality of second augmentation operations on the training image to obtain a second augmented image;”)
computing a contrastive loss value between a first representation based on the first augmented input and a second representation based on the second augmented input, (see claim 1 “evaluating a loss function that evaluates a difference between the first projected representation and the second projected representation” and “respectively processing, with a base encoder neural network, the first augmented image and the second augmented image to respectively generate a first intermediate representation for the first augmented image and a second intermediate representation for the second augmented image; respectively processing, with a projection head neural network comprising a plurality of layers, the first intermediate representation and the second intermediate representation to respectively generate a first projected representation for the first augmented image and a second projected representation for the second augmented image”)
wherein computing the contrastive loss value comprises:
processing, with a base encoder neural network, the first augmented input to generate a first intermediate representation for the first augmented input; (see claim 1 “respectively processing, with a base encoder neural network, the first augmented image and the second augmented image to respectively generate a first intermediate representation for the first augmented image and a second intermediate representation for the second augmented image”)
and processing, with a projection head neural network, the first intermediate representation to obtain the first representation (see claim 1 “respectively processing, with a projection head neural network comprising a plurality of layers, the first intermediate representation and the second intermediate representation to respectively generate a first projected representation for the first augmented image and a second projected representation for the second augmented image”);
and modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the contrastive loss value (see claim 1 “evaluating a loss function that evaluates a difference between the first projected representation and the second projected representation; modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the loss function”).
Claim 1 does not expressly disclose wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input;
Bachman et al discloses wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input; (see section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations );
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of claim 1. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 with Bachman to reach the aforementioned advantage.
Re claim 12 claim 1 discloses generating a classification model from the base encoder neural network and the projection head neural network, the classification model comprising some but not all of the plurality of layers of the projection head neural network (see claim 1 “generating an image classification model from the base encoder neural network and the projection head neural network, the image classification model comprising some but not all of the plurality of layers of the projection head neural network”).
Claims 4-6, 8, 9, 13-15 17 and 18 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. US 11386302 B2 in view of Bachman et al “Learning Representations by Maximizing Mutal Information Across Views” 2019 arXiv:1906.00910,in view of Chen et al “Improved Baselines with Momentum Contrastive Learning” 9 March 2020 arXiv:2003.04297.
Re claim 4 Claim 1 does not expressly disclose herein the projection head neural network transforms the first intermediate representation using a nonlinear transformation. Chen discloses wherein the projection head neural network transforms the first intermediate representation using a nonlinear transformation (see section 3 MLP head “ (hidden layer2048-d, with ReLU” note that Relu is not linear). The motivation to combine is “an MLP projection head and stronger data augmentation, are orthogonal to the frameworks of MoCo and SimCLR, and when used with MoCo they lead to better image classification and object detection transfer learning results” (see section 1). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 Bachman and Chen to reach the aforementioned advantage.
Re claim 5 claim 1 does not expressly disclose wherein the projection head neural network comprises a multilayer perceptron Chen discloses wherein the projection head neural network comprises a multilayer perceptron. (See section 3 MLP head, the examiner notes that MLP stands for multi-layer perceptron). The motivation to combine is “an MLP projection head and stronger data augmentation, are orthogonal to the frameworks of MoCo and SimCLR, and when used with MoCo they lead to better image classification and object detection transfer learning results” (see section 1). Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 Bachman and Chen to reach the aforementioned advantage.
Re claim 6 claim 1 does not expressly discloses wherein the one or more first augmentation operations comprise a blur operation.(see section 3 Augmentation “We extend the original augmentation in [6] by including the blur augmentation”.) Chen discloses wherein the one or more first augmentation operations comprise a blur operation.(see section 3 Augmentation “We extend the original augmentation in [6] by including the blur augmentation”). The motivation to combine is “an MLP projection head and stronger data augmentation, are orthogonal to the frameworks of MoCo and SimCLR, and when used with MoCo they lead to better image classification and object detection transfer learning results” (see section 1). Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 Bachman and Chen to reach the aforementioned advantage.
Re claim 8 Claim 1 does not expressly disclose wherein the one or more first augmentation operations comprise a first resize operation that resizes the training input.
Bachman discloses wherein the one or more first augmentation operations comprise a first resize operation that resizes the training input. (See section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations);
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of claim 1. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 with Bachman and Chen to reach the aforementioned advantage.
Re claim 9 claim 1 further does not disclose wherein the one or more first augmentation operations comprises a first random flip operation that randomly flips the training input.
Bachman discloses wherein the one or more first augmentation operations comprises a first random flip operation that randomly flips the training input.
(see section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations );
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of claim 1. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 with Bachman and Chen to reach the aforementioned advantage.
Re claim 13 Claim 1 does not expressly disclose herein the projection head neural network transforms the first intermediate representation using a nonlinear transformation. Chen discloses wherein the projection head neural network transforms the first intermediate representation using a nonlinear transformation (see section 3 MLP head “ (hidden layer2048-d, with ReLU” note that Relu is not linear). The motivation to combine is “an MLP projection head and stronger data augmentation, are orthogonal to the frameworks of MoCo and SimCLR, and when used with MoCo they lead to better image classification and object detection transfer learning results” (see section 1). Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 Bachman and Chen to reach the aforementioned advantage.
Re claim 14 claim 1 does not expressly disclose wherein the projection head neural network comprises a multilayer perceptron Chen discloses wherein the projection head neural network comprises a multilayer perceptron. (see section 3 MLP head, the examiner notes that MLP stands for multi-layer perceptron). The motivation to combine is “an MLP projection head and stronger data augmentation, are orthogonal to the frameworks of MoCo and SimCLR, and when used with MoCo they lead to better image classification and object detection transfer learning results” (see section 1). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 Bachman and Chen to reach the aforementioned advantage.
Re claim 15 claim 1 does not expressly discloses wherein the one or more first augmentation operations comprise a blur operation (see section 3 Augmentation “We extend the original augmentation in [6] by including the blur augmentation”.) Chen discloses wherein the one or more first augmentation operations comprise a blur operation (see section 3 Augmentation “We extend the original augmentation in [6] by including the blur augmentation”). The motivation to combine is “an MLP projection head and stronger data augmentation, are orthogonal to the frameworks of MoCo and SimCLR, and when used with MoCo they lead to better image classification and object detection transfer learning results” (see section 1). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 Bachman and Chen to reach the aforementioned advantage.
Re claim 17 Claim 1 does not expressly disclose wherein the one or more first augmentation operations comprise a first resize operation that resizes the training input.
Bachman discloses wherein the one or more first augmentation operations comprise a first resize operation that resizes the training input. (See section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations);
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of claim 1. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 with Bachman and Chen to reach the aforementioned advantage.
Re claim 18 claim 1 further does not disclose wherein the one or more first augmentation operations comprises a first random flip operation that randomly flips the training input.
Bachman discloses wherein the one or more first augmentation operations comprises a first random flip operation that randomly flips the training input.
(see section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations );
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of claim 1. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 with Bachman and Chen to reach the aforementioned advantage.
Claim 2, and 11 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim of U.S. Patent No. US 11386302 B2 in view of Bachman et al “Learning Representations by Maximizing Mutal Information Across Views” 2019 in view of Lin US 20190355102 A1.
Re claim 2, claim 1 does not expressly disclose the encoder neural network comprises an attention mechanism. Lin discloses the encoder neural network comprises an attention mechanism. (see paragraph 70 note the encoder contains an attention layer) The motivation to combine is see paragraph 71 “As discussed in the following, the contextual attention pathway of the described system overcomes this limitation—the contextual attention pathway is capable of identifying features for content generation that are from distant spatial locations of the image under consideration”. One of ordinary skill in the art could have easily combine claim 1 with Lin to add an attention layer to the encoders of claim 1. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 Bachman and Lin to reach the aforementioned advantage.
Re claim 11, claim 1 does not expressly disclose the encoder neural network comprises an attention mechanism. Lin discloses the encoder neural network comprises an attention mechanism. (see paragraph 70 note the encoder contains an attention layer) The motivation to combine is see paragraph 71 “As discussed in the following, the contextual attention pathway of the described system overcomes this limitation—the contextual attention pathway is capable of identifying features for content generation that are from distant spatial locations of the image under consideration”. One of ordinary skill in the art could have easily combine claim 1 with Lin to add an attention layer to the encoders of Chen. Therefore, it would have been obvious one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 Bachman and Lin to reach the aforementioned advantage.
Claims 19 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. US 11386302 B2 in view of Bachman et al “Learning Representations by Maximizing Mutal Information Across Views” 2019 arXiv:1906.00910, in view of Skala US 2021/0049346.
Re claim 19, claim 1 of the patent discloses
A computer-implemented method, comprising: (see claim 1 “A computer-implemented method for performing semi-supervised contrastive learning of visual representations, the method comprising”)
obtaining one or more training inputs; (see claim 1 “obtaining a training image in a set of one or more unlabeled training images”)
performing one or more first augmentation operations on at least one of the training inputs to obtain a first augmented input; separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the at least one of the training inputs to obtain a second augmented input; ( see claim 1 “performing a plurality of first augmentation operations on the training image to obtain a first augmented image; separate from performing the plurality of first augmentation operations, performing a plurality of second augmentation operations on the training image to obtain a second augmented image;”)
computing a contrastive loss value between a first representation based on the first augmented input and a second representation based on the second augmented input, (see claim 1 “evaluating a loss function that evaluates a difference between the first projected representation and the second projected representation” and “respectively processing, with a base encoder neural network, the first augmented image and the second augmented image to respectively generate a first intermediate representation for the first augmented image and a second intermediate representation for the second augmented image; respectively processing, with a projection head neural network comprising a plurality of layers, the first intermediate representation and the second intermediate representation to respectively generate a first projected representation for the first augmented image and a second projected representation for the second augmented image”)
wherein computing the contrastive loss value comprises:
processing, with a base encoder neural network, the first augmented input to generate a first intermediate representation for the first augmented input; (see claim 1 “respectively processing, with a base encoder neural network, the first augmented image and the second augmented image to respectively generate a first intermediate representation for the first augmented image and a second intermediate representation for the second augmented image”)
and processing, with a projection head neural network, the first intermediate representation to obtain the first representation (see claim 1 “respectively processing, with a projection head neural network comprising a plurality of layers, the first intermediate representation and the second intermediate representation to respectively generate a first projected representation for the first augmented image and a second projected representation for the second augmented image”);
and modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the contrastive loss value (see claim 1 “evaluating a loss function that evaluates a difference between the first projected representation and the second projected representation; modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the loss function”).
Claim 1 does not expressly disclose wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input;
Bachman et al discloses wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input; (see section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations );
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of claim 1. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 with Bachman to reach the aforementioned advantage.
Claim 1 and Bachman do not expressly disclose One or more non-transitory computer-readable media that collectively store a neural network. Skala discloses One or more non-transitory computer-readable media that collectively store a neural network (*see paragraph 69 note that the neural network is stored on a computer readable medium). One or ordinary skill in the art could have easily used a computer readable medium as disclosed in Skala to store the neural network trained by the combination to Claim 1 and Bachman and the results would merely be the neural network is stored on a computer readable medium and very predictable. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Claim 1 Bachman and Skala to reach the aforementioned advantage.
Claims 20, 21 23 and 24 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. US 11386302 B2 in view of Bachman et al “Learning Representations by Maximizing Mutal Information Across Views” 2019 arXiv:1906.00910, in view of Skala US 2021/0049346 in view of Chen et al “Improved Baselines with Momentum Contrastive Learning” 9 March 2020 arXiv:2003.04297.
Re claim 20 Claim 1 does not expressly disclose wherein the one or more non-transitory computer-readable media collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations comprising: receiving an input; processing the input using the neural network to generate an output representation from the neural network; and generating, based on the output representation, an output value
Skala disclose wherein the one or more non-transitory computer-readable media collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations comprising (see paragraph 28) One or ordinary skill in the art could have easily used a computer readable medium as disclosed in Skala to computer instruction to implement Claim 1 and Bachman and the results would merely be storing a program to implement the instructions on a computer readable medium and very predictable. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Claim 1 Bachman and Skala to reach the aforementioned advantage.
Chen discloses: receiving an input; processing the input using the neural network to generate an output representation from the neural network; and generating, based on the output representation, an output value (see section 3 setting “Unsupervised learning is conducted on the 1.28MImageNet [3] training set. We follow two common protocols for evaluation. (i) ImageNet linear classification: features are frozen and a supervised linear classifier is trained (ii) Transferring to VOC object detection [4]: a Faster R-CNN detector [14] (C4-backbone) is fine-tuned end-to-end on the VOC 07+12 trainval set1 and evaluated on the VOC 07 testset using the COCO suite of metrics [10]. We use the same hyper-parameters (except when noted) and codebase as MoCo [6]. All results use a standard-size ResNet-50”). Note that after the unsupervised training a test set is applied to the neural network test a classification using a linear classifier). The motivation is to train a classifier (see section 3 setting “Unsupervised learning is conducted on the 1.28MImageNet [3] training set. We follow two common protocols for evaluation. (i) ImageNet linear classification: features are frozen and a supervised linear classifier is trained (ii) Transferring to VOC object detection [4]: a Faster R-CNN detector [14] (C4-backbone) is fine-tuned end-to-end on the VOC 07+12 trainval set1 and evaluated on the VOC 07 testset using the COCO suite of metrics [10]. We use the same hyper-parameters (except when noted) and codebase as MoCo [6]. All results use a standard-size ResNet-50”). Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Claim 1 Bachman and Skala with Chen to reach the aforementioned advantage
Re claim 21 Chen further discloses wherein the input comprises image data, and wherein generating the output value comprises: generating, based on the output representation, at least one of: a classification prediction, a detection prediction, a recognition prediction, or a segmentation prediction. (see section 3 setting “Unsupervised learning is conducted on the 1.28MImageNet [3] training set. We follow two common protocols for evaluation. (i) ImageNet linear classification: features are frozen and a supervised linear classifier is trained (ii) Transferring to VOC object detection [4]: a Faster R-CNN detector [14] (C4-backbone) is fine-tuned end-to-end on the VOC 07+12 trainval set1 and evaluated on the VOC 07 testset using the COCO suite of metrics [10]. We use the same hyper-parameters (except when noted) and codebase as MoCo [6]. All results use a standard-size ResNet-50”). Note that after the unsupervised training a test set is applied to the neural network test a classification using a linear classifier). Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Claim 1 Bachman and Skala with Chen to reach the aforementioned advantage.
Re claim 23 claim 1 does not expressly disclose wherein the projection head neural network comprises a multilayer perceptron Chen discloses wherein the projection head neural network comprises a multilayer perceptron. (see section 3 MLP head, the examiner notes that MLP stands for multi-layer perceptron). The motivation to combine is “an MLP projection head and stronger data augmentation, are orthogonal to the frameworks of MoCo and SimCLR, and when used with MoCo they lead to better image classification and object detection transfer learning results” (see section 1). Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 Bachman Skala and Chen to reach the aforementioned advantage.
Re claim 24 claim 1 does not expressly discloses wherein the one or more first augmentation operations comprise a blur operation.(see section 3 Augmentation “We extend the original augmentation in [6] by including the blur augmentation”.) Chen discloses wherein the one or more first augmentation operations comprise a blur operation.(see section 3 Augmentation “We extend the original augmentation in [6] by including the blur augmentation”). The motivation to combine is “an MLP projection head and stronger data augmentation, are orthogonal to the frameworks of MoCo and SimCLR, and when used with MoCo they lead to better image classification and object detection transfer learning results” (see section 1). Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 Bachman Skala and Chen to reach the aforementioned advantage.
Claim 1, 4-7, 10, 13-16 and 19 rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1,8 and 6 of U.S. Patent No. US 11847571 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because recite all the features of the instant claim.
Re claim 1
A computer-implemented method, comprising: (see claim 1 “instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising”)
obtaining one or more training inputs; (see claim 1 “obtaining a training image;”)
performing one or more first augmentation operations on at least one of the training inputs to obtain a first augmented input; separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the at least one of the training inputs to obtain a second augmented input; wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input; ( see claim 1 “performing one or more first augmentation operations on the training image to obtain a first augmented image, wherein the one or more first augmentation operations comprise one or both of: a first random crop operation that randomly crops the training image and a first random color distortion operation that randomly modifies color values of the training image to the training image; separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the training image to obtain a second augmented image, wherein the one or more second augmentation operations comprise one or both of: a second random crop operation that randomly crops the training image and a second random color distortion operation that randomly modifies color values of the training image”)
computing a contrastive loss value between a first representation based on the first augmented input and a second representation based on the second augmented input(see claim 1 “evaluating a loss function that evaluates a difference between the first projected representation and the second projected representation” and “respectively processing, with the base encoder neural network, the first augmented image and the second augmented image to respectively generate a first intermediate representation for the first augmented image and a second intermediate representation for the second augmented image; respectively processing, with the projection head neural network, the first intermediate representation and the second intermediate representation to respectively obtain a first projected representation for the first augmented image and a second projected representation for the second augmented image” note this loss is contrastive), wherein computing the contrastive loss value comprises:
processing, with a base encoder neural network, the first augmented input to generate a first intermediate representation for the first augmented input; and processing, with a projection head neural network, the first intermediate representation to obtain the first representation; (see claim 1 “processing, with the base encoder neural network, the first augmented image and the second augmented image to respectively generate a first intermediate representation for the first augmented image and a second intermediate representation for the second augmented image; respectively processing, with the projection head neural network, the first intermediate representation and the second intermediate representation to respectively obtain a first projected representation for the first augmented image and a second projected representation for the second augmented image”)
and modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the contrastive loss value. (see claim 1 “modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the loss function.”)
Re claim 4 claim 1 discloses wherein the projection head neural network transforms the first intermediate representation using a nonlinear transformation. (see claim 1 “wherein to generate the projected representation, the projection head neural network is configured to perform at least one non-linear transformation”)
Re claim 5 claim 8 of the patent discloses wherein the projection head neural network comprises a multilayer perceptron (see claim 8 “wherein the projection head neural network comprises a multi-layer perceptron that comprises one hidden layer and a rectified linear unit non-linear activation function.”)
Re claim 6 claim 6 of the patent discloses wherein the one or more first augmentation operations comprise a blur operation.(See claim 6 “wherein the one or more first augmentation operations further comprises a first random Gaussian blur operation that randomly applies a Gaussian blur to the training image and wherein the one or more second augmentation operations further comprises a second random Gaussian blur operation that randomly applies a Gaussian blur to the training image.”)
Re claim 7 claim 6 of the patent discloses wherein the blur operation applies a Gaussian blur based on a randomly sampled value (see claim 6 “wherein the one or more first augmentation operations further comprises a first random Gaussian blur operation that randomly applies a Gaussian blur to the training image and wherein the one or more second augmentation operations further comprises a second random Gaussian blur operation that randomly applies a Gaussian blur to the training image.”)
Re claim 10
A computing system comprising: one or more processors; and instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising (see claim 1 “A computing system to perform contrastive learning of visual representations, the computing system comprising: one or more processors; and one or more non-transitory computer-readable media that collectively store… and instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising”)
obtaining one or more training inputs; (see claim 1 “obtaining a training image;”)
performing one or more first augmentation operations on at least one of the training inputs to obtain a first augmented input; separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the at least one of the training inputs to obtain a second augmented input; wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input; ( see claim 1 “performing one or more first augmentation operations on the training image to obtain a first augmented image, wherein the one or more first augmentation operations comprise one or both of: a first random crop operation that randomly crops the training image and a first random color distortion operation that randomly modifies color values of the training image to the training image; separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the training image to obtain a second augmented image, wherein the one or more second augmentation operations comprise one or both of: a second random crop operation that randomly crops the training image and a second random color distortion operation that randomly modifies color values of the training image”)
computing a contrastive loss value between a first representation based on the first augmented input and a second representation based on the second augmented input(see claim 1 “evaluating a loss function that evaluates a difference between the first projected representation and the second projected representation” and “respectively processing, with the base encoder neural network, the first augmented image and the second augmented image to respectively generate a first intermediate representation for the first augmented image and a second intermediate representation for the second augmented image; respectively processing, with the projection head neural network, the first intermediate representation and the second intermediate representation to respectively obtain a first projected representation for the first augmented image and a second projected representation for the second augmented image” note this loss is contrastive), wherein computing the contrastive loss value comprises:
processing, with a base encoder neural network, the first augmented input to generate a first intermediate representation for the first augmented input; and processing, with a projection head neural network, the first intermediate representation to obtain the first representation; (see claim 1 “processing, with the base encoder neural network, the first augmented image and the second augmented image to respectively generate a first intermediate representation for the first augmented image and a second intermediate representation for the second augmented image; respectively processing, with the projection head neural network, the first intermediate representation and the second intermediate representation to respectively obtain a first projected representation for the first augmented image and a second projected representation for the second augmented image”)
and modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the contrastive loss value. (see claim 1 “modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the loss function.”)
Re claim 13 claim 1 discloses wherein the projection head neural network transforms the first intermediate representation using a nonlinear transformation. (see claim 1 “wherein to generate the projected representation, the projection head neural network is configured to perform at least one non-linear transformation”)
Re claim 14 claim 8 of the patent discloses wherein the projection head neural network comprises a multilayer perceptron. (see claim 8 “wherein the projection head neural network comprises a multi-layer perceptron that comprises one hidden layer and a rectified linear unit non-linear activation function.”)
Re claim 15 claim 6 of the patent discloses wherein the one or more first augmentation operations comprise a blur operation.(See claim 6 “wherein the one or more first augmentation operations further comprises a first random Gaussian blur operation that randomly applies a Gaussian blur to the training image and wherein the one or more second augmentation operations further comprises a second random Gaussian blur operation that randomly applies a Gaussian blur to the training image.”)
Re claim 16 claim 6 of the patent discloses wherein the blur operation applies a Gaussian blur based on a randomly sampled value (see claim 6 “wherein the one or more first augmentation operations further comprises a first random Gaussian blur operation that randomly applies a Gaussian blur to the training image and wherein the one or more second augmentation operations further comprises a second random Gaussian blur operation that randomly applies a Gaussian blur to the training image.”)
Re claim 19
One or more non-transitory computer-readable media that collectively store a neural network trained by: (see claim 1 “ A computing system to perform contrastive learning of visual representations, the computing system comprising: one or more processors; and one or more non-transitory computer-readable media that collectively store: a base encoder neural network configured to process an input image to generate an intermediate representation of the input image; a projection head neural network configured to process the intermediate representation of the input image to generate a projected representation of the input image, wherein to generate the projected representation, the projection head neural network is configured to perform at least one non-linear transformation; and instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising” note that process comprises training and the trained network is stored in the medium)
obtaining one or more training inputs; (see claim 1 “obtaining a training image;”)
performing one or more first augmentation operations on at least one of the training inputs to obtain a first augmented input; separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the at least one of the training inputs to obtain a second augmented input; wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input; ( see claim 1 “performing one or more first augmentation operations on the training image to obtain a first augmented image, wherein the one or more first augmentation operations comprise one or both of: a first random crop operation that randomly crops the training image and a first random color distortion operation that randomly modifies color values of the training image to the training image; separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the training image to obtain a second augmented image, wherein the one or more second augmentation operations comprise one or both of: a second random crop operation that randomly crops the training image and a second random color distortion operation that randomly modifies color values of the training image”)
computing a contrastive loss value between a first representation based on the first augmented input and a second representation based on the second augmented input(see claim 1 “evaluating a loss function that evaluates a difference between the first projected representation and the second projected representation” and “respectively processing, with the base encoder neural network, the first augmented image and the second augmented image to respectively generate a first intermediate representation for the first augmented image and a second intermediate representation for the second augmented image; respectively processing, with the projection head neural network, the first intermediate representation and the second intermediate representation to respectively obtain a first projected representation for the first augmented image and a second projected representation for the second augmented image” note this loss is contrastive), wherein computing the contrastive loss value comprises:
processing, with a base encoder neural network, the first augmented input to generate a first intermediate representation for the first augmented input; and processing, with a projection head neural network, the first intermediate representation to obtain the first representation; (see claim 1 “processing, with the base encoder neural network, the first augmented image and the second augmented image to respectively generate a first intermediate representation for the first augmented image and a second intermediate representation for the second augmented image; respectively processing, with the projection head neural network, the first intermediate representation and the second intermediate representation to respectively obtain a first projected representation for the first augmented image and a second projected representation for the second augmented image”)
and modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the contrastive loss value. (see claim 1 “modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the loss function.”)
Claim 2, and 11 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. US 11847571 in view of Lin US 20190355102 A1.
Re claim 2, claim 1 does not expressly disclose the encoder neural network comprises an attention mechanism. Lin discloses the encoder neural network comprises an attention mechanism. (see paragraph 70 note the encoder contains an attention layer) The motivation to combine is see paragraph 71 “As discussed in the following, the contextual attention pathway of the described system overcomes this limitation—the contextual attention pathway is capable of identifying features for content generation that are from distant spatial locations of the image under consideration”. One of ordinary skill in the art could have easily combine claim 1 with Lin to add an attention layer to the encoders of claim 1. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 and Lin to reach the aforementioned advantage.
Re claim 11, claim 1 does not expressly disclose the encoder neural network comprises an attention mechanism. Lin discloses the encoder neural network comprises an attention mechanism. (see paragraph 70 note the encoder contains an attention layer) The motivation to combine is see paragraph 71 “As discussed in the following, the contextual attention pathway of the described system overcomes this limitation—the contextual attention pathway is capable of identifying features for content generation that are from distant spatial locations of the image under consideration”. One of ordinary skill in the art could have easily combine claim 1 with Lin to add an attention layer to the encoders of claim 1. Therefore, it would have been obvious one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 and Lin to reach the aforementioned advantage.
Claim 8 ,9 17 and 18 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 6 of U.S. Patent No. US 11847571 B2 in view of Bachman et al “Learning Representations by Maximizing Mutal Information Across Views” 2019 arXiv:1906.00910
Re claim 8 Claim 6 does not expressly disclose wherein the one or more first augmentation operations comprise a first resize operation that resizes the training input.
Bachman discloses wherein the one or more first augmentation operations comprise a first resize operation that resizes the training input. (See section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations);
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of claim 6. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 6 with Bachman to reach the aforementioned advantage.
Re claim 9 claim 6 further does not disclose wherein the one or more first augmentation operations comprises a first random flip operation that randomly flips the training input.
Bachman discloses wherein the one or more first augmentation operations comprises a first random flip operation that randomly flips the training input.
(see section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations );
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of claim 6. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 6 with Bachman to reach the aforementioned advantage.
Re claim 17 Claim 6 does not expressly disclose wherein the one or more first augmentation operations comprise a first resize operation that resizes the training input.
Bachman discloses wherein the one or more first augmentation operations comprise a first resize operation that resizes the training input. (See section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations).
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of claim 6. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 6 with Bachman to reach the aforementioned advantage.
Re claim 18 claim 6 further does not disclose wherein the one or more first augmentation operations comprises a first random flip operation that randomly flips the training input.
Bachman discloses wherein the one or more first augmentation operations comprises a first random flip operation that randomly flips the training input.(see section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations );
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of claim 6. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 6 with Bachman to reach the aforementioned advantage.
Claim 20, 21 23 and 24 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 8 and 6 of U.S. Patent No. 11847571 B2 in view of Chen et al “Improved Baselines with Momentum Contrastive Learning” 9 March 2020 arXiv:2003.04297.
Re claim 20 Claim 20, Claim 1 Further discloses wherein the one or more non-transitory computer-readable media collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations comprising (see claim 1 “one or more non-transitory computer-readable media that collectively store: a base encoder neural network configured to process an input image to generate an intermediate representation of the input image; a projection head neural network configured to process the intermediate representation of the input image to generate a projected representation of the input image, wherein to generate the projected representation, the projection head neural network is configured to perform at least one non-linear transformation; and instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising” )
Claim 1 does not expressly disclose receiving an input; processing the input using the neural network to generate an output representation from the neural network; and generating, based on the output representation, an output value
Chen discloses receiving an input; processing the input using the neural network to generate an output representation from the neural network; and generating, based on the output representation, an output value (see section 3 setting “Unsupervised learning is conducted on the 1.28MImageNet [3] training set. We follow two common protocols for evaluation. (i) ImageNet linear classification: features are frozen and a supervised linear classifier is trained (ii) Transferring to VOC object detection [4]: a Faster R-CNN detector [14] (C4-backbone) is fine-tuned end-to-end on the VOC 07+12 trainval set1 and evaluated on the VOC 07 testset using the COCO suite of metrics [10]. We use the same hyper-parameters (except when noted) and codebase as MoCo [6]. All results use a standard-size ResNet-50”). Note that after the unsupervised training a test set is applied to the neural network test a classification using a linear classifier). The motivation is to train a classifier (see section 3 setting “Unsupervised learning is conducted on the 1.28MImageNet [3] training set. We follow two common protocols for evaluation. (i) ImageNet linear classification: features are frozen and a supervised linear classifier is trained (ii) Transferring to VOC object detection [4]: a Faster R-CNN detector [14] (C4-backbone) is fine-tuned end-to-end on the VOC 07+12 trainval set1 and evaluated on the VOC 07 testset using the COCO suite of metrics [10]. We use the same hyper-parameters (except when noted) and codebase as MoCo [6]. All results use a standard-size ResNet-50”). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Claim 1 and Chen to reach the aforementioned advantage.
Re claim 21 Chen further discloses wherein the input comprises image data, and wherein generating the output value comprises: generating, based on the output representation, at least one of: a classification prediction, a detection prediction, a recognition prediction, or a segmentation prediction. (see section 3 setting “Unsupervised learning is conducted on the 1.28MImageNet [3] training set. We follow two common protocols for evaluation. (i) ImageNet linear classification: features are frozen and a supervised linear classifier is trained (ii) Transferring to VOC object detection [4]: a Faster R-CNN detector [14] (C4-backbone) is fine-tuned end-to-end on the VOC 07+12 trainval set1 and evaluated on the VOC 07 testset using the COCO suite of metrics [10]. We use the same hyper-parameters (except when noted) and codebase as MoCo [6]. All results use a standard-size ResNet-50”). Note that after the unsupervised training a test set is applied to the neural network test a classification using a linear classifier).
Re claim 23 claim 8 discloses wherein the projection head neural network comprises a multilayer perceptron (see claim 8 “wherein the projection head neural network comprises a multi-layer perceptron that comprises one hidden layer and a rectified linear unit non-linear activation function.”).
Re claim 24, claim 6 of the patent discloses wherein the one or more first augmentation operations comprise a blur operation. (see claim 6 “wherein the plurality of first augmentation operations further comprises a first random Gaussian blur operation that randomly applies a Gaussian blur to the training image and wherein the plurality of second augmentation operations further comprises a second random Gaussian blur operation that randomly applies a Gaussian blur to the training image.”)
Claim 22 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. 11847571 B2 in view of Chen et al “Improved Baselines with Momentum Contrastive Learning” 9 March 2020 arXiv:2003.04297 in further view of Lin US 20190355102 A1.
Re claim 22, claim 1 does not expressly disclose the encoder neural network comprises an attention mechanism. Lin discloses the encoder neural network comprises an attention mechanism. (see paragraph 70 note the encoder contains an attention layer) The motivation to combine is see paragraph 71 “As discussed in the following, the contextual attention pathway of the described system overcomes this limitation—the contextual attention pathway is capable of identifying features for content generation that are from distant spatial locations of the image under consideration”. One of ordinary skill in the art could have easily combine claim 1 with Lin to add an attention layer to the encoders of claim 1. Therefore, it would have been obvious before the effective filing date of the claimed invention to combine claim 1, Chen and Lin to reach the aforementioned advantage.
Claim 1, 4-7, 10, 13-16 and 19 rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1,8 and 6 U.S. Patent No. US 12254413 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because recite all the features of the instant claim.
Re claim 1 Claim 1 of the patent discloses A computer-implemented method, comprising: (see claim 1 “instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising” note that the operations are the method)
obtaining one or more training inputs (see claim 1 “obtaining one or more training inputs”);
performing one or more first augmentation operations on at least one of the training inputs to obtain a first augmented input; separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the at least one of the training inputs to obtain a second augmented input;(see claim 1 “performing one or more first augmentation operations on at least one of the training inputs to obtain a first augmented input; separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the at least one of the training inputs to obtain a second augmented input”)
wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input; (see claim 1 “wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input”)
computing a contrastive loss value between a first representation based on the first augmented input and a second representation based on the second augmented input, (see claim 1 “evaluating a loss function that evaluates a difference between the first projected representation and the second projected representation” and “respectively processing, with the base encoder neural network, the first augmented input and the second augmented input to respectively generate a first intermediate representation for the first augmented input and a second intermediate representation for the second augmented input; respectively processing, with the projection head neural network, the first intermediate representation and the second intermediate representation to respectively obtain a first projected representation for the first augmented input and a second projected representation for the second augmented input”) wherein computing the contrastive loss value comprises:
processing, with a base encoder neural network, the first augmented input to generate a first intermediate representation for the first augmented input; and processing, with a projection head neural network, the first intermediate representation to obtain the first representation; (see claim 1 “respectively processing, with the base encoder neural network, the first augmented input and the second augmented input to respectively generate a first intermediate representation for the first augmented input and a second intermediate representation for the second augmented input; respectively processing, with the projection head neural network, the first intermediate representation and the second intermediate representation to respectively obtain a first projected representation for the first augmented input and a second projected representation for the second augmented input”)
and modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the contrastive loss value (see claim 1 “modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the loss function”).
Re claim 4 claim 8 discloses wherein the projection head neural network transforms the first intermediate representation using a nonlinear transformation. (see claim 8 “wherein the projection head neural network comprises a multi-layer perceptron that comprises one hidden layer and a rectified linear unit non-linear activation function”)
Re claim 5 claim 8 of the patent discloses wherein the projection head neural network comprises a multilayer perceptron. (see claim 8 “wherein the projection head neural network comprises a multi-layer perceptron that comprises one hidden layer and a rectified linear unit non-linear activation function.”)
Re claim 6 claim 6 of the patent discloses wherein the one or more first augmentation operations comprise a blur operation.(See claim 6 “wherein the one or more first augmentation operations further comprises a first random Gaussian blur operation that randomly applies a Gaussian blur to the training image and wherein the one or more second augmentation operations further comprises a second random Gaussian blur operation that randomly applies a Gaussian blur to the training image.”)
Re claim 7 claim 6 of the patent discloses wherein the blur operation applies a Gaussian blur based on a randomly sampled value (see claim 6 “wherein the one or more first augmentation operations further comprises a first random Gaussian blur operation that randomly applies a Gaussian blur to the training image and wherein the one or more second augmentation operations further comprises a second random Gaussian blur operation that randomly applies a Gaussian blur to the training image.”)
Re claim 10 Claim 1 of the patent discloses A computing system comprising: one or more processors; and instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising (see claim 1 “one or more processors; and one or more non-transitory computer-readable media that collectively store… and instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising”)
obtaining one or more training inputs (see claim 1 “obtaining one or more training inputs”);
performing one or more first augmentation operations on at least one of the training inputs to obtain a first augmented input; separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the at least one of the training inputs to obtain a second augmented input;(see claim 1 “performing one or more first augmentation operations on at least one of the training inputs to obtain a first augmented input; separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the at least one of the training inputs to obtain a second augmented input”)
wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input; (see claim 1 “wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input”)
computing a contrastive loss value between a first representation based on the first augmented input and a second representation based on the second augmented input, (see claim 1 “evaluating a loss function that evaluates a difference between the first projected representation and the second projected representation” and “respectively processing, with the base encoder neural network, the first augmented input and the second augmented input to respectively generate a first intermediate representation for the first augmented input and a second intermediate representation for the second augmented input; respectively processing, with the projection head neural network, the first intermediate representation and the second intermediate representation to respectively obtain a first projected representation for the first augmented input and a second projected representation for the second augmented input”) wherein computing the contrastive loss value comprises:
processing, with a base encoder neural network, the first augmented input to generate a first intermediate representation for the first augmented input; and processing, with a projection head neural network, the first intermediate representation to obtain the first representation; (see claim 1 “respectively processing, with the base encoder neural network, the first augmented input and the second augmented input to respectively generate a first intermediate representation for the first augmented input and a second intermediate representation for the second augmented input; respectively processing, with the projection head neural network, the first intermediate representation and the second intermediate representation to respectively obtain a first projected representation for the first augmented input and a second projected representation for the second augmented input”)
and modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the contrastive loss value (see claim 1 “modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the loss function”).
Re claim 13 claim 8 discloses wherein the projection head neural network transforms the first intermediate representation using a nonlinear transformation. (see claim 8 “wherein the projection head neural network comprises a multi-layer perceptron that comprises one hidden layer and a rectified linear unit non-linear activation function”).
Re claim 14 claim 8 of the patent discloses wherein the projection head neural network comprises a multilayer perceptron. (See claim 8 “wherein the projection head neural network comprises a multi-layer perceptron that comprises one hidden layer and a rectified linear unit non-linear activation function.”)
Re claim 15 claim 6 of the patent discloses wherein the one or more first augmentation operations comprise a blur operation.(See claim 6 “wherein the one or more first augmentation operations further comprises a first random Gaussian blur operation that randomly applies a Gaussian blur to the training image and wherein the one or more second augmentation operations further comprises a second random Gaussian blur operation that randomly applies a Gaussian blur to the training image.”)
Re claim 16 claim 6 of the patent discloses wherein the blur operation applies a Gaussian blur based on a randomly sampled value (see claim 6 “wherein the one or more first augmentation operations further comprises a first random Gaussian blur operation that randomly applies a Gaussian blur to the training image and wherein the one or more second augmentation operations further comprises a second random Gaussian blur operation that randomly applies a Gaussian blur to the training image.”)
Re claim 19 claim 1 discloses One or more non-transitory computer-readable media that collectively store a neural network trained by: (see claim 1 “ A computing system to perform contrastive learning of visual representations, the computing system comprising: one or more processors; and one or more non-transitory computer-readable media that collectively store: a base encoder neural network configured to process an input image to generate an intermediate representation of the input image; a projection head neural network configured to process the intermediate representation of the input image to generate a projected representation of the input image, wherein to generate the projected representation, the projection head neural network is configured to perform at least one non-linear transformation; and instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising” note that process comprises training and the trained network is stored in the medium)
obtaining one or more training inputs (see claim 1 “obtaining one or more training inputs”);
performing one or more first augmentation operations on at least one of the training inputs to obtain a first augmented input; separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the at least one of the training inputs to obtain a second augmented input;(see claim 1 “performing one or more first augmentation operations on at least one of the training inputs to obtain a first augmented input; separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the at least one of the training inputs to obtain a second augmented input”)
wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input; (see claim 1 “wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input”)
computing a contrastive loss value between a first representation based on the first augmented input and a second representation based on the second augmented input, (see claim 1 “evaluating a loss function that evaluates a difference between the first projected representation and the second projected representation” and “respectively processing, with the base encoder neural network, the first augmented input and the second augmented input to respectively generate a first intermediate representation for the first augmented input and a second intermediate representation for the second augmented input; respectively processing, with the projection head neural network, the first intermediate representation and the second intermediate representation to respectively obtain a first projected representation for the first augmented input and a second projected representation for the second augmented input”) wherein computing the contrastive loss value comprises:
processing, with a base encoder neural network, the first augmented input to generate a first intermediate representation for the first augmented input; and processing, with a projection head neural network, the first intermediate representation to obtain the first representation; (see claim 1 “respectively processing, with the base encoder neural network, the first augmented input and the second augmented input to respectively generate a first intermediate representation for the first augmented input and a second intermediate representation for the second augmented input; respectively processing, with the projection head neural network, the first intermediate representation and the second intermediate representation to respectively obtain a first projected representation for the first augmented input and a second projected representation for the second augmented input”)
and modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the contrastive loss value (see claim 1 “modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the loss function”).
Claim 2, and 11 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. US 12254413 B2 in view of Lin US 20190355102 A1.
Re claim 2, claim 1 does not expressly disclose the encoder neural network comprises an attention mechanism. Lin discloses the encoder neural network comprises an attention mechanism. (see paragraph 70 note the encoder contains an attention layer) The motivation to combine is see paragraph 71 “As discussed in the following, the contextual attention pathway of the described system overcomes this limitation—the contextual attention pathway is capable of identifying features for content generation that are from distant spatial locations of the image under consideration”. One of ordinary skill in the art could have easily combine claim 1 with Lin to add an attention layer to the encoders of claim 1. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 and Lin to reach the aforementioned advantage.
Re claim 11, claim 1 does not expressly disclose the encoder neural network comprises an attention mechanism. Lin discloses the encoder neural network comprises an attention mechanism. (see paragraph 70 note the encoder contains an attention layer) The motivation to combine is see paragraph 71 “As discussed in the following, the contextual attention pathway of the described system overcomes this limitation—the contextual attention pathway is capable of identifying features for content generation that are from distant spatial locations of the image under consideration”. One of ordinary skill in the art could have easily combine claim 1 with Lin to add an attention layer to the encoders of claim 1. Therefore, it would have been obvious one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 and Lin to reach the aforementioned advantage.
Claim 8 ,9 17 and 18 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 6 of U.S. Patent No. US12254413 B2 in view of Bachman et al “Learning Representations by Maximizing Mutal Information Across Views” 2019 arXiv:1906.00910
Re claim 8 Claim 6 does not expressly disclose wherein the one or more first augmentation operations comprise a first resize operation that resizes the training input.
Bachman discloses wherein the one or more first augmentation operations comprise a first resize operation that resizes the training input. (See section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations);
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of claim 6. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 6 with Bachman to reach the aforementioned advantage.
Re claim 9 claim 6 further does not disclose wherein the one or more first augmentation operations comprises a first random flip operation that randomly flips the training input.
Bachman discloses wherein the one or more first augmentation operations comprises a first random flip operation that randomly flips the training input.
(see section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations);
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of claim 6. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 6 with Bachman to reach the aforementioned advantage.
Re claim 17 Claim 6 does not expressly disclose wherein the one or more first augmentation operations comprise a first resize operation that resizes the training input.
Bachman discloses wherein the one or more first augmentation operations comprise a first resize operation that resizes the training input. (see section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations);
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of claim 6. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 6 with Bachman to reach the aforementioned advantage.
Re claim 18 claim 6 further does not disclose wherein the one or more first augmentation operations comprises a first random flip operation that randomly flips the training input.
Bachman discloses wherein the one or more first augmentation operations comprises a first random flip operation that randomly flips the training input (see section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations);
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of claim 6. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 6 with Bachman to reach the aforementioned advantage.
Claim 20, 21 23 and 24 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 8 and 6 of U.S. Patent No. 12254413 B2 in view of Chen et al “Improved Baselines with Momentum Contrastive Learning” 9 March 2020 arXiv:2003.04297.
Re claim 20 Claim 20, Claim 1 Further discloses wherein the one or more non-transitory computer-readable media collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations comprising (see claim 1 “one or more non-transitory computer-readable media that collectively store: a base encoder neural network configured to process an input image to generate an intermediate representation of the input image; a projection head neural network configured to process the intermediate representation of the input image to generate a projected representation of the input image, wherein to generate the projected representation, the projection head neural network is configured to perform at least one non-linear transformation; and instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising” )
Claim 1 does not expressly disclose receiving an input; processing the input using the neural network to generate an output representation from the neural network; and generating, based on the output representation, an output value
Chen discloses receiving an input; processing the input using the neural network to generate an output representation from the neural network; and generating, based on the output representation, an output value (see section 3 setting “Unsupervised learning is conducted on the 1.28MImageNet [3] training set. We follow two common protocols for evaluation. (i) ImageNet linear classification: features are frozen and a supervised linear classifier is trained (ii) Transferring to VOC object detection [4]: a Faster R-CNN detector [14] (C4-backbone) is fine-tuned end-to-end on the VOC 07+12 trainval set1 and evaluated on the VOC 07 testset using the COCO suite of metrics [10]. We use the same hyper-parameters (except when noted) and codebase as MoCo [6]. All results use a standard-size ResNet-50”). Note that after the unsupervised training a test set is applied to the neural network test a classification using a linear classifier). The motivation is to train a classifier (see section 3 setting “Unsupervised learning is conducted on the 1.28MImageNet [3] training set. We follow two common protocols for evaluation. (i) ImageNet linear classification: features are frozen and a supervised linear classifier is trained (ii) Transferring to VOC object detection [4]: a Faster R-CNN detector [14] (C4-backbone) is fine-tuned end-to-end on the VOC 07+12 trainval set1 and evaluated on the VOC 07 testset using the COCO suite of metrics [10]. We use the same hyper-parameters (except when noted) and codebase as MoCo [6]. All results use a standard-size ResNet-50”). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Claim 1 and Chen to reach the aforementioned advantage.
Re claim 21 Chen further discloses wherein the input comprises image data, and wherein generating the output value comprises: generating, based on the output representation, at least one of: a classification prediction, a detection prediction, a recognition prediction, or a segmentation prediction. (see section 3 setting “Unsupervised learning is conducted on the 1.28MImageNet [3] training set. We follow two common protocols for evaluation. (i) ImageNet linear classification: features are frozen and a supervised linear classifier is trained (ii) Transferring to VOC object detection [4]: a Faster R-CNN detector [14] (C4-backbone) is fine-tuned end-to-end on the VOC 07+12 trainval set1 and evaluated on the VOC 07 testset using the COCO suite of metrics [10]. We use the same hyper-parameters (except when noted) and codebase as MoCo [6]. All results use a standard-size ResNet-50”). Note that after the unsupervised training a test set is applied to the neural network test a classification using a linear classifier).
Re claim 23 claim 8 discloses wherein the projection head neural network comprises a multilayer perceptron (see claim 8 “wherein the projection head neural network comprises a multi-layer perceptron that comprises one hidden layer and a rectified linear unit non-linear activation function.”).
Re claim 24, claim 6 of the patent discloses wherein the one or more first augmentation operations comprise a blur operation. (see claim 6 “wherein the plurality of first augmentation operations further comprises a first random Gaussian blur operation that randomly applies a Gaussian blur to the training image and wherein the plurality of second augmentation operations further comprises a second random Gaussian blur operation that randomly applies a Gaussian blur to the training image.”)
Claim 22 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. 12254413 B2 in view of Chen et al “Improved Baselines with Momentum Contrastive Learning” 9 March 2020 arXiv:2003.04297 in further view of Lin US 20190355102 A1.
Re claim 22, claim 1 does not expressly disclose the encoder neural network comprises an attention mechanism. Lin discloses the encoder neural network comprises an attention mechanism. (see paragraph 70 note the encoder contains an attention layer) The motivation to combine is see paragraph 71 “As discussed in the following, the contextual attention pathway of the described system overcomes this limitation—the contextual attention pathway is capable of identifying features for content generation that are from distant spatial locations of the image under consideration”. One of ordinary skill in the art could have easily combine claim 1 with Lin to add an attention layer to the encoders of claim 1. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1, Chen and Lin to reach the aforementioned advantage.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 4-6 8-10, 13-15 17-21 23 and 24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al “Improved Baselines with Momentum Contrastive Learning” 9 March 2020 arXiv:2003.04297 in view of Bachman et al “Learning Representations by Maximizing Mutal Information Across Views” 2019 arXiv:1906.00910.
Re claim 1 Chen discloses A computer-implemented method, comprising:
obtaining one or more training inputs (see section 3 “Settings” section note that the image net training set is used);
performing one or more first augmentation operations on at least one of the training inputs to obtain a first augmented input; separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the at least one of the training inputs to obtain a second augmented input (see section 2 first paragraph a query and a key form a positive pair if they are data-augmented version of the same image” note that data augmented version of the same image are used);
computing a contrastive loss value between a first representation based on the first augmented input and a second representation based on the second augmented input, wherein computing the contrastive loss value comprises: (see figure 1 and equations 1 and section 2 contrastive learning last paragraph “In the MoCo mechanism(Fig. 1b) [6], the negative keys are maintained in a queue, and only the queries and positive keys are encoded in each training batch. A momentum encoder is adopted to improve the representation consistency between the current and earlier keys. MoCo decouples the batch size from the number of negatives” note that a loss in calculated between a momentum encoder and an encoder for images which are data augmented version of the same image);
processing, with a base encoder neural network, the first augmented input to generate a first intermediate representation for the first augmented input (see figure 1 and equations 1 and section 2 contrastive learning last paragraph “In the MoCo mechanism(Fig. 1b) [6], the negative keys are maintained in a queue, and only the queries and positive keys are encoded in each training batch. A momentum encoder is adopted to improve the representation
consistency between the current and earlier keys. MoCo decouples the batch size from the number of negatives” note that a loss in calculated between a momentum encoder and an encoder for images which are data augmented version of the same image);
and processing, with a projection head neural network, the first intermediate representation to obtain the first representation;(see section 2 improved designs and MLP head section note that that a projection head is used)
and modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the contrastive loss value (see section 3 MLP head “Using the default τ = 0.07 [16, 6], pre-training with the MLP head improves from 60.6% to 62.9%;” See also section 2 “The contrastive loss (1) can be minimized by various mechanisms that differ in how the keys are maintained [6].
In an end-to-end mechanism (Fig. 1a) [13, 8, 17, 1, 9, 2], the negative keys are from the same batch and updated end to-end by back-propagation. SimCLR [2] is based on this mechanism and requires a large batch to provide a large set of negatives. In the MoCo mechanism (Fig. 1b) [6], the negative keys are maintained in a queue, and only the queries
and positive keys are encoded in each training batch. A momentum encoder is adopted to improve the representation consistency between the current and earlier keys. MoCo
decouples the batch size from the number of negatives” note that the networks are trained ).
Chen does not expressly disclose wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input;
Bachman et al discloses wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input; (see section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations );
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of Chen. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Chen with Bachman to reach the aforementioned advantage.
Re claim 4 Chen discloses wherein the projection head neural network transforms the first intermediate representation using a nonlinear transformation (see section 3 MLP head “ (hidden layer2048-d, with ReLU” note that Relu is not linear).
Re claim 5 Chen discloses wherein the projection head neural network comprises a multilayer perceptron. (see section 3 MLP head, the examiner notes that MLP stands for multi-layer perceptron).
Re claim 6 Chen discloses wherein the one or more first augmentation operations comprise a blur operation.(see section 3 Augmentation “We extend the original augmentation in [6] by including the blur augmentation”)
Re claim 8 Chen further does not expressly disclose wherein the one or more first augmentation operations comprise a first resize operation that resizes the training input.
Bachman discloses wherein the one or more first augmentation operations comprise a first resize operation that resizes the training input. (see section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations );
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of Chen. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 with Bachman to reach the aforementioned advantage.
Re claim 9 Chen further does not disclose wherein the one or more first augmentation operations comprises a first random flip operation that randomly flips the training input.
Bachman discloses wherein the one or more first augmentation operations comprises a first random flip operation that randomly flips the training input.
(see section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations );
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of Chen. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Chen with Bachman to reach the aforementioned advantage.
Re claim 10 A computing system comprising: one or more processors; and instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising (see table 3 and section 3 computational cost note that the invention is implemented on a GPU and stored in memory).
obtaining one or more training inputs (see section 3 “Settings” section note that the image net training set is used);
performing one or more first augmentation operations on at least one of the training inputs to obtain a first augmented input; separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the at least one of the training inputs to obtain a second augmented input (see section 2 first paragraph a query and a key form a positive pair if they are data-augmented version of the same image” note that data augmented version of the same image are used);
computing a contrastive loss value between a first representation based on the first augmented input and a second representation based on the second augmented input, wherein computing the contrastive loss value comprises: (see figure 1 and equations 1 and section 2 contrastive learning last paragraph “In the MoCo mechanism(Fig. 1b) [6], the negative keys are maintained in a queue, and only the queries and positive keys are encoded in each training batch. A momentum encoder is adopted to improve the representation consistency between the current and earlier keys. MoCo decouples the batch size from the number of negatives” note that a loss in calculated between a momentum encoder and an encoder for images which are data augmented version of the same image);
processing, with a base encoder neural network, the first augmented input to generate a first intermediate representation for the first augmented input (see figure 1 and equations 1 and section 2 contrastive learning last paragraph “In the MoCo mechanism(Fig. 1b) [6], the negative keys are maintained in a queue, and only the queries and positive keys are encoded in each training batch. A momentum encoder is adopted to improve the representation
consistency between the current and earlier keys. MoCo decouples the batch size from the number of negatives” note that a loss in calculated between a momentum encoder and an encoder for images which are data augmented version of the same image);
and processing, with a projection head neural network, the first intermediate representation to obtain the first representation;(see section 2 improved designs and MLP head section note that that a projection head is used)
and modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the contrastive loss value (see section 3 MLP head “Using the default τ = 0.07 [16, 6], pre-training with the MLP head improves from 60.6% to 62.9%;” See also section 2 “The contrastive loss (1) can be minimized by various mechanisms that differ in how the keys are maintained [6].
In an end-to-end mechanism (Fig. 1a) [13, 8, 17, 1, 9, 2], the negative keys are from the same batch and updated end to-end by back-propagation. SimCLR [2] is based on this mechanism and requires a large batch to provide a large set of negatives. In the MoCo mechanism(Fig. 1b) [6], the negative keys are maintained in a queue, and only the queries
and positive keys are encoded in each training batch. A momentum encoder is adopted to improve the representation consistency between the current and earlier keys. MoCo
decouples the batch size from the number of negatives” note that the networks are trained).
Chen does not expressly disclose wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input;
Bachman et al discloses wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input; (see section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations );
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of Chen. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Chan with Bachman to reach the aforementioned advantage.
Re claim 13 Chen discloses wherein the projection head neural network transforms the first intermediate representation using a nonlinear transformation (see section 3 MLP head “ (hidden layer2048-d, with ReLU” note that Relu is not linear).
Re claim 14 Chen discloses wherein the projection head neural network comprises a multilayer perceptron. (see section 3 MLP head, the examiner notes that MLP stands for multi-layer perceptron).
Re claim 15 Chen discloses wherein the one or more first augmentation operations comprise a blur operation.(see section 3 Augmentation “We extend the original augmentation in [6] by including the blur augmentation”)
Re claim 17 Chen further does not expressly disclose wherein the one or more first augmentation operations comprise a first resize operation that resizes the training input.
Bachman discloses wherein the one or more first augmentation operations comprise a first resize operation that resizes the training input. (see section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations );
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of Chen. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 with Bachman to reach the aforementioned advantage.
Re claim 18 Chen further does not disclose wherein the one or more first augmentation operations comprises a first random flip operation that randomly flips the training input.
Bachman discloses wherein the one or more first augmentation operations comprises a first random flip operation that randomly flips the training input.
(see section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations );
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of Chen. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 1 with Bachman to reach the aforementioned advantage.
Re claim 19. One or more non-transitory computer-readable media that collectively store a neural network trained by (see table 3 and section 3 computational cost note that the invention is implemented on a GPU and stored in memory).
obtaining one or more training inputs (see section 3 “Settings” section note that the image net training set is used);
performing one or more first augmentation operations on at least one of the training inputs to obtain a first augmented input; separate from performing the one or more first augmentation operations, performing one or more second augmentation operations on the at least one of the training inputs to obtain a second augmented input (see section 2 first paragraph a query and a key form a positive pair if they are data-augmented version of the same image” note that data augmented version of the same image are used);
computing a contrastive loss value between a first representation based on the first augmented input and a second representation based on the second augmented input, wherein computing the contrastive loss value comprises: (see figure 1 and equations 1 and section 2 contrastive learning last paragraph “In the MoCo mechanism(Fig. 1b) [6], the negative keys are maintained in a queue, and only the queries and positive keys are encoded in each training batch. A momentum encoder is adopted to improve the representation consistency between the current and earlier keys. MoCo decouples the batch size from the number of negatives” note that a loss in calculated between a momentum encoder and an encoder for images which are data augmented version of the same image);
processing, with a base encoder neural network, the first augmented input to generate a first intermediate representation for the first augmented input (see figure 1 and equations 1 and section 2 contrastive learning last paragraph “In the MoCo mechanism(Fig. 1b) [6], the negative keys are maintained in a queue, and only the queries and positive keys are encoded in each training batch. A momentum encoder is adopted to improve the representation
consistency between the current and earlier keys. MoCo decouples the batch size from the number of negatives” note that a loss in calculated between a momentum encoder and an encoder for images which are data augmented version of the same image);
and processing, with a projection head neural network, the first intermediate representation to obtain the first representation;(see section 2 improved designs and MLP head section note that that a projection head is used)
and modifying one or more values of one or more parameters of one or both of the base encoder neural network and the projection head neural network based at least in part on the contrastive loss value (see section 3 MLP head “Using the default τ = 0.07 [16, 6], pre-training with the MLP head improves from 60.6% to 62.9%;” See also section 2 “The contrastive loss (1) can be minimized by various mechanisms that differ in how the keys are maintained [6].
In an end-to-end mechanism (Fig. 1a) [13, 8, 17, 1, 9, 2], the negative keys are from the same batch and updated end to-end by back-propagation. SimCLR [2] is based on this mechanism and requires a large batch to provide a large set of negatives. In the MoCo mechanism(Fig. 1b) [6], the negative keys are maintained in a queue, and only the queries
and positive keys are encoded in each training batch. A momentum encoder is adopted to improve the representation consistency between the current and earlier keys. MoCo
decouples the batch size from the number of negatives” note that the networks are trained).
Chen does not expressly disclose wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input;
Bachman et al discloses wherein at least one of the one or more first augmentation operations or the one or more second augmentation operations comprise one or both of: a random crop operation that randomly crops the training input and a random color distortion operation that randomly modifies color values of the training input; (see section 3.4 data augmentation note data augmentation can include random resized cropping and random jitter and random horizontal flips in the color space see figure 1 elements a note that two training images are used with different augmentations );
One of ordinary skill in the art could have used the more detailed data augmentation of Bachman to modify the data augmentation of Chen. The motivation to combine is that “Our model extends local DIM in three key ways: it predicts features across independently-augmented versions of each input, it predicts features simultaneously across multiple scales, and it uses a more powerful encoder. Each of these modifications provides improvements over local DIM. Predicting across independently-augmented copies of an input and predicting at multiple scales are two simple ways of producing multiple views of the context provided by a single image.” (see section 1 3rd paragraph). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 6 with Bachman to reach the aforementioned advantage.
Re claim 20 Chen discloses wherein the one or more non-transitory computer-readable media collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations comprising: (see table 3 and section 3 computational cost note that the invention is implemented on a GPU and stored in memory). receiving an input; processing the input using the neural network to generate an output representation from the neural network; and generating, based on the output representation, an output value (see section 3 setting “Unsupervised learning is conducted on the 1.28MImageNet [3] training set. We follow two common protocols for evaluation. (i) ImageNet linear classification: features are frozen and a supervised linear classifier is trained (ii) Transferring to VOC object detection [4]: a Faster R-CNN detector [14] (C4-backbone) is fine-tuned end-to-end on the VOC 07+12 trainval set1 and evaluated on the VOC 07 testset using the COCO suite of metrics [10]. We use the same hyper-parameters (except when noted) and codebase as MoCo [6]. All results use a standard-size ResNet-50”). Note that after the unsupervised training a test set is applied to the neural network test a classification using a linear classifier).
Re claim 21 Chen discloses wherein the input comprises image data, and wherein generating the output value comprises: generating, based on the output representation, at least one of: a classification prediction, a detection prediction, a recognition prediction, or a segmentation prediction. (see section 3 setting “Unsupervised learning is conducted on the 1.28MImageNet [3] training set. We follow two common protocols for evaluation. (i) ImageNet linear classification: features are frozen and a supervised linear classifier is trained (ii) Transferring to VOC object detection [4]: a Faster R-CNN detector [14] (C4-backbone) is fine-tuned end-to-end on the VOC 07+12 trainval set1 and evaluated on the VOC 07 testset using the COCO suite of metrics [10]. We use the same hyper-parameters (except when noted) and codebase as MoCo [6]. All results use a standard-size ResNet-50”). Note that after the unsupervised training a test set is applied to the neural network test a classification using a linear classifier).
Re claim 23 Chen discloses wherein the projection head neural network comprises a multilayer perceptron. (see section 3 MLP head, the examiner notes that MLP stands for multi-layer perceptron).
Re claim 24 Chen discloses wherein the one or more first augmentation operations comprise a blur operation. (see section 3 Augmentation “We extend the original augmentation in [6] by including the blur augmentation”)
Claim(s) 2, 11 and 22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al “Improved Baselines with Momentum Contrastive Learning” 9 March 2020 arXiv:2003.04297 Bachman et al “Learning Representations by Maximizing Mutal Information Across Views” 2019 arXiv:1906.00910 in further view of Lin US 20190355102 A1.
Re claim 2 Chen and Bachman do not expressly disclose the encoder neural network comprises an attention mechanism. Lin discloses the encoder neural network comprises an attention mechanism. (see paragraph 70 note the encoder contains an attention layer) The motivation to combine is see paragraph 71 “As discussed in the following, the contextual attention pathway of the described system overcomes this limitation—the contextual attention pathway is capable of identifying features for content generation that are from distant spatial locations of the image under consideration”. One of ordinary skill in the art could have easily combine Chen and Bachman with Lin to add an attention layer to the encoders of Chen. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Chen Bachman and Lin to reach the aforementioned advantage.
Re claim 11 Chen and Bachman do not expressly disclose the encoder neural network comprises an attention mechanism. Lin discloses the encoder neural network comprises an attention mechanism. (see paragraph 70 note the encoder contains an attention layer) The motivation to combine is see paragraph 71 “As discussed in the following, the contextual attention pathway of the described system overcomes this limitation—the contextual attention pathway is capable of identifying features for content generation that are from distant spatial locations of the image under consideration”. One of ordinary skill in the art could have easily combine Chen and Bachman with Lin to add an attention layer to the encoders of Chen. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Chen Bachman and Lin to reach the aforementioned advantage.
Re claim 22 Chen and Bachman do not expressly disclose the encoder neural network comprises an attention mechanism. Lin discloses the encoder neural network comprises an attention mechanism. (see paragraph 70 note the encoder contains an attention layer) The motivation to combine is see paragraph 71 “As discussed in the following, the contextual attention pathway of the described system overcomes this limitation—the contextual attention pathway is capable of identifying features for content generation that are from distant spatial locations of the image under consideration”. One of ordinary skill in the art could have easily combine Chen and Bachman with Lin to add an attention layer to the encoders of Chen. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Chen Bachman and Lin to reach the aforementioned advantage.
Allowable Subject Matter
Claim 3 and 7 would be allowable if a terminal disclaimer was field to overcome the rejection(s) with respect to double patenting, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
Re claim 3 the prior art of record does not expressly disclose “generating a classification model from the base encoder neural network and the projection head neural network, the classification model comprising some but not all of the plurality of layers of the projection head neural network”
Re claim 7 the prior art of record does not expressly disclose “generating a classification model from the base encoder neural network and the projection head neural network, the classification model comprising some but not all of the plurality of layers of the projection head neural network”
Claim 12 and 16 would be allowable if a terminal disclaimer was field to overcome the rejection(s) with respect to double patenting, set forth in this Office action, and if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
Re claim 12 the prior art of record does not expressly disclose “generating a classification model from the base encoder neural network and the projection head neural network, the classification model comprising some but not all of the plurality of layers of the projection head neural network”
Re claim 16 the prior art of record does not expressly disclose “generating a classification model from the base encoder neural network and the projection head neural network, the classification model comprising some but not all of the plurality of layers of the projection head neural network”
Cited Art
The following is a listing of prior art considered relevant but not applied in a rejection above:
Zadeh US 20200184278 A1 discloses “In one embodiment, as for example depicted in FIG. 187, a sample generator is used to provide data (e.g., images) for training.: rendering unit renders the data according to one or more models (e.g., functional, tabular, and/or heuristic) and the corresponding model parameters governing the instantiation of the model by the rendering unit. In one embodiment, at least a subset of model parameters are generated stochastically (or via a deterministic sequential algorithm) by a randomizer unit, which for example, uses applicable probability model(s) and/or model rules to generate the subset of model parameters within given ranges or constraints. In one embodiment, the training of the network (e.g., a deep belief network based on Boltzmann machines) is done repeatedly generating training data samples via the sample generator to teed to the V layer of a network being trained. In one embodiment, the training is done one hidden layer at the time (e.g., until H.sup.(3)). In one embodiment, the training of hidden layers is done unsupervised (i.e., without supplying labeled training samples). In one embodiment, an autoencoder is setup (e.g., as shown in FIG. 187) and fine tuned using back propagation. In one embodiment, a correlation or associative layer is added to learn the correlation between the features of the data and the labels (L.sub.M), where the labels are supplied by the sample generator (along with the rendered data). In one embodiment, for example as depicted in FIG. 188, multiple L.sub.M layers (e.g., in parallel) are used to represent various classes of (e.g., independent) models. In one embodiment, the relevant weights between C layer and an L.sub.M layer are fixed for one class of model(s) while training another class of model(s) through the same C layer. In one embodiment, the cross correlation between two models is determined, via cross correlation (e.g., through layer C) between the labels associates with both models. For example, by a subset of labels from L.sub.M1 layer is clamped and sampled generated from top-down reconstruction from layer C to layer LM2 are used to determine such cross correlation. In one embodiment, states on layer C are stochastically run to derive the reconstruction in both L.sub.M1 and L.sub.M2 layers for determining a correlation between the reconstructions samples. In one embodiment, the units in layer C are derived (e.g., through inference) from V layer (by inputting data), and labels are reconstructed. In layers L.sub.M1 and L.sub.M2. In one embodiment, the levels of conformity or correlation of data supplied to V units (or a subset of V units) with models(s) are obtained for each model based on relative probabilities and energy of states. In comparing on model to another, the weights associated with one model are not used in determining energy or probability associated with the other model (for such comparison).” (see paragraph 1779).
Park US 20210334994 A1 discloses A multiple instance learning device for analyzing 3D images, comprises a memory in which a multiple instance learning model is stored and at least one processor electrically connected to the memory, wherein the multiple instance learning model comprises a convolution block configured to derive a feature map for each of 2D instances of a 3D image inputted to the multiple instance learning model, a spatial attention block configured to derive spatial attention maps of the instances from the feature maps derived from the convolution block, an instance attention block configured to receive a result of combining the feature maps and the spatial attention maps and derive an attention score for each instance, and derive an aggregated embedding for the 3D image by aggregating embeddings of the instances according to the attention scores. (see abstract)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SEAN T MOTSINGER whose telephone number is (571)270-1237. The examiner can normally be reached 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SEAN T MOTSINGER/Primary Examiner, Art Unit 2673