Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The abstract of the disclosure is objected to because of the inclusion of “[Figure 3]” on line 11. A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP § 608.01(b).
Claim Interpretation
Note that according to the Federal Circuit’s 2004 Superguide v. DirecTV decision, “at least one of … and … “ requires at least one instance of each and every item listed. Claim 12 contain such limitations, however, the specification supports a disjunctive interpretation (see example tasks of page 15, lines 16-17). For examination purposes, the limitations be interpreted under the broader disjunctive interpretation, requiring at least one instance of any of the items listed.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2-3, 7-8, 10, and 15-16 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 2, lines 4-6 and 9-11, recites “updating the first patch tokens by analyzing the other first patch tokens with an attention mechanism to generate the first set of embeddings comprising first updated patch tokens” and “updating the second patch tokens by analyzing the other second patch tokens with an attention mechanism to generate the second set of embeddings comprising second updated patch tokens”, respectively, which is indefinite. The claim terms “the other first patch tokens” and “the other second patch tokens” lacks antecedent basis; thus, it is unclear how each of the first and second patch tokens are updated. For example, is the claim meant to require all of the first and second patch tokens being analyzed to generate their respective set of embeddings or only part of the first and second patch tokens being analyzed to generate their respective set of embeddings. For examination purposes, the claim limitations will be interpreted to mean that updating the first and second patch tokens requires analyzing one or more of the first patch tokens and one or more of the second patch tokens, respectively, to generate their respective set of embeddings.
Claim 3 recites “a said input image” which lacks antecedent basis. It is unclear if the limitation is meant to refer to “the first input image” and/or “the second input images” as previously recited in the claims, or if the limitation is meant to refer to a new element. Similarly, claim 3 recites “a plurality of said patch tokens”, which lacks antecedent basis. It is unclear if the limitation is mean to refer to “the first patch tokens”, “the second patch tokens”, “the other first patch tokens”, “the other second patch tokens”, “first updated patch tokens” and/or “second updated patch tokens” as previously recited in the claims, or if is meant to refer to a new element. For examination purposes, the claim will be interpreted to mean dividing any input image into a plurality of patches including a grid of non-overlapping contiguous patch tokens.
Claim 7, lines 3, recites “the reference image”, which lacks antecedent basis. The claim requires that the first and second input images comprises (1) first and second augmentations of a reference image or (2) the reference image and an augmentation thereof. In other words, one or the other applies. In the instance (2) applies, there is no antecedents in the claims for “the reference image”. For examination purposes, the limitation will be interpreted as “a reference image”.
Claim 7 further recites “adjusting the at least one network weight of the first image encoder network to reduce or bring to or towards zero the difference between the first and second representations”, which is indefinite. The listing of elements is unclear. For example, the interpretation of the claim element “adjusting the at least one network weight… to reduce the difference” is definite, however, claim elements “adjusting the at least one network weight… bring to the difference” and “adjusting the at least one network weight… towards zero the difference” create ambiguity as to how the weight is adjusted with respect to the difference. For examination purposes, the limitation will be interpreted as “adjusting the at least one network weight of the first image encoder network to reduce or bring towards zero the difference between the first and second representations”.
Claim 8 recites “a said augmentation comprises…” which lacks antecedent basis. It is unclear if the limitation is meant to refer to either of the first and second augmentations as previously recited in the claims, or if the limitation is meant to refer to a new element. For examination purposes, the limitation will be interpreted to as referring to either of the first or second augmentations.
Claim 10 recites “the updates to the first image encoder network”, which lacks antecedent basis. It is unclear what the claim limitation “the updates” is meant to refer to in the iterative training process of claim 9. For example, is updates meant to refer to updates generated for each iteration of the training process or updates to the network in some other form. For examination purposes, the limitation will be interpreted as distilling any updates to the first image encoder network after a number of iterations of the training process to the second image encoder network.
Claim 15 recites “selecting k nearest neighbors for each said updated patch token and including nodes in the NNG corresponding to the said updated patch tokens”, which lacks antecedent basis. It is unclear if the limitations “each said updated patch token” and “the said updated patch tokens” are meant to refer to as claim 1, of which claim 15 is dependent on, does not recite any updated patch tokens. Examiner believes the intent of Applicant was to make claim 15 dependent from claim 2, thus for examination purposes the claim will be interpreted as if dependent from claim 2. However, similar to the arguments made above corresponding to claim 3, the claim limitations would still lack clear antecedent basis as “each said updated patch token” does not properly refer to either of the previously recited “the first patch tokens” or “the second patch tokens”. Furthermore, it is unclear if the claim limitation “the NNG” is meant to refer to the first or second NNG. For examination purposes, the claim will be interpreted to mean that k nearest neighbors are selected for any of the updated first patch tokens or the updated second patch tokens of claim 2, and that nodes are included in either the first or second NNG depending on which updated patch is selected.
Claim 16 recites “a given patch token” and “the patch tokens concerned”, which lacks antecedent basis. It is unclear if the limitations “a given patch token” and “the patch tokens concerned” are meant to refer to either of the previously recited “the first patch tokens” or “the second patch tokens” or if they are meant to refer to a new element. For examination purposes, the claim will be interpreted to mean selecting the k nearest neighbors for any of the updated first patch tokens or the updated second patch tokens of claim 2 comprises selecting k patch tokens among any of the first patch tokens, the updated first patch tokens, the second patch tokens, or the second updated patch tokens which has the smallest distance from the updated patch that is selected.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 19 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter (i.e., computer program per se. See MPEP § 2106.03).
Claim 19 recites “A computer program which, when run on a computer, causes the computer to carry out a method comprising…”. The broadest reasonable interpretation of computer program includes software. Software expressed as code or a set of instructions detached from any medium is an idea without physical embodiment. Thus, a claim to a software program that does not also contain at least one structural limitation (such as a "means plus function" limitation) has no physical or tangible form, and thus does not fall within any statutory category: processes, machines, manufactures or compositions of matter (See MPEP 2106.03). Therefore, claim 19 is rejected under 35 U.S.C. 101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4, 7-9, 11-13, and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Kefato et al ("Jointly learnable data augmentations for self-supervised gnns.", arXiv preprint arXiv:2108.10420, 2021), (hereinafter Kefato) in view of Tang et al. (hereinafter Tang), (“Unifying Visual Contrastive Learning for Object Recognition from a Graph Perspective”, European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022).
Regarding claim 1, Kefato teaches a computer-implemented method comprising performing a training process (Kefato, “In this study, we propose an SSL model for GNNs based on the Siamese architecture called GRAPHSURGEON (self-supervised GNN that jointly learns to augment)... We design an SSL architecture in such a way that data augmentation is jointly learned with the graph representation.”, pg. 2, 3rd full paragraph, lines 1-4), the training process comprising:
using first and second image encoder networks, generating first and second sets of embeddings based on a pair of first and second input images, respectively; using at least one graph neural network, GNN, extracting first and second representations; and adjusting at least one network weight of the first image encoder network based on a difference between the first and second representations (Kefato, “To augment a given node v, we simply use two augmentation heads
v
1
=
f
θ
1
(
v
)
and
v
2
=
f
θ
2
(
v
)
, parameterized by two separate sets of weights
θ
1
and
θ
2
, which produce two views
v
1
and
v
2
of v. For example,
f
θ
1
(
v
)
and
f
θ
2
(
v
)
could be simple MLP heads.
v
1
and
v
2
are fed to a shared GNN,
h
θ
(
∙
)
, parameterized by a set of weights
θ
. Here, the key idea is that the parameters of the augmentation heads,
θ
1
and
θ
2
, are jointly learned with
θ
. Because the two networks are symmetric and equivalent, we use the terms left and right, instead of student and teacher.”, pg. 2, 4th full paragraph, “Our goal in an SSL-GNN framework is to maximize the agreement between these two representations. To this end, we closely follow Laplacian Eigenmaps Belkin and Niyogi (2003) and minimize the mean squared error between the normalized representations (unit vectors) of two data points. Though in Laplacian Eigenmaps the two data points correspond to different objects (e.g., two different nodes, images), in our case, these are just the unit embedding vectors of the two augmented views, which are
Z
-
1
and
Z
-
2
(z1_unit and z2_unit in the pseudocode).”, pg. 5, 2nd full paragraph, lines 1-5, “Finally, when training GRAPHSURGEON, gradients are back propagated on both the left and right networks. As a result, all the parameters are updated according to the loss incurred with respect to the signal from the graph. This in turn, allows the parameters of both the encoder,
h
θ
, and the augmenters,
f
θ
1
and
f
θ
2
, to be governed by the graph signal.”, pg. 5, 7th full paragraph, lines 1-3, A training process in implemented using a Siamese architecture described as having two parallel processing paths, referred to as ‘left’ and ‘right” networks. These networks are used to generate respective embeddings, Z1 and Z2, from different augmented views of an input graph. The embeddings are then used to compute a loss based on their agreement, and the weights of the networks are adjusted based on this difference.).
Kefato does not teach generating first and second nearest neighbor graphs, NNGs, based on the first and second sets of embeddings, respectively; and extracting first and second representations from the first and second NNGs, respectively.
However, Tang teaches generating first and second nearest neighbor graphs, NNGs, based on the first and second sets of embeddings, respectively; and extracting first and second representations from the first and second NNGs, respectively (Tang, “Give
z
1
and
z
2
from Step 1, we respectively construct the fully connected graph
G
(
z
1
)
and
G
(
z
2
)
, where nodes in
G
(
z
1
)
and
G
(
z
2
)
are K nearest neighbors of
z
1
and
z
2
in support queue S, respectively. Then we implement typical graph augmentations (Sec. 3.3) to generate the augmented graphs
G
~
(
z
1
)
and
G
~
(
z
2
)
.”, pg. 5, Step2: Graph Construction and Augmentation, “Recent states-of-the-art methods treats the sample and its K nearest neighbors in the feature space as the positive samples. Given
F
l
=
(
f
l
0
,
f
l
1
,
…
,
f
l
k
)
. The K nearest neighbor predictor can be presented as (see eq. (8)).”, pg. 8, 2nd full paragraph, Nearest neighbor graphs (NNGs) are constructed based on embedding produced from two views of an input image. A GCN predictor is then used to extract representations from the NNGs.).
Kefato teaches implementing a training process using a Siamese architecture with two encoding networks used to generate respective embeddings from different augmented views of an input graph (Kefato, pg. 2, 4th full paragraph and pg. 5, 2nd full paragraph, lines 1-5). Kefato further teaches using the embeddings to determine a difference for adjusting model weights (Kefato, pg. 5, 7th full paragraph, lines 1-3) but does not teach generating nearest neighbor graphs (NNGs) based on the embeddings or extracting representations from these NNGs. Tang teaches constructing NNGs from model produced embeddings to extract representations of input images (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the Siamese architecture of Kefato to include the NNG representation extraction as taught by Tang (Tang, pg. 5, Step2: Graph Construction and Augmentation and pg. 8, 2nd full paragraph). The motivation for doing so would have been to enable neighborhood aggregation, thereby increasing the linear evaluation performance (as suggested by Tang, “Different from self-supervised learning methods that do not use supervision from other samples, neighborhood based methods uses K nearest neighbors as their positive samples. This can be achieved by using the neighborhood aggregation term in Eq. 3 in GCN predictor. As shown in Tab. 3, we have three findings. First, the linear evaluation performances of using neighboring information on the target branch are significantly higher than those self-supervised learning methods by a considerable 2.1% gain by comparing Exp.(b) and Exp. (f).”, pg. 11, 2nd full paragraph, lines 1-8). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Kefato with Tang to obtain the invention as specified in claim 1.
Regarding claim 4, Kefato in view of Tang teaches the computer-implemented method as claimed in claim 1, wherein the training process comprises computing the difference between the first and second representations (Kefato, “Our goal in an SSL-GNN framework is to maximize the agreement between these two representations. To this end, we closely follow Laplacian Eigenmaps Belkin and Niyogi (2003) and minimize the mean squared error between the normalized representations (unit vectors) of two data points. Though in Laplacian Eigenmaps the two data points correspond to different objects (e.g., two different nodes, images), in our case, these are just the unit embedding vectors of the two augmented views, which are
Z
-
1
and
Z
-
2
(z1_unit and z2_unit in the pseudocode).”, pg. 5, 2nd full paragraph, lines 1-5).
Regarding claim 7, Kefato in view of Tang teaches the computer-implemented method as claimed in claim 1, wherein the first and second input images comprise first and second augmentations of a reference image or the reference image and an augmentation thereof, and wherein adjusting the at least one network weight of the first image encoder network comprises adjusting the at least one network weight of the first image encoder network to reduce or bring to or towards zero the difference between the first and second representations (Kefato, “Our goal in an SSL-GNN framework is to maximize the agreement between these two representations. To this end, we closely follow Laplacian Eigenmaps Belkin and Niyogi (2003) and minimize the mean squared error between the normalized representations (unit vectors) of two data points. Though in Laplacian Eigenmaps the two data points correspond to different objects (e.g., two different nodes, images), in our case, these are just the unit embedding vectors of the two augmented views, which are
Z
-
1
and
Z
-
2
(z1_unit and z2_unit in the pseudocode).”, pg. 5, 2nd full paragraph, lines 1-5, “we modify E.q. 2 and incorporate an orthonormality constraint inspired by the Laplacian Eigenmaps. Moreover, we want to jointly optimize the parameters of the augmenters. To achieve these goals we update E.q. 2 and formulate it as a constrained optimization objective as in Eq. 3. (see Eq. 3) Eq. 3 encourages positive pairs across
Z
-
1
and
Z
-
2
to be similar to each other, and the orthonormality constraint ensures that each row in
Z
-
1
and
Z
-
2
is similar to itself and orthonormal to other rows.”, pg. 5, 3rd and 4th full paragraphs, Input images include two augmented views. The parameters for the augmenters are updated jointly by encouraging the difference between the representations to be similar to each other.).
Regarding claim 8, Kefato in view of Tang teaches the computer-implemented method as claimed in claim 7, wherein a said augmentation comprises any of: a recoloring of the reference image; a brightness adjustment of the reference image; cropping the reference image; blurring the reference image; flipping the reference image; and a rotation of the reference image (Kefato, “Our goal in an SSL-GNN framework is to maximize the agreement between these two representations. To this end, we closely follow Laplacian Eigenmaps Belkin and Niyogi (2003) and minimize the mean squared error between the normalized representations (unit vectors) of two data points. Though in Laplacian Eigenmaps the two data points correspond to different objects (e.g., two different nodes, images), in our case, these are just the unit embedding vectors of the two augmented views, which are
Z
-
1
and
Z
-
2
(z1_unit and z2_unit in the pseudocode).”, pg. 5, 2nd full paragraph, lines 1-5, Augmentation of input images includes producing two different augmented views. This would include standard image augmentation like resizing, flipping, or rotating.).
Regarding claim 9, Kefato in view of Tang teaches the computer-implemented method as claimed in claim 1, comprising performing the training process for a plurality of iterations with different pairs of input images for each iteration (Kefato, “For this reason, we train the model using sampled neighborhood subgraphs Hamilton et al. (2018) instead of full-batch, and both the model and the linear head are trained for 100 epochs.” pg. 9, 4th full paragraph, Input images used for training correspond to a number of nodes in the training batch where each node is used to define different pairs (views) of the input images. This training process is repeated for each sample in the training set.).
Regarding claim 11, Kefato in view of Tang teaches the computer-implemented method as claimed in claim 1, further comprising using the first image encoder network in an image processing task after performing the training process (Kefato, “We validate the practical use of GRAPHSURGEON using 14 publicly available datasets, ranging from small to large-scale graphs. All of the datasets are collected from PyTorch Geometric (PyG) 1, and grouped as… Social (Facebook, Flickr, GitHub, Reddit, and Yelp): Facebook contains a page to page graph of verified Facebook sites, and we want to classify pages into their categories Rozemberczki et al. (2021). Flickr contains a network of images based on common properties (e.g., geo-location) along with their description, and the task is to predict a unique tag of an image Zeng et al. (2020).”, pg. 7, 1st paragraph, 5th bullet point).
Regarding claim 12, Kefato in view of Tang teaches the computer-implemented method as claimed in claim 11, wherein the image processing task comprises at least one of: visual question answering, VQA; object detection; image classification; image segmentation; and image retrieval (Kefato, “We validate the practical use of GRAPHSURGEON using 14 publicly available datasets, ranging from small to large-scale graphs. All of the datasets are collected from PyTorch Geometric (PyG) 1, and grouped as… Social (Facebook, Flickr, GitHub, Reddit, and Yelp): Facebook contains a page to page graph of verified Facebook sites, and we want to classify pages into their categories Rozemberczki et al. (2021). Flickr contains a network of images based on common properties (e.g., geo-location) along with their description, and the task is to predict a unique tag of an image Zeng et al. (2020).”, pg. 7, 1st paragraph, 5th bullet point, Predicting a unique tag of an image can be considered a form of image classification.).
Regarding claim 13, Kefato in view of Tang teaches the computer-implemented method as claimed in claim 1, wherein the first and second image encoder networks each comprises a transformer-based architecture (Kefato, “Furthermore, because of the flexibility of the learnable augmenters, we introduce an alternative new strategy called post-augmentation. Post-augmentation applies augmentation in a latent space. That is, we first encode (e.g., using a GNN, CNN, or Transformer) and augment the encoded representations.”, pg. 2, 4th full paragraph, lines 1-2 and 5th full paragraph, lines 1-3, In the post-augmentation, the input is first encoded using a transformer followed by augmentation using the left and right networks. Thus, both networks can be considered to have a transformer-based architecture).
Regarding claim 17, Kefato in view of Tang teaches the computer-implemented method as claimed in claim 1, wherein extracting the first and second representations comprises generating first and second graph embeddings of the of first and second NNGs, respectively (Tang, “Step 3 : GCN Predictor (Sec. 3.1). The augmented graphs
G
~
z
1
and
G
~
z
2
are respectively transformed to prediction features
q
1
and
q
2
through GCN predictors
P
*
,
ξ
and
P
'
*
,
ξ
'
.
”, pg. 5, 3rd full paragraph, Step 3).
Regarding claim 18, Kefato in view of Tang teaches the computer-implemented method as claimed in claim 1, wherein the training process comprises computing an embedding difference between the first and second sets of embeddings, and wherein the adjustment of the at least one network weight of the first image encoder network is based on the embedding difference (Kefato, “Our goal in an SSL-GNN framework is to maximize the agreement between these two representations. To this end, we closely follow Laplacian Eigenmaps Belkin and Niyogi (2003) and minimize the mean squared error between the normalized representations (unit vectors) of two data points. Though in Laplacian Eigenmaps the two data points correspond to different objects (e.g., two different nodes, images), in our case, these are just the unit embedding vectors of the two augmented views, which are
Z
-
1
and
Z
-
2
(z1_unit and z2_unit in the pseudocode).”, pg. 5, 2nd full paragraph, lines 1-5, The networks are used to generate respective embeddings, Z1 and Z2, from different augmented views of an input graph. The embeddings are then used to compute a loss based on their agreement, and the weights of the networks are adjusted based on this difference).
Claim 19 corresponds to claim 1, with the addition of a computer program which, when run on a computer, causes the computer to carry out a method comprising the steps according to claim 1. Kefato in view of Tang teaches the addition of computer program which, when run on a computer, causes the computer to carry out a method comprising the steps according to claim 1 (Kefato, “For all the datasets we have three splits, training, validation and test. For some of them, we use the splits provided by PyTorch Geometric, and for the rest, we randomly split them into 5% training, 15% validation and 80% test sets. We tune the hyperparameters of all the algorithms using Bayesian optimization 2, however for a fair comparison, we fix the representation dimension to 128. In addition, we run all the models for 500 epochs and take the epoch with the best validation score.”, pg. 8, 1st full paragraph, a computer program is required for obtaining PyTorch training datasets and implementing the model during training and testing.). As indicated in the analysis of claim 1, Kefato in view of Tang teaches all the limitations according to claim 1. Therefore, claim 19 is rejected for the same reasons of obviousness as claim 1.
Claim 20 corresponds to claim 1, with the addition of an information processing apparatus comprising a memory and processor connected to the memory, wherein the processor is configured perform the steps according to claim 1. Kefato in view of Tang teaches the addition of an information processing apparatus comprising a memory and processor connected to the memory, wherein the processor is configured perform the steps according to claim 1 (Kefato, “For all the datasets we have three splits, training, validation and test. For some of them, we use the splits provided by PyTorch Geometric, and for the rest, we randomly split them into 5% training, 15% validation and 80% test sets. We tune the hyperparameters of all the algorithms using Bayesian optimization 2, however for a fair comparison, we fix the representation dimension to 128. In addition, we run all the models for 500 epochs and take the epoch with the best validation score.”, pg. 8, 1st full paragraph, a memory and processor is required for obtaining and storing PyTorch training datasets and implementing the model during training and testing). As indicated in the analysis of claim 1, Kefato in view of Tang teaches all the limitations according to claim 1. Therefore, claim 20 is rejected for the same reasons of obviousness as claim 1.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Kefato et al ("Jointly learnable data augmentations for self-supervised gnns.", arXiv preprint arXiv:2108.10420, 2021) in view of Tang et al. (“Unifying Visual Contrastive Learning for Object Recognition from a Graph Perspective”, European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022) and further in view of Mou et al. (“Dynamic attentive graph learning for image restoration”, Proceedings of the IEEE/CVF international conference on computer vision. 2021.), (hereinafter Mou).
Regarding claim 2, Kefato in view of Tang teaches the computer-implemented method as claimed in claim 1. Kefato in view of Tang does not teach wherein: generating the first set of embeddings comprises dividing the first input image into a plurality of first patch tokens and, using the first image encoder network, updating the first patch tokens by analyzing the other first patch tokens with an attention mechanism to generate the first set of embeddings comprising first updated patch tokens; and generating the second set of embeddings comprises dividing the second input image into a plurality of second patch tokens and, using the second image encoder network, updating the second patch tokens by analyzing the other second patch tokens with an attention mechanism to generate the second set of embeddings comprising second updated patch tokens.
However, Mou teaches wherein: generating the first set of embeddings comprises dividing the first input image into a plurality of first patch tokens and, using the first image encoder network, updating the first patch tokens by analyzing the other first patch tokens with an attention mechanism to generate the first set of embeddings comprising first updated patch tokens; and generating the second set of embeddings comprises dividing the second input image into a plurality of second patch tokens and, using the second image encoder network, updating the second patch tokens by analyzing the other second patch tokens with an attention mechanism to generate the second set of embeddings comprising second updated patch tokens (Mou, “An overview of our proposed model (DAGL) is shown
in Fig. 1, mainly composed of two components: feature extraction module (FEM) and graph-based feature aggregation module (GFAM) with multiple heads (M-GFAM)… The graph-based feature aggregation module is the core of our proposed DAGL, which is implemented based on graph attention networks (GAT) [38].”, pg. 4330, 1st column, 1st full paragraph, “The graph nodes in
our method are directly assigned by feature patches in
G
'
'
:
V
=
G
'
'
.
In establishing graph connections, we select a dynamic number of neighbors for each node based on the nearest principle. For this purpose, we design a dynamic KNN module to generate an adaptive threshold for each node to select neighbors whose similarities are above the threshold.”, pgs. 4330 and 4331, 2nd column, last paragraph, lines 1-2, and 1st column, lines 1-5, respectively, “Guided by the adjacency matrix
A, the feature aggregation process is a weighted sum of all connected neighbors, which is represented as: (see eq. (10)). Then we extract all feature patches from the graph and utilize the fold operation to combine this array of updated local patches into a feature map, which can be viewed as the inverse of the unfold operation.”, pg. 4331, 2nd column, 1st full paragraph, lines 1-8, see Fig. 2).
Kefato in view of Tang teaches a Siamese network including two encoder networks which process images to generate embeddings for each network to extract feature representation in order to perform comparison for model training (pg. 5, 2nd full paragraph, lines 1-5 and pg. 5, 7th full paragraph, lines 1-3, see Fig. 1 and listing 1). Kefato in view of Tang does not teach generating the embeddings by performing patch-based processing on the image or applying an attention mechanism to update embeddings of the patches. Mou teaches applying a graph attention network to perform patch-wise graph convolution, which divides input images into patches and updates those patches by aggregating information from other patches via an attention mechanism to form embeddings (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the embedding generation for each network of Kefato in view of Tang to include the patch-wise processing and attention-based updating as taught by Mou (pg. 4330, 1st column, 1st full paragraph, pgs. 4330 and 4331, 2nd column, last paragraph, lines 1-2, and 1st column, lines 1-5, respectively, and pg. 4331, 2nd column, 1st full paragraph, lines 1-8, see Fig. 2). The motivation for doing so would have been to make correlations based on feature patches, thereby increasing image understanding as opposed to pixel-wise methods (as suggested by Mou, “Correctly, we replace the graph modules in our DAGL with non-local neural networks with one head (NL) and multiple heads (MHNL). The results are presented in Table 5. One can see that our patch-wise non-local method obviously outperforms the commonly used pixel-wise non-local method [40].”, pg. 4334, 2nd column, 1st full paragraph, lines 5-10). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Kefato in view of Tang with Mou to obtain the invention as specified in claim 2.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Kefato et al ("Jointly learnable data augmentations for self-supervised gnns.", arXiv preprint arXiv:2108.10420, 2021) in view of Tang et al. (“Unifying Visual Contrastive Learning for Object Recognition from a Graph Perspective”, European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022) and further in view of Mou et al. (“Dynamic attentive graph learning for image restoration”, Proceedings of the IEEE/CVF international conference on computer vision. 2021.) and Lee et al. (“KNNLocal Attention for Image Restoration”, Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.), (hereinafter Lee).
Regarding claim 3, Kefato in view of Tang and further in view of Mou teaches the computer-implemented method as claimed in claim 2. Kefato in view of Tang and further in view of Mou does not teach wherein dividing a said input image into a plurality of said patch tokens comprises dividing the said input image into a grid of non-overlapping contiguous patch tokens.
However, Lee teaches wherein dividing a said input image into a plurality of said patch tokens comprises dividing the said input image into a grid of non-overlapping contiguous patch tokens (Lee, “The overall framework for image restoration is shown in Fig. 2. To restore a degraded image, we first conduct three convolutions to a degraded input image Id, and then pass it through three stages of the encoder network and the decoder network. Each stage is comprised of the patch partition, k NNTransformer Blocks (KTB), and an interpolation layer. The patch partition operation splits the input feature map X into non-overlapping patches with the patch size r… In the KTB, the split patches are normalized and fed to k-NN local attention (KLA) for non-local aggregation.”, pgs. 2141 and 2142, 2nd column, last paragraph, lines 1-4, and 1st column, lines 1-7, respectively, see Fig. 3).
Kefato in view of Tang and further in view of Mou teaches dividing input images into overlapping patch tokens to apply attention-based patch token updates for embedding generation (Mou, “Then we extract all feature patches from the graph and utilize the fold operation to combine this array of updated local patches into a feature map, which can be viewed as the inverse of the unfold operation. Since there exist overlaps between feature patches, we use the average operation to deal with the overlapped areas.”, pg. 4331, 2nd column, 1st full paragraph, lines 5-10). Kefato in view of Tang and further in view of Mou does not teach non-overlapping patch tokens. Lee teaches dividing input images into a grid of non-overlapping contiguous patch tokens for attention processing (see above). Before the effective filing date of the claimed invention, it would have been obvious to have modified the patch-wise processing of Kefato in view of Tang and further in view of Mou to be applied to non-overlapping patch tokens as taught by Lee (Lee, pgs. 2141 and 2142, 2nd column, last paragraph, lines 1-4, and 1st column, lines 1-7, respectively, see Fig. 3). The motivation for doing so would have been to remove the averaging operation for overlapping patches, thereby simplifying the architecture and reducing computational complexity. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Kefato in view of Tang and further in view of Mou with Lee to obtain the invention as specified in claim 3.
Claims 5 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Kefato et al ("Jointly learnable data augmentations for self-supervised gnns.", arXiv preprint arXiv:2108.10420, 2021) in view of Tang et al. (“Unifying Visual Contrastive Learning for Object Recognition from a Graph Perspective”, European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022) and further in view of Zeng et al. (“RG-GCN: A random graph based on graph convolution network for point cloud semantic segmentation”, Remote Sensing 14.16, 2022), (hereinafter Zeng).
Regarding claim 5, Kefato in view of Tang teaches the computer-implemented method as claimed in claim 4. Kefato in view of Tang does not teach wherein computing the difference between the first and second representations comprises: performing a pooling operation on the first representation to generate first pooled features and performing a pooling operation on the second representation to generate second pooled features; and computing a difference between the first and second pooled features.
However, Zeng teaches wherein computing the difference between the first and second representations comprises: performing a pooling operation on the first representation to generate first pooled features and performing a pooling operation on the second representation to generate second pooled features; and computing a difference between the first and second pooled features (Zeng, “The max pooling unit can extract the most important features in k dimensions. In the max pooling unit,
P
i
k
and
F
i
k
are aggregated into a set of local fusion information (
F
^
i
). The implementation is as follows: (see eq. (5)) Eventually, the local fusion information can be encoded by the max pooling unit using a new set of multidimensional features as
F
^
i
, which can represent the local significant features.”, pg. 7, 1st paragraph, “After the features are input to four consecutive PGCN modules, four different levels of cached local features are obtained. All cached local features are concatenated for the MLP layer, that is, (256
→
1024), to obtain features with dimensions N X 1024. Then, they are input to the max pooling layer to obtain a 1 X 1024 global descriptor.”, pg. 7, 6th paragraph, see Fig. 3, A pooling operation is performed on feature representations to aggregate local information into a fused representation.).
Kefato in view of Tang teaches adjusting weights of the network by computing a difference between two representations of the two encoder networks using a loss function, such as the mean square error between the normalized representations (Kefato, pg. 5, 2nd full paragraph). Kefato in view of Tang does not teach performing a pooling operation on the two representations before computing this difference. Zeng teaches implementing a max pooling layer to process feature representations to generates pooled features which represent local significance (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the network of Kefato in view of Tang to include a max pooling layer as taught by Zeng (Zeng, pg. 7, 1st paragraph and pg. 7, 6th paragraph, see Fig. 3). The motivation for doing so would have been to summarize the representations according to significant local features before computing the difference, thereby improving the loss calculations. The combination of Kefato in view of Tang and further in view of Zeng would include applying max pooling to the representations produced by the two networks and then computing the difference between the resulting pooled features. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Kefato in view of Tang with Zeng to obtain the invention as specified in claim 5.
Regarding claim 6, Kefato in view of Tang and further in view of Zeng teaches the computer-implemented method as claimed in claim 5, wherein computing the difference between the first and second pooled features comprises computing a cosine difference or a Euclidean difference between the first and second pooled features (Zeng, “To this end, we closely follow Laplacian Eigenmaps Belkin and Niyogi (2003) and minimize the mean squared error between the normalized representations (unit vectors) of two data points.”, pg. 5, 2nd full paragraph, lines 1-3, The mean squared error is mathematically equivalent to the average of the squared Euclidean distance between the two representation vectors1. Therefore, the computed difference is based on the Euclidean distance between the representations, which satisfies computing a Euclidean difference.).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Kefato et al ("Jointly learnable data augmentations for self-supervised gnns.", arXiv preprint arXiv:2108.10420, 2021) in view of Tang et al. (“Unifying Visual Contrastive Learning for Object Recognition from a Graph Perspective”, European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022) and further in view of He et al. ("Momentum contrast for unsupervised visual representation learning" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.), (hereinafter He).
Regarding claim 10, Kefato in view of Tang teaches the computer-implemented method as claimed in claim 9. Kefato in view of Tang does not teach distilling the updates to the first image encoder network after a number of iterations of the training process to the second image encoder network using a momentum encoder or using an exponential moving average-based update process.
However, He teaches distilling the updates to the first image encoder network after a number of iterations of the training process to the second image encoder network using a momentum encoder or using an exponential moving average-based update process (He, “We present Momentum Contrast (MoCo) as a way of building large and consistent dictionaries for unsupervised learning with a contrastive loss (Figure 1).”, pg. 9729, 2nd column, 2nd full paragraph, lines 1-3, “From the above perspective, contrastive learning is a way of building a discrete dictionary on high-dimensional continuous inputs such as images. The dictionary is dynamic in the sense that the keys are randomly sampled, and that the key encoder evolves during training. Our hypothesis is that good features can be learned by a large dictionary that covers a rich set of negative samples, while the encoder for the dictionary keys is kept as consistent as possible despite its evolution… At the core of our approach is maintaining the dictionary as a queue of data samples. This allows us to reuse the encoded keys from the immediate preceding mini-batches.”, pg. 9731, 1st column, section 3.2 Momentum Contrast, lines 1-14).
Kefato in view of Tang teaches a training process for a Siamese network, including updating parameters of the two encoder networks based on a difference between the two networks representations (Kefato, pg. 5, 2nd full paragraph). Kefato in view of Tang does not teach distilling updates of the network using a momentum encoder or an exponential moving average-based updated process. He teaches implementing a momentum encoder to distill updates from a query encoder to a key encoder through an iterative training process (see above). Before ethe effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the network of Kefato in view of Tang to include a momentum encoder training process as taught by He (He, pg. 9731, 1st column, section 3.2 Momentum Contrast, lines 1-14). The motivation for doing so would have been enhance the consistency of the contrastive learning process, particularly for large dictionaries of negative samples, thereby improving the stability and performance of the model (as suggested by He, “Our hypothesis is that Good features can be learned by a large dictionary that covers a rich set of negative samples, while the encoder for the dictionary keys is kept as consistent as possible despite its evolution. Based on this motivation, we present Momentum Contrast as described next.”, pg. 9731, 1st column, 1st paragraph, lines 5-10). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Kefato in view of Tang with He to obtain the invention as specified in claim 10.
Allowable Subject Matter
Claims 15 and 16 would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims and by overcome the 35 U.S.C. 112(b) rejection outlined above.
Note the interpretation of claim 15 under 35 U.S.C. 112(b), requires the claim to be interpreted as if being dependent from claim 2. Accordingly, any allowable subject matter for claims 15 and 16 would necessarily incorporate the limitations of claim 2 as interpreted.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CONNOR LEVI HANSEN whose telephone number is (703)756-5533. The examiner can normally be reached Monday-Friday 9:00-5:00 (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CONNOR L HANSEN/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
1 basilisk et al. “What Is the Difference between Euclidean Distance and RMSE?” Data Science Stack Exchange, https://stackoverflow.com/#organization, 1 July 1963, datascience.stackexchange.com/questions/63186/what-is-the-difference-between-euclidean-distance-and-rmse.
Prosecution Insights