DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This communication is in response to the Application Filed on 02/21/2024.
Claims 1-20 are pending in this application.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 06/06/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Specification
The disclosure is objected to because of the following informalities:
At paragraphs [0053] and [0056] it states, “(-1,-1) (labeled with “C”)”. Perhaps the proper coordinate value should be (-1,1) as seen in Fig. 2A, label “C” in constellation set 120A and 120B.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claims 1-20 are rejected under 35 U.S.C. 103(a) as being unpatentable over Seo et al. (“Attentive Semantic Alignment with Offset-Aware Correlation Kernels”, 2018, hereinafter, “Seo”) in view of Garegrat et al. (US 2022/0222318 A1, hereinafter, “Garegrat”).
Regarding claim 1, Seo discloses [a processing system comprising: one or more memories comprising processor-executable instructions; and one or more processors configured to execute the processor-executable instructions and cause the processing system to:]
access a reference tensor corresponding to a reference image (See Seo, Pg. 370, [3.1 Feature Extractor], lines 1-2, Given source and target images, we first extract their image feature maps fsrc, ftrg. Examiner considers fsrc as the reference tensor as shown below in Fig. 2 for clarity);
access a target tensor corresponding to a target image (See Seo, Pg. 370, [3.1 Feature Extractor], lines 1-2, Given source and target images, we first extract their image feature maps fsrc, ftrg. Examiner considers ftrg as the target tensor as shown below in Fig. 2 for clarity); and
PNG
media_image1.png
470
905
media_image1.png
Greyscale
use a first constellation set (See Seo, Pg. 372, caption of [Fig. 3], Offset-aware correlation kernel at different source locations: (a) at (0,0) and (b) at (0,1). Examiner considers the different source locations as shown in Fig. 3 to be the first constellation set. The offset-aware correlation (OAC) show in Fig. 3 is also seen in Fig. 2 (see figure below) as a part of the overall architecture, following the extraction of the two tensors as seen below for clarity of OAC’s location) to perform feature matching for the reference and target tensors while processing data using an artificial neural network (See Seo, Pg. 370, [3 Deep Attentive Semantic Alignment Network], lines 1-2, a deep neural network architecture for semantic alignment incorporating an attention process with a novel offset-aware correlation kernel),
PNG
media_image2.png
470
905
media_image2.png
Greyscale
wherein the first constellation set comprises a set of offsets (See Seo, Pg. 372, caption of [Fig. 3], Offset-aware correlation kernel at different source locations: (a) at (0,0) and (b) at (0,1). Examiner considers the off-set aware correlation as constellation/pixel mapping. See Fig. 3 below for clarity),
PNG
media_image3.png
563
929
media_image3.png
Greyscale
and wherein, to perform the feature matching (See Seo, Pg. 371, [Correlation Layer], lines 1-2, The correlation layer computes correlations of all pairs of features from the source and the target images), [the one or more processors are configured to execute the processor-executable instructions and cause the processing system to:]
for each respective offset of the first constellation set (See Seo, Pg. 372, caption of [Fig. 3], Offset-aware correlation kernel at different source locations: (a) at (0,0) and (b) at (0,1)):
shift the target tensor based on the respective offset (See Seo, Pg. 371, [Offset-Aware Correlation Kernels], lines 3-5, kernels naturally capture the displacement of a source feature in the target feature map by aligning kernel weights based on the offset between the source and target features. Examiner considers the displacement as “shift”));
generate a respective intermediate tensor (See Seo, Pg. 372, indented par., lines 3-5, We use 128 OAC kernels resulting in a feature displacement map h ∈ R,128×15×15 encoding the displacement of each source feature. Examiner considers the feature displacement map as the intermediate tensor) [based on elementwise multiplying the reference tensor and the shifted target tensor;] and
generate a respective flattened tensor based on flattening the respective intermediate tensor (See Seo, Pg. 373, par. 1, lines 8-10, Then, we use a 1 × 1 convolutional layer to compute the dense feature representation from the raw displacement information captured in the reordered correlation map. Examiner considers the use of a 1x1 convolutional layer as flattening to get the dense feature representation, which is considered to be a flattened tensor); and
aggregate the respective flattened tensors (See Seo, Pg. 373, par. 1, lines 10-12, Note that this process significantly reduces the number of channels by compactly encoding various aspects of the local displacements into dense representations. Examiner considers compactly encoding as aggregating) to generate a correlation tensor for the reference and target tensors (See Seo, Pg. 373, par. 1, lines 2-3, We first compute the normalized correlation map reordered based on the offsets between the locations of the source and target features. Examiner considers the normalized correlation map as the correlation tensor).
However, Seo does not explicitly disclose a processing system comprising: one or more memories comprising processor-executable instructions; and one or more processors configured to execute the processor-executable instructions and cause the processing system to;
[and wherein, to perform the feature matching] the one or more processors are configured to execute the processor-executable instructions and cause the processing system to; and
[generate a respective intermediate] based on elementwise multiplying the reference tensor and the shifted target tensor.
Garegrat teaches a processing system (See Garegrat, [0112] FIG. 16 illustrates a neural network processing system) comprising: one or more memories comprising processor-executable instructions; and one or more processors configured to execute the processor-executable instructions and cause the processing system to (See Garegrat, [0112] NN processors 1611(1)-1611(N) and 1612(1)-1612(N) may include a variety of configurations of functional processing blocks and memory optimized for neural network processing);
[and wherein, to perform the feature matching] the one or more processors are configured to execute the processor-executable instructions and cause the processing system (See Garegrat, [0112] NN processors 1611(1)-1611(N) and 1612(1)-1612(N) may include a variety of configurations of functional processing blocks and memory optimized for neural network processing) to and
[generate a respective intermediate] based on elementwise multiplying the reference tensor and the shifted target tensor (See Garegrat, Fig. 4; [0049] As shown, matrix 405 is shifted to the right by one element. Thus, element F0 in matrix 405 is multiplied with element E4 in matrix 400).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Seo’s reference to wherein a processing system comprising: one or more memories comprising processor-executable instructions; and one or more processors configured to execute the processor-executable instructions and cause the processing system to; and elementwise multiplying the reference tensor and the shifted target tensor based on the method of Garegrat’s reference. The suggestion/motivation would have been for hardware architectures optimized for neural network computations as suggested by Garegrat at [0112] and to improve the efficiency of performing those tensor operations by reducing the number of reads necessary to retrieve elements of the 3D tensors and/or reducing reading the same data from memory multiple times as suggested by Garegrat at [0022].
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Garegrat with Seo to obtain the invention as specified in claim 1.
Regarding claim 2, in which claim 1 in incorporated, Seo discloses [wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system] to generate a feature matching output for the target image and the reference image based on processing the correlation tensor using an artificial neural network filter of the artificial neural network (See Seo, Pg. 370, caption of [Fig. 2], Overall architecture of the proposed network. It consists of three main components: feature extractor, local transformation encoder, and attentive global transformation estimator. Examiner considers the correlation map to be in the ‘local transformation encoder’ area which is used for subsequent processing to yield an output of an attended feature map with noise features filtered in which higher probabilities of matching are brighter. This process is part of the overall architecture as shown in Fig. 2 below for clarity).
PNG
media_image4.png
470
905
media_image4.png
Greyscale
However, Seo does not explicitly disclose wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system to.
Garegrat teaches wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system (See Garegrat, [0112] NN processors 1611(1)-1611(N) and 1612(1)-1612(N) may include a variety of configurations of functional processing blocks and memory optimized for neural network processing).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Seo’s reference to wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system [to generate] based on the method of Garegrat’s reference. The suggestion/motivation would have been for hardware architectures optimized for neural network computations as suggested by Garegrat at [0112].
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Garegrat with Seo to obtain the invention as specified in claim 2.
Regarding claim 3, in which claim 2 is incorporated, Seo discloses [wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system] to select the artificial neural network filter from a plurality of artificial neural network filters (See Seo, Pg. 371, [Correlation Layer], lines 1-2, The correlation layer computes correlations of all pairs of features from the source and the target images [13]. Specifically, the correlation layer takes two feature maps as its inputs and constructs a correlation map. Examiner considers the correlation layer, a type of neural network filter among many other layers in the neural network, to have been chosen based on the correlation kernel and its purpose for matching) based on the first constellation set (See Seo, Pg. 372, caption of [Fig. 3], Offset-aware correlation kernel at different source locations: (a) at (0,0) and (b) at (0,1). Examiner considers the different source locations as shown in Fig. 3 to be the first constellation set).
However, Seo does not explicitly disclose wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system.
Garegrat teaches wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system (See Garegrat, [0112] NN processors 1611(1)-1611(N) and 1612(1)-1612(N) may include a variety of configurations of functional processing blocks and memory optimized for neural network processing).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Seo’s reference to wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system [to select] based on the method of Garegrat’s reference. The suggestion/motivation would have been for hardware architectures optimized for neural network computations as suggested by Garegrat at [0112].
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Garegrat with Seo to obtain the invention as specified in claim 3.
Regarding claim 4, in which claim 1 is incorporated, Seo discloses [wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system] to select the first constellation set from a plurality of constellation sets based on at least one of the reference tensor or the target tensor (See Seo, Pg. 372, caption of [Fig. 3], Offset-aware correlation kernel at different source locations: (a) at (0,0) and (b) at (0,1)… Note that kernel weights are associated with different correlation pairs when source locations vary. Examiner considers the constellation set, or the pairs to vary depending on the source location. This can mean there are multiple locations or pixels to map to).
However, Seo does not explicitly disclose wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system.
Garegrat teaches wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system (See Garegrat, [0112] NN processors 1611(1)-1611(N) and 1612(1)-1612(N) may include a variety of configurations of functional processing blocks and memory optimized for neural network processing).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Seo’s reference to wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system [to select] based on the method of Garegrat’s reference. The suggestion/motivation would have been for hardware architectures optimized for neural network computations as suggested by Garegrat at [0112].
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Garegrat with Seo to obtain the invention as specified in claim 4.
Regarding claim 5, in which claim 4 is incorporated, Seo discloses wherein, to select the first constellation set (See Seo, Pg. 372, caption of [Fig. 3], Offset-aware correlation kernel at different source locations: (a) at (0,0) and (b) at (0,1)… Note that kernel weights are associated with different correlation pairs when source locations vary. Examiner considers the constellation set, or the pairs to vary depending on the source location. This can mean there are multiple locations or pixels to map to), [the one or more processors are configured to execute the processor-executable instructions and cause the processing system to:]
determine a coherency based at least in part on the reference tensor and the target tensor (See Seo, Pg. 379, caption of [Fig. 5], Qualitative results of the attentive semantic alignment. Examiner considers the points, as seen in the last column of images (Transformed (affine+TPS)) shown in Fig. 5 below for clarity, to be coherent based on points from the source and target image, which results in the source and target tensors used); and
PNG
media_image5.png
726
928
media_image5.png
Greyscale
select the first constellation set based on the coherency (See Seo, Pg. 379, caption of [Fig. 5], The model attends to the objects to match. Examiner considers for the matching process to occur, the different source locations or points from both images must be coherent to be selected and used for matching).
However, Seo does not explicitly disclose the one or more processors are configured to execute the processor-executable instructions and cause the processing system to.
Garegrat teaches the one or more processors are configured to execute the processor-executable instructions and cause the processing system to (See Garegrat, [0112] NN processors 1611(1)-1611(N) and 1612(1)-1612(N) may include a variety of configurations of functional processing blocks and memory optimized for neural network processing).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Seo’s reference to wherein the one or more processors are configured to execute the processor-executable instructions and cause the processing system based on the method of Garegrat’s reference. The suggestion/motivation would have been for hardware architectures optimized for neural network computations as suggested by Garegrat at [0112].
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Garegrat with Seo to obtain the invention as specified in claim 5.
Regarding claim 6, in which claim 5 is incorporated, Seo discloses the first constellation set of the plurality of constellation sets corresponds to a first receptive field (See Seo, Pg. 379, caption of [Fig. 5], Qualitative results of the attentive semantic alignment. Examiner considers the first receptive field to be the area encompassing the object of interest mapped in the source image),
a second constellation set of the plurality of constellation sets corresponds to a second receptive field larger than the first receptive field (See Seo, Pg. 379, caption of [Fig. 5], Qualitative results of the attentive semantic alignment. Examiner considers the second receptive field to be points the area encompassing the object of interest mapped in the target image. The second receptive field can be larger than the first receptive field as seen in the second row in the first column (source image) and the third column (target image) of Fig. 5 as shown below for clarity), and
PNG
media_image6.png
726
928
media_image6.png
Greyscale
[the one or more processors are configured to execute the processor-executable instructions and cause the processing system to] select the first constellation set to perform the feature matching, over the second constellation set, in response to determining that the coherency (See Seo, Pg. 379, caption of [Fig. 5], The model attends to the objects to match. Examiner considers for the matching process to occur, the sets must be as close to matching, or coherent, as possible) satisfies one or more coherency criteria (See Seo, Pg. 371, [Correlation Layer], lines 9-11, each element of a correlation vector maintains the correspondence likelihood of a source feature onto a certain location in the target feature map. Examiner considers the likelihood to be the coherency criteria).
However, Seo does not explicitly disclose the one or more processors are configured to execute the processor-executable instructions and cause the processing system to.
Garegrat teaches the one or more processors are configured to execute the processor-executable instructions and cause the processing system to (See Garegrat, [0112] NN processors 1611(1)-1611(N) and 1612(1)-1612(N) may include a variety of configurations of functional processing blocks and memory optimized for neural network processing).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Seo’s reference to wherein the one or more processors are configured to execute the processor-executable instructions and cause the processing system based on the method of Garegrat’s reference. The suggestion/motivation would have been for hardware architectures optimized for neural network computations as suggested by Garegrat at [0112].
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Garegrat with Seo to obtain the invention as specified in claim 6.
Regarding claim 7, in which claim 6 is incorporated, Seo discloses wherein the coherency criteria comprise one or more threshold values (See Seo, Pg. 376, [Evaluation], lines 12-15, A transformed source key point is considered correct if its distance to its corresponding target key point is less than α · max(h,w), where α =0.1, and h and w are height and width of the object bounding box. Examiner considers the distance to its corresponding target key as a threshold value), and wherein, to determine that the coherency satisfies the one or more coherency criteria (See Seo, Pg. 371, [Correlation Layer], lines 9-11, each element of a correlation vector maintains the correspondence likelihood of a source feature onto a certain location in the target feature map. Examiner considers the likelihood to be the coherency criteria)., [the one or more processors are configured to execute the processor-executable instructions and cause the processing system to] determine that the coherency meets or exceeds the one or more threshold values (See Seo, Pg. 376, [Evaluation], lines 17-19, returns 1 if the expression inside brackets is true and 0 otherwise. Examiner considers the threshold to be met if 1 is returned)
However, Seo does not explicitly disclose the one or more processors are configured to execute the processor-executable instructions and cause the processing system to.
Garegrat teaches the one or more processors are configured to execute the processor-executable instructions and cause the processing system to (See Garegrat, [0112] NN processors 1611(1)-1611(N) and 1612(1)-1612(N) may include a variety of configurations of functional processing blocks and memory optimized for neural network processing).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Seo’s reference to wherein the one or more processors are configured to execute the processor-executable instructions and cause the processing system based on the method of Garegrat’s reference. The suggestion/motivation would have been for hardware architectures optimized for neural network computations as suggested by Garegrat at [0112].
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Garegrat with Seo to obtain the invention as specified in claim 7.
Regarding claim 8, in which claim 4 is incorporated, Seo discloses wherein, to select the first constellation set (See Seo, Pg. 372, caption of [Fig. 3], Offset-aware correlation kernel at different source locations: (a) at (0,0) and (b) at (0,1)… Note that kernel weights are associated with different correlation pairs when source locations vary. Examiner considers the constellation set, or the pairs to vary depending on the source location. This can mean there are multiple locations or pixels to map to), [the one or more processors are configured to execute the processor-executable instructions and cause the processing system to] process the at least one of the reference tensor or the target tensor using a preliminary portion of the artificial neural network (See Seo, Pg. 370, caption of [Fig. 2], Overall architecture of the proposed network. It consists of three main components: feature extractor, local transformation encoder, and attentive global transformation estimator Examiner considers fsrc, source tensor, and ftrg, target tensor, to be processed during feature extraction, before the encoding process, or preliminary portion, of the neural network).
However, Seo does not explicitly disclose the one or more processors are configured to execute the processor-executable instructions and cause the processing system to.
Garegrat teaches the one or more processors are configured to execute the processor-executable instructions and cause the processing system to (See Garegrat, [0112] NN processors 1611(1)-1611(N) and 1612(1)-1612(N) may include a variety of configurations of functional processing blocks and memory optimized for neural network processing).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Seo’s reference to wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system based on the method of Garegrat’s reference. The suggestion/motivation would have been for hardware architectures optimized for neural network computations as suggested by Garegrat at [0112].
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Garegrat with Seo to obtain the invention as specified in claim 8.
Regarding claim 9, in which claim 1 is incorporated, Seo discloses wherein the set of offsets of the first constellation set was learned during training of the artificial neural network (See Seo, Pg. 375, [Network Training], lines 12-13, During training, the pretrained feature extractor is fixed and only the other parts of the network are finetuned. Examiner considers all other parts of the overall architecture, shown in Fig. 2 below with an arrow pointed to the enclosed area, to be finetuned or learned during a training process. The offset-aware correlation kernels, which include a set of offsets, is a part of the structure that is learned during training; thus, the offsets are learned during training).
PNG
media_image7.png
470
905
media_image7.png
Greyscale
Regarding claim 10, in which claim 1 is incorporated, Seo discloses, wherein the set of offsets of the first constellation set is specified as a hyperparameter of the artificial neural network (See Seo, Pg. 371, [3.2 Local Transformation Encoder], lines 3-4, The encoding is given by introducing a novel offset-aware correlation (OAC) kernel. Examiner considers the parameter of offsetting to be input into the encoder for processing of the neural network).
Regarding claim 11, in which claim 1 is incorporated, Seo discloses wherein each respective flattened tensor (See Seo, Pg. 373, par. 1, lines 8-10, Then, we use a 1 × 1 convolutional layer to compute the dense feature representation from the raw displacement information captured in the reordered correlation map. Examiner considers the use of a 1x1 convolutional layer as flattening to get the dense feature representation, which is considered to be a flattened tensor) is configured to be respectively weighted prior to the aggregation of the respective flattened tensors (See Seo, Pg. 372, caption of [Fig. 3], w,i,j represents a kernel weight for the dotted line; Pg. 372, last par., lines 2-3, a set of the proposed kernels produce a dense feature representation of feature displacement for each source feature. Examiner considers the flattened tensors to be weighted if the target tensor was shifted with weighted kernels, which is then used to ultimately get the flattened tensor, which is considered to be the dense feature representation).
Regarding claim 12, in which claim 1 is incorporated, Seo discloses wherein, to flatten the respective intermediate tensor (See Seo, Pg. 373, par. 1, lines 8-10 Then, we use a 1 × 1 convolutional layer to compute the dense feature representation from the raw displacement information captured in the reordered correlation map. Examiner considers the use of a 1x1 convolutional layer as flattening to get the dense feature representation, which is considered to be a flattened tensor), [the one or more processors are configured to execute the processor-executable instructions and cause the processing system to average values of the respective intermediate tensor across a depth dimension.]
However, Seo does not explicitly disclose the one or more processors are configured to execute the processor-executable instructions and cause the processing system to average values of the respective intermediate tensor across a depth dimension.
Garegrat teaches the one or more processors are configured to execute the processor-executable instructions and cause the processing system (See Garegrat, [0112] NN processors 1611(1)-1611(N) and 1612(1)-1612(N) may include a variety of configurations of functional processing blocks and memory optimized for neural network processing) to average values of the respective intermediate tensor across a depth dimension (See Garegrat, [0046] FIGS. 4A-41 illustrate example matrix operations performed on matrices stored in memory based on depth; [0048] the first set of operations can be represented as the sum of the products of corresponding elements in matrices 400 and 405. Examiner considers the sum of elements in the matrix to then be combined, or averaged, for a physical representation as seen in 400 based on depth, shown in Fig. 4A below for clarity).
PNG
media_image8.png
539
791
media_image8.png
Greyscale
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Seo’s reference wherein the one or more processors are configured to execute the processor-executable instructions and cause the processing system to average values of the respective intermediate tensor across a depth dimension based on the method of Garegrat’s reference. The suggestion/motivation would have been for hardware architectures optimized for neural network computations as suggested by Garegrat at [0112] and to improve the efficiency of performing those tensor operations by reducing the number of reads necessary to retrieve elements of the 3D tensors and/or reducing reading the same data from memory multiple times as suggested by Garegrat at [0022].
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Garegrat with Seo to obtain the invention as specified in claim 12.
Regarding claim 13, in which claim 1 is incorporated, Seo discloses [wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system] to take one or more actions based on the correlation tensor (See Seo, Pg. 368, first indented par., lines 9-11, correlation map between all feature pairs across images is used to predict such a global transformation. Examiner considers the correlation map used to predict as an action taken).
However, Seo does not explicitly disclose wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system.
Garegrat teaches wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system (See Garegrat, [0112] NN processors 1611(1)-1611(N) and 1612(1)-1612(N) may include a variety of configurations of functional processing blocks and memory optimized for neural network processing).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Seo’s reference to wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system based on the method of Garegrat’s reference. The suggestion/motivation would have been for hardware architectures optimized for neural network computations as suggested by Garegrat at [0112].
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Garegrat with Seo to obtain the invention as specified in claim 13.
Regarding claim 14, in which claim 13 is incorporated, Seo discloses wherein, to take the one or more actions (See Seo, Pg. 368, first indented par., lines 9-11, correlation map between all feature pairs across images is used to predict such a global transformation. Examiner considers the correlation map used to predict as an action taken), [the one or more processors are configured to execute the processor-executable instructions and cause the processing system to] perform, based on the correlation tensor, at least one of: (i) tracking of a visual object in a video, (ii) motion estimation, or (iii) depth estimation (See Seo, Pg. 375, [Network Training], lines 4-6, measures the distance from the predicted transformation parameters θ to the ground-truth transformation parameters θGT; Pg. 368, first indented par., lines 4-5, estimate parameters of a global transformation model that densely aligns one image to the other. Examiner considers the measure of a distance from points to be motion estimation. The prediction of global transformations using a correlation map is used for this estimation).
However, Seo does not explicitly teach the one or more processors are configured to execute the processor-executable instructions and cause the processing system to.
Garegrat teaches the one or more processors are configured to execute the processor-executable instructions and cause the processing system to (See Garegrat, [0112] NN processors 1611(1)-1611(N) and 1612(1)-1612(N) may include a variety of configurations of functional processing blocks and memory optimized for neural network processing).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Seo’s reference to wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system based on the method of Garegrat’s reference. The suggestion/motivation would have been for hardware architectures optimized for neural network computations as suggested by Garegrat at [0112].
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Garegrat with Seo to obtain the invention as specified in claim 14.
Regarding claim 15, Seo discloses [a processor-implemented method for an artificial neural network, comprising:]
accessing a reference tensor corresponding to a reference image (See Seo, Pg. 370, [3.1 Feature Extractor], lines 1-2, Given source and target images, we first extract their image feature maps fsrc, ftrg. Examiner considers fsrc as the reference tensor as shown below in Fig. 2 for clarity);
accessing a target tensor corresponding to a target image (See Seo, Pg. 370, [3.1 Feature Extractor], lines 1-2, Given source and target images, we first extract their image feature maps fsrc, ftrg. Examiner considers ftrg as the target tensor as shown below in Fig. 2 for clarity); and
PNG
media_image1.png
470
905
media_image1.png
Greyscale
using a first constellation set (See Seo, Pg. 372, caption of [Fig. 3], Offset-aware correlation kernel at different source locations: (a) at (0,0) and (b) at (0,1). Examiner considers the different source locations as shown in Fig. 3 to be the first constellation set. The offset-aware correlation (OAC) show in Fig. 3 is also seen in Fig. 2 (see figure below) as a part of the overall architecture, following the extraction of the two tensors as seen below for clarity of OAC’s location) to perform feature matching for the reference and target tensors while processing data using an artificial neural network (See Seo, Pg. 370, [3 Deep Attentive Semantic Alignment Network], lines 1-2, a deep neural network architecture for semantic alignment incorporating an attention process with a novel offset-aware correlation kernel),
PNG
media_image2.png
470
905
media_image2.png
Greyscale
wherein the first constellation set comprises a set of offsets (See Seo, Pg. 372, caption of [Fig. 3], Offset-aware correlation kernel at different source locations: (a) at (0,0) and (b) at (0,1). See Fig. 3 below for clarity),
PNG
media_image3.png
563
929
media_image3.png
Greyscale
the feature matching (See Seo, Pg. 371, [Correlation Layer], lines 1-2, The correlation layer computes correlations of all pairs of features from the source and the target images) comprising:
for each respective offset of the first constellation set (See Seo, Pg. 372, caption of [Fig. 3], Offset-aware correlation kernel at different source locations: (a) at (0,0) and (b) at (0,1)):
shift the target tensor based on the respective offset (See Seo, Pg. 371, [Offset-Aware Correlation Kernels], lines 3-5, kernels naturally capture the displacement of a source feature in the target feature map by aligning kernel weights based on the offset between the source and target features. Examiner considers the displacement as “shift”));
generate a respective intermediate tensor (See Seo, Pg. 372, indented par., lines 3-5, We use 128 OAC kernels resulting in a feature displacement map h ∈ R,128×15×15 encoding the displacement of each source feature. Examiner considers the feature displacement map as the intermediate tensor) [based on elementwise multiplying the reference tensor and the shifted target tensor;] and
generate a respective flattened tensor based on flattening the respective intermediate tensor (See Seo, Pg. 373, par. 1, lines 8-10, Then, we use a 1 × 1 convolutional layer to compute the dense feature representation from the raw displacement information captured in the reordered correlation map. Examiner considers the use of a 1x1 convolutional layer as flattening to get the dense feature representation, which is considered to be a flattened tensor); and
aggregate the respective flattened tensors (See Seo, Pg. 373, par. 1, lines 10-12, Note that this process significantly reduces the number of channels by compactly encoding various aspects of the local displacements into dense representations. Examiner considers compactly encoding as aggregating) to generate a correlation tensor for the reference and target tensors (See Seo, Pg. 373, par. 1, lines 2-3, We first compute the normalized correlation map reordered based on the offsets between the locations of the source and target features. Examiner considers the normalized correlation map as the correlation tensor).
However, Seo does not explicitly disclose a processor-implemented method for an artificial neural network, and [generate a respective intermediate tensor] based on elementwise multiplying the reference tensor and the shifted target tensor.
Garegrat teaches a processor-implemented method for an artificial neural network (See Garegrat, [0112] FIG. 16 illustrates a neural network processing system; [0112] NN processors 1611(1)-1611(N) and 1612(1)-1612(N) may include a variety of configurations of functional processing blocks and memory optimized for neural network processing), and
[generate a respective intermediate] based on elementwise multiplying the reference tensor and the shifted target tensor (See Garegrat, Fig. 4; [0049] As shown, matrix 405 is shifted to the right by one element. Thus, element F0 in matrix 405 is multiplied with element E4 in matrix 400).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Seo’s reference for a processor-implemented method for an artificial neural network and elementwise multiplying the reference tensor and the shifted target tensor based on the method of Garegrat’s reference. The suggestion/motivation would have been for hardware architectures optimized for neural network computations as suggested by Garegrat at [0112] and to improve the efficiency of performing those tensor operations by reducing the number of reads necessary to retrieve elements of the 3D tensors and/or reducing reading the same data from memory multiple times as suggested by Garegrat at [0022].
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Garegrat with Seo to obtain the invention as specified in claim 15.
Regarding claim 16, in which claim 15 is incorporated, Seo discloses generating a feature matching output for the target image and the reference image based on processing the correlation tensor using an artificial neural network filter of the artificial neural network (See Seo, Pg. 370, caption of [Fig. 2], Overall architecture of the proposed network. It consists of three main components: feature extractor, local transformation encoder, and attentive global transformation estimator. Examiner considers the correlation map to be in the ‘local transformation encoder’ area which is used for subsequent processing to yield an output of an attended feature map with noise features filtered in which higher probabilities of matching are brighter. This process is part of the overall architecture as shown below in Fig. 2 for clarity).
PNG
media_image4.png
470
905
media_image4.png
Greyscale
Regarding claim 17, in which claim 16 is incorporated, Seo discloses selecting the artificial neural network filter from a plurality of artificial neural network filters (See Seo, Pg. 371, [Correlation Layer], lines 1-2, The correlation layer computes correlations of all pairs of features from the source and the target images [13]. Specifically, the correlation layer takes two feature maps as its inputs and constructs a correlation map. Examiner considers the correlation layer, a type of neural network filter among many other layers in the neural network, to have been chosen based on the correlation kernel and its purpose for matching) based on the first constellation set (See Seo, Pg. 372, caption of [Fig. 3], Offset-aware correlation kernel at different source locations: (a) at (0,0) and (b) at (0,1). Examiner considers the different source locations as shown in Fig. 3 to be the first constellation set).
Regarding claim 18, in which claim 15 is incorporated, Seo discloses selecting the first constellation set from a plurality of constellation sets based on at least one of the reference tensor or the target tensor (See Seo, Pg. 372, caption of [Fig. 3], Offset-aware correlation kernel at different source locations: (a) at (0,0) and (b) at (0,1)… Note that kernel weights are associated with different correlation pairs when source locations vary. Examiner considers the constellation set, or the pairs to vary depending on the source location. This can mean there are multiple locations or pixels to map to).
Regarding claim 19, in which claim 18 is incorporated, Seo discloses wherein selecting the first constellation set (See Seo, Pg. 372, caption of [Fig. 3], Offset-aware correlation kernel at different source locations: (a) at (0,0) and (b) at (0,1)… Note that kernel weights are associated with different correlation pairs when source locations vary. Examiner considers the constellation set, or the pairs to vary depending on the source location. This can mean there are multiple locations or pixels to map to), comprises:
determine a coherency based at least in part on the reference tensor and the target tensor (See Seo, Pg. 379, caption of [Fig. 5], Qualitative results of the attentive semantic alignment. Examiner considers the points, as seen in the last column of images (Transformed (affine+TPS)) shown below in Fig. 5 for clarity, to be coherent based on points from the source and target image, which results in the source and target tensors used); and
PNG
media_image5.png
726
928
media_image5.png
Greyscale
selecting the first constellation set based on the coherency (See Seo, Pg. 379, caption of [Fig. 5], The model attends to the objects to match. Examiner considers for the matching process to occur, the different source locations or points from both images must be coherent to be selected and used for matching).
Regarding claim 20, in which claim 15 is incorporated, Seo discloses performing, based on the correlation tensor, at least one of: (i) tracking of a visual object in a video, (ii) motion estimation, or (iii) depth estimation (See Seo, Pg. 375, [Network Training], lines 9-11, measures the distance from the predicted transformation parameters θ to the ground-truth transformation parameters θGT; Pg. 368, first indented par., lines 4-5, estimate parameters of a global transformation model that densely aligns one image to the other. Examiner considers the measure of a distance from points to be motion estimation. The prediction of global transformations using a correlation map is used for this estimation).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Trzcinski et al. (“SConE: Siamese Constellation Embedding Descriptor for Image Matching”, 2018) discloses a Siamese neural network architecture that learns feature embedding of constellation points by maximizing the distance between pairs that do not match from the matched image patches. The architecture also minimizes the distance for correct matches. The proposed method improves the precision of the matching step by comparing more representations of key point constellations.
Schmid et al. (“Building Descriptors From Local Feature Neighborhoods for Applications in Semi-Dense 3D Reconstruction”, 2018) discloses a method of using other extracted features withing the local constellations present to help define a unique descriptor used for matching. Using location, magnitude, and orientation of features can help build a set of oriented feature constellation descriptors. Higher number of matches are possible while lowering the time of processing.
Burl et al. (“Autonomous Driving with Deep Learning: A Survey of State-of-Art Technologies”, 2020) discloses a survey of autonomous driving technologies with deep learning methods. Some major fields of self-driving systems are mapping, localization, predicting, and perception. One deep learning method for autonomous driving transforms points into canonical coordinates to learn spatial features, which are combined with global semantic features of each point, for improved feature extraction.
Lin et al. (US 20220391702 A1) discloses a method of expanding kernels and obtaining tensors for a convolution operation. The method aims to improve efficiency and compatibility with a wider range of kernel sizes. Tensor and kernel movements are considered by a defined offset value that is set by the overall system.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jasmin Marcelino Hernandez whose telephone number is (571) 270-0211. The examiner can normally be reached 7am-3pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASMIN MARCELINO HERNAND/Examiner, Art Unit 2676
/SHEFALI D GORADIA/Primary Patent Examiner, Art Unit 2676
