DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on September 11, 2023, is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Specification
The disclosure is objected to because of the following informalities:
[0006]: "a the plurality" should read "the plurality"; "wherein the each" should read "wherein each"; "involve and generating" should read "involve generating"; "a principal hypervectors" should read "principal hypervectors"; "each principal hypervectors" should read "each principal hypervector"
[0007]: "vector is based" should read "vector based"; "one or the following" should read "one of the following"; "the approach factorize" should read "the approach to factorize"; "when a predetermined number of iterations" should read "when a predetermined number of iterations is reached"
[0009]: "processors. to perform" should read "processors to perform"
[0021]: "may be sparse hypervector" should read "may be a sparse hypervector"
[0024]: "may enable to represent" should read "may enable representing"
[0025]: "may enable to perform" should read "may enable performing"
[0038]: "may thus preserved" should read "may thus be preserved"
[0042]: "scarification" should read "sparsification"
[0046]: "of the of the" should read "of the"
[0049]: "that's is" should read "that is"; "all finite length block" should read "all finite length blocks"
[0051]: "respectively memories" should read "respective memories"
[0057]: “may enable to obtain” should read “may enable obtaining”
[0062]: "may enable to identify" should read "may enable identifying"
[0065]: "brute force approach." should read "brute force approach may be applicable."
[0076]: "chosen fort the weighted bundling" should read "chosen for the weighted bundling"
Appropriate correction is required.
Claim Objections
Claims 1-20 are objected to because of the following informalities:
Claim 1: "a the plurality of concepts" should read "the plurality of concepts"; the space before “:” should be omitted; “unbinding, by the processor” should read “unbinding, by a processor”; "the each of the plurality of unbound hypervectors" should read "each of the plurality of unbound hypervectors"; "a principal hypervectors" should read "principal hypervectors"; "each principal hypervectors" should read "each principal hypervector"
Claim 6: "further comprising;" should read "further comprising:"; "one or the following" should read "one of the following"
Claim 8: "computer implemented" should read "computer-implemented"; "a the plurality of concepts" should read "the plurality of concepts"
Claim 9: “the convergence criteria” should read “the convergence criterion”
Claim 10: “the convergence criteria” should read “the convergence criterion”; "when a predefined number of iterations." should read "when a predefined number of iterations is reached."
Claim 11: "a the plurality of concepts" should read "the plurality of concepts"; "unbind by the input hypervector, into" should read "unbind the input hypervector into"; "the each of the plurality of unbound hypervectors" should read "each of the plurality of unbound hypervectors"; "a principal hypervectors" should read "principal hypervectors"; "each principal hypervectors" should read "each principal hypervector"
Claims 12-15: "system of claim 1" should read "system of claim 11"
Claim 16: "system of claim 1" should read "system of claim 11"; "further comprising;" should read "further comprising:"; "one or the following" should read "one of the following"
Claim 17: "applying, by the processor, the plurality" should read "apply the plurality"
Claim 18: "system of claim 1" should read "system of claim 11"; "a the plurality of concepts" should read "the plurality of concepts"
Claim 19: “the convergence criteria” should read “the convergence criterion”
Claim 20: "a the plurality of concepts" should read "the plurality of concepts"; "unbind by the input hypervector, into" should read "unbind the input hypervector into"; "the each of the plurality of unbound hypervectors" should read "each of the plurality of unbound hypervectors"; "a principal hypervectors" should read "principal hypervectors"; "each principal hypervectors" should read "each principal hypervector"
Claims 2-5 and 7 are objected to due to dependency on an objected-to base claim.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 4, 9, 14, and 19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 4 and 14 recite the limitation “the similarity vector of the hypervector”. There is insufficient antecedent basis for this limitation in the claim. The claims previously recite “one or more similarity vectors for each of the unbound hypervectors”, however there is no singular similarity vector of a hypervector previously introduced, thus it is unclear what similarity vector this limitation is meant to refer to.
Claims 9 and 19 recite the limitation "the plurality of similarity scores”. There is insufficient antecedent basis for this limitation in the claim. It is unclear if this limitation is meant to refer to the “one or more similarity vectors” of claims 1 and 11, or to a different plurality of similarity scores.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 20 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because it is directed to “A computer program product…comprising: one or more computer readable storage devices” which encompasses signals per se. Examiner recommends amending the claim to recite “one or more non-transitory computer readable storage devices” or “one or more computer readable storage media” in order to overcome this rejection, given that the specification defines “computer readable storage medium” as excluding signals per se (see [0094]).
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The analysis of the claims will follow the 2019 Revised Patent
Subject Matter Eligibility Guidance (2019 PEG”).
Claim 1
Step 1: The claim is directed to a computer-implemented method and thus is directed to the statutory category of processes.
Step 2A Prong 1: The claim recites, inter alia:
unbinding…the input hypervector, into a plurality of unbound hypervectors, wherein the each of the plurality of unbound hypervectors corresponds to a single concept from the plurality of concepts of the data structure: This limitation encompasses mentally unbinding the input hypervector into a plurality of unbound hypervectors, such as by mentally determining the factors of an input hypervector.
generating… one or more similarity vectors for each of the unbound hypervectors, wherein each of the one or more similarity vectors is based on the similarity of the unbound hypervector and one or more candidate code hypervectors representing each of the plurality of concepts: This limitation encompasses mentally generating one or more similarity vectors for each of the unbound hypervectors, such as by mentally comparing the unbound hypervector to each of the candidate code hypervectors and determining a vector of similarity scores.
generating… a plurality of a principal hypervectors, wherein each principal hypervectors is an estimate which represents one concept of the plurality of concepts, based at least in part on the one or more similarity vectors corresponding to the one concept: This limitation encompasses mentally generating a plurality of principal hypervectors based on the one or more similarity vectors.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites “receiving an input hypervector representing a data structure comprised of a plurality of concepts”, however this limitation amounts to the insignificant extra-solution activity of mere data gathering (MPEP 2106.05(g)). The claim additionally recites “A computer-implemented method to factorize an input hypervector representing a plurality of concepts into individual hypervectors each representing a concept from a the plurality of concepts, through iterative processing of a resonator network” and that each step is performed “by the processor”, however these limitations amount to mere instructions to apply the judicial exception on a generic computer programmed with a generic class of computer algorithms (MPEP 2106.05(f)).
Step 2B: The claim does not contain significantly more than the judicial exception. The “receiving an input hypervector” limitation, in addition to reciting insignificant extra-solution activity, is also directed to the well-understood, routine, and conventional activity of receiving or transmitting data over a network (MPEP 2106.05(d)(II)(i) buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network)). Otherwise, the analysis at this step mirrors that of Step 2A Prong 2 above. As an ordered whole, the claim is directed to an abstract idea of factorizing an input hypervector by unbinding the input hypervector, generating one or more similarity vectors for each unbound hypervector based on candidate code hypervectors, and generating a plurality of principal hypervectors based on the similarity vectors. Nothing in the claim provides significantly more than this. As such, the claim is not patent eligible.
Claim 2
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
wherein generating a similarity vector is based on one of the following: a dot product, L1 norm, L2 norm, or L^∞ norm: This limitation encompasses a mathematical calculation of performing a dot product, L1 norm, L2 norm, or L^∞ norm to generate a similarity vector.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. No further additional elements are recited, see analysis of claim 1.
Step 2B: The claim does not contain significantly more than the judicial exception. No further additional elements are recited, see analysis of claim 1.
Claim 3
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
wherein unbinding the input hypervector is based on a circular convolution or an addition and modulo operation: This limitation encompasses the mathematical calculation of unbinding the input hypervector using circular convolution or an addition and modulo operation.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. No further additional elements are recited, see analysis of claim 1.
Step 2B: The claim does not contain significantly more than the judicial exception. No further additional elements are recited, see analysis of claim 1.
Claim 4
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
adding… noise to the similarity vector of the hypervector, wherein the noise is gaussian noise or uniform noise: This limitation encompasses the mathematical calculation of adding noise to a vector.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites that the adding is performed “by the processor”, however this limitation amounts to mere instructions to apply a judicial exception on a generic computer (MPEP 2106.05(f)).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step mirrors that of Step 2A Prong 2 above.
Claim 5
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
separating… the similarity vectors, based on a softmax operation or an identity operation: This limitation encompasses the mathematical calculation of applying a softmax operation or an identity operation to the similarity vectors.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites that the separating is performed “by the processor”, however this limitation amounts to mere instructions to apply a judicial exception on a generic computer (MPEP 2106.05(f)).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step mirrors that of Step 2A Prong 2 above.
Claim 6
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
sparsifying… one or more elements of the similarity vectors, wherein sparsifying is based on one or the following: a pre-determined threshold, a dynamic threshold, a Top-A operation, or an absolute value larger than a mean of all: This limitation encompasses mentally sparsifying one or more elements of the similarity vectors, such as by mentally selecting one or more elements of the similarity vectors to set to zero based on a pre-determined threshold, a dynamic threshold, a Top-A operation, or an absolute value larger than a mean of all.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites that the sparsifying is performed “by the processor”, however this limitation amounts to mere instructions to apply a judicial exception on a generic computer (MPEP 2106.05(f)).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step mirrors that of Step 2A Prong 2 above.
Claim 7
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
combining… through a linear combination each of the candidate code hypervectors with weights to generate a plurality of bundled weights, based on the sparsified similarity vector corresponding to the candidate code hypervector: This limitation encompasses a mathematical calculation of performing a linear combination.
applying… the plurality of bundled weights to a selection function: This limitation encompasses mentally applying the plurality of bundled weights to a selection function, such as by mentally selecting elements of the plurality of bundled weights based on a selection function.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites that the sparsifying and applying is performed “by the processor”, however this limitation amounts to mere instructions to apply a judicial exception on a generic computer (MPEP 2106.05(f)).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step mirrors that of Step 2A Prong 2 above.
Claim 8
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the same judicial exception as claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites that the iterative processing of a resonator network “performs a plurality of iterations until a convergence criterion is fulfilled”, however this limitation amounts to mere instructions to apply a judicial exception on a generic computer programmed with a generic class of computer algorithms (MPEP 2106.05(f)).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step mirrors that of Step 2A Prong 2 above.
Claim 9
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the same judicial exception as claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites “wherein the convergence criteria is when a value of at least one element of each of the plurality of similarity scores exceeds a threshold”, however this merely further limits the convergence criteria of the iterative processing of a resonator network and still amounts to mere instructions to apply a judicial exception on a generic computer programmed with a generic class of computer algorithms (MPEP 2106.05(f)).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step mirrors that of Step 2A Prong 2 above.
Claim 10
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the same judicial exception as claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites “wherein the convergence criteria is when a predefined number of iterations”, however this merely further limits the convergence criteria of the iterative processing of a resonator network and still amounts to mere instructions to apply a judicial exception on a generic computer programmed with a generic class of computer algorithms (MPEP 2106.05(f)).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step mirrors that of Step 2A Prong 2 above.
Claims 11-19
Step 1: The claims are directed to a computer system and thus are directed to the statutory category of machines.
Step 2A Prong 1: Claims 11-19 recite the same judicial exception as claims 1-9, respectively,
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The analysis at this step mirrors that of claims 1-9, respectively, except insofar as claims 11-19 further recite “A computer system…comprising: one or more computer processors; one or more computer readable storage devices; program instructions stored on the one or more computer readable storage devices for execution by at least one of the one or more computer processors, the program instructions comprising: [the method]”, however this limitation amounts to mere instructions to apply a judicial exception on a generic computer programmed with a generic class of computer algorithms (MPEP 2106.05(f)).
Step 2B: The claims do not contain significantly more than the judicial exception. The computer system limitation amounts to mere instructions to apply a judicial exception on a generic computer programmed with a generic class of computer algorithms (MPEP 2106.05(f)), as stated above. Otherwise, the analysis at this step mirrors that of claims 1-9, respectively.
Claim 20
Step 1: The claim is directed to non-statutory subject matter, however for the purposes of the abstract idea rejection, Examiner will assume it is directed to the statutory category of articles of manufacture.
Step 2A Prong 1: The claim recites the same judicial exception as claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The analysis at this step mirrors that of claim 1, except insofar as claim 20 further recites “A computer program product…comprising: one or more computer readable storage devices; program instructions stored on the one or more computer readable storage devices, wherein the program instructions are executable by a computer processor to perform one or more operations, the operations comprising: [the method]”, however this limitation amounts to mere instructions to apply a judicial exception on a generic computer programmed with a generic class of computer algorithms (MPEP 2106.05(f)).
Step 2B: The claim does not contain significantly more than the judicial exception. The computer program product limitation amounts to mere instructions to apply a judicial exception on a generic computer programmed with a generic class of computer algorithms (MPEP 2106.05(f)), as stated above. Otherwise, the analysis at this step mirrors that of claim 1.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 2, 6, 8, 10, 12, 16, 18, and 20 are rejected under 35 U.S.C. 102(a)(1)/102(a)(2) as being anticipated by Hersche et al. (US20230206056) (hereinafter “Hersche”).
The applied reference has a common applicant/joint inventor with the instant application. Based upon the earlier effectively filed date of the reference, it constitutes prior art under 35 U.S.C. 102(a)(2). This rejection under 35 U.S.C. 102(a)(2) might be overcome by: (1) a showing under 37 CFR 1.130(a) that the subject matter disclosed in the reference was obtained directly or indirectly from the inventor or a joint inventor of this application and is thus not prior art in accordance with 35 U.S.C. 102(b)(2)(A); (2) a showing under 37 CFR 1.130(b) of a prior public disclosure under 35 U.S.C. 102(b)(2)(B) if the same invention is not being claimed; or (3) a statement pursuant to 35 U.S.C. 102(b)(2)(C) establishing that, not later than the effective filing date of the claimed invention, the subject matter disclosed in the reference and the claimed invention were either owned by the same person or subject to an obligation of assignment to the same person or subject to a joint research agreement.
Regarding claim 1, Hersche discloses “A computer-implemented method ([0061]: “FIG. 6 is a block diagram depicting components of a computing device, generally designated 600, suitable for performing a method for factorizing hypervectors in a resonator network in accordance with at least one embodiment of the present invention. Computing device 600 includes one or more processor(s) 604 (including one or more computer processors), communications fabric 602, memory 606 including, RAM 616 and cache 618, persistent storage 608, communications unit 612, I/O interface(s) 614, display 622, and external device(s) 620”) to factorize an input hypervector representing a plurality of concepts into individual hypervectors each representing a concept from a the plurality of concepts, through iterative processing of a resonator network ([0002]: “Given a hypervector formed from an element-wise product of two or more atomic hypervectors (each from a fixed codebook), a resonator network may find its factors. The resonator network may iteratively search over the alternatives for each factor individually rather than all possible combinations until a set of factors is found that agrees with the input hypervector”), the computer-implemented method comprising :
receiving an input hypervector representing a data structure comprised of a plurality of concepts ([0044]: “For simplicity purposes, resonator network computing environment 100 is configured to execute a resonator network to decode hypervectors that are encoded in a vector space defined by three concepts” and [0046]: “In an embodiment, an input hypervector 101 named s is received by resonator network computing environment 100. The input hypervector s may be the result of encoding a data structure such as a coloured image comprising MNIST digits. The encoding may be performed by a VSA technique. At t = 0, the resonator network computing environment 100 initializes an estimate of the hypervector that represents each concept of the set of concepts as a superposition of all candidate code hypervectors of said concept as follows: x̂(0) = sign(
∑
i
=
1
,
.
.
.
,
M
x
x
i
), ŷ(0) = sign(
∑
j
=
1
,
.
.
.
,
M
y
y
j
) and ẑ(0) = sign(
∑
k
=
1
,
.
.
.
,
M
z
z
k
)”);
unbinding, by the processor, the input hypervector, into a plurality of unbound hypervectors, wherein the each of the plurality of unbound hypervectors corresponds to a single concept from the plurality of concepts of the data structure ([0047]: “In an embodiment, the three network nodes may compute the first estimates x̃(t), ỹ(t), and z̃(t) of the hypervectors that represent the set of concepts respectively as follows: x̃(t) = s⊙ ŷ(t)⊙ ẑ(t), ỹ(t) = s⊙ x̂(t)⊙ ẑ(t) and z̃(t) = s⊙ x̂(t)⊙ ŷ(t), where ⊙ refers to elementwise multiplication. This may be referred to as an inference step. That is, the nodes may perform the inference step on respective input triplets”; Examiner notes that x̃(t), ỹ(t), and z̃(t) correspond to a plurality of unbound hypervectors, which are obtained by unbinding input hypervector s);
generating, by the processor, one or more similarity vectors for each of the unbound hypervectors, wherein each of the one or more similarity vectors is based on the similarity of the unbound hypervector and one or more candidate code hypervectors representing each of the plurality of concepts ([0044]: “The codebooks/matrices representing the set of concepts may be referred to as X, Y and Z respectively” and [0048]: “In an embodiment, the similarity of the first estimate x̃(t) with each of the Mx code hypervectors
x
i
. . .
x
M
x
is computed using the codebook X stored in memory 104x as follows:
a
x
t
=
X
T
x
~
t
∈
R
M
x
for multiplying the hypervector
x
~
t
by the matrix
X
T
. In an embodiment, the similarity of the first estimate ỹ(t) with each of the My code hypervectors
y
j
. . .
y
M
y
is computed using the codebook Y stored in memory 104y as follows:
a
y
t
=
Y
T
ỹ
(
t
)
∈
R
M
y
for multiplying the hypervector ỹ(t) by the matrix
Y
T
. In an embodiment, the similarity of the first estimate z̃(t) with each of the Mz code hypervectors
z
k
. . .
z
M
z
is computed using the codebook Z stored in memory 104z as follows:
a
z
t
=
Z
T
z
̃
(
t
)
∈
R
M
z
for multiplying the hypervector z̃(t) by the matrix
Z
T
. The resulting vectors
a
x
t
,
a
y
t
a
n
d
a
z
t
may be named similarity vectors or attention vectors”); and
generating, by the processor, a plurality of a principal hypervectors, wherein each principal hypervectors is an estimate which represents one concept of the plurality of concepts, based at least in part on the one or more similarity vectors corresponding to the one concept ([0050]: “In an embodiment, after obtaining the modified similarity vectors
a
'
x
t
,
a
'
y
t
a
n
d
a
'
z
t
, a weighted superposition of the modified similarity vectors
a
'
x
t
,
a
'
y
t
a
n
d
a
'
z
t
is performed using the codebooks
X
T
,
Y
T
and
Z
T
stored in memories 108x, 108y, and 108z, respectively. This may be performed by the following matrix vector multiplications: X
a
'
x
t
, Y
a
'
y
t
and Z
a
'
z
t
. The resulting hypervectors X
a
'
x
t
, Y
a
'
y
t
and Z
a
'
z
t
are fed to the sign units 110x, 110y and 110z, respectively. This results in obtaining the following: x̂(t+1)=sign(X
a
'
x
t
), ŷ(t+1)=sign(Y
a
'
y
t
) and ẑ(t+1)=sign(Z
a
'
z
t
), respectively, which subsequently results in obtaining the estimate of the hypervectors x̂(t+1), ŷ(t+1) and ẑ(t+1) respectively for the next iteration t+1”; Examiner notes that x̂(t+1), ŷ(t+1) and ẑ(t+1) correspond to a plurality of principal hypervectors).
Regarding claim 2, the rejection of claim 1 is incorporated. Hersche further discloses “wherein generating a similarity vector is based on one of the following: a dot product, L1 norm, L2 norm, or L^∞ norm” ([0028]: “The similarity of each code hypervector of the given concept may be computed as a dot product of the codebook that represents the given concept by the first estimate of the hypervector, resulting in a similarity vector ax(t), ay(t) and az(t), respectively”).
Regarding claim 6, the rejection of claim 1 is incorporated. Hersche further discloses “sparsifying, by the processor, one or more elements of the similarity vectors, wherein sparsifying is based on one or the following: a pre-determined threshold, a dynamic threshold, a Top-A operation, or an absolute value larger than a mean of all” ([0049]: “In an embodiment, after computing the similarity vectors, the similarity vectors ax(t), ay(t) and az(t) are sparsified using the activation function kact implemented by activation units 106x, 106y and 106z respectively. In an embodiment, the sparsification of the similarity vector is performed by activating a portion of the elements of the similarity vector” and [0030]: “In a first embodiment, the activation function kact may only activate the top K absolute values in each of the similarity vectors in each of the of the similarity vectors ax(t), ay(t) and az(t) where K<<Mz and deactivate the rest of the elements by setting them to a given value”).
Regarding claim 8, the rejection of claim 1 is incorporated. Hersche further discloses “wherein the computer implemented method to factorize an input hypervector representing a plurality of concepts into individual hypervectors each representing a concept from a the plurality of concepts, through iterative processing of a resonator network performs a plurality of iterations until a convergence criterion is fulfilled” ([0051]: “In an embodiment, the iterative process may stop if a stopping criterion is fulfilled”).
Regarding claim 10, the rejection of claim 8 is incorporated. Hersche further discloses “wherein the convergence criteria is when a predefined number of iterations” ([0051]: “In an embodiment, the iterative process may stop if a stopping criterion is fulfilled. The stopping criterion may, for example, require… or that a maximum number of iterations is reached”).
Regarding claim 11, Hersche discloses “A computer system to factorize an input hypervector representing a plurality of concepts into individual hypervectors each representing a concept from a the plurality of concepts, through iterative processing of a resonator network ([0002]: “Given a hypervector formed from an element-wise product of two or more atomic hypervectors (each from a fixed codebook), a resonator network may find its factors. The resonator network may iteratively search over the alternatives for each factor individually rather than all possible combinations until a set of factors is found that agrees with the input hypervector”), the computer system comprising: one or more computer processors ([0061]: “FIG. 6 is a block diagram depicting components of a computing device, generally designated 600, suitable for performing a method for factorizing hypervectors in a resonator network in accordance with at least one embodiment of the present invention. Computing device 600 includes one or more processor(s) 604 (including one or more computer processors), communications fabric 602, memory 606 including, RAM 616 and cache 618, persistent storage 608, communications unit 612, I/O interface(s) 614, display 622, and external device(s) 620”); one or more computer readable storage devices ([0063]: “Memory 606 and persistent storage 608 are computer readable storage media”); program instructions stored on the one or more computer readable storage devices for execution by at least one of the one or more computer processors ([0064]: “Program instructions for performing a method for factorizing hypervectors in a resonator network in accordance with at least one embodiment of the present invention can be stored in persistent storage 608”), the program instructions comprising:
receive an input hypervector representing a data structure comprised of a plurality of concepts ([0044]: “For simplicity purposes, resonator network computing environment 100 is configured to execute a resonator network to decode hypervectors that are encoded in a vector space defined by three concepts” and [0046]: “In an embodiment, an input hypervector 101 named s is received by resonator network computing environment 100. The input hypervector s may be the result of encoding a data structure such as a coloured image comprising MNIST digits. The encoding may be performed by a VSA technique. At t = 0, the resonator network computing environment 100 initializes an estimate of the hypervector that represents each concept of the set of concepts as a superposition of all candidate code hypervectors of said concept as follows: x̂(0) = sign(
∑
i
=
1
,
.
.
.
,
M
x
x
i
), ŷ(0) = sign(
∑
j
=
1
,
.
.
.
,
M
y
y
j
) and ẑ(0) = sign(
∑
k
=
1
,
.
.
.
,
M
z
z
k
)”);
unbind by the input hypervector, into a plurality of unbound hypervectors, wherein the each of the plurality of unbound hypervectors corresponds to a single concept from the plurality of concepts of the data structure ([0047]: “In an embodiment, the three network nodes may compute the first estimates x̃(t), ỹ(t), and z̃(t) of the hypervectors that represent the set of concepts respectively as follows: x̃(t) = s⊙ ŷ(t)⊙ ẑ(t), ỹ(t) = s⊙ x̂(t)⊙ ẑ(t) and z̃(t) = s⊙ x̂(t)⊙ ŷ(t), where ⊙ refers to elementwise multiplication. This may be referred to as an inference step. That is, the nodes may perform the inference step on respective input triplets”; Examiner notes that x̃(t), ỹ(t), and z̃(t) correspond to a plurality of unbound hypervectors, which are obtained by unbinding input hypervector s);
generate one or more similarity vectors for each of the unbound hypervectors, wherein each of the one or more similarity vectors is based on the similarity of the unbound hypervector and one or more candidate code hypervectors representing each of the plurality of concepts ([0044]: “The codebooks/matrices representing the set of concepts may be referred to as X, Y and Z respectively” and [0048]: “In an embodiment, the similarity of the first estimate x̃(t) with each of the Mx code hypervectors
x
i
. . .
x
M
x
is computed using the codebook X stored in memory 104x as follows:
a
x
t
=
X
T
x
~
t
∈
R
M
x
for multiplying the hypervector
x
~
t
by the matrix
X
T
. In an embodiment, the similarity of the first estimate ỹ(t) with each of the My code hypervectors
y
j
. . .
y
M
y
is computed using the codebook Y stored in memory 104y as follows:
a
y
t
=
Y
T
ỹ
(
t
)
∈
R
M
y
for multiplying the hypervector ỹ(t) by the matrix
Y
T
. In an embodiment, the similarity of the first estimate z̃(t) with each of the Mz code hypervectors
z
k
. . .
z
M
z
is computed using the codebook Z stored in memory 104z as follows:
a
z
t
=
Z
T
z
̃
(
t
)
∈
R
M
z
for multiplying the hypervector z̃(t) by the matrix
Z
T
. The resulting vectors
a
x
t
,
a
y
t
a
n
d
a
z
t
may be named similarity vectors or attention vectors”); and
generate a plurality of a principal hypervectors, wherein each principal hypervectors is an estimate which represents one concept of the plurality of concepts, based at least in part on the one or more similarity vectors corresponding to the one concept ([0050]: “In an embodiment, after obtaining the modified similarity vectors
a
'
x
t
,
a
'
y
t
a
n
d
a
'
z
t
, a weighted superposition of the modified similarity vectors
a
'
x
t
,
a
'
y
t
a
n
d
a
'
z
t
is performed using the codebooks
X
T
,
Y
T
and
Z
T
stored in memories 108x, 108y, and 108z, respectively. This may be performed by the following matrix vector multiplications: X
a
'
x
t
, Y
a
'
y
t
and Z
a
'
z
t
. The resulting hypervectors X
a
'
x
t
, Y
a
'
y
t
and Z
a
'
z
t
are fed to the sign units 110x, 110y and 110z, respectively. This results in obtaining the following: x̂(t+1)=sign(X
a
'
x
t
), ŷ(t+1)=sign(Y
a
'
y
t
) and ẑ(t+1)=sign(Z
a
'
z
t
), respectively, which subsequently results in obtaining the estimate of the hypervectors x̂(t+1), ŷ(t+1) and ẑ(t+1) respectively for the next iteration t+1”; Examiner notes that x̂(t+1), ŷ(t+1) and ẑ(t+1) correspond to a plurality of principal hypervectors).
Regarding claim 12, the rejection of claim 11 is incorporated. Claim 12 is a computer system claim corresponding to method claim 2, and the rejection follows the same rationale as the rejection of claim 2 above.
Regarding claim 16, the rejection of claim 11 is incorporated. Claim 16 is a computer system claim corresponding to method claim 6, and the rejection follows the same rationale as the rejection of claim 6 above.
Regarding claim 18, the rejection of claim 11 is incorporated. Claim 18 is a computer system claim corresponding to method claim 8, and the rejection follows the same rationale as the rejection of claim 8 above.
Regarding claim 20, Hersche discloses “A computer program product to factorize an input hypervector representing a plurality of concepts into individual hypervectors each representing a concept from a the plurality of concepts, through iterative processing of a resonator network ([0002]: “Given a hypervector formed from an element-wise product of two or more atomic hypervectors (each from a fixed codebook), a resonator network may find its factors. The resonator network may iteratively search over the alternatives for each factor individually rather than all possible combinations until a set of factors is found that agrees with the input hypervector”), the computer program product comprising: one or more computer readable storage devices ([0063]: “Memory 606 and persistent storage 608 are computer readable storage media”); program instructions stored on the one or more computer readable storage devices, wherein the program instructions are executable by a computer processor to perform one or more operations ([0064]: “Program instructions for performing a method for factorizing hypervectors in a resonator network in accordance with at least one embodiment of the present invention can be stored in persistent storage 608, or more generally, any computer readable storage media, for execution by one or more of the respective computer processor(s) 604 via one or more memories of memory 606”), the operations comprising:
receive an input hypervector representing a data structure comprised of a plurality of concepts ([0044]: “For simplicity purposes, resonator network computing environment 100 is configured to execute a resonator network to decode hypervectors that are encoded in a vector space defined by three concepts” and [0046]: “In an embodiment, an input hypervector 101 named s is received by resonator network computing environment 100. The input hypervector s may be the result of encoding a data structure such as a coloured image comprising MNIST digits. The encoding may be performed by a VSA technique. At t = 0, the resonator network computing environment 100 initializes an estimate of the hypervector that represents each concept of the set of concepts as a superposition of all candidate code hypervectors of said concept as follows: x̂(0) = sign(
∑
i
=
1
,
.
.
.
,
M
x
x
i
), ŷ(0) = sign(
∑
j
=
1
,
.
.
.
,
M
y
y
j
) and ẑ(0) = sign(
∑
k
=
1
,
.
.
.
,
M
z
z
k
)”);
unbind by the input hypervector, into a plurality of unbound hypervectors, wherein the each of the plurality of unbound hypervectors corresponds to a single concept from the plurality of concepts of the data structure ([0047]: “In an embodiment, the three network nodes may compute the first estimates x̃(t), ỹ(t), and z̃(t) of the hypervectors that represent the set of concepts respectively as follows: x̃(t) = s⊙ ŷ(t)⊙ ẑ(t), ỹ(t) = s⊙ x̂(t)⊙ ẑ(t) and z̃(t) = s⊙ x̂(t)⊙ ŷ(t), where ⊙ refers to elementwise multiplication. This may be referred to as an inference step. That is, the nodes may perform the inference step on respective input triplets”; Examiner notes that x̃(t), ỹ(t), and z̃(t) correspond to a plurality of unbound hypervectors, which are obtained by unbinding input hypervector s);
generate one or more similarity vectors for each of the unbound hypervectors, wherein each of the one or more similarity vectors is based on the similarity of the unbound hypervector and one or more candidate code hypervectors representing each of the plurality of concepts ([0044]: “The codebooks/matrices representing the set of concepts may be referred to as X, Y and Z respectively” and [0048]: “In an embodiment, the similarity of the first estimate x̃(t) with each of the Mx code hypervectors
x
i
. . .
x
M
x
is computed using the codebook X stored in memory 104x as follows:
a
x
t
=
X
T
x
~
t
∈
R
M
x
for multiplying the hypervector
x
~
t
by the matrix
X
T
. In an embodiment, the similarity of the first estimate ỹ(t) with each of the My code hypervectors
y
j
. . .
y
M
y
is computed using the codebook Y stored in memory 104y as follows:
a
y
t
=
Y
T
ỹ
(
t
)
∈
R
M
y
for multiplying the hypervector ỹ(t) by the matrix
Y
T
. In an embodiment, the similarity of the first estimate z̃(t) with each of the Mz code hypervectors
z
k
. . .
z
M
z
is computed using the codebook Z stored in memory 104z as follows:
a
z
t
=
Z
T
z
̃
(
t
)
∈
R
M
z
for multiplying the hypervector z̃(t) by the matrix
Z
T
. The resulting vectors
a
x
t
,
a
y
t
a
n
d
a
z
t
may be named similarity vectors or attention vectors”); and
generate a plurality of a principal hypervectors, wherein each principal hypervectors is an estimate which represents one concept of the plurality of concepts, based at least in part on the one or more similarity vectors corresponding to the one concept ([0050]: “In an embodiment, after obtaining the modified similarity vectors
a
'
x
t
,
a
'
y
t
a
n
d
a
'
z
t
, a weighted superposition of the modified similarity vectors
a
'
x
t
,
a
'
y
t
a
n
d
a
'
z
t
is performed using the codebooks
X
T
,
Y
T
and
Z
T
stored in memories 108x, 108y, and 108z, respectively. This may be performed by the following matrix vector multiplications: X
a
'
x
t
, Y
a
'
y
t
and Z
a
'
z
t
. The resulting hypervectors X
a
'
x
t
, Y
a
'
y
t
and Z
a
'
z
t
are fed to the sign units 110x, 110y and 110z, respectively. This results in obtaining the following: x̂(t+1)=sign(X
a
'
x
t
), ŷ(t+1)=sign(Y
a
'
y
t
) and ẑ(t+1)=sign(Z
a
'
z
t
), respectively, which subsequently results in obtaining the estimate of the hypervectors x̂(t+1), ŷ(t+1) and ẑ(t+1) respectively for the next iteration t+1”; Examiner notes that x̂(t+1), ŷ(t+1) and ẑ(t+1) correspond to a plurality of principal hypervectors).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 3, 7, 13, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Hersche in view of Hersche et al. (Factorizers for Distributed Sparse Block Codes) (hereinafter “Hersche2”).
Regarding claim 3, the rejection of claim 1 is incorporated. Hersche does not appear to explicitly disclose the further limitations of the claim.
However, Hersche2 discloses “wherein unbinding the input hypervector is based on a circular convolution or an addition and modulo operation” (Hersche2, IV. A. Generalized sparse block codes (GSBCs): “a) Binding/Unbinding: We exploit general binding and unbinding operations in blockwise circular convolution and correlation to support arbitrary block representations. Specifically, if both operands have blockwise unit ℓ1-norm, the result does as well”).
Hersche2 and the instant application both relate to factorizing vectors and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified Hersche with the teachings of Hersche2 such that unbinding the input hypervector is based on a circular convolution, and one would have been motivated to do so. Doing so would allow for supporting arbitrary block representations (see Hersche2, IV. A. Generalized sparse block codes (GSBCs)).
Regarding claim 7, the rejection of claim 6 is incorporated. Hersche further discloses “combining, by the processor, through a linear combination each of the candidate code hypervectors with weights to generate a plurality of bundled weights, based on the sparsified similarity vector corresponding to the candidate code hypervector (Hersche, [0032]: “In other words, the superposition step generates each of the estimates x̂(t+1), ŷ(t+1), ẑ(t+1) representing the respective concept by a linear combination of the candidate code hypervectors (provided in respective matrices X, Y, and Z), with weights given by the respective sparsified similarity vectors
a
'
x
t
,
a
'
y
t
,
a
'
z
t
…
"
); and
applying, by the processor, the plurality of bundled weights to a… [non-linear] function” ([0032]: “…followed by the application of the non-linear function g”).
Hersche doesn’t appear to explicitly disclose that the non-linear function is a “selection” function.
However, Hersche2 discloses “applying… [summed vectors] to a selection function” (Hersche2, III. VSA Preliminary: “The bundling of two or more vectors is defined as their elementwise addition, followed by a selection function that retains the sparsity by setting the largest element of each block to 1 and the remaining elements to 0”).
Hersche2 and the instant application both relate to factorizing vectors and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified Hersche with the teachings of Hersche2 such that the function is a selection function, and one would have been motivated to do so. Doing so would ensure that sparsity is retained (see Hersche2, III. VSA Preliminary).
Regarding claim 13, the rejection of claim 11 is incorporated. Claim 13 is a computer system claim corresponding to method claim 3, and the rejection follows the same rationale as the rejection of claim 3 above.
Regarding claim 17, the rejection of claim 16 is incorporated. Claim 17 is a computer system claim corresponding to method claim 7, and the rejection follows the same rationale as the rejection of claim 7 above.
Claims 4, 9, 14, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Hersche in view of Langenegger et al. (In-memory factorization of holographic perceptual representations) (hereinafter “Langenegger”).
Regarding claim 4, the rejection of claim 1 is incorporated. Hersche does not appear to explicitly disclose the further limitations of the claim.
However, Langenegger discloses “adding… noise to the similarity vector of the hypervector, wherein the noise is gaussian noise or uniform noise” (Langenegger, III. DISCUSSION, Software simulations of the in-memory factorizer: “The stochasticity is the key enabler of the in-memory factorizer. Adding some stochasticity helps to diverge from the limit cycles as each solution becomes unique. For the experiments reported in Fig. 3, this important aspect is modeled in software by simulating the noisy behaviour of the MVMs on the crossbar as an additive Gaussian noise with zero mean:
α
i
'
= f(αi)=αi+n, where αi is a single entry of the output vector, and n is normally distributed with n ∼ N (0, σ2). In total, we simulated F ×(M+D) additive Gaussian noise sources: M on the similarity vector and D on the projection vector. The noisy similarity vector is required to break free of the limit cycles. Due to the random distribution of the similarity values, there is always a chance of activating none of the similarity values if they do not cross the activation threshold. Adding noise on top of the projection prevents such an all-zero estimation by randomly initializing the vector prior to the bipolarization”).
Langenegger and the instant application both relate to factorizing vectors and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified Hersche with the teachings of Langenegger to include “adding, by the processor, noise to the similarity vector of the hypervector, wherein the noise is gaussian noise or uniform noise,” and one would have been motivated to do so. Doing so would allow for diverging from limit cycles and preventing an all-zero estimation (see Langengger, III. DISCUSSION, Software simulations of the in-memory factorizer).
Regarding claim 9, the rejection of claim 8 is incorporated. Hersche does not appear to explicitly disclose the further limitations of the claim.
However, Langenegger discloses “wherein the convergence criteria is when a value of at least one element of each of the plurality of similarity scores exceeds a threshold” (Langengger, III. DISCUSSION, Detection of convergence in the in-memory factorizer: “To detect this convergence, we define a novel early convergence detection algorithm. The in-memory factorizer is said to be converged if a single similarity value across all the factors surpasses a convergence detection threshold”).
Langenegger and the instant application both relate to factorizing vectors and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified Hersche with the teachings of Langenegger such that the convergence criteria is when the convergence criteria is when a value of at least one element of each of the plurality of similarity scores exceeds a threshold, and one would have been motivated to do so. Doing so would counteract parasitic codevectors and decrease resource usage by eliminating the need to store the history of prior estimates to detect convergence (see Langengger, SUPPLEMENTARY NOTES, III. DISCUSSION, Detection of convergence in the in-memory factorizer).
Regarding claim 14, the rejection of claim 11 is incorporated. Claim 14 is a computer system claim corresponding to method claim 4, and the rejection follows the same rationale as the rejection of claim 4 above.
Regarding claim 19, the rejection of claim 18 is incorporated. Claim 19 is a computer system claim corresponding to method claim 9, and the rejection follows the same rationale as the rejection of claim 9 above.
Claims 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Hersche in view of
Renner et al. (Neuromorphic Visual Scene Understanding with Resonator Networks) (hereinafter “Renner”).
Regarding claim 5, the rejection of claim 1 is incorporated. Hersche does not appear to explicity disclose the further limitations of the claim.
However, Renner discloses “separating… the similarity vectors, based on a softmax operation or an identity operation” (Renner, Methods, Details of simulation experiments: “Beyond the dynamics described in equation (10), we included some modifications to improve performance…A second modification encouraged sparsity in the solutions by including a non-linearity during the cleanup that encourages sparsity, such as a polynomial exponent or softmax function, i.e.:
PNG
media_image1.png
42
876
media_image1.png
Greyscale
”; Examiner notes that eq(12) depicts function p(x) being applied to the similarity vector, where p(x) may be a softmax function).
Renner and the instant application both relate to factorizing hypervectors using resonator networks and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified Hersche with the teachings of Renner to include a step of “separating, by the processor, the similarity vectors, based on a softmax operation or an identity operation”, and one would have been motivated to do so. Doing so would improve performance by encouraging sparsity (see Renner, Methods, Details of simulation experiments).
Regarding claim 15, the rejection of claim 11 is incorporated. Claim 15 is a computer system claim corresponding to method claim 5, and the rejection follows the same rationale as the rejection of claim 5 above.
Claims 1, 2, 5, 8, 11, 12, 15, 18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Frady et al. (Resonator Networks, 1: An Efficient Solution for Factoring High-Dimensional, Distributed Representations of Data Structures) in view of Renner et al. (Neuromorphic Visual Scene Understanding with Resonator Networks) (hereinafter “Renner”).
Regarding claim 1, Frady discloses: “A… method to factorize an input hypervector representing a plurality of concepts into individual hypervectors each representing a concept from a the plurality of concepts, through iterative processing of a resonator network (Frady, 4.2 Visual Scene Analysis as a Factorization Problem: “The scene can then be decoded by iterating through the resonator network”), the… method comprising :
receiving an input hypervector representing a data structure comprised of a plurality of concepts (Frady, 4.2 Visual Scene Analysis as a Factorization Problem: “Any given scene can have between one and three of these objects, which are allowed to partially occlude one another. We generate symbolic vectors cblue, cgreen, ..., cwhite to encode color; d0, d1, ..., d9 to encode shape; vtop, vmiddle, vbottom to encode vertical position; and hleft, hcenter, hright to encode horizontal position, which are stored in respective codebooks, C, D, V, H… The VSA approach to represent a scene like this is to form the conjunction of each of the four factors with the binding operation and superposing multiple objects together to form a single high-dimensional vector that constitutes a distributed representation of the entire scene. This encoding is depicted in Figure 3, and as in the previous examples, the encoding provides a flexible data structure such that aspects of the scene can be individually queried… Supervised learning via backpropagation is used to train the network to output the VSA representation of the entire scene from the image pixels as input…When we present the scene vector s to a resonator network, it automatically hones in on a particular one of these composites, finding its factors”; Examiner notes that scene vector s corresponds to an input hypervector representing a data structure comprised of a plurality of concepts (color, shape, vertical position, horizontal position), see Figure 3);
unbinding… the input hypervector, into a plurality of unbound hypervectors, wherein the each of the plurality of unbound hypervectors corresponds to a single concept from the plurality of concepts of the data structure (Frady, 3 Factorization via Search in Superposition: “A resonator network combines the strategy of superposition and cleanup memory to efficiently search over the combinatorially large space of possible factorizations. The vectors ˆx, ˆy, and ˆz represent the current estimate for each factor… A particular factor can then be inferred from s based on the estimates for the other two—for example, ˆz(1) = s ⊙ ˆx(0) ⊙ ˆy(0)” and 4.2 Visual Scene Analysis as a Factorization Problem:
PNG
media_image2.png
312
650
media_image2.png
Greyscale
; Examiner notes that (s ⊙ ˆd(t) ⊙ ˆv(t) ⊙ ˆh(t)) represents the unbound hypervector corresponding to color, (s ⊙ ˆc(t) ⊙ ˆv(t) ⊙ ˆh(t)) represents the unbound hypervector corresponding to digit, and so on);
generating… one or more similarity vectors for each of the unbound hypervectors, wherein each of the one or more similarity vectors is based on the similarity of the unbound hypervector and one or more candidate code hypervectors representing each of the plurality of concepts (Frady, 2 VSA Preliminaries: “Dot product (·) is the conventional vector inner product, x · s =
∑
i
x
i
s
i
, which is used to measure the similarity between vectors. This is used to decode the result of a VSA computation by comparing the vector to the set of vectors in the codebook: a =
X
T
s. Here, X is the codebook of atomic vectors, and s is a high-dimensional vector resulting from a VSA computation. The result of a VSA computation can be a single symbol indicated by the largest component of a. Alternatively, the coefficients a can be considered as a weighted sum, where each entry indicates a confidence level, probability, or intensity value” and 3 Factorization via Search in Superposition: “The inference process, however, is noisy if many guesses are tested simultaneously. This noise results from cross talk of many quasi-orthogonal vectors and can be reduced through a clean-up memory. This is built from the codebooks, which contain all the vectors that are possible factors of the input s. Each clean-up memory projects the initial noisy estimate onto the span of the codebook. This computes a measure of confidence for whether each element in the codebook is a factor” and 4.2 Visual Scene Analysis as a Factorization Problem: “Any given scene can have between one and three of these objects, which are allowed to partially occlude one another. We generate symbolic vectors cblue, cgreen, ..., cwhite to encode color; d0, d1, ..., d9 to encode shape; vtop, vmiddle, vbottom to encode vertical position; and hleft, hcenter, hright to encode horizontal position, which are stored in respective codebooks, C, D, V, H…
PNG
media_image2.png
312
650
media_image2.png
Greyscale
; Examiner notes that
C
T
(s ⊙ ˆd(t) ⊙ ˆv(t) ⊙ ˆh(t)) corresponds to a similarity vector for the unbound hypervector corresponding to color (see that this expression is analogous to a =
X
T
s), wherein the similarity vector is based on the similarity of the unbound color hypervector to the candidate code hypervectors in codebook C,
D
T
(s ⊙ ˆc(t) ⊙ ˆv(t) ⊙ ˆh(t)) corresponds to a similarity vector for the unbound hypervector corresponding to digit, wherein the similarity vector is based on the similarity of the unbound digit hypervector to the candidate code hypervectors in codebook D, and so on); and
generating… a plurality of a principal hypervectors, wherein each principal hypervectors is an estimate which represents one concept of the plurality of concepts, based at least in part on the one or more similarity vectors corresponding to the one concept (Frady, 3 Factorization via Search in Superposition: “The result of the inference and clean-up leads to a new estimate for each factor. The new estimate is formed by a sum of dictionary items weighted by the confidence levels. This produces a better guess for each one of the factors” and 4.2 Visual Scene Analysis as a Factorization Problem:
PNG
media_image2.png
312
650
media_image2.png
Greyscale
; Examiner notes that ˆc(
t
+
1
), ˆd(
t
+
1
), ˆv(
t
+
1
), and ˆh(
t
+
1
) correspond to a plurality of principal hypervectors, which are generated based on the similarity vectors).
Frady does not appear to explicitly disclose that the method is “computer-implemented” or that the steps are performed “by the processor”.
However, Renner discloses a computer-implemented method and a processor (Renner, METHODS, Details of hardware implementation: “On Loihi, we implemented a smaller model that solves a 28x28x3 factorization task. We implemented the network on Intel’s neuromorphic research chip Loihi…The Loihi board was interfaced to a system with an Intel Xeon CPU E5-2670 @ 2.60GHz and 128GiB of RAM running Ubuntu 20.04.4 LTS”).
Renner and the instant application both relate to factorizing hypervectors using resonator networks and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the method of Frady with the teachings of Renner such that the method is computer-implemented and the method steps are performed by a processor, and one would have been motivated to do so. Doing so would accelerate computing time (see Renner, Introductions, paragraph 2).
Regarding claim 2, the rejection of claim 1 is incorporated. Frady as modified by Renner further discloses “wherein generating a similarity vector is based on one of the following: a dot product, L1 norm, L2 norm, or L^∞ norm” (Frady, 2 VSA Preliminaries: “Dot product (·) is the conventional vector inner product, x · s =
∑
i
x
i
s
i
, which is used to measure the similarity between vectors. This is used to decode the result of a VSA computation by comparing the vector to the set of vectors in the codebook: a =
X
T
s”).
Regarding claim 5, the rejection of claim 1 is incorporated. Frady as modified by Renner further discloses “separating, by the processor, the similarity vectors, based on a softmax operation or an identity operation” (Renner, Methods, Details of simulation experiments: “Beyond the dynamics described in equation (10), we included some modifications to improve performance…A second modification encouraged sparsity in the solutions by including a non-linearity during the cleanup that encourages sparsity, such as a polynomial exponent or softmax function, i.e.:
PNG
media_image1.png
42
876
media_image1.png
Greyscale
”; Examiner notes that eq(12) depicts function p(x) being applied to the similarity vector, where p(x) may be a softmax function).
Renner and the instant application both relate to factorizing hypervectors using resonator networks and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the method of Frady with the teachings of Renner to include a step of “separating, by the processor, the similarity vectors, based on a softmax operation or an identity operation”, and one would have been motivated to do so. Doing so would improve performance by encouraging sparsity (see Renner, Methods, Details of simulation experiments).
Regarding claim 8, the rejection of claim 1 is incorporated. Frady as modified by Renner further discloses “wherein the computer implemented method to factorize an input hypervector representing a plurality of concepts into individual hypervectors each representing a concept from a the plurality of concepts, through iterative processing of a resonator network performs a plurality of iterations until a convergence criterion is fulfilled” (Frady, 3 Factorization via Search in Superposition: “The result of the inference and clean-up leads to a new estimate for each factor. The new estimate is formed by a sum of dictionary items weighted by the confidence levels. This produces a better guess for each one of the factors. The inference can then be repeated with better guesses, which reduces cross-talk noise even further. By iteratively applying this procedure, the inference and clean-up stages cooperate to successively reduce cross talk noise until the solution is found… The set of equations in (3.2) defines a nonlinear dynamical system that has interesting empirical and theoretical properties, which we thoroughly examine through simulation experiments in Kent et al. (2020), the companion article in this issue. Empirically, the system bounces around in state space until the correct solution appears to resonate with the network dynamics, popping out as if in a moment of insight. We find that while there is no Lyapunov function governing these dynamics and no guarantee for convergence, the resonator network empirically converges to the correct solution with high probability as long as the number of product combinations to be searched is within the network’s operational capacity”).
Regarding claim 11, Frady discloses: “A… [method] to factorize an input hypervector representing a plurality of concepts into individual hypervectors each representing a concept from a the plurality of concepts, through iterative processing of a resonator network (Frady, 4.2 Visual Scene Analysis as a Factorization Problem: “The scene can then be decoded by iterating through the resonator network”) …
receive an input hypervector representing a data structure comprised of a plurality of concepts (Frady, 4.2 Visual Scene Analysis as a Factorization Problem: “Any given scene can have between one and three of these objects, which are allowed to partially occlude one another. We generate symbolic vectors cblue, cgreen, ..., cwhite to encode color; d0, d1, ..., d9 to encode shape; vtop, vmiddle, vbottom to encode vertical position; and hleft, hcenter, hright to encode horizontal position, which are stored in respective codebooks, C, D, V, H… The VSA approach to represent a scene like this is to form the conjunction of each of the four factors with the binding operation and superposing multiple objects together to form a single high-dimensional vector that constitutes a distributed representation of the entire scene. This encoding is depicted in Figure 3, and as in the previous examples, the encoding provides a flexible data structure such that aspects of the scene can be individually queried… Supervised learning via backpropagation is used to train the network to output the VSA representation of the entire scene from the image pixels as input…When we present the scene vector s to a resonator network, it automatically hones in on a particular one of these composites, finding its factors”; Examiner notes that scene vector s corresponds to an input hypervector representing a data structure comprised of a plurality of concepts (color, shape, vertical position, horizontal position), see Figure 3);
unbind by the input hypervector, into a plurality of unbound hypervectors, wherein the each of the plurality of unbound hypervectors corresponds to a single concept from the plurality of concepts of the data structure (Frady, 3 Factorization via Search in Superposition: “A resonator network combines the strategy of superposition and cleanup memory to efficiently search over the combinatorially large space of possible factorizations. The vectors ˆx, ˆy, and ˆz represent the current estimate for each factor… A particular factor can then be inferred from s based on the estimates for the other two—for example, ˆz(1) = s ⊙ ˆx(0) ⊙ ˆy(0)” and 4.2 Visual Scene Analysis as a Factorization Problem:
PNG
media_image2.png
312
650
media_image2.png
Greyscale
; Examiner notes that (s ⊙ ˆd(t) ⊙ ˆv(t) ⊙ ˆh(t)) represents the unbound hypervector corresponding to color, (s ⊙ ˆc(t) ⊙ ˆv(t) ⊙ ˆh(t)) represents the unbound hypervector corresponding to digit, and so on);
generate one or more similarity vectors for each of the unbound hypervectors, wherein each of the one or more similarity vectors is based on the similarity of the unbound hypervector and one or more candidate code hypervectors representing each of the plurality of concepts (Frady, 2 VSA Preliminaries: “Dot product (·) is the conventional vector inner product, x · s =
∑
i
x
i
s
i
, which is used to measure the similarity between vectors. This is used to decode the result of a VSA computation by comparing the vector to the set of vectors in the codebook: a =
X
T
s. Here, X is the codebook of atomic vectors, and s is a high-dimensional vector resulting from a VSA computation. The result of a VSA computation can be a single symbol indicated by the largest component of a. Alternatively, the coefficients a can be considered as a weighted sum, where each entry indicates a confidence level, probability, or intensity value” and 3 Factorization via Search in Superposition: “The inference process, however, is noisy if many guesses are tested simultaneously. This noise results from cross talk of many quasi-orthogonal vectors and can be reduced through a clean-up memory. This is built from the codebooks, which contain all the vectors that are possible factors of the input s. Each clean-up memory projects the initial noisy estimate onto the span of the codebook. This computes a measure of confidence for whether each element in the codebook is a factor” and 4.2 Visual Scene Analysis as a Factorization Problem: “Any given scene can have between one and three of these objects, which are allowed to partially occlude one another. We generate symbolic vectors cblue, cgreen, ..., cwhite to encode color; d0, d1, ..., d9 to encode shape; vtop, vmiddle, vbottom to encode vertical position; and hleft, hcenter, hright to encode horizontal position, which are stored in respective codebooks, C, D, V, H…
PNG
media_image2.png
312
650
media_image2.png
Greyscale
; Examiner notes that
C
T
(s ⊙ ˆd(t) ⊙ ˆv(t) ⊙ ˆh(t)) corresponds to a similarity vector for the unbound hypervector corresponding to color (see that this expression is analogous to a =
X
T
s), wherein the similarity vector is based on the similarity of the unbound color hypervector to the candidate code hypervectors in codebook C,
D
T
(s ⊙ ˆc(t) ⊙ ˆv(t) ⊙ ˆh(t)) corresponds to a similarity vector for the unbound hypervector corresponding to digit, wherein the similarity vector is based on the similarity of the unbound digit hypervector to the candidate code hypervectors in codebook D, and so on); and
generate a plurality of a principal hypervectors, wherein each principal hypervectors is an estimate which represents one concept of the plurality of concepts, based at least in part on the one or more similarity vectors corresponding to the one concept (Frady, 3 Factorization via Search in Superposition: “The result of the inference and clean-up leads to a new estimate for each factor. The new estimate is formed by a sum of dictionary items weighted by the confidence levels. This produces a better guess for each one of the factors” and 4.2 Visual Scene Analysis as a Factorization Problem:
PNG
media_image2.png
312
650
media_image2.png
Greyscale
; Examiner notes that ˆc(
t
+
1
), ˆd(
t
+
1
), ˆv(
t
+
1
), and ˆh(
t
+
1
) correspond to a plurality of principal hypervectors, which are generated based on the similarity vectors).
Frady does not appear to explicitly disclose “A computer system… the computer system comprising: one or more processors; one or more computer readable storage devices; program instructions stored on the one or more computer readable storage devices for execution by at least one of the one or more computer processors”.
However, Renner discloses “A computer system… the computer system comprising: one or more processors; one or more computer readable storage devices; program instructions stored on the one or more computer readable storage devices for execution by at least one of the one or more computer processors” (Renner, METHODS, Details for simulation experiments: “Simulation experiments using the resonator network were implemented in Python using Numpy and PyTorch” and Details of hardware implementation: “On Loihi, we implemented a smaller model that solves a 28x28x3 factorization task. We implemented the network on Intel’s neuromorphic research chip Loihi…The Loihi board was interfaced to a system with an Intel Xeon CPU E5-2670 @ 2.60GHz and 128GiB of RAM running Ubuntu 20.04.4 LTS”).
Renner and the instant application both relate to factorizing hypervectors using resonator networks and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the method of Frady with the teachings of Renner such that the method is implemented by a computer system comprising one or more computer processors, one or more computer readable storage devices, and program instructions comprising the method steps, and one would have been motivated to do so. Doing so would accelerate computing time (see Renner, Introductions, paragraph 2).
Regarding claim 12, the rejection of claim 11 is incorporated. Claim 12 is a computer system claim corresponding to method claim 2, and the rejection of claim 12 follows the same rationale as the rejection of claim 2 above.
Regarding claim 15, the rejection of claim 11 is incorporated. Claim 15 is a computer system claim corresponding to method claim 5, and the rejection of claim 15 follows the same rationale as the rejection of claim 5 above.
Regarding claim 18, the rejection of claim 11 is incorporated. Claim 18 is a computer system claim corresponding to method claim 8, and the rejection of claim 18 follows the same rationale as the rejection of claim 8 above.
Regarding claim 20, Frady discloses: “A… [method] to factorize an input hypervector representing a plurality of concepts into individual hypervectors each representing a concept from a the plurality of concepts, through iterative processing of a resonator network (Frady, 4.2 Visual Scene Analysis as a Factorization Problem: “The scene can then be decoded by iterating through the resonator network”) …
receive an input hypervector representing a data structure comprised of a plurality of concepts (Frady, 4.2 Visual Scene Analysis as a Factorization Problem: “Any given scene can have between one and three of these objects, which are allowed to partially occlude one another. We generate symbolic vectors cblue, cgreen, ..., cwhite to encode color; d0, d1, ..., d9 to encode shape; vtop, vmiddle, vbottom to encode vertical position; and hleft, hcenter, hright to encode horizontal position, which are stored in respective codebooks, C, D, V, H… The VSA approach to represent a scene like this is to form the conjunction of each of the four factors with the binding operation and superposing multiple objects together to form a single high-dimensional vector that constitutes a distributed representation of the entire scene. This encoding is depicted in Figure 3, and as in the previous examples, the encoding provides a flexible data structure such that aspects of the scene can be individually queried… Supervised learning via backpropagation is used to train the network to output the VSA representation of the entire scene from the image pixels as input…When we present the scene vector s to a resonator network, it automatically hones in on a particular one of these composites, finding its factors”; Examiner notes that scene vector s corresponds to an input hypervector representing a data structure comprised of a plurality of concepts (color, shape, vertical position, horizontal position), see Figure 3);
unbind by the input hypervector, into a plurality of unbound hypervectors, wherein the each of the plurality of unbound hypervectors corresponds to a single concept from the plurality of concepts of the data structure (Frady, 3 Factorization via Search in Superposition: “A resonator network combines the strategy of superposition and cleanup memory to efficiently search over the combinatorially large space of possible factorizations. The vectors ˆx, ˆy, and ˆz represent the current estimate for each factor… A particular factor can then be inferred from s based on the estimates for the other two—for example, ˆz(1) = s ⊙ ˆx(0) ⊙ ˆy(0)” and 4.2 Visual Scene Analysis as a Factorization Problem:
PNG
media_image2.png
312
650
media_image2.png
Greyscale
; Examiner notes that (s ⊙ ˆd(t) ⊙ ˆv(t) ⊙ ˆh(t)) represents the unbound hypervector corresponding to color, (s ⊙ ˆc(t) ⊙ ˆv(t) ⊙ ˆh(t)) represents the unbound hypervector corresponding to digit, and so on);
generate one or more similarity vectors for each of the unbound hypervectors, wherein each of the one or more similarity vectors is based on the similarity of the unbound hypervector and one or more candidate code hypervectors representing each of the plurality of concepts (Frady, 2 VSA Preliminaries: “Dot product (·) is the conventional vector inner product, x · s =
∑
i
x
i
s
i
, which is used to measure the similarity between vectors. This is used to decode the result of a VSA computation by comparing the vector to the set of vectors in the codebook: a =
X
T
s. Here, X is the codebook of atomic vectors, and s is a high-dimensional vector resulting from a VSA computation. The result of a VSA computation can be a single symbol indicated by the largest component of a. Alternatively, the coefficients a can be considered as a weighted sum, where each entry indicates a confidence level, probability, or intensity value” and 3 Factorization via Search in Superposition: “The inference process, however, is noisy if many guesses are tested simultaneously. This noise results from cross talk of many quasi-orthogonal vectors and can be reduced through a clean-up memory. This is built from the codebooks, which contain all the vectors that are possible factors of the input s. Each clean-up memory projects the initial noisy estimate onto the span of the codebook. This computes a measure of confidence for whether each element in the codebook is a factor” and 4.2 Visual Scene Analysis as a Factorization Problem: “Any given scene can have between one and three of these objects, which are allowed to partially occlude one another. We generate symbolic vectors cblue, cgreen, ..., cwhite to encode color; d0, d1, ..., d9 to encode shape; vtop, vmiddle, vbottom to encode vertical position; and hleft, hcenter, hright to encode horizontal position, which are stored in respective codebooks, C, D, V, H…
PNG
media_image2.png
312
650
media_image2.png
Greyscale
; Examiner notes that
C
T
(s ⊙ ˆd(t) ⊙ ˆv(t) ⊙ ˆh(t)) corresponds to a similarity vector for the unbound hypervector corresponding to color (see that this expression is analogous to a =
X
T
s), wherein the similarity vector is based on the similarity of the unbound color hypervector to the candidate code hypervectors in codebook C,
D
T
(s ⊙ ˆc(t) ⊙ ˆv(t) ⊙ ˆh(t)) corresponds to a similarity vector for the unbound hypervector corresponding to digit, wherein the similarity vector is based on the similarity of the unbound digit hypervector to the candidate code hypervectors in codebook D, and so on); and
generate a plurality of a principal hypervectors, wherein each principal hypervectors is an estimate which represents one concept of the plurality of concepts, based at least in part on the one or more similarity vectors corresponding to the one concept (Frady, 3 Factorization via Search in Superposition: “The result of the inference and clean-up leads to a new estimate for each factor. The new estimate is formed by a sum of dictionary items weighted by the confidence levels. This produces a better guess for each one of the factors” and 4.2 Visual Scene Analysis as a Factorization Problem:
PNG
media_image2.png
312
650
media_image2.png
Greyscale
; Examiner notes that ˆc(
t
+
1
), ˆd(
t
+
1
), ˆv(
t
+
1
), and ˆh(
t
+
1
) correspond to a plurality of principal hypervectors, which are generated based on the similarity vectors).
Frady does not appear to explicitly disclose “A computer program product… the computer program product comprising: one or more computer readable storage devices; program instructions stored on the one or more computer readable storage devices, wherein the program instructions are executable by a computer processor to perform one or more operations”.
However, Renner discloses “A computer program product… the computer program product comprising: one or more computer readable storage devices; program instructions stored on the one or more computer readable storage devices, wherein the program instructions are executable by a computer processor to perform one or more operations” (Renner, METHODS, Details for simulation experiments: “Simulation experiments using the resonator network were implemented in Python using Numpy and PyTorch” and Details of hardware implementation: “On Loihi, we implemented a smaller model that solves a 28x28x3 factorization task. We implemented the network on Intel’s neuromorphic research chip Loihi…The Loihi board was interfaced to a system with an Intel Xeon CPU E5-2670 @ 2.60GHz and 128GiB of RAM running Ubuntu 20.04.4 LTS”).
Renner and the instant application both relate to factorizing hypervectors using resonator networks and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the method of Frady with the teachings of Renner such that the method is implemented by a computer program product comprising one or more computer readable storage devices and program instructions stored on the one or more computer readable storage devices, and one would have been motivated to do so. Doing so would accelerate computing time (see Renner, Introductions, paragraph 2).
Claims 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Frady in view of Renner, and further in view of Snaider et al. (Modular Composite Representation) (hereinafter “Snaider”).
Regarding claim 3, the rejection of claim 1 is incorporated. Neither Frady nor Renner appear to explicitly disclose the further limitations of the claim.
However, Snaider discloses “wherein unbinding the input hypervector is based on a circular convolution or an addition and modulo operation” (Snaider, pages 517-518, Basic Operations of MCR: “The binding (or multiplication) of modular integer vectors is defined as the modular sum in each dimension… The unbinding operation is simply the modular subtraction in each dimension, or the modular sum of the first operand with the complement of the second operand in each dimension”).
Snaider and the instant application both relate to high-dimensional vector representations and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention to have modified the combination of Frady/Renner with the teachings of Snaider such that the unbinding of the input hypervector is based on an addition and modulo operation, and one would have been motivated to do so. Doing so would preserve the distance between vectors, ensuring that the distance between vectors correlates with the distance between the concepts they represent (see Snaider, page 516, paragraph 5 and page 518, paragraph 5).
Regarding claim 13, the rejection of claim 11 is incorporated. Claim 13 is a computer system claim corresponding to method claim 3, and the rejection of claim 13 follows the same rationale as the rejection of claim 3 above.
Claims 4 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Frady in view of Renner, and further in view of Wan et al. (A compute-in-memory chip based on resistive random-access memory) (hereinafter “Wan”).
Regarding claim 4, the rejection of claim 1 is incorporated. Neither Frady nor Renner appear to explicitly disclose the further limitations of the claim.
However, Wan discloses “adding… noise to… [model weights] wherein the noise is gaussian noise or uniform noise” (Wan, Methods, Noise-resilient neural-network training: “During noise-resilient neural-network training, we inject noise into weights of all fully connected and convolutional layers during the forwards pass of neural-network training to emulate the effects of RRAM conductance relaxation and read noises. The distribution of the injected noise is obtained by RRAM characterization. We used the iterative write–verify technique to program RRAM cells into different initial conductance states and measure their conductance relaxation after 30 min. Extended Data Fig. 3d shows that measured conductance relaxation has an absolute value of mean at all conductance states. The highest standard deviation is 3.87 μS, about 10% of the gmax 40 μS, found at about 12 μS initial conductance state. Therefore, to simulate such conductance relaxation behaviour during inference, we inject a Gaussian noise with a zero mean and a standard deviation equal to 10% of the maximum weights of a layer”).
Wan and the instant application both relate to neural networks and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the combination of Frady/Renner with the teachings of Wan to include a step of “adding, by the processor, noise to the similarity vector of the hypervector, wherein the noise is gaussian noise or uniform noise” and one would have been motivated to do so. Doing so would allow for emulating the effects of noise on in-memory matrix vector multiplication (see Wan, Methods, Noise-resilient neural-network training).
Regarding claim 14, the rejection of claim 11 is incorporated. Claim 14 is a computer system claim corresponding to method claim 4, and the rejection of claim 14 follows the same rationale as the rejection of claim 4 above.
Claims 6, 7, 16, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Frady in view of Renner, and further in view of Zhao et al. (Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection) (hereinafter “Zhao”).
Regarding claim 6, the rejection of claim 1 is incorporated. Neither Frady nor Renner appear to explicitly disclose the further limitations of the claim.
However, Zhao discloses “sparsifying… one or more elements of… [an attention matrix], wherein sparsifying is based on one or the following: a pre-determined threshold, a dynamic threshold, a Top-A operation, or an absolute value larger than a mean of all” (Zhao, 2 Explicit Sparse Transformer: “Explicit Sparse Transformer is still based on the Transformer framework. The difference is in the implementation of self-attention. The attention is degenerated to the sparse attention through top-k selection. In this way, the most contributive components for attention are reserved and the other irrelevant information are removed. This selective method is effective in preserving important information and removing noise… Then the model evaluates the values of the scores P based on the hypothesis that scores with larger values demonstrate higher relevance. The sparse attention masking operation M(·) is implemented upon P in order to select the top-k contributive elements”).
Zhao and the instant application both relate to neural networks and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the combination of Frady/Renner with the teachings of Zhao to include “sparsifying, by the processor, one or more elements of the similarity vectors, wherein sparsifying is based on one or the following: a pre-determined threshold, a dynamic threshold, a Top-A operation, or an absolute value larger than a mean of all”, and one would have been motivated to do so. Doing so would allow for preserving important information while removing noise/irrelevant information (see Zhao, 2 Explicit Sparse Transformer), which in turn would increase computational efficiency by reducing the amount of downstream computations required.
Regarding claim 7, the rejection of claim 6 is incorporated. Frady as modified by Zhao further discloses “wherein generating the plurality of principal hypervectors comprises:
combining, by the processor, through a linear combination each of the candidate code hypervectors with weights to generate a plurality of bundled weights, based on the sparsified similarity vector corresponding to the candidate code hypervector (Frady, 3 Factorization via Search in Superposition: “The result of the inference and clean-up leads to a new estimate for each factor. The new estimate is formed by a sum of dictionary items weighted by the confidence levels. This produces a better guess for each one of the factors”); and
applying, by the processor, the plurality of bundled weights to a… function” (Frady, 3 Factorization via Search in Superposition: “The procedure described above, for all three factors, is specified by the following set of equations (see Figure 1):
PNG
media_image3.png
132
650
media_image3.png
Greyscale
where the function g prevents runaway positive feedback by thresholding the elements of each vector to ±1”).
Neither Frady nor Zhao appear to explicitly disclose that the function is a “selection” function.
However, Renner discloses “applying, by the processor, the plurality of bundled weights to a selection function” (Renner, METHODS, Details of performance benchmarking: “To benchmark the different model variants, we report an accuracy measure based on the percentage of correct classifications of the letter identity. We chose this measure as it is straightforward to calculate and compare and because getting the letter identity correct usually also means that the other factors are estimated well (the opposite is not true). The letter output of the resonator is the argmax of the state readout (product of the letter factor codebook and the factor state)”; Examiner notes that “argmax” corresponds to a selection function).
Renner and the instant application both relate to factorizing hypervectors using resonator networks and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the combination of Frady/Zhao with the teachings of Renner such that the function is a “selection” function, and one would have been motivated to do so. Doing so would allow for determining an accuracy measure of the resonator network in estimating the correct factors (see Renner, METHODS, Details of performance benchmarking).
Regarding claim 16, the rejection of claim 11 is incorporated. Claim 16 is a computer system claim corresponding to method claim 6, and the rejection of claim 16 follows the same rationale as the rejection of claim 6 above.
Regarding claim 17, the rejection of claim 16 is incorporated. Claim 17 is a computer system claim corresponding to method claim 7, and the rejection of claim 17 follows the same rationale as the rejection of claim 7 above.
Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Frady in view Renner, and further in view of Langenegger et al. (In-memory factorization of holographic perceptual representations) (hereinafter “Langenegger”).
Regarding claim 9, the rejection of claim 8 is incorporated. Neither Frady nor Renner appear to explicitly disclose the further limitations of the claim.
However, Langenegger discloses “wherein the convergence criteria is when a value of at least one element of each of the plurality of similarity scores exceeds a threshold” (Langengger, III. DISCUSSION, Detection of convergence in the in-memory factorizer: “To detect this convergence, we define a novel early convergence detection algorithm. The in-memory factorizer is said to be converged if a single similarity value across all the factors surpasses a convergence detection threshold”).
Langenegger and the instant application both relate to factorizing vectors and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the combination of Frady/Renner with the teachings of Langenegger such that the convergence criteria is when a value of at least one element of each of the plurality of similarity scores exceeds a threshold, and one would have been motivated to do so. Doing so would counteract parasitic codevectors and decrease resource usage by eliminating the need to store the history of prior estimates to detect convergence (see Langengger, SUPPLEMENTARY NOTES, III. DISCUSSION, Detection of convergence in the in-memory factorizer).
Regarding claim 19, the rejection of claim 18 is incorporated. Claim 19 is a computer system claim corresponding to method claim 9, and the rejection of claim 19 follows the same rationale as the rejection of claim 9 above.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Frady in view Renner, and further in view of Kleyko et al. (Integer Factorization with Compositional Distributed Representations) (hereinafter “Kleyko”).
Regarding claim 10, the rejection of claim 8 is incorporated. Neither Frady nor Renner appear to explicitly disclose the further limitations of the claim.
However, Kleyko discloses “wherein the convergence criteria is when a predefined number of iterations” (Kleyko, 3.3 Factorization of semiprimes with the resonator network: “Once the resonator network converges or reaches the maximum number of iterations, the most recent estimates of the resonator network are used to obtain the predictions ˆx and ˆy… In the experiments, the maximum number of iterations was 100”).
Kleyko and the instant application both relate to factoring hypervectors with resonator networks and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the combination of Frady/Renner with the teachings of Kleyko such that the convergence criteria is when a predefined number of iterations has been reached, and one would have been motivated to do so. Doing so would improve computational efficiency by preventing the network from searching indefinitely for a solution.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GWYNEVERE A DETERDING whose telephone number is (571)272-7657. The examiner can normally be reached Mon-Fri. 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/G.A.D./Examiner, Art Unit 2125
/KAMRAN AFSHAR/Supervisory Patent Examiner, Art Unit 2125
Prosecution Insights