DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This office action is in response to the amendment filed on 05/10/2026. Claims 1-2, 4-7, and 9-10 remain pending in the application. Claims 1 and 6 are independent.
Specification
Applicant's amendment to Abstract and Specification corrects previous objections; therefore, the previous objections are withdrawn.
Claim Objections
Applicant's amendment to claims corrects previous objections; therefore, the previous objections are withdrawn.
Claim Rejections - 35 USC § 101
Applicant's amendment to claims corrects previous rejections; therefore, previous rejections are withdrawn.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4, 6-7, and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Qi (US 2022/0114413 A1, pub. date: 04/14/2022), hereinafter Qi in view of Kleyko et al. ("Density Encoding Enables Resource-Efficient Randomly Connected Neural Networks", IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 32, NO. 8, Aug. 24, 2020, pp. 3777-3783), hereinafter Kleyko and IDELSON (US 2023/0090262 A1, filed on 06/24/2021), hereinafter IDELSON.
Independent Claims 1 and 6
Qi discloses an encoding method (Qi, ¶¶ [0002]-[0003]: to achieve real-time and low power in deep convolution neural network (DCNN) inference, the models are compressed in terms of memory size and the number of computation operation when deployed in limited-resource devices; most current mechanisms require a non-trivial normalization stages before and after the convolution operation; the hardware cost of the normalization stage exceeds that of the convolution operation; a possible solution is presented for quantization implementation that utilizes only an extra shifter for the normalization, thus significantly reducing the hardware cost; the proposed solution transforms the multiplication-based normalization into a shift-only one by limiting the quantization in a power-of-two scale; ¶ [0024]: allow a convolution to be determined utilizing minimal hardware resources while retaining an inference accuracy comparable to a corresponding floating point model; specifically, a proposed hardware implementation for the fused convolution layer operation utilizing only a shifter in the normalization stage is proposed; the proposed method and system may support multiple different quantization configurations such as symmetric or asymmetric, layer-wise or channel-wise, and multiple activation functions such as rectified linear unit (ReLU), parametric rectified linear unit (pReLU), filtered rectified linear unit (ReLUx), and hard-sigmoid; the proposed solution may provide zero-numerical-error hardware deployment by utilizing a quantization library; the library may directly quantize individual tensors in a computational graph from public deep learning (DL) frameworks without modifying the kernel implementation of operations from the original framework), comprising:
performing linear conversion on an input into a first vector based on a weight by a convolution layer (Qi, ¶ [0005]: quantizing an input tensor into a first power-of-two value, quantizing a weight tensor into a second power-of-two value, performing a convolution based on the quantized input tensor and the quantized weight tensor, quantizing a bias tensor into a third power-of-two value, bias-adding an output of the convolution and the quantized bias tensor and outputting a bias-addition; ¶¶ [0025]-[0042]: A typical 2D convolution layer in the convolutional neural network (CNN) may be described in following equations: P = W*Ai-1 + b (1), Ai = σ(P) (2), where Ai is the output of ith layer in shape of Ci × Hi × Wi, W is the weight matrix of the layer in shape of Ci × C i-1 × KH × Kw, b is the bias vector of the layer in shape of Ci × 1, P is the intermediate multiply-accumulation result of the layer in shape of Ci × Hi × Wi, σ is the non-linear activation function, Ci is the number of channel of ith layer, Hi, Wi, is the height and width of the feature map of ith layer, * denotes the 2D convolution operation; to limit the data precision for computation efficiency, an affine mapping is commonly used to linearly quantize the floating-point value to a finite bit-width integer representation;
v
~
=
s
v
-
-
z
-
(3), where
v
~
is the dequantized value in floating point that approximate the real value v,
v
-
is the integer representation of the quantized value,
s
,
z
-
are the quantization parameters where s is the normalization factor of v in floating-point format and
z
-
the zero point of
v
-
in integer format, these tuples may be computed offline; convolution layer operations are given by Equations (1) and (2), with quantized variables and illustrate integer-based implementations; the offline-computed parameters may be in a lower-bit format to reduce model and memory footprint, the heavy computation kernel (convolution) may also be lower-bit integer-based for low-cost hardware implementation and the offline-computed parameters may be fused when possible while minimizing the induced bias in the fusion; plugging equation (3) into equation (1) with respective quantization parameters (s, z), the equivalent integer-based 2D convolution operation is given by equation (4); separating the integer operations and floating-point operations in equation (5); the intermediate result of the integer-based multiply-accumulation operation is designated as
P
-
in equations (6)-(7); ¶¶ [0059]-[0064] with 310 and 314 in FIG. 3: a multiply-accumulate layer 310 outputs an intermediate result of the multiply-accumulate
P
-
314 as an integer variable; asymmetric quantization is supported in the proposed data path when using ReLU activation function; Weight quantization: since the number of MACs operations may be double, it is recommended to avoid quantizing weight asymmetrically; Activation functions: when using symmetric quantization or partial asymmetric quantization (asymmetric weight and symmetric activation), several activation functions are feasible: ReLU (a=0, d=0), ReLU6 (x=6×
2
n
A
n
w
), leakyReLU (a≈0.125, d=0), pReLU (p≈2n), hard-sigmoid (with slope 2-n); ¶¶ [0065]-[0066] with 410-424 and 432-438 FIG. 4: the software quantization library may compute the parameters by inserting quasi-quantization nodes into the original floating-point computational graph in the DL framework without replacing the original math-related operation kernels (2D convolutional layer (Conv2D), bias addition (BiasAdd), Non-linear), as shown in FIG. 4; an example fused convolutional method includes (a) quantizing 414 an input tensor 410 Ai-1 434 into a first power-of-two value; (b) quantizing 416 a weight tensor 412 W 432 into a second power-of-two value and performing a convolution 418 based on the quantized input tensor and the quantized weight tensor; (c) quantizing 422 a bias tensor 420 b 436 into a third power-of-two value, bias-adding 424 an output of the convolution and the quantized bias tensor and outputting a bias-addition
P
~
438; ¶ [0068] with 510-518 in FIG. 5: quantizing 510 an input tensor into a first power-of-two value, quantizing 512 a weight tensor into a second power-of-two value and performing 514 a convolution based on the quantized input tensor and the quantized weight tensor; quantizing 516 a bias tensor into a third power-of two value, bias-adding 518 an output of the convolution and the quantized bias tensor and outputting a bias-addition);
comparing the first vector generated from the convolution layer with a reference value to generate a second vector by an activation function; binding the second vector generated by the activation function (Qi, ¶¶ [0043]-[0059] and [0004] with 312-332 and in FIG. 3: considering the final non-linear operation in equation (2) and the output expressed in quantized integer-format to cascade to the next layer as equation (8); most non-linear activation functions may be relaxed or approximated by a Parametric Rectifier Linear Unit (PReLU) with upper bound clamping as equation (9); plugging equation (9) into equation (8), and rounding the remaining floating-point parts to integer or converting them to fixed-point representation yields equations (10)-(11); the operation is formulated as in equation (6) and (10); by constraining the floating-point scaling factor s and a to power-of-two values as in equation (12); the normalization stage may be simplified as equations (13)-(14); under either one of the following conditions, the addition operation in the normalization stage may be avoided;
z
A
i
= 0 indicates that activation may be symmetrically quantized; ReLU (a=0) is used as the activation function a(x); in this case,
z
A
i
may be fused in the bias-stage as equations (15)-(17); the data path is illustrated in FIG. 3; accumulator precision, to simulate the precision of the mantissa of float 32, a 24-bit integer may be used for the intermediate accumulation
P
-
; as long as the accumulation operation does not overflow, the result may be bit-exact compared to software quantization, which implies that the weights and input activation data may be properly normalized, which is satisfied when using a batch normalization layer in the model; the precision of fused bias
b
~
as the bias term may be added into the accumulator, a reasonable maximum precision may be designated as 24-bit; in equation (15), the first two terms are in the same scale as the accumulation result
P
-
; no overflow may occur if the normalization requirement is satisfied; the third term indicates a conversion of the original bias vector b to fixed-point and normalization on the scale of the accumulation result
P
-
; when quantizing the original bias vector b in the software library, no affine mapping is needed; the fixed-point bit-precision and Q factor may be designed to maintain bit-exactness between software quasi-quantization and a hardware quantization implementation; the last term may be completely fused without introducing numerical error, as it remains the lower-bit precision of
z
A
i
; the precision of the fused zero-point
z
~
may impact the subsequent non-linear operations, for the entries of
P
-
close to
z
~
; the induced bias may be amplified in the output activation and may be propagated in the subsequent layers; the intermediate multiply-accumulate is routed to a comparator 318 that also receives a first zero point z 316; a first multiplexer 320 is coupled to the comparator 318, the first multiplexer 320 receives multiple of power-of-two exponent values ni 322; a shift normalizer 324 which normalizes based on shift-only is coupled to the first multiplexer 320; the shift normalizer 324 receives the multiply-accumulation result 314 and the multiple power-of-two exponent values 322; the shift normalizer 324 limits a quantization of the multiply-accumulation result 314 to a power-of-two scale; a second multiplexer 326 is coupled to an output of the shift normalizer 324, the first multiplexer 320 and receives a second zero point
z
A
i
328 and outputs an activation
A
i
-
332; the multiply-accumulation result 314 may be an integer variable; the first zero point 316 may be fused or an approximated integer variable; the activation 332 may be an integer variable; ¶ [0063]: Activation functions: when using symmetric quantization or partial asymmetric quantization (asymmetric weight and symmetric activation), several activation functions are feasible: ReLU (a=0, d=0), ReLU6 (x=6×
2
n
A
n
w
), leakyReLU (a≈0.125, d=0), pReLU (p≈2n), hard-sigmoid (with slope 2-n); ¶¶ [0065]-[0067] with 426-430 and 439-440 in FIG. 4: non-linearizes 426 the output of the bias-addition and quantizes 428 the output of the non-linearization into an activation Ai taking a form of a fourth power-of-two value output tensor 430; ¶ [0068] with 520-522 in FIG. 5: non-linearizing 520 the output of the bias-addition and quantizing 522 the output of the non-linearization into an activation taking a form of a fourth power-of-two value output tensor), wherein, when the input is a 32-bit floating point input, the first vector is a floating point vector, wherein the second vector and the output vector are both finite bit-width integer vectors, wherein the output vector is stored in a memory device (Qi, ¶¶ [0025]-[0058]: A typical 2D convolution layer in the convolutional neural network (CNN) may be described in following equations: P = W*Ai-1 + b (1), Ai = σ(P) (2), where Ai is the output of ith layer in shape of Ci × Hi × Wi, W is the weight matrix of the layer in shape of Ci × C i-1 × KH × Kw, b is the bias vector of the layer in shape of Ci × 1, P is the intermediate multiply-accumulation result of the layer in shape of Ci × Hi × Wi, σ is the non-linear activation function, Ci is the number of channel of ith layer, Hi, Wi, is the height and width of the feature map of ith layer, * denotes the 2D convolution operation; to limit the data precision for computation efficiency, an affine mapping is commonly used to linearly quantize the floating-point value to a finite bit-width integer representation;
v
~
=
s
v
-
-
z
-
(3), where
v
~
is the dequantized value in floating point that approximate the real value v,
v
-
is the integer representation of the quantized value,
s
,
z
-
are the quantization parameters where s is the normalization factor of v in floating-point format and
z
-
the zero point of
v
-
in integer format, these tuples may be computed offline; convolution layer operations are given by Equations (1) and (2), with quantized variables and illustrate integer-based implementations; the offline-computed parameters may be in a lower-bit format to reduce model and memory footprint, the heavy computation kernel (convolution) may also be lower-bit integer-based for low-cost hardware implementation and the offline-computed parameters may be fused when possible while minimizing the induced bias in the fusion; plugging equation (3) into equation (1) with respective quantization parameters (s, z), the equivalent integer-based 2D convolution operation is given by equation (4); separating the integer operations and floating-point operations in equation (5); the intermediate result of the integer-based multiply-accumulation operation is designated as
P
-
in equations (6)-(7); considering the final non-linear operation in equation (2) and the output expressed in quantized integer-format to cascade to the next layer as shown in equation (8), wherein the input to the non-linear activation function σ is floating number
P
~
obtained after applying 2D convolution layer shown in equation (1); most non-linear activation functions may be relaxed or approximated by a Parametric Rectifier Linear Unit (PReLU) with upper bound clamping as shown in equation (9); plugging equation (11) into equation (10), and rounding the remaining floating-point parts to integer or converting them to fixed-point representation yields equations (10)-(11); accumulator precision, to simulate the precision of the mantissa of float 32, a 24-bit integer may be used for the intermediate accumulation P; it is well-known in the art that a 32-bit floating number is represents by the mantissa part using a 24-bit integer (including 1-bit sign and 23-bit integer) and the exponent part using a 8-bit integer1; the precision of the fused zero-point
z
~
may impact the subsequent non-linear operations, for the entries of
P
-
close to
z
~
; the induced bias may be amplified in the output activation and may be propagated in the subsequent layers; ¶ [0067] with FIG. 2: quantizing 422 a bias tensor 420 b 436 into a third power-of-two value, bias-adding 424 an output of the convolution and the quantized bias tensor and outputting a bias-addition
P
~
438 (i.e., floating number); the method non-linearizes 426 the output of the bias-addition and quantizes 428 the output of the non-linearization into an activation Ai taking a form of a fourth power-of-two value output tensor 430).
Qi further discloses an encoding circuit coupled to a memory device, the encoding circuit comprising: a convolution layer circuit coupled to the memory device; an activation circuit coupled to the convolution layer circuit; a binding circuit coupled to the activation circuit; an adding circuit coupled to the binding circuit; and a Signum function and normalization circuit coupled to the adding circuit, wherein the output vector is written into the memory device (Qi, ¶¶ [0015]-[0023] with FIGS. 1-2: implement neural nets associated with the operation of one or more portions or steps of process 500; the processors associated with the hybrid system comprise a field programmable gate army (FPGA) 122, a graphical processor unit (GPU) 120 and a central processing unit (CPU) 118, which have the capability of providing a neural net; an FPGA is a field programmable device, it has the ability to be reconfigured and perform in hardwired circuit fashion any function that may be programmed into a CPU or GPU; there are other types of processors that the system may encompass such as an accelerated processing unit (APUs) which comprise a CPU with GPU elements on chip and digital signal processors (DSPs) which are designed for performing high speed numerical data processing. Application specific integrated circuits (ASICs) may also perform the hardwired functions of an FPGA; the graphical processor unit 120, central processing unit 118 and field programmable gate arrays 122 are connected and are connected to a memory interface controller 112; the FPGA is connected to the memory interface through a programmable logic circuit to memory interconnect 130; this additional device is utilized due to the fact that the FPGA is operating with a very large bandwidth and to minimize the circuitry utilized from the FPGA to perform memory tasks. The memory and interface controller 112 is additionally connected to persistent memory disk 110, system memory 114 and read only memory (ROM) 116; the processors associated with the hybrid system comprise a field programmable gate array (FPGA) 210 and a central processing unit (CPU) 220; the FPGA is electrically connected to an FPGA controller 212 which interfaces with a direct memory access (DMA) 218. The DMA is connected to input buffer 214 and output buffer 216, which are coupled to the FPGA to buffer data into and out of the FPGA respectively; the DMA 218 includes of two first in first out (FIFO) buffers one for the host CPU and the other for the FPGA, the DMA allows data to be written to and read from the appropriate buffer; on the CPU side of the DMA are a main switch228 which shuttles data and commands to the DMA; the DMA is also connected to an SD RAM controller 224 which allows data to be shuttled to and from the FPGA to the CPU 220, the SDRAM controller is also connected to external SDRAM 226 and the CPU 220; the main switch 228 is connected to the peripherals interface 230. A flash controller 222 controls persistent memory and is connected to the CPU 220; ¶ [0059] with FIG. 3: a fused convolutional layer 300, having a shift-only normalizer; a multiply-accumulate layer 310 outputs an intermediate result of the multiply-accumulate 314 as an integer variable; the intermediate multiply-accumulate is routed to a comparator 318 that also receives a first zero point 316; a first multiplexer 320 is coupled to the comparator 318, the first multiplexer 320 receives multiple of power-of-two exponent values, 322; a shift normalizer 324 which normalizes based on shift-only is coupled to the first multiplexer 320; the shift normalizer 324 receives the multiply-accumulation result 314 and the multiple power-of-two exponent values 322; the shift normalizer 324 limits a quantization of the multiply-accumulation result 314 to a power-of-two scale; a second multiplexer 326 is coupled to an output of the shift normalizer 324, the first multiplexer 320 and receives a second zero point 328 and outputs an activation 332).
Qi fails to explicitly discloses (1) binding the second vector generated by the activation function with a random vector to generate a plurality of binding results; (2) wherein the second vector and the output vector are both binary vectors; and (3) wherein the output vector is used in a similarity search in the memory device.
Kleyko teaches a system a method relating to machine learning algorithms for neural networks (Abstract in Page 3777), wherein binding the second vector generated by the activation function with a random vector to generate a plurality of binding results; adding the binding results to generate an adding result (Qi, Abstract and Section I of Page 3777: resource-efficient randomly connected neural networks known as random vector functional link (RVFL) networks since their simple design and extremely fast training time make them very attractive for solving many applied classification tasks; propose to represent input features via the density-based encoding known in the area of stochastic computing and use the operations of binding and bundling from the area of hyperdimensional computing for obtaining the activations of the hidden neurons; the proposed approach demonstrates higher average accuracy than the conventional RVFL; also demonstrate that it is possible to represent the readout matrix using only integers in a limited range with minimal loss in the accuracy; in this case, the proposed approach operates only on small n-bits integers, which results in a computationally efficient architecture; finally, through hardware field-programmable gate array (FPGA) implementations, show that such an approach consumes approximately 11 times less energy than that of the conventional RVFL; RFVLs provide a universal approximation for continuous maps and functional approximations that converge in the Kullback–Leibler divergence when the target function is a probability density function; present an approach for an order of magnitude increase of the resource-efficiency (memory footprint, computational complexity, and energy consumption) of RVFLs operations; the proposed approach combines techniques from two fields of computer science: stochastic computing and hyperdimensional computing; the fundamental idea is in the realization of activations of the hidden layer with the computationally simple operations of hyperdimensional computing and the usage of the density-based encoding of the input features as in stochastic computing; moreover, enhance this approach with the integer-only readout matrix; this combination allows us to use integer arithmetics end-to-end; Section II.A with FIG. 1 of Pages 3777-3778: Fig. 1 depicts the architecture of the conventional RVFL, which includes three layers of neurons; in general, the connectivity of an RVFL is described by two matrices and a vector; a matrix Win [Symbol font/0xCE] [N × K] describes connections between the input layer neurons and the hidden layer neurons; this matrix projects the given input features to the hidden layer; each neuron in the hidden layer has a parameter called a bias; biases of the hidden layer are stored in a vector and denoted as b [Symbol font/0xCE] [N × 1]; the other matrix of readout connections Wout [Symbol font/0xCE] [L × N] between the hidden and output layers transforms the current activations in the hidden layer stored in h into the network’s output y; the main feature of the RVFL is that matrix Win and vector b are randomly generated at the network initialization and stay fixed during the network’s lifetime; there are no strict limitations for the generation of Win and b; they are usually randomly drawn from either normal or uniform distributions; here, both Win and b are generated from a uniform distribution; the range for Win is [−1, 1], while the range for b is [−0.1, 0.1]; since Win and b are fixed, the process of training RVFL is focused on learning the values of the readout matrix Wout; the main advantage of training only Wout is that the corresponding optimization problem is strictly convex; thus, the solution could be found in a single analytical step; the activations of the network’s hidden layer h are described by the following equation h = g(Win x + b), where g(x) is a nonlinear activation function applied to each neuron; here, the sigmoid function g(x) = (1/1 + e−x ) is used; the predictions issued by the output layer are calculated as y = Wout h ; the standard way of acquiring weights of the trainable connections between the hidden and the output layers in the Wout matrix is via solving the ridge regression (which is a special case of the Tikhonov regularization) problem, which minimizes the mean square error between predictions (2) and the ground truth; the activations of the hidden layer hT for each training example are collected together in matrix H [Symbol font/0xCE] [M × N]; matrix Y [Symbol font/0xCE] [M × L] stores the corresponding ground-truth classifications using one-hot encodings; given H and Y, Wout is calculated as equation (3); Section II.B with FIG. 2 of Pages 3778-3779: stochastic computing operates with scalars between 0 and 1, which are represented as random bit vectors where the scalar being encoded determines the probability of generating ones; thus, the density of ones in the obtained bit vector encodes the scalar; hence, such a representation method is called the density-based encoding; generating random streams is important because the independence of two vectors is a prerequisite for using the Boolean operations to implement the arithmetics on them (e.g., AND for multiplication); note that, for the proposed approach, no arithmetic operations will be performed with the density-based encodings of scalars; therefore, the randomness of representations for encoding scalars is not compulsory in this study; in fact, from the simplicity point of view, it is more advantageous to use a structured version of the density-based encoding, which does not require a source of randomness; use the structured version of the density-based encoding also known under the name thermometric encoding for the rest of this brief; Fig. 2 illustrates all possible values, which could be encoded when the dimensionality of the representation5 is set to N = 4; Fig. 2 indicates that using the density-based encoding, it is possible to represent N + 1 different values; the most convenient way of denoting these values is by using integers in the range [0, N] (nodes on the left in the figure); in this case, in order to obtain the encoding of a given value v, it is necessary to set v leftmost positions of the vector to “one” (hashed red nodes in the figure), while the rest of the vector is set to “zero” (filled green nodes); in the case of bipolar representations used in the following, “one” corresponds to −1, while “zero” corresponds to 1; recall, however, that input features are not integers in the range [0, N]; instead, it is assumed that a feature xi is represented by a real number in the range [0, 1; the task is to represent the current value of the feature as a vector f [Symbol font/0xCE] [N × 1] using the abovementioned density-based encoding; since the encoding requires a finite set of values between 0 and N, real numbers are first discretized using a fixed quantization step, which is determined by N; given the current value of the feature, it is quantized to the closest integer as equation (4); the obtained v will determine the density-based encoding f; the presented procedure allows generating density-based encodings for the whole feature vector x; Matrix F [Symbol font/0xCE] [N × K], where K denotes the number of features, contains the density-based encodings f of the current values of x; Section II.C with of Page-3779: hyperdimensional computing also known as vector symbolic architectures is a family of bioinspired methods of representing and manipulating concepts for cognitive architectures and their meanings in a high-dimensional space; vectors of high (but fixed) dimensionality (denoted as N) are the basis for representing information in hyperdimensional computing; the information is distributed across the HD vector’s positions; therefore, HD vectors use distributed representations; distributed representations are contrary to the localist representations since any subset of the positions can be interpreted; this is very relevant to the density-based encoding introduced in Section II-B since the encoding in f is also distributed; in the scope of this brief paper, columns of Win matrix are interpreted as HD vectors, which are generated randomly; these HD vectors are bipolar (Win [Symbol font/0xCE] {−1,+1}[N×K]) and random with equal probabilities for +1 and −1. It is worth noting that an important property of high-dimensional spaces is that with an extremely high probability, all random HD vectors are dissimilar to each other (quasi-orthogonal); in order to manipulate HD vectors, hyperdimensional computing defines operations on them; in this brief, implicitly use only two key operations: binding and bundling; the binding operation is used to associate two HD vectors together; the result of binding is another HD vector; here, the result of binding (denoted as z) two vectors x and y is calculated as follows: z = x [Symbol font/0xC4] y, where the notation [Symbol font/0xC4] for the Hadamard product is used to denote the binding operation since this brief uses position-wise multiplication for binding; an important property of the binding operation is that the resultant HD vector z is quasi-orthogonal to the HD vectors being bound; the second operation is called bundling; the bundling operation combines several HD vectors into a single HD vector; its simplest realization is a position-wise addition; however, when using the position-wise addition, the vector space becomes unlimited; therefore, it is practical to limit the values of the result; this could be achieved with, e.g., a clipping function [denoted as fκ(*)]; in the clipping function, κ is a configurable threshold parameter; thus, in this brief, the bundling operation is implemented via position-wise addition limited via the clipping function; e.g., the result (denoted as a) of bundling HD vectors x and y is simply a = fκ(x + y); in contrast to the binding operation, the resultant HD vector a is similar to all bundled HD vectors, which allows, e.g., storing information in HD vectors; Section III with FIGS. 3-4 in Pages 3779-3780: presents an architecture of the RVFL utilizing the density-based encoding; the approach is illustrated in Fig. 3; the major difference is that the proposed approach is illustrated with four layers of neurons: input layer (x, K neurons); density-based representation layer (F, N × K neurons); hidden layer (h, N neurons); and output layer (y, L neurons); thus, in contrast to the conventional RVFL, the hidden layer is not connected directly to the input layer; instead, each input feature is first transformed into a row of neurons storing its density-based encodings; these vectors constitute the density-based representation layer, which, in turn, is connected to the hidden layer; note also that the input and density-based representation layers are not fully connected; each neuron in the input layer is only connected to N neurons in the corresponding row of the next layer; moreover, these connections (blue lines in Fig. 3) are called “feature-dependent” because the activation of the ith input neuron xi will be quantized to the closest integer v according to (4); in turn, v determines the number of the rightmost connections, which transmits −1, the remaining connections from that neuron transmit +1; since each neuron in the density-based representation layer has only one incoming connection, the input activations are projected in the form of the bipolar matrix F; it is also important to mention that the density-based representation and hidden layers are not fully connected; in fact, each neuron in the density-based representation layer has only one outgoing connection; therefore, the matrix Win describing the fixed random connections to the hidden layer is still Win [Symbol font/0xCE] [N × K]; moreover, these connections have a clear structure; in Fig. 3, the connections are structured in such a way that each column in F is connected to one of the hidden layer neurons; it explains why the number of hidden neurons N also determines the dimensionality of the density-based encoding of features: each hidden neuron has its corresponding column in F (see Fig. 4); similar to the conventional RVFL, the values of Win are also generated randomly; however, the values are drawn equiprobably from {−1,+1}; thus, similar to F, Win is also a bipolar matrix; when reflecting to the ideas of hyperdimensional computing, Win should be interpreted as K N-dimensional bipolar HD vectors; i.e., each feature is assigned with the corresponding HD vector; thus, a conceptual intermediate step before getting input values of the hidden neurons is the binding operation between features’ HD vectors and their current density-based encoding; finally, the proposed approach uses different nonlinear activation function in the hidden layer; the clipping function (5) is used instead of the sigmoid function. The clipping function is characterized by the threshold value κ regulating nonlinear behavior of the neurons and limiting the range of activation values. Summarizing the aforementioned differences, activations of the hidden layer h are obtained as follows: h = fκ (Σ F [Symbol font/0xC4] Win), where Σ is a column-wise summation; note that in contrast to (1), there is no bias term since it has been found empirically that its presence does not improve classification performance; in order to make operations of the proposed approach more intuitive, Fig. 4 presents a numerical example of acquiring the activations of the hidden layer; first, the input layer with K = 5 neurons sets the values of the current feature vector; the quantized values determine the neurons of the density-based encoding, which are set to −1 (the rest is +1); the bottom left figure shows a randomly generated Win; once F is obtained, we calculate the Hadamard product F [Symbol font/0xC4] Win, which is denoted as “bound representations” in Fig. 4; the row-wise summation of the resultant matrix represents the input values of the hidden layer; finally, the clipping function (κ = 2 in Fig. 4) is used in the hidden layer to get h); and
wherein the second vector and the output vector are both binary vectors (Kleyko, Section III with FIGS. 3-4 in Pages 3779-3780: presents an architecture of the RVFL utilizing the density-based encoding; the approach is illustrated in Fig. 3; the major difference is that the proposed approach is illustrated with four layers of neurons: input layer (x, K neurons); density-based representation layer (F, N × K neurons); hidden layer (h, N neurons); and output layer (y, L neurons); thus, in contrast to the conventional RVFL, the hidden layer is not connected directly to the input layer; instead, each input feature is first transformed into a row of neurons storing its density-based encodings; these vectors constitute the density-based representation layer, which, in turn, is connected to the hidden layer; note also that the input and density-based representation layers are not fully connected; each neuron in the input layer is only connected to N neurons in the corresponding row of the next layer; moreover, these connections (blue lines in Fig. 3) are called “feature-dependent” because the activation of the ith input neuron xi will be quantized to the closest integer v according to (4); in turn, v determines the number of the rightmost connections, which transmits −1, the remaining connections from that neuron transmit +1; since each neuron in the density-based representation layer has only one incoming connection, the input activations are projected in the form of the bipolar matrix F; it is also important to mention that the density-based representation and hidden layers are not fully connected; in fact, each neuron in the density-based representation layer has only one outgoing connection; therefore, the matrix Win describing the fixed random connections to the hidden layer is still Win [Symbol font/0xCE] [N × K]; moreover, these connections have a clear structure; in Fig. 3, the connections are structured in such a way that each column in F is connected to one of the hidden layer neurons; it explains why the number of hidden neurons N also determines the dimensionality of the density-based encoding of features: each hidden neuron has its corresponding column in F (see Fig. 4); similar to the conventional RVFL, the values of Win are also generated randomly; however, the values are drawn equiprobably from {−1,+1}; thus, similar to F, Win is also a bipolar matrix; when reflecting to the ideas of hyperdimensional computing, Win should be interpreted as K N-dimensional bipolar HD vectors; i.e., each feature is assigned with the corresponding HD vector; thus, a conceptual intermediate step before getting input values of the hidden neurons is the binding operation between features’ HD vectors and their current density-based encoding; finally, the proposed approach uses different nonlinear activation function in the hidden layer; the clipping function (5) is used instead of the sigmoid function. The clipping function is characterized by the threshold value κ regulating nonlinear behavior of the neurons and limiting the range of activation values. Summarizing the aforementioned differences, activations of the hidden layer h are obtained as follows: h = fκ (Σ F [Symbol font/0xC4] Win), where Σ is a column-wise summation; note that in contrast to (1), there is no bias term since it has been found empirically that its presence does not improve classification performance; in order to make operations of the proposed approach more intuitive, Fig. 4 presents a numerical example of acquiring the activations of the hidden layer; first, the input layer with K = 5 neurons sets the values of the current feature vector; the quantized values determine the neurons of the density-based encoding, which are set to −1 (the rest is +1) (i.e., 1-bit integer – binary representation); the bottom left figure shows a randomly generated Win; once F is obtained, we calculate the Hadamard product F [Symbol font/0xC4] Win, which is denoted as “bound representations” in Fig. 4; the row-wise summation of the resultant matrix represents the input values of the hidden layer; finally, the clipping function (κ = 2 in Fig. 4) is used in the hidden layer to get h).
Qi and Kleyko are analogous art because they are from the same field of endeavor, a system a method relating to machine learning algorithms for neural networks. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of Kleyko to Qi. Motivation for doing so would improve accuracy and enhance resource-efficienc.
Qi in view of Kleyko fails to explicitly disclose wherein the output vector is used in a similarity search in the memory device.
DELSON teaches a system and a method using and encoder (DELSON, ABSTARCT), wherein the output vector is used in a similarity search in the memory device (DELSON, ¶¶ [0003]-[0014] with FIGS. 1 and 2: similarity search is a general term used for a range of techniques which share the principle of searching, typically, very large object sets, where the only available comparator is the similarity between any pair of objects; the definition of the Nearest Neighbor (NN) search is to retrieve candidate items close to a given sample item, from a database of candidate items; distance or proximity is defined by a distance metric such as Euclidean distance-for example L2-distance, or angular distance or, for example, cosine similarity; K-Nearest-Neighbor (KNN) search is the retrieval of the K nearest neighbors to an item and is either used as is for example to present results in a web search or as a prediction algorithm for classification using a voting method, or for regression using an averaging method; data first needs to be transformed to d-dimensional feature vectors that have a meaningful distance metric in the feature space for the task at hand; these real-valued feature vector representations are often called feature embeddings, and hopefully hold desired semantic information of the input data, so that semantically similar inputs fall close to one another in the embedding space; such feature vectors in the same embedding space can be compared using a distance (or similarity) metric, such as cosine similarity or L2 distance; the process of converting raw input data to feature vectors, or feature embeddings, is known as feature extraction, embedding, or X2Vec (i.e., from some kind of data X to a feature vector) and there are many methods, including deep neural networks, domain specific feature engineering, and other machine learning methods that are used for this process; for KNN in large retrieval databases, the search in the feature representation space often requires significant computation and memory resources and imposes a performance bottleneck; as data volumes become increasingly large, and content search becomes a widely required task, methods for fast Approximate-Nearest-Neighbor (ANN) search, which trades off a slight loss in accuracy for large performance gains, have become the focus of extensive research; there are a number of ANN techniques, including graph-based methods, clustering methods, and Hashing methods; hashing methods aim to map data points into low-dimensional representations, or compact bit-codes, for efficient comparison and reduction of memory space; one of the most popular hashing methods is locality sensitive hashing (LSH) which maps high-dimensional points to lowdimensional points using a set of random projections; one main group of data-dependent methods for ANN is based on binary hashing, which maps data points in the original floating-point, feature vector representation space into binary codes in the Hamming space for compact representation and fast search; similarity search in the Hamming space is measured using Hamming distance or Hamming similarity; Hamming distance between two binary strings of equal length is the number of positions at which the corresponding symbols are different; in other words, it measures the minimum number of substitutions needed to change one binary string into the other; hamming similarity is the inverse of Hamming distance and is the number of positions at which the corresponding symbols are similar; an ANN search system 20 has a data encoder 12, similar to that in KNN search system 10 in FIG. 1, a floating-point to binary encoder 22, and an ANN searcher 24; data encoder 12 encodes raw data d, into vectors fvi then floating-point to binary encoder 22 converts floating-point data vectors fvi, into binary encoded data vectors bvi; ANN searcher 24 then performs an approximate similarity search on a set of binary vectors; one implementation of floating-point to binary encoder 22 uses a shallow neural network to encode vectors; it should be noted that, in vector to binary conversion, it is essential that the similarity relationships between the binary encoded vectors and the similarity relationships between the original floating-point vectors are preserved as best as possible, while finding the balance between loss in accuracy and resources, such as memory and search-time; ¶¶ [0063]-[0094] with FIGS.4A-4E: the speed and quality of any approximate nearest-neighbor (ANN) similarity search may be increased by improving the preservation of the locality relations between the original floating-point data vectors and the locality relations between the output binary vectors during binary encoding by a neural binary hashing (NPH) encoder; the performance of such a NPH encoder may be improved by a novel calculation of the loss function, by a comparison of an output pseudo-bipolar Hamming probability distribution to an input cosine probability distribution; for optimizing the encoding of binary output vectors at inference time, during training they may be represented as floating point pseudo-bipolar vectors; the training process trains the NPH encoder so as to achieve output pseudo-bipolar values which strive to either -1 or + 1 (i.e., 1-bit integer – binary data); as mentioned hereinabove, the system architecture of the NPH encoder during training may be different to that of the NPH encoder used during inference, which in tum will encode to real bipolar or binary vectors; in the training scenario, training vector generator 41 may generate a sample floating-point vector q and a set N of k floating-point reference vectors vi, which are input into neural network under training 42; neural network under training 42 may encode input vector q and input vector set N into a pseudo-bipolar encoded sample vector f(q) and a set of pseudo-bipolar encoded reference vectors f(N), respectively; PDLF generator 43 may generate a loss function LKL for neural network under training 42 as a function of the pseudo-bipolar Hamming space probability distribution between pseudo-bipolar encoded output sample vector f(q) and pseudo-bipolar encoded output reference vector set f(N), and the cosine space probability distribution between input sample vector q and input reference vector set N, as explained hereinbelow; as mentioned hereinabove, loss function Liu may then be used to adjust the weights of neural network under training 42 at each iteration of the training process; however, during inference, neural network under training 42 may be reconfigured to output true binary vectors, as explained hereinbelow; details neural network under training 42 comprising a configurable number of hidden layers 421, a final embedding layer 422, and a bipolar simulator layer 423; each hidden layer 421 may comprise a dense layer (FC) 4211 with a configurable number of units, a batch normalization (BN) layer 4212, and a ReLU activation layer 4213; embedding layer 422 may comprise a dense layer (FC) 4221 with n number of units, n being the final desired code-length of the binary vectors, a BN layer 4222 and an L2-normalizer layer 4223; neural network 42 may be designed to output a pseudo-bipolar output vector, which may be used in the training phase; embedding layer 422 may create an L2-normalized, floating-point representation vector as an output; In order to simulate a bipolar vector {-1, + 1}, the output embedding from embedding layer 422 may then undergo a relaxation of the non-differential sgn function, using a β-scaled function 4231 and a hyperbolic tangent function (tan h) 4232, in bipolar simulator layer 423; the chosen code length n is configurable and should take into consideration the tradeoff between a number of factors, such as database size, required accuracy, memory resources and required search time throughput and latency; details PDLF generator 43 comprises a cosine space probability distribution calculator 431, a pseudo-bipolar Hamming space probability distribution calculator 432 and a loss function calculator 433; in left column, FIG. 4D shows the operations on set N of associated reference vectors vi; for sample vector q and in right column, FIG. 4D shows the operations on the pseudo-bipolar encoded output set f(N) of associated reference vectors f(vi) for pseudo-bipolar encoded sample vector f(q); in the training scenario, cosine distribution calculator 431 may determine the probability distribution of the cosine similarities Sci; between sample floating-point vector q and its k associated floating-point reference vectors vi; in set N; cosine distribution calculator 431 may then convert the cosine similarities Sci; (shown as a vector of different width elements in the second row of FIG. 4D) to a similarity probability distribution PDc for sample q by first defining the probabilities Pci of q over its associated reference set N; as shown in the third row of FIG. 4D, the input probability distribution PDc is a vector of the k per-reference vector probabilities Pci, each determined in a manner similar to a "softmax" function, as shown in equation 3; where the softmax function takes k real numbers and normalizes them into a probability distribution consisting of k probabilities proportional to the exponentials of the real numbers; in order to preserve relational similarities of the encoded vectors in a binary Hamming space, rather than in a cosine space, during training, normalized pseudo-Hamming similarities may be used to simulate a real Hamming similarity; the pseudo-Hamming similarities may be calculated from the pseudo-bipolar vectors in a differential manner using the inner product, the pseudo-Hamming similarity SH, for two encoded output vectors f(q) and f(vi) in the pseudo-bipolar { -1, + 1} space, may be defined by the inner product of the pseudo-bipolar vectors f(q) and f(vi), as provided in equation 4, where n is the binary code length of the encoded vectors, which is the dimension of the output pseudo-bipolar vectors during training; to ensure that the similarities of the original input space and the output space are within the same range of values, assuming pseudo-bipolar values, the output similarity scores may be normalized to the range of (-1, + 1 ), using the binary code length n; hence, the normalized pseudo-Hamming similarity SHi between a pseudo-bipolar encoded vector of sample q, f (q), and a pseudo-bipolar encoded vector of reference point vi, f(vi), is defined as in equation 5; Hamming distribution calculator 432 may then utilize the pseudo-Hamming similarity SHi; to construct a pseudo-Hamming similarity probability distribution for encoded pseudo-bipolar vector f(q) of sample q by first defining the probabilities PHi of q over its associated reference set N; the vector of probabilities PHi defines the pseudo-bipolar Hamming probability distribution PDH and is shown on the right side of the third row of FIG. 4D; each pseudo-bipolar Hamming probability 1s determined according to equation 6; the input cosine distribution PDc and the output pseudo-bipolar Hamming distribution PDH may be used by probability distribution loss calculator 433 to calculate the probability distribution loss function LKL, using the Kullback-Leibler divergence, D, also called the relative entropy; Kullback-Leibler divergence is a measure of how the output probability distribution is different from the input or "target" probability distribution; in equation 7, loss function LKL for sample q is defined using the Kullback-Leibler divergence, which, as shown in FIG. 4D, utilizes the input and output probability distributions PCi and PHi vis-a-vis query q; this minimization objective for creating binary, locality preserving vectors may be described as an explicit, multi-wise normalized KL divergence loss, where "explicit" refers to using relations between distances rather than implicitly using some space partitioning method, such as labels; "multi-wise" refers to using multiple reference points which preserve relations among more than two or three items; and "normalized" refers to similarity-to-similarity divergence minimization; to generate a meaningful similarity distribution around any query q, using meaningful points around q, such as its k nearest neighbors, as reference points to q may be most useful; using the k nearest neighbors to q to form a distribution may exploit relative information that may preserve ranking between data points during the training process, and may avoid the noise of irrelevant, far away points, such as might be present if all vectors are used as reference vectors to each sample vector q; a subset of 'proxy' points, enough to form a representation of the data distribution (e.g., around 10% ), may be randomly sampled from the training set to create a 'proxy' vector set from which to extract nearest-neighbors as reference sets; training vector generator 41 comprising a training data vector store 411, a proxy vector set generator 412, a sample vector selector 413 and a KNN vector set generator 414; for each iteration, sample vector selector 413 may select a batch size b of sample vectors qj from the training set T; KNN vector set generator 414 may determine the set N of k nearest neighbor vectors vi, to each vector q).
Qi in view of Kleyko, and DELSON are analogous art because they are from the same field of endeavor, a system and a method using and encoder. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of DELSON to Qi in view of Kleyko. Motivation for doing so would improve
Claims 2 and 7
Qi in view of Kleyko and DELSON discloses all the elements as stated in Claims 1 and 6 further discloses wherein the convolution layer performs the linear conversion on the input into the first vector based on the weight and a bias value (Qi, ¶¶ [0025]-[0042]: A typical 2D convolution layer in the convolutional neural network (CNN) may be described in following equations: P = W*Ai-1 + b (1), Ai = σ(P) (2), where Ai is the output of ith layer in shape of Ci × Hi × Wi, W is the weight matrix of the layer in shape of Ci × C i-1 × KH × Kw, b is the bias vector of the layer in shape of Ci × 1, P is the intermediate multiply-accumulation result of the layer in shape of Ci × Hi × Wi, σ is the non-linear activation function, Ci is the number of channel of ith layer, Hi, Wi, is the height and width of the feature map of ith layer, * denotes the 2D convolution operation; to limit the data precision for computation efficiency, an affine mapping is commonly used to linearly quantize the floating-point value to a finite bit-width integer representation;
v
~
=
s
v
-
-
z
-
(3), where
v
~
is the dequantized value in floating point that approximate the real value v,
v
-
is the integer representation of the quantized value,
s
,
z
-
are the quantization parameters where s is the normalization factor of v in floating-point format and
z
-
the zero point of
v
-
in integer format, these tuples may be computed offline; convolution layer operations are given by Equations (1) and (2), with quantized variables and illustrate integer-based implementations; the offline-computed parameters may be in a lower-bit format to reduce model and memory footprint, the heavy computation kernel (convolution) may also be lower-bit integer-based for low-cost hardware implementation and the offline-computed parameters may be fused when possible while minimizing the induced bias in the fusion; plugging equation (3) into equation (1) with respective quantization parameters (s, z), the equivalent integer-based 2D convolution operation is given by equation (4); separating the integer operations and floating-point operations in equation (5); the intermediate result of the integer-based multiply-accumulation operation is designated as
P
-
in equations (6)-(7)).
Claims 4 and 9
Qi in view of Kleyko and DELSON discloses all the elements as stated in Claims 1 and 6 further discloses wherein in a training stage, the activation function is a hyperbolic tangent function; and in an inference stage, the activation function is a Signum function (Qi, ¶ [0063]: Activation functions: when using symmetric quantization or partial asymmetric quantization (asymmetric weight and symmetric activation), several activation functions are feasible: ReLU (a=0, d=0), ReLU6 (x=6×
2
n
A
n
w
), leakyReLU (a≈0.125, d=0), pReLU (p≈2n), hard-sigmoid (with slope 2-n)) (DELSON, ¶¶ [0069]-[0070]: details neural network under training 42 comprising a configurable number of hidden layers 421, a final embedding layer 422, and a bipolar simulator layer 423; each hidden layer 421 may comprise a dense layer (FC) 4211 with a configurable number of units, a batch normalization (BN) layer 4212, and a ReLU activation layer 4213; embedding layer 422 may comprise a dense layer (FC) 4221 with n number of units, n being the final desired code-length of the binary vectors, a BN layer 4222 and an L2-normalizer layer 4223; neural network 42 may be designed to output a pseudo-bipolar output vector, which may be used in the training phase; embedding layer 422 may create an L2-normalized, floating-point representation vector as an output; In order to simulate a bipolar vector {-1, + 1}, the output embedding from embedding layer 422 may then undergo a relaxation of the non-differential sgn function, using a β-scaled function 4231 and a hyperbolic tangent function (tan h) 4232, in bipolar simulator layer 423; the chosen code length n is configurable and should take into consideration the tradeoff between a number of factors, such as database size, required accuracy, memory resources and required search time throughput and latency).
Claims 5 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Qi in view of Kleyko and DELSON as applied to Claims 1 and 6 respectively above, and further in view of Rosing et al. (US 2022/0019441 A1, pub. date: 01/20/2022), hereinafter Rosing.
Claims 5 and 10
Qi in view of Kleyko and DELSON discloses all the elements as stated in Claims 1 and 6 except failing to explicitly disclose wherein the second vector is bound with the random vector by an XOR logic operation or wherein the binding circuit is an XOR logic gate.
Rosing teaches a system and a method relating to deep learning (Rosing, ¶ [0004]), wherein the second vector is bound with the random vector by an XOR logic operation or wherein the binding circuit is an XOR logic gate (Rosing, ¶¶ [0147]-[0151] and [0154]-[0160] with FIGS. 11(a): the human brain is more capable of recognizing patterns than calculating with numbers; this fact motivates us to simulate the process of brain's computing with points in high-dimensional space; these points can effectively model the neural activity patterns of the brain's circuits; this capability makes hyperdimensional vectors very helpful in many real-world tasks; a new hyper-vector can be based on vector or Boolean operations, such as binding that forms a new hyper-vector which associates two base hyper-vectors, and bundling that combines several hyper-vectors into a single composite hyper-vector; component-wise XOR: bind two hyper-vectors A and B by component-wise XOR and denote the operation as A[Symbol font/0xC4]B. The result of this operation is a new hyper-vector that is dissimilar to its constituents (i.e., d(A[Symbol font/0xC4]B; A)≈D/2), where d() is the Hamming distance; hence XOR can be used to associate two hyper-vectors; ¶¶ [0284]-[0286] with FIGS. 31(a)-(c): the encoder, shown in FIG. 31a, implements bitwise XOR operations between hyper-vectors P and L over different features, and thresholds the results; enable in-memory XOR operations by making a small modification to the sense amplifier of the crossbar memory, as shown in FIG. 31b; FIG. 31c shows the sense amplifier designed to implement the majority function).
Qi in view of Kleyko and DELSON, and Rosing are analogous art because they are from the same field of endeavor, a system and a method relating to deep learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of Rosing to Qi in view of Kleyk. Motivation for doing so would be more e.
Response to Arguments
Applicant’s arguments filed on 05/10/2026 with respect to Claims 1 and 6 have been fully considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in some of arguments. Some of arguments are not persuasive.
Applicant argues on Pages 18-20 of the Remarks that P output by CNN of Qi is an integer not a 32-bit floating-point input.
In response, examiner respectfully disagrees. Qi teaches in ¶¶ [0025]-[0037] that (A) a typical 2D convolution layer in the convolutional neural network (CNN) may be described in following equations: P = W*Ai-1 + b (1), Ai = σ(P) (2), where Ai is the output of ith layer, W is the weight matrix of the layer, b is the bias vector of the layer, P is the intermediate multiply-accumulation result of the layer, σ is the non-linear activation function, * denotes the 2D convolution operation; (B) to limit the data precision for computation efficiency, an affine mapping is commonly used to linearly quantize the floating-point value to a finite bit-width integer representation:
v
~
=
s
v
-
-
z
-
(3), where
v
~
is the dequantized value in floating point that approximate the real value v,
v
-
is the integer representation of the quantized value,
s
,
z
-
are the quantization parameters where s is the normalization factor of v in floating-point format and
z
-
the zero point of
v
-
in integer format. Qi further discloses in ¶¶ [0041]-[0049] that (A) separating the integer operations and floating-point operations in equation (5); the intermediate result of the integer-based multiply-accumulation operation is designated as
P
-
in equations (6)-(7); (B) considering the final non-linear operation in equation (2) and the output expressed in quantized integer-format to cascade to the next layer as shown in equation (8), wherein the input to the non-linear activation function σ is floating number
P
~
obtained after applying 2D convolution layer shown in equation (1); (C) most non-linear activation functions may be relaxed or approximated by a Parametric Rectifier Linear Unit (PReLU) with upper bound clamping as shown in equation (9); and (D) plugging equation (9) into equation (8), and rounding the remaining floating-point parts to integer or converting them to fixed-point representation yields equations (10)-(11). In other words,
P
~
(the first vector) is a floating number which is output from 2D convolution layer shown in equation (1) and input into the non-linear activation function σ in equation (8). It is well-known in the art that a 32-bit floating number is represents by the mantissa part using a 24-bit integer (including 1-bit sign and 23-bit integer) and the exponent part using an 8-bit integer2, and therefore it is common to simulate the precision of the mantissa of float 32 (i.e., 32-bit floating number) as a 24-bit integer as described in ¶ [0056] of Qi. This can be further confirmed in ¶ [0067] with FIG. 4 that quantizing 422 a bias tensor 420 b 436 into a third power-of-two value, bias-adding 424 an output of the convolution and the quantized bias tensor and outputting a bias-addition
P
~
438 (i.e., floating number); the method non-linearizes 426 the output of the bias-addition and quantizes 428 the output of the non-linearization into an activation Ai taking a form of a fourth power-of-two value output tensor 430. Therefore, Qi teaches wherein, when the input is a 32-bit floating point input, the first vector is a floating point vector, wherein the second vector and the output vector are both finite-bit width integer vectors. In other words, Qi only fails to explicitly disclose wherein "finite-bit width integer" vectors includes "binary" vectors. Kleyko teaches this missing limitations in FIG. 4 that input features are floating numbers which are quantized into integer quantized features and then encoding into bipolar HD vectors (i.e., binary vectors). Therefore, the combination of Qi and Kleyko teaches wherein, when the input is a 32-bit floating point input, the first vector is a floating point vector, wherein the second vector and the output vector are both binary vectors.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Rosing et al. (US 2022/0019441 A1, pub. date: 01/20/2022) discloses in ¶¶ [0005] and [0100]-[0136] with FIGS. 1 and 7(a)-(b) that (1) encoding is the first and major operation involved in both training and inference of hyperdimensional (HD); (2) varied HD encoding techniques with different accuracy-performance trade-off have been proposed; (3) in contrast to the deep neural networks that comprise non-linear operations that somewhat cover up the details of raw input, HD operations are fairly reversible, leaving it zero privacy; i.e., the input can be reconstructed from the encoded hyper-vector; (4) previous work on HD computing have introduced the concept of model quantization for compression and energy efficiency, where both encoding and class hyper-vectors are quantized at the cost of significant accuracy loss; (5) we, however, target quantizing the encoding hyper-vectors since the sensitivity is merely determined by the L2 norm of encoding; (6) equation (1-13) shows the 1-bit quantization of encoding in (1-2a); and (7) the original scalar-vector product, as well as the accumulation, is performed in full-precision, and only the final hyper-vector is quantized; (8) for the 1-bit bipolar quantization, a basic approach is adding up all bits of the same dimension, followed by a final sign/threshold operation, which is equivalent to a majority operation between '-1's and '+1's; i.e., we can represent -1 by 0, and + 1 by 1 in hardware, as it does not change the logic. Rosing further discloses in ¶¶ [0137]-[0161] with FIGS. 10 and 11a-11b that (1) HyperRec is based on hyperdimensional computing; (2) user and item information can be preserved nearly loseless for identifying similarity; (3) it is a binary encoding method and only relies on Boolean operations; (4) since the basic operations of hyperdimensional vectors are component-wise operations and associative search, this design can be accelerated in hardware; (4) HyperRec provides a three-stage pipeline: encoding, similarity check and recommendation; (5) in HyperRec, users, items and ratings are included with hyperdimensional binary vectors, wherein the overview of the framework is shown in FIG. 10; (5) all users, items and ratings are included using hyperdimensional vectors; (6) our goal is to discover and preserve users' and items' information based on their historical interactions; (7) the generating process of rating hyper-vectors is shown in FIG. 11a; (8) by this means, consecutive ratings are close in terms of Hamming distance; (9) if two ratings are numerically different from each other by a large margin, the Hamming distance between their hyper-vectors is large; (10) compute the characterization hyper-vector of each user and each item, wherein the process is shown in FIG. 11b; (11) by this approach, we can capture the difference between users' consuming behaviors and their rating patterns; (12) after we obtain the characterization hyper-vectors of users and items, we use Hamming distance to identify similarity; (13) in order to compute the rating that user u will give to item v, we first identify the k-nearest items of item v based the ratings they received and denote this set as Nk(v); for each of the k-nearest item v', we also identify k'-nearest users of user u in the set Bv', based on the ratings they give, and denote this as Nk'(u, v'); (14) the compute the predicted rating of user u for item v' as shown in equation (2-1); (15) then compute the predicted rating of user u for item v as in equation (2-2); and (16) after we obtain the predicted ratings of user u for all the items he/she did not buy before, we can recommend the user u with the items with highest predicted ratings.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HWEI-MIN LU whose telephone number is (313)446-4913. The examiner can normally be reached Mon - Fri: 9:00 AM - 6:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela D. Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HWEI-MIN LU/Primary Examiner, Art Unit 2142
1 See https://web.archive.org/web/20111128141229/http://en.wikipedia.org/wiki/Single-precision_floating-point_format (recorded by Wayback Machine on 11/28/2011).
2 See https://web.archive.org/web/20111128141229/http://en.wikipedia.org/wiki/Single-precision_floating-point_format (recorded by Wayback Machine on 11/28/2011).
Prosecution Insights