DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 1/2/2026 has been entered.
Response to Amendment
(Submitted 1/2/2026)
The examiner notes that the applicant amended the claims 2-4, 12-14 and 21.
In regard to 103 rejections
The examiner submits that the applicant argument is MOOT as a result of the amended claims within the context of the invention focus to specify weights of the layers.
- On Page 13, the applicant specifically that the reference “BORSE” does not teach the amended limitation “weights of a layer in the weight tensor” and further argues that the reference “BORSE” does not teach “ the setting current values for next training step equal to the scaled updates” and
“ receiving a layer input by the layer and processing the layer input by the layer in accordance with the weights of the layer that have the scaled updated values to generate a layer output”.
Examiner Response
It appears that the entire argument of the applicant is to use the context of a “weights of the layer in weight tensor”. The examiner respectfully disagrees with the applicant’s arguments. In fact, the reference “BORSE” teaches weights of the layer in weight tensor. BORSE teaches in [0044] “ a fully connected layer may change the size between an input X (e.g., a tensor) and output Y (e.g., another tensor)” , in [0062] “ The output of activation function 406 is weight tensor 407, which also has dimensionality of 1×N. Weight tensor 407 is then used as input to fusion function 408” and in [0031] “ a node in a first layer may be connected to a limited number of nodes in the second layer. More generally, a locally connected layer of the locally connected neural network 104 may be configured so that each node in a layer will have the same or a similar connectivity pattern, but with connections strengths (or weights) that may have different values”. Without conceding the applicant’s argument, the examiner submits that the entire argument is MOOT as a result of using new grounds of rejection under RCE. The examiner notes that the applicant’s argument above is with the context of using weights of a layer in the weight tensor” and “neural network scaling engine”. The examiner has used three new references “ROSE”, and “LI”, “Narkhede” and “Abraham” in the Office Action.
In CONCLISION, the examiner rejects claims independent claims 2, 12 and 21 and all dependent claims 3-11, and 13-20 under 103 as NON FINAL REJECTION under RCE.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2-3, 11 -13 and 21 are rejected under 35 U.S.C. 103 unpatentable over J
James Christopher ROSEWARNE et.al. (hereinafter ROSE) US 2026/0075209 A1,
in view of Xinlin LI et.al. (hereinafter LI) US 2020/0097818 A1.
In view of Meenal V. Narkhede et.al. (hereinafter Narkhede), Artificial Intelligence Review (2022) 55:291–322.
In regard to claim 2: (Currently Amended)
ROSE discloses:
- A method performed by one or more computers and for training a neural network to perform one or more machine learning tasks, wherein the method comprises repeatedly performing the following for a weight tensor that includes weights of a layer of the neural
network:
[0004]:
CNNs typically include many layers, such as convolution layers and fully connected layers, with data passing from one layer to the next in the form of ‘tensors’
[0062]:
At the step 610 the CNN backbone 114 receives one frame of the video frame data 113 from the video source 112 and performs specific early layers of an overall CNN, such as layers corresponding to the ‘backbone’ of the CNN. The step 610 outputs the tensors 115
[0070]:
Although the modules 170 and 174 receive tensors one at a time, incoming tensors may be grouped into batches of a size greater than one. Increasing the batch size can improve training as each weight update step is influenced by a variety of inputs. [BRI: In a CNN, the tensors do represent weight tensors that are learnable parameters inside each later where the weight acts as a filter to extract feature from the input data]
[0068]:
A forward pass through the modules 170 and 174 is performed using weights currently present in the modules 170 and 174.
[0070]:
At the step 650 a weight update is performed in the modules 170 and 174 using a process of ‘back propagation’ whereby weights are updated based on a process, such as stochastic gradient descent (SGD), attempting to minimise the measured loss 181. The trainable bottleneck encoder 170 and the trainable bottleneck decoder 174, by virtue of ongoing weight updating due to back propagation, are able to adapt to such changes in statistics of the tensors 115, resulting in the potential for achievement of a lower MSE 181 compared to the MSE 179.
- generating respective gradient-based updates to the weights of the layer in the weight tensor, comprising:
[0062]:
At the step 610 the CNN backbone 114 receives one frame of the video frame data 113 from the video source 112 and performs specific early layers of an overall CNN, such as layers corresponding to the ‘backbone’ of the CNN. The step 610 outputs the tensors 115
[0070]:
a weight update is performed in the modules 170 and 174 using a process of ‘back propagation’ whereby weights are updated based on a process, such as stochastic gradient descent (SGD), attempting to minimise the measured loss 181.
- performing, using a plurality of training examples, a training step that comprises a plurality of forward passes through the neural network based on the weights of the layer in the weight tensor to obtain respective gradients of a loss function with respect to the weights of the layer in the weight tensor;
[0068]:
A forward pass through the modules 170 and 174 is performed using weights currently present in the modules 170 and 174.
[0005]:
An iteration of the entire training dataset forms an ‘epoch’ of training, and training typically requires multiple epochs to achieve a high level of performance for the task
[0062]:
At the step 610 the CNN backbone 114 receives one frame of the video frame data 113 from the video source 112 and performs specific early layers of an overall CNN, such as layers corresponding to the ‘backbone’ of the CNN. The step 610 outputs the tensors 115 [BRI: In a CNN, the tensors do represent weight tensors that are learnable parameters inside each later where the weight acts as a filter to extract feature from the input data]
- and generating, by using an optimizer and based on the respective gradients, respective gradient-based updates to the weights of the layer in the weight tensor;
[0062]:
At the step 610 the CNN backbone 114 receives one frame of the video frame data 113 from the video source 112 and performs specific early layers of an overall CNN, such as layers corresponding to the ‘backbone’ of the CNN. The step 610 outputs the tensors 115
[0070]:
a weight update is performed in the modules 170 and 174 using a process of ‘back propagation’ whereby weights are updated based on a process, such as stochastic gradient descent (SGD), attempting to minimise the measured loss 181.
[0007]:
When training a network, the value of the batch dimension may be increased so that multiple frames are passed through the network in each batch before the network weights are updated, according to a predetermined ‘batch size
[0070]:
Moreover, for video data the statistical variety in consecutive frames or consecutive tensors from the backbone 114 is less pronounced, reducing the benefit of using a larger batch size. Other processes to update weights may also be used, such as the ‘Adam W’ optimiser which utilises momentum and scaling and decouples weight decay from gradient update. [BRI: in according to the focus of the invention in this application]
- applying the respective gradient-based updates generated by using the optimizer to the weights of the layer in the weight tensor to generate initial updated values of the weights of the layer in the weight tensor without applying any weight decay updates to the weights of the layer in the weight tensor
[0004]:
CNNs typically include many layers, such as convolution layers and fully connected layers, with data passing from one layer to the next in the form of ‘tensors’
[0062]:
At the step 610 the CNN backbone 114 receives one frame of the video frame data 113 from the video source 112 and performs specific early layers of an overall CNN, such as layers corresponding to the ‘backbone’ of the CNN. The step 610 outputs the tensors 115
[0070]:
Although the modules 170 and 174 receive tensors one at a time, incoming tensors may be grouped into batches of a size greater than one. Increasing the batch size can improve training as each weight update step is influenced by a variety of inputs. [BRI: In a CNN, the tensors do represent weight tensors that are learnable parameters inside each later where the weight acts as a filter to extract feature from the input data]
[0070]:
Other processes to update weights may also be used, such as the ‘Adam W’ optimiser which utilises momentum and scaling and decouples weight decay from gradient update
- receiving a layer input by the layer and processing the layer input by the layer in accordance with the weights of the layer that have the scaled updated values to generate a layer output.
[0063]:
The step 615 receives as input at least one tensor where the neural network for a machine task has been partially performed (that is the backbone network has been implemented) and produces tensors 117.
[0063]:
the step 615 receives as input at least one tensor where the neural network for a machine task has been partially performed (that is the backbone network has been implemented) and produces tensors 117.
[0128]:
A concatenation module 524 performs a channel-wise concatenation of the tensors 505, 523, 523a, and 523b to produce concatenated tensor 525, of dimensions h, w, 512. The concatenated tensor 525 is passed to a squeeze and excitation (SE) module 526 to produce a tensor 527. The SE module 526 sequentially performs a global pooling, a fully-connected layer with reduction in channel count, a rectified linear unit activation, a second fully-connected layer restoring the channel count, and a sigmoid activation function to produce a scaling tensor. [BRI: the input data received as tensor does receive layer input and the context of ML task uses layer for mathematical transformation to produce output tensor (via weight and activation function) and channel concatenation combine multiple inputs along the channel dimension ]
- and processing the layer input by the layer in accordance with the weights of the layer that have the scaled updated values to generate a layer output.
[0055]:
The mask head includes two convolutional layers and produces a segmentation map for each ‘region of interest’
[0055]:
MaskRCNN may be used to perform both object detection and instance segmentation, with additional complexity in the network head due to use of mask heads. [BRI: producing segmentation map does represent processing the layer as the layers are responsible for generating precise segmentation masks in the image]
ROSE does not explicitly disclose:
- and applying neural network weight value regularization to the weights of the layer in the weight tensor that have the initial updated values, comprising
However, LI discloses in:
- and applying neural network weight value regularization to the weights of the layer in the weight tensor that have the initial updated values, comprising:
[0017]:
a method for training a neural network (NN) block in a NN by applying a trainable scaling factor on output of a binary convolution, which may help to save computational cost significantly and improve computation accuracy to approximate to a full-precision NN. A regularization function with respect to an estimated real-valued weight tensor including the scaling factor and a real-valued weight tensor is included in a loss function of the NN.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine ROSE and LI.
ROSE teaches weights of the layer in a weight tensor, generating gradients and performing machine learning tasks and receiving a layer input by the layer and processing the layer that have the scaled updated values to generate a layer output.
LI teaches applying the respective gradient-based updates without applying any weight decay updates to the weights of the layer in the weight tensor and weight regularization and training with the initial scaled updated value as a startup weight.
One of ordinary skill would have motivation to combine ROSE and LI that improved the stability of the NN with greater accuracy using scaling (LI [0017]).
ROSE and LI do not explicitly disclose:
- scaling, by using a neural network weight scaling engine, the initial updated values of the weights of the layer in the weight tensor to generate scaled updated values that are different from the initial updated values;
- and setting current values of the weights of the weights of the layer in the weight tensor for a next training step to be equal to the scaled updated values, such that in the next training step comprises:
However, Narkhede discloses:
- scaling, by using a neural network weight scaling engine
[1 Introduction, Page 294]:
The accelerated libraries like NVIDIA Compute Unified Device Architecture (CUDA), NVIDIA CUDA Deep Neural Network Library (cuDNN), Intel Math Kernel Library (MKL), Open Computing Language (OpenCL) provide optimized functions to get advantage of parallelism of GPUs. These libraries are used by deep learning frameworks like Tensorflow, [BRI: TensorFlow does contain a scaling engine to operate efficieinly accors heterogenous computing environments)
- the initial updated values of the weights of the layer in the weight tensor to generate scaled updated values that are different from the initial updated values;
[1 Introduction, Page 292]:
Iteratively training a neural network is solving a non-convex optimization problem parameterized over all the network weights and is done by the backpropagation,
[1 Introduction, Page 292]:
the training process to be consisting of two passes—forward and backward pass. The output is predicted using the input and the initialized weights during the forward pass. A simple thought to initialize these weights may be zero initialization. But because of this, all the layers would learn the same and during backpropagation, all the weights would get the same update.
[1 Introduction, Page 293]:
Implementing a task in deep learning involves the following steps:
• Preparing data The data is collected, balanced, pre-processed and split into a training, validation and test set.
• Choosing and building an appropriate model A model suitable for the task is selected.
• Training The model is initialized and trained on the dataset to minimize the selected loss function.
• Evaluation The performance of trained model is checked on the test dataset.
• Parameter tuning This step may be performed to improve the results if the model does not give expected performance after evaluation. The hyperparameters of the model can be tuned in this step and the model has to be retrained
[1 Introduction, Page 294]:
• Inference Once the trained model is ready, it can be used for inference.
The training step, which is given in the above steps, involves two approaches: training the model from scratch or adopting a transfer learning approach. Training deep networks from scratch is a computationally-intensive process and requires specialized hardware that is present in accelerated computing environments like Graphics Processing Units (GPUs)
[2 Initialization without pre-training, Page 296]:
2.1 Random initialization This section discusses the initialization strategies in which weights are either initialized randomly or strategies in which weights are first randomly chosen and later updated by some method or algorithm
[2.1.2 Variance scaling based initialization, Page 298]:
various weight initialization techniques in which weights are initially selected from random distributions but later scaled so that variance of input and output layer is maintained or variance of the output layer is maintained to a desired value or variance of gradients is maintained while training.
[2.1.2 Variance scaling based initialization, Page 298]:
Yang et al. (2020) have proposed an initialization technique that introduces a norm on parameter space, which gives constraints on parameter updates. A scalar value has been derived based on norm equality and the variance of a vector of learnable parameters has been determined by using this scalar and as given by Glorot and Bengio (2010)
[2.1.2 Variance scaling based initialization, Page 298]:
Ioffe et al. (2015) have proposed a batch normalizing technique for normalizing layer activations, which brought a significant speedup in the training process.
[3 Initialization with pre-traininng, Page 304]:
Larochelle et al. (2009) have also discussed after experimentation that unsupervised weight initialization helped to select the parameters in a region of suitable local minimum and also acted as a regularization that offered good generalization.
- and setting current values of the weights of the weights of the layer in the weight tensor for a next training step to be equal to the scaled updated values, such that in the next training step comprises:
[1 Introduction, Page 293]:
Implementing a task in deep learning involves the following steps:
• Preparing data The data is collected, balanced, pre-processed and split into a training, validation and test set.
• Choosing and building an appropriate model A model suitable for the task is selected.
• Training The model is initialized and trained on the dataset to minimize the selected loss function.
• Evaluation The performance of trained model is checked on the test dataset.
• Parameter tuning This step may be performed to improve the results if the model does not give expected performance after evaluation. The hyperparameters of the model can be tuned in this step and the model has to be retrained
[1 Introduction, Page 294]:
• Inference Once the trained model is ready, it can be used for inference.
[3 Initialization with pre-training, Page 305]:
Li et al. (1993) have proposed a delta pre-training method to obtain initial weights for the backpropagation algorithm. The pre-training started by assuming initial weights to be zero. The weights obtained after pre-training the network were set as initial weights. The average number of epochs was used as a measure for convergence. The authors have
discussed that this method of weight initialization has very less probability of getting stuck in local minima as compared to random weight initialization.
[2.1.2 Variance scaling based initialization, Page 298]:
various weight initialization techniques in which weights are initially selected from random distributions but later scaled so that variance of input and output layer is maintained or variance of the output layer is maintained to a desired value or variance of gradients is maintained while training.
[2.1.1 Interval based initialization, Page 297]:
Drago et al. (1992) have obtained an expression for initializing the weights of a FFNN in an optimal way to ensure that the network did not get stuck in local minima or to avoid saturation of neurons while training. The weights were initially picked from a random uni form distribution and were then scaled with a scale factor. The authors have obtained a scale factor proportional to the magnitude of weights by computer simulations. That value of scale factor was selected where the paralyzed neuron percentage value is optimal and expected epochs to convergence were minimum. [BRI: this is using a scaled model for the next training iteration that are optimized through training and improve the performance for improved convergence)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine ROSE, LI and Narkhede.
ROSE teaches weights of the layer in a weight tensor, generating gradients and performing machine learning tasks and receiving a layer input by the layer and processing the layer that have the scaled updated values to generate a layer output.
LI teaches applying the respective gradient-based updates without applying any weight decay updates to the weights of the layer in the weight tensor and weight regularization and training with the initial scaled updated value as a startup weight.
Narkhede teaches scaled updated values of the weights of the layers different from the initial updated values.
One of ordinary skill would have motivation to combine ROSE, LI and Narkhede that uses various techniques for weight initialization such that the training is accelerated and the performance is improved (Narkhede[Abstract, Page 291])
In regard to claim 3: (Currently Amended)
ROSE and LI do not explicitly disclose:
wherein layer includes no biases.
However, Nakhede discloses:
- wherein layer includes no biases.
[3, Page 305]:
Paine et al. (2014) have proposed a zero-bias convolutional autoencoder (CAE) to learn the parameters from the input data in an unsupervised way.
[3, Page 305]:
Ruiz-Garcia et al. (2017) have employed stacked convolutional autoencoder (SCAE), which pre-trained the weights for CNN for the task
[3, Page 305]:
It was observed that the pre-training method using SCAE gave better accuracy and also reduced the training time.
[3, Page 305]:
Wiehman et al. (2016) have employed unsupervised pre-training on U-net architecture, which is a fully convolutional architecture for image segmentation task.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine ROSE, LI and Narkhede.
ROSE teaches weights of the layer in a weight tensor, generating gradients and performing machine learning tasks and receiving a layer input by the layer and processing the layer that have the scaled updated values to generate a layer output.
LI teaches applying the respective gradient-based updates without applying any weight decay updates to the weights of the layer in the weight tensor and weight regularization and training with the initial scaled updated value as a startup weight.
Narkhede teaches scaled updated values of the weights of the layers different from the initial updated values.
One of ordinary skill would have motivation to combine ROSE, LI and Narkhede that uses various techniques for weight initialization such that the training is accelerated and the performance is improved (Narkhede[Abstract, Page 291])
In regard to claim 11: (Previously Presented)
ROSE does not explicitly disclose:
- wherein training the one or more training tasks comprises: (i) a text processing task, (ii) an image processing task, (iii) an audio processing task, (iv) a video processing task, or (v) a multi-modal task involving two or more of (i)-(iv).
However, LI discloses in:
- wherein training the one or more training tasks comprises:
(i) a text processing task,
[0231]:
By using word embedding model, text data, like sentences and articles, can be converted into a sequence of fixed-length vectors so DNN models can be trained on the top of embedded data to predict sentiment label of the text and solved the sentiment analysis problem. FIG. 24 shows sentiment analysis diagram.
(ii) an image processing task,
[0156]:
Image Classification
[0157] :
Facial recognition is a technology that capable of identifying or verifying a person from an image
(iii) an audio processing task,
[0155]:
In some implementations, the NN block trained by a method of the present disclosure may perform inference tasks in various applications. The inferences tasks may include facial recognition, object detections, image classification, machine translation, or text-to-speech transition.
(iv) a video processing task, or
[0157]:
Facial recognition is a technology that capable of identifying or verifying a person from an image or a video.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine ROSE and LI.
ROSE teaches weights of the layer in a weight tensor, generating gradients and performing machine learning tasks and receiving a layer input by the layer and processing the layer that have the scaled updated values to generate a layer output.
LI teaches applying the respective gradient-based updates without applying any weight decay updates to the weights of the layer in the weight tensor and weight regularization and training with the initial scaled updated value as a startup weight.
One of ordinary skill would have motivation to combine ROSE and LI that improved the stability of the NN with greater accuracy using scaling (LI [0017]).
In regard to claim 12: (Currently Amended)
ROSE discloses:
- A system comprising one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a neural network to perform one or more machine learning tasks, wherein the operations comprise repeatedly performing the following for a weight tensor that includes weights of the neural network:
[0021]
[0004]:
CNNs typically include many layers, such as convolution layers and fully connected layers, with data passing from one layer to the next in the form of ‘tensors’
[0062]:
At the step 610 the CNN backbone 114 receives one frame of the video frame data 113 from the video source 112 and performs specific early layers of an overall CNN, such as layers corresponding to the ‘backbone’ of the CNN. The step 610 outputs the tensors 115
[0070]:
Although the modules 170 and 174 receive tensors one at a time, incoming tensors may be grouped into batches of a size greater than one. Increasing the batch size can improve training as each weight update step is influenced by a variety of inputs.
[0068]:
A forward pass through the modules 170 and 174 is performed using weights currently present in the modules 170 and 174.
[0070]:
At the step 650 a weight update is performed in the modules 170 and 174 using a process of ‘back propagation’ whereby weights are updated based on a process, such as stochastic gradient descent (SGD), attempting to minimise the measured loss 181. The trainable bottleneck encoder 170 and the trainable bottleneck decoder 174, by virtue of ongoing weight updating due to back propagation, are able to adapt to such changes in statistics of the tensors 115, resulting in the potential for achievement of a lower MSE 181 compared to the MSE 179.
- generating respective gradient-based updates to the weights of the layer in the weight tensor, comprising:
[0062]:
At the step 610 the CNN backbone 114 receives one frame of the video frame data 113 from the video source 112 and performs specific early layers of an overall CNN, such as layers corresponding to the ‘backbone’ of the CNN. The step 610 outputs the tensors 115
[0070]:
a weight update is performed in the modules 170 and 174 using a process of ‘back propagation’ whereby weights are updated based on a process, such as stochastic gradient descent (SGD), attempting to minimise the measured loss 181.
- performing, using a plurality of training examples, a training step that comprises a plurality of forward passes through the neural network based on the weights of the layer in the weight tensor to obtain respective gradients of a loss function with respect to the weights of the layer in the weight tens
[0068]:
A forward pass through the modules 170 and 174 is performed using weights currently present in the modules 170 and 174.
[0005]:
An iteration of the entire training dataset forms an ‘epoch’ of training, and training typically requires multiple epochs to achieve a high level of performance for the task
[0062]:
At the step 610 the CNN backbone 114 receives one frame of the video frame data 113 from the video source 112 and performs specific early layers of an overall CNN, such as layers corresponding to the ‘backbone’ of the CNN. The step 610 outputs the tensors 115
- and generating, by using an optimizer and based on the respective gradients, respective gradient-based updates to the weights of the layer in the weight tensor;
[0062]:
At the step 610 the CNN backbone 114 receives one frame of the video frame data 113 from the video source 112 and performs specific early layers of an overall CNN, such as layers corresponding to the ‘backbone’ of the CNN. The step 610 outputs the tensors 115
[0070]:
a weight update is performed in the modules 170 and 174 using a process of ‘back propagation’ whereby weights are updated based on a process, such as stochastic gradient descent (SGD), attempting to minimise the measured loss 181.
[0007]:
When training a network, the value of the batch dimension may be increased so that multiple frames are passed through the network in each batch before the network weights are updated, according to a predetermined ‘batch size
[0070]:
Moreover, for video data the statistical variety in consecutive frames or consecutive tensors from the backbone 114 is less pronounced, reducing the benefit of using a larger batch size. Other processes to update weights may also be used, such as the ‘Adam W’ optimiser which utilises momentum and scaling and decouples weight decay from gradient update. [BRI: in according to the focus of the invention in this application]
- applying the respective gradient-based updates generated by using the optimizer to the weights of the layer in the weight tensor to generate initial updated values of the weights of the layer in the weight tensor without applying any weight decay updates to the weights of the layer in the weight tensor
[0004]:
CNNs typically include many layers, such as convolution layers and fully connected layers, with data passing from one layer to the next in the form of ‘tensors’
[0062]:
At the step 610 the CNN backbone 114 receives one frame of the video frame data 113 from the video source 112 and performs specific early layers of an overall CNN, such as layers corresponding to the ‘backbone’ of the CNN. The step 610 outputs the tensors 115
[0070]:
Although the modules 170 and 174 receive tensors one at a time, incoming tensors may be grouped into batches of a size greater than one. Increasing the batch size can improve training as each weight update step is influenced by a variety of inputs.
[0070]:
Other processes to update weights may also be used, such as the ‘Adam W’ optimiser which utilises momentum and scaling and decouples weight decay from gradient update
- receiving a layer input by the layer and processing the layer input by the layer in accordance with the weights of the layer that have the scaled updated values to generate a layer output.
[0063]:
The step 615 receives as input at least one tensor where the neural network for a machine task has been partially performed (that is the backbone network has been implemented) and produces tensors 117.
[0063]:
the step 615 receives as input at least one tensor where the neural network for a machine task has been partially performed (that is the backbone network has been implemented) and produces tensors 117.
[0128]:
A concatenation module 524 performs a channel-wise concatenation of the tensors 505, 523, 523a, and 523b to produce concatenated tensor 525, of dimensions h, w, 512. The concatenated tensor 525 is passed to a squeeze and excitation (SE) module 526 to produce a tensor 527. The SE module 526 sequentially performs a global pooling, a fully-connected layer with reduction in channel count, a rectified linear unit activation, a second fully-connected layer restoring the channel count, and a sigmoid activation function to produce a scaling tensor.
- and processing the layer input by the layer in accordance with the weights of the layer that have the scaled updated values to generate a layer output.
[0055]:
The mask head includes two convolutional layers and produces a segmentation map for each ‘region of interest’
[0055]:
MaskRCNN may be used to perform both object detection and instance segmentation, with additional complexity in the network head due to use of mask heads. [BRI: producing segmentation map does represent processing the layer as the layers are responsible for generating precise segmentation masks in the image]
ROSE does not explicitly disclose:
- and applying neural network weight value regularization to the weights of the layer in the weight tensor that have the initial updated values, comprising
- and setting current values of the weights of the weights of the layer in the weight tensor for a next training step to be equal to the scaled updated values, such that in the next training step comprises:
However, LI discloses in:
- and applying neural network weight value regularization to the weights of the layer in the weight tensor that have the initial updated values, comprising:
[0017]:
a method for training a neural network (NN) block in a NN by applying a trainable scaling factor on output of a binary convolution, which may help to save computational cost significantly and improve computation accuracy to approximate to a full-precision NN. A regularization function with respect to an estimated real-valued weight tensor including the scaling factor and a real-valued weight tensor is included in a loss function of the NN.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine ROSE and LI.
ROSE teaches weights of the layer in a weight tensor, generating gradients and performing machine learning tasks and receiving a layer input by the layer and processing the layer that have the scaled updated values to generate a layer output.
LI teaches applying the respective gradient-based updates without applying any weight decay updates to the weights of the layer in the weight tensor and weight regularization and training with the initial scaled updated value as a startup weight.
One of ordinary skill would have motivation to combine ROSE and LI that improved the stability of the NN with greater accuracy using scaling (LI [0017]).
ROSE and LI do not explicitly disclose:
- scaling, by using a neural network weight scaling engine, the initial updated values of the weights of the layer in the weight tensor to generate scaled updated values that are different from the initial updated values;
- and setting current values of the weights of the weights of the layer in the weight tensor for a next training step to be equal to the scaled updated values, such that in the next training step comprises:
However, Narkhede discloses:
- scaling, by using a neural network weight scaling engine
[1 Introduction, Page 294]:
The accelerated libraries like NVIDIA Compute Unified Device Architecture (CUDA), NVIDIA CUDA Deep Neural Network Library (cuDNN), Intel Math Kernel Library (MKL), Open Computing Language (OpenCL) provide optimized functions to get advantage of parallelism of GPUs. These libraries are used by deep learning frameworks like Tensorflow, [BRI: TensorFlow does contain a scaling engine to operate efficieinly accors heterogenous computing environments)
- the initial updated values of the weights of the layer in the weight tensor to generate scaled updated values that are different from the initial updated values;
[1 Introduction, Page 292]:
Iteratively training a neural network is solving a non-convex optimization problem parameterized over all the network weights and is done by the backpropagation,
[1 Introduction, Page 292]:
the training process to be consisting of two passes—forward and backward pass. The output is predicted using the input and the initialized weights during the forward pass. A simple thought to initialize these weights may be zero initialization. But because of this, all the layers would learn the same and during backpropagation, all the weights would get the same update.
[1 Introduction, Page 293]:
Implementing a task in deep learning involves the following steps:
• Preparing data The data is collected, balanced, pre-processed and split into a training, validation and test set.
• Choosing and building an appropriate model A model suitable for the task is selected.
• Training The model is initialized and trained on the dataset to minimize the selected loss function.
• Evaluation The performance of trained model is checked on the test dataset.
• Parameter tuning This step may be performed to improve the results if the model does not give expected performance after evaluation. The hyperparameters of the model can be tuned in this step and the model has to be retrained
[1 Introduction, Page 294]:
• Inference Once the trained model is ready, it can be used for inference.
The training step, which is given in the above steps, involves two approaches: training the model from scratch or adopting a transfer learning approach. Training deep networks from scratch is a computationally-intensive process and requires specialized hardware that is present in accelerated computing environments like Graphics Processing Units (GPUs)
[2 Initialization without pre-training, Page 296]:
2.1 Random initialization This section discusses the initialization strategies in which weights are either initialized randomly or strategies in which weights are first randomly chosen and later updated by some method or algorithm
[2.1.2 Variance scaling based initialization, Page 298]:
various weight initialization techniques in which weights are initially selected from random distributions but later scaled so that variance of input and output layer is maintained or variance of the output layer is maintained to a desired value or variance of gradients is maintained while training.
[2.1.2 Variance scaling based initialization, Page 298]:
Yang et al. (2020) have proposed an initialization technique that introduces a norm on parameter space, which gives constraints on parameter updates. A scalar value has been derived based on norm equality and the variance of a vector of learnable parameters has been determined by using this scalar and as given by Glorot and Bengio (2010)
[2.1.2 Variance scaling based initialization, Page 298]:
Ioffe et al. (2015) have proposed a batch normalizing technique for normalizing layer activations, which brought a significant speedup in the training process.
[3 Initialization with pre-traininng, Page 304]:
Larochelle et al. (2009) have also discussed after experimentation that unsupervised weight initialization helped to select the parameters in a region of suitable local minimum and also acted as a regularization that offered good generalization.
- and setting current values of the weights of the weights of the layer in the weight tensor for a next training step to be equal to the scaled updated values, such that in the next training step comprises:
[1 Introduction, Page 293]:
Implementing a task in deep learning involves the following steps:
• Preparing data The data is collected, balanced, pre-processed and split into a training, validation and test set.
• Choosing and building an appropriate model A model suitable for the task is selected.
• Training The model is initialized and trained on the dataset to minimize the selected loss function.
• Evaluation The performance of trained model is checked on the test dataset.
• Parameter tuning This step may be performed to improve the results if the model does not give expected performance after evaluation. The hyperparameters of the model can be tuned in this step and the model has to be retrained
[1 Introduction, Page 294]:
• Inference Once the trained model is ready, it can be used for inference.
[3 Initialization with pre-training, Page 305]:
Li et al. (1993) have proposed a delta pre-training method to obtain initial weights for the backpropagation algorithm. The pre-training started by assuming initial weights to be zero. The weights obtained after pre-training the network were set as initial weights. The average number of epochs was used as a measure for convergence. The authors have
discussed that this method of weight initialization has very less probability of getting stuck in local minima as compared to random weight initialization.
[2.1.2 Variance scaling based initialization, Page 298]:
various weight initialization techniques in which weights are initially selected from random distributions but later scaled so that variance of input and output layer is maintained or variance of the output layer is maintained to a desired value or variance of gradients is maintained while training.
[2.1.1 Interval based initialization, Page 297]:
Drago et al. (1992) have obtained an expression for initializing the weights of a FFNN in an optimal way to ensure that the network did not get stuck in local minima or to avoid saturation of neurons while training. The weights were initially picked from a random uni form distribution and were then scaled with a scale factor. The authors have obtained a scale factor proportional to the magnitude of weights by computer simulations. That value of scale factor was selected where the paralyzed neuron percentage value is optimal and expected epochs to convergence were minimum. [BRI: this is using a scaled model for the next training iteration that are optimized through training and improve the performance for improved convergence)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine ROSE, LI and Narkhede.
ROSE teaches weights of the layer in a weight tensor, generating gradients and performing machine learning tasks and receiving a layer input by the layer and processing the layer that have the scaled updated values to generate a layer output.
LI teaches applying the respective gradient-based updates without applying any weight decay updates to the weights of the layer in the weight tensor and weight regularization and training with the initial scaled updated value as a startup weight.
Narkhede teaches scaled updated values of the weights of the layers different from the initial updated values.
One of ordinary skill would have motivation to combine ROSE, LI and Narkhede that uses various techniques for weight initialization such that the training is accelerated and the performance is improved (Narkhede[Abstract, Page 291])
In regard to claim 13: (Currently Amended)
ROSE and LI do not explicitly disclose:
wherein layer includes no biases.
However, Nakhede discloses:
- wherein layer includes no biases.
[3, Page 305]:
Paine et al. (2014) have proposed a zero-bias convolutional autoencoder (CAE) to learn the parameters from the input data in an unsupervised way.
[3, Page 305]:
Ruiz-Garcia et al. (2017) have employed stacked convolutional autoencoder (SCAE), which pre-trained the weights for CNN for the task
[3, Page 305]:
It was observed that the pre-training method using SCAE gave better accuracy and also reduced the training time.
[3, Page 305]:
Wiehman et al. (2016) have employed unsupervised pre-training on U-net architecture, which is a fully convolutional architecture for image segmentation task.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine ROSE, LI and Narkhede.
ROSE teaches weights of the layer in a weight tensor, generating gradients and performing machine learning tasks and receiving a layer input by the layer and processing the layer that have the scaled updated values to generate a layer output.
LI teaches applying the respective gradient-based updates without applying any weight decay updates to the weights of the layer in the weight tensor and weight regularization and training with the initial scaled updated value as a startup weight.
Narkhede teaches scaled updated values of the weights of the layers different from the initial updated values.
One of ordinary skill would have motivation to combine ROSE, LI and Narkhede that uses various techniques for weight initialization such that the training is accelerated and the performance is improved (Narkhede[Abstract, Page 291])
In regard to claim 21: (Currently Amended)
Sinha discloses:
- One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a neural network to perform one or more machine learning tasks, wherein the operations comprise repeatedly performing the following for a weight tensor that includes weights of the neural network
[0021]; [0022]
[0004]:
CNNs typically include many layers, such as convolution layers and fully connected layers, with data passing from one layer to the next in the form of ‘tensors’
[0062]:
At the step 610 the CNN backbone 114 receives one frame of the video frame data 113 from the video source 112 and performs specific early layers of an overall CNN, such as layers corresponding to the ‘backbone’ of the CNN. The step 610 outputs the tensors 115
[0070]:
Although the modules 170 and 174 receive tensors one at a time, incoming tensors may be grouped into batches of a size greater than one. Increasing the batch size can improve training as each weight update step is influenced by a variety of inputs.
[0068]:
A forward pass through the modules 170 and 174 is performed using weights currently present in the modules 170 and 174.
[0070]:
At the step 650 a weight update is performed in the modules 170 and 174 using a process of ‘back propagation’ whereby weights are updated based on a process, such as stochastic gradient descent (SGD), attempting to minimise the measured loss 181. The trainable bottleneck encoder 170 and the trainable bottleneck decoder 174, by virtue of ongoing weight updating due to back propagation, are able to adapt to such changes in statistics of the tensors 115, resulting in the potential for achievement of a lower MSE 181 compared to the MSE 179.
- generating respective gradient-based updates to the weights of the layer in the weight tensor, comprising:
[0062]:
At the step 610 the CNN backbone 114 receives one frame of the video frame data 113 from the video source 112 and performs specific early layers of an overall CNN, such as layers corresponding to the ‘backbone’ of the CNN. The step 610 outputs the tensors 115
[0070]:
a weight update is performed in the modules 170 and 174 using a process of ‘back propagation’ whereby weights are updated based on a process, such as stochastic gradient descent (SGD), attempting to minimise the measured loss 181.
- performing, using a plurality of training examples, a training step that comprises a plurality of forward passes through the neural network based on the weights of the layer in the weight tensor to obtain respective gradients of a loss function with respect to the weights of the layer in the weight tensor;
[0068]:
A forward pass through the modules 170 and 174 is performed using weights currently present in the modules 170 and 174.
[0005]:
An iteration of the entire training dataset forms an ‘epoch’ of training, and training typically requires multiple epochs to achieve a high level of performance for the task
[0062]:
At the step 610 the CNN backbone 114 receives one frame of the video frame data 113 from the video source 112 and performs specific early layers of an overall CNN, such as layers corresponding to the ‘backbone’ of the CNN. The step 610 outputs the tensors 115
- and generating, by using an optimizer and based on the respective gradients, respective gradient-based updates to the weights of the layer in the weight tensor;
[0062]:
At the step 610 the CNN backbone 114 receives one frame of the video frame data 113 from the video source 112 and performs specific early layers of an overall CNN, such as layers corresponding to the ‘backbone’ of the CNN. The step 610 outputs the tensors 115
[0070]:
a weight update is performed in the modules 170 and 174 using a process of ‘back propagation’ whereby weights are updated based on a process, such as stochastic gradient descent (SGD), attempting to minimise the measured loss 181.
[0007]:
When training a network, the value of the batch dimension may be increased so that multiple frames are passed through the network in each batch before the network weights are updated, according to a predetermined ‘batch size
[0070]:
Moreover, for video data the statistical variety in consecutive frames or consecutive tensors from the backbone 114 is less pronounced, reducing the benefit of using a larger batch size. Other processes to update weights may also be used, such as the ‘Adam W’ optimiser which utilises momentum and scaling and decouples weight decay from gradient update.
- applying the respective gradient-based updates generated by using the optimizer to the weights of the layer in the weight tensor to generate initial updated values of the weights of the layer in the weight tensor without applying any weight decay updates to the weights of the layer in the weight tensor
[0004]:
CNNs typically include many layers, such as convolution layers and fully connected layers, with data passing from one layer to the next in the form of ‘tensors’
[0062]:
At the step 610 the CNN backbone 114 receives one frame of the video frame data 113 from the video source 112 and performs specific early layers of an overall CNN, such as layers corresponding to the ‘backbone’ of the CNN. The step 610 outputs the tensors 115
[0070]:
Although the modules 170 and 174 receive tensors one at a time, incoming tensors may be grouped into batches of a size greater than one. Increasing the batch size can improve training as each weight update step is influenced by a variety of inputs.
[0070]:
Other processes to update weights may also be used, such as the ‘Adam W’ optimiser which utilises momentum and scaling and decouples weight decay from gradient update
- receiving a layer input by the layer and processing the layer input by the layer in accordance with the weights of the layer that have the scaled updated values to generate a layer output.
[0063]:
The step 615 receives as input at least one tensor where the neural network for a machine task has been partially performed (that is the backbone network has been implemented) and produces tensors 117.
[0063]:
the step 615 receives as input at least one tensor where the neural network for a machine task has been partially performed (that is the backbone network has been implemented) and produces tensors 117.
[0128]:
A concatenation module 524 performs a channel-wise concatenation of the tensors 505, 523, 523a, and 523b to produce concatenated tensor 525, of dimensions h, w, 512. The concatenated tensor 525 is passed to a squeeze and excitation (SE) module 526 to produce a tensor 527. The SE module 526 sequentially performs a global pooling, a fully-connected layer with reduction in channel count, a rectified linear unit activation, a second fully-connected layer restoring the channel count, and a sigmoid activation function to produce a scaling tensor.
- and processing the layer input by the layer in accordance with the weights of the layer that have the scaled updated values to generate a layer output.
[0055]:
The mask head includes two convolutional layers and produces a segmentation map for each ‘region of interest’
[0055]:
MaskRCNN may be used to perform both object detection and instance segmentation, with additional complexity in the network head due to use of mask heads.
ROSE does not explicitly disclose:
- and applying neural network weight value regularization to the weights of the layer in the weight tensor that have the initial updated values, comprising
- and setting current values of the weights of the weights of the layer in the weight tensor for a next training step to be equal to the scaled updated values, such that in the next training step comprises:
However, LI discloses in:
- and applying neural network weight value regularization to the weights of the layer in the weight tensor that have the initial updated values, comprising:
[0017]:
a method for training a neural network (NN) block in a NN by applying a trainable scaling factor on output of a binary convolution, which may help to save computational cost significantly and improve computation accuracy to approximate to a full-precision NN. A regularization function with respect to an estimated real-valued weight tensor including the scaling factor and a real-valued weight tensor is included in a loss function of the NN.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine ROSE and LI.
ROSE teaches weights of the layer in a weight tensor, generating gradients and performing machine learning tasks and receiving a layer input by the layer and processing the layer that have the scaled updated values to generate a layer output.
LI teaches applying the respective gradient-based updates without applying any weight decay updates to the weights of the layer in the weight tensor and weight regularization and training with the initial scaled updated value as a startup weight.
One of ordinary skill would have motivation to combine ROSE and LI that improved the stability of the NN with greater accuracy using scaling (LI [0017]).
ROSE and LI do not explicitly disclose:
- scaling, by using a neural network weight scaling engine, the initial updated values of the weights of the layer in the weight tensor to generate scaled updated values that are different from the initial updated values;
- and setting current values of the weights of the weights of the layer in the weight tensor for a next training step to be equal to the scaled updated values, such that in the next training step comprises:
However, Narkhede discloses:
- scaling, by using a neural network weight scaling engine
[1 Introduction, Page 294]:
The accelerated libraries like NVIDIA Compute Unified Device Architecture (CUDA), NVIDIA CUDA Deep Neural Network Library (cuDNN), Intel Math Kernel Library (MKL), Open Computing Language (OpenCL) provide optimized functions to get advantage of parallelism of GPUs. These libraries are used by deep learning frameworks like Tensorflow, [BRI: TensorFlow does contain a scaling engine to operate efficiently across heterogenous computing environments)
- the initial updated values of the weights of the layer in the weight tensor to generate scaled updated values that are different from the initial updated values;
[1 Introduction, Page 292]:
Iteratively training a neural network is solving a non-convex optimization problem parameterized over all the network weights and is done by the backpropagation,
[1 Introduction, Page 292]:
the training process to be consisting of two passes—forward and backward pass. The output is predicted using the input and the initialized weights during the forward pass. A simple thought to initialize these weights may be zero initialization. But because of this, all the layers would learn the same and during backpropagation, all the weights would get the same update.
[1 Introduction, Page 293]:
Implementing a task in deep learning involves the following steps:
• Preparing data The data is collected, balanced, pre-processed and split into a training, validation and test set.
• Choosing and building an appropriate model A model suitable for the task is selected.
• Training The model is initialized and trained on the dataset to minimize the selected loss function.
• Evaluation The performance of trained model is checked on the test dataset.
• Parameter tuning This step may be performed to improve the results if the model does not give expected performance after evaluation. The hyperparameters of the model can be tuned in this step and the model has to be retrained
[1 Introduction, Page 294]:
• Inference Once the trained model is ready, it can be used for inference.
The training step, which is given in the above steps, involves two approaches: training the model from scratch or adopting a transfer learning approach. Training deep networks from scratch is a computationally-intensive process and requires specialized hardware that is present in accelerated computing environments like Graphics Processing Units (GPUs)
[2 Initialization without pre-training, Page 296]:
[2.1 Random initialization, Page 296]:
This section discusses the initialization strategies in which weights are either initialized randomly or strategies in which weights are first randomly chosen and later updated by some method or algorithm
[2.1.2 Variance scaling based initialization, Page 298]:
various weight initialization techniques in which weights are initially selected from random distributions but later scaled so that variance of input and output layer is maintained or variance of the output layer is maintained to a desired value or variance of gradients is maintained while training.
[2.1.2 Variance scaling based initialization, Page 298]:
Yang et al. (2020) have proposed an initialization technique that introduces a norm on parameter space, which gives constraints on parameter updates. A scalar value has been derived based on norm equality and the variance of a vector of learnable parameters has been determined by using this scalar and as given by Glorot and Bengio (2010)
[2.1.2 Variance scaling based initialization, Page 298]:
Ioffe et al. (2015) have proposed a batch normalizing technique for normalizing layer activations, which brought a significant speedup in the training process.
[3 Initialization with pre-traininng, Page 304]:
Larochelle et al. (2009) have also discussed after experimentation that unsupervised weight initialization helped to select the parameters in a region of suitable local minimum and also acted as a regularization that offered good generalization.
- and setting current values of the weights of the weights of the layer in the weight tensor for a next training step to be equal to the scaled updated values, such that in the next training step comprises:
[1 Introduction, Page 293]:
Implementing a task in deep learning involves the following steps:
• Preparing data The data is collected, balanced, pre-processed and split into a training, validation and test set.
• Choosing and building an appropriate model A model suitable for the task is selected.
• Training The model is initialized and trained on the dataset to minimize the selected loss function.
• Evaluation The performance of trained model is checked on the test dataset.
• Parameter tuning This step may be performed to improve the results if the model does not give expected performance after evaluation. The hyperparameters of the model can be tuned in this step and the model has to be retrained
[1 Introduction, Page 294]:
• Inference Once the trained model is ready, it can be used for inference.
[3 Initialization with pre-training, Page 305]:
Li et al. (1993) have proposed a delta pre-training method to obtain initial weights for the backpropagation algorithm. The pre-training started by assuming initial weights to be zero. The weights obtained after pre-training the network were set as initial weights. The average number of epochs was used as a measure for convergence. The authors have
discussed that this method of weight initialization has very less probability of getting stuck in local minima as compared to random weight initialization.
[2.1.2 Variance scaling based initialization, Page 298]:
various weight initialization techniques in which weights are initially selected from random distributions but later scaled so that variance of input and output layer is maintained or variance of the output layer is maintained to a desired value or variance of gradients is maintained while training.
[2.1.1 Interval based initialization, Page 297]:
Drago et al. (1992) have obtained an expression for initializing the weights of a FFNN in an optimal way to ensure that the network did not get stuck in local minima or to avoid saturation of neurons while training. The weights were initially picked from a random uni form distribution and were then scaled with a scale factor. The authors have obtained a scale factor proportional to the magnitude of weights by computer simulations. That value of scale factor was selected where the paralyzed neuron percentage value is optimal and expected epochs to convergence were minimum. [BRI: this is using a scaled model for the next training iteration that are optimized through training and improve the performance for improved convergence)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine ROSE, LI and Narkhede.
ROSE teaches weights of the layer in a weight tensor, generating gradients and performing machine learning tasks and receiving a layer input by the layer and processing the layer that have the scaled updated values to generate a layer output.
LI teaches applying the respective gradient-based updates without applying any weight decay updates to the weights of the layer in the weight tensor and weight regularization and training with the initial scaled updated value as a startup weight.
Narkhede teaches scaled updated values of the weights of the layers different from the initial updated values.
One of ordinary skill would have motivation to combine ROSE, LI and Narkhede that uses various techniques for weight initialization such that the training is accelerated and the performance is improved (Narkhede[Abstract, Page 291])
Claims 7-9 and 17-19 are rejected under 35 U.S.C. 103 unpatentable over
James Christopher ROSEWARNE et.al. (hereinafter ROSE) US 2026/0075209 A1,
in view of Xinlin LI et.al. (hereinafter LI) US 2020/0097818 A1.
in view of Meenal V. Narkhede et.al. (hereinafter Narkhede), Artificial Intelligence Review (2022) 55:291–322.
further in view of Dan Popescu et.al. (hereinafter Popescu) US 2024/0256875 A1.
In regard to claim 7: (Previously Presented)
ROSE, LI and Narkhede do not explicitly disclose:
- wherein applying the optimizer to the respective gradients to generate respective gradient-
based updates comprises: to generate respective gradient-based updates comprises, for each weight in the weight tensor: computing the respective gradient-based update to the weight based on one or more moments for the weight.
However, Popescu discloses in:
- computing the respective gradient-based update to the weight based on one or more moments for the weight.
[0055]:
In the recurrent form 307, the dense architecture is modified to include as input intermediary weights up to time t for t=1, . . . , T, as shown in FIG. 3. The output of the last layer is the sought after T × k tensor of weights, such as portfolio weights in some embodiments,
[0062]:
the optimization model 302 can perform a model fitting process, such as the Adam optimization algorithm.
[0062]:
The model fitting process used by the optimization model 302, however, can adapt the parameter learning rates in real time based on the average of the first and second moments by calculating an exponential moving average of the gradient as well as the squared gradient. For example, the aggregate gradients at time t, m.sub.t can be expressed as shown in Equation (1) below.
PNG
media_image1.png
75
447
media_image1.png
Greyscale
where L is the loss function and θ.sub.t is the parameter in the neural network.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine ROSE, LI , Narkhede and Popescu.
ROSE teaches weights of the layer in a weight tensor, generating gradients and performing machine learning tasks and receiving a layer input by the layer and processing the layer that have the scaled updated values to generate a layer output.
LI teaches applying the respective gradient-based updates without applying any weight decay updates to the weights of the layer in the weight tensor and weight regularization and training with the initial scaled updated value as a startup weight.
Narkhede teaches scaled updated values of the weights of the layers different from the initial updated values.
Popescu teaches moment, and exponential decay rate.
One of ordinary skill would have motivation to combine ROSE, LI, Narkhede and Popescu that can provide a local optimization to reduce the computation cost (Popescu [0087]).
In regard to claim 8: (Previously Presented)
ROSE, LI and Narkhede do not explicitly disclose:
- wherein computing the respective gradient-based update to the weight based on one or more moments for the weight comprises: computing a square root over a difference between one and a square of a first exponential decay rate;
However, Popescu discloses in:
- computing a square root over a difference between one and a square of a first exponential decay rate;
[0055]:
In the recurrent form 307, the dense architecture is modified to include as input intermediary weights up to time t for t=1, . . . , T, as shown in FIG. 3. The output of the last layer is the sought after T × k tensor of weights, such as portfolio weights in some embodiments,
[0062]:
the optimization model 302 can perform a model fitting process, such as the Adam optimization algorithm.
[0062]:
The model fitting process used by the optimization model 302, however, can adapt the parameter learning rates in real time based on the average of the first and second moments by calculating an exponential moving average of the gradient as well as the squared gradient. For example, the aggregate gradients at time t, m.sub.t can be expressed as shown in Equation (1) below.
PNG
media_image1.png
75
447
media_image1.png
Greyscale
where L is the loss function and θ.sub.t is the parameter in the neural network.
[0063]:
In addition to the cumulative sum of gradients, the model fitting process also takes the moving weighted average of squared gradients v.sub.t which can be expressed as shown in Equation (2) below.
PNG
media_image2.png
77
463
media_image2.png
Greyscale
[0064]:
The parameters β.sub.1 and β.sub.2 can control the decay rates of both moving averages. In some embodiments, default values can be used, such as β.sub.1=0.9 and β.sub.1=0.999. Since, m.sub.t and v.sub.t can both be initialized as 0, they can be biased towards 0 as both β.sub.1 and β.sub.2 close to 1. The optimization model 302 addresses this problem by computing bias-corrected m.sub.t and v.sub.t. This bias-correction is also performed to control the weights while reaching the global minimum to prevent high oscillations when near the global minimum, which can be expressed as shown in Equation (3) below.
PNG
media_image3.png
95
511
media_image3.png
Greyscale
[0065]:
The optimization model 302, as part of the model fitting process, then takes the bias-corrected weight parameters {circumflex over (m)}.sub.t and î.sub.t to update parameters in the neural networks, which can be expressed as shown in Equation (4) below.
PNG
media_image4.png
88
485
media_image4.png
Greyscale
where lr is the learning rate, which can have a default of 0.001, and ε is a small positive constant, such as 10.sup.−8, for numerical stability,
[0067]:
the third and fourth layers, the loss layer 308 and the metric layer 310, are custom parameter-free layers used to record the loss function value and any relevant metrics, respectively. Incidentally, these custom layers are included in the optimization model 302 to allow for flexibility in defining arbitrary tensor-based loss functions and metrics operating on weights (such as portfolio weights) or differences in weights corresponding to the optimization objective and constraints without changing the architecture of the neural network.
[0066]:
The optimization model 302 executes backpropagation coupled with the above optimization process in order to find the neural network trainable weights that minimize its loss function. The loss function of the optimization model 304 is set to the objective function {tilde over (f)}. The output of the neural network is the set of portfolio weights W*=argmin.sub.w {tilde over (f)}(W). In some embodiments, input information, such as market input information, is already embedded in {tilde over (f)} and, therefore, the optimization model 302 fits a neural network specific to the objective.
In regard to claim 9: (Previously Presented)
ROSE, LI and Narkhede do not explicitly disclose:
- wherein computing the square root over the difference between one and the square of the first exponential decay rate comprises: determining a value of the first exponential decay rate based on a predetermined schedule
However, Popescu discloses:
- determining a value of the first exponential decay rate based on a predetermined schedule
[0064]:
The parameters β.sub.1 and β.sub.2 can control the decay rates of both moving averages. In some embodiments, default values can be used, such as β.sub.1=0.9 and β.sub.1=0.999. Since, m.sub.t and v.sub.t can both be initialized as 0, they can be biased towards 0 as both β.sub.1 and β.sub.2 close to 1. The optimization model 302 addresses this problem by computing bias-corrected m.sub.t and v.sub.t. This bias-correction is also performed to control the weights while reaching the global minimum to prevent high oscillations,
[0115]:
in some embodiments, performing the model fitting includes adapting parameter learning rates in real time based on an average of first and second moments by calculating an exponential moving average of a gradient and a moving average of a squared gradient, controlling decay rates of the exponential moving average of the gradient and the moving average of the squared gradient, bias correcting one or more weight parameters, and updating the initial weights using the bias corrected one or more weight parameters. As just one example of this disclosure, performing the training and optimization process can include performing a multi-period portfolio optimization, wherein the plurality of inputs includes returns data, risks data, volumes data, and initial weights from one or more feeder models, and wherein the goal of the optimization model is to output multi-period weights for a defined time period that maximize returns for a given forecasting based on market impact predictions.
[0066]:
let this input x.sub.0 vary per optimization problem with the number of periods in the multi-period optimization, so the capacity of the network can organically increase with the difficulty of the problem (more time periods can use a more complicated network). Therefore, the input is defined to be the time vector for the problem corresponding to the T periods in the multi-period optimization, thus x.sub.0=(1, . . . , T). This is akin to inputting a vector of states to the optimization problem, where the only state information is the time period. As optimization problems increase in difficulty, additional information can be incorporated into these states.
In regard to claim 17: (Previously Presented)
ROSE, LI and Narkhede do not explicitly disclose:
- wherein applying the optimizer to the respective gradients to generate respective gradient-
based updates comprises: to generate respective gradient-based updates comprises, for each weight in the weight tensor: computing the respective gradient-based update to the weight based on one or more moments for the weight.
However, Popescu discloses:
- computing the respective gradient-based update to the weight based on one or more moments for the weight.
[0055]:
In the recurrent form 307, the dense architecture is modified to include as input intermediary weights up to time t for t=1, . . . , T, as shown in FIG. 3. The output of the last layer is the sought after T × k tensor of weights, such as portfolio weights in some embodiments,
[0062]:
the optimization model 302 can perform a model fitting process, such as the Adam optimization algorithm.
[0062]:
The model fitting process used by the optimization model 302, however, can adapt the parameter learning rates in real time based on the average of the first and second moments by calculating an exponential moving average of the gradient as well as the squared gradient. For example, the aggregate gradients at time t, m.sub.t can be expressed as shown in Equation (1) below.
PNG
media_image1.png
75
447
media_image1.png
Greyscale
where L is the loss function and θ.sub.t is the parameter in the neural network.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine ROSE, LI , Narkhede and Popescu.
ROSE teaches weights of the layer in a weight tensor, generating gradients and performing machine learning tasks and receiving a layer input by the layer and processing the layer that have the scaled updated values to generate a layer output.
LI teaches applying the respective gradient-based updates without applying any weight decay updates to the weights of the layer in the weight tensor and weight regularization and training with the initial scaled updated value as a startup weight.
Narkhede teaches scaled updated values of the weights of the layers different from the initial updated values.
Popescu teaches moment, and exponential decay rate.
One of ordinary skill would have motivation to combine ROSE, LI, Narkhede and Popescu that can provide a local optimization to reduce the computation cost (Popescu [0087]).
In regard to claim 18: (Previously Presented)
ROSE and LI do not explicitly disclose:
- wherein computing the respective gradient-based update to the weight based on one or more moments for the weight comprises: computing a square root over a difference between one and a square of a first exponential decay rate;
However, Popescu discloses:
- computing a square root over a difference between one and a square of a first exponential decay rate;
[0055]:
In the recurrent form 307, the dense architecture is modified to include as input intermediary weights up to time t for t=1, . . . , T, as shown in FIG. 3. The output of the last layer is the sought after T × k tensor of weights, such as portfolio weights in some embodiments,
[0062]:
the optimization model 302 can perform a model fitting process, such as the Adam optimization algorithm.
[0062]:
The model fitting process used by the optimization model 302, however, can adapt the parameter learning rates in real time based on the average of the first and second moments by calculating an exponential moving average of the gradient as well as the squared gradient. For example, the aggregate gradients at time t, m.sub.t can be expressed as shown in Equation (1) below.
PNG
media_image1.png
75
447
media_image1.png
Greyscale
where L is the loss function and θ.sub.t is the parameter in the neural network.
[0063]:
In addition to the cumulative sum of gradients, the model fitting process also takes the moving weighted average of squared gradients v.sub.t which can be expressed as shown in Equation (2) below.
PNG
media_image2.png
77
463
media_image2.png
Greyscale
[0064]:
The parameters β.sub.1 and β.sub.2 can control the decay rates of both moving averages. In some embodiments, default values can be used, such as β.sub.1=0.9 and β.sub.1=0.999. Since, m.sub.t and v.sub.t can both be initialized as 0, they can be biased towards 0 as both β.sub.1 and β.sub.2 close to 1. The optimization model 302 addresses this problem by computing bias-corrected m.sub.t and v.sub.t. This bias-correction is also performed to control the weights while reaching the global minimum to prevent high oscillations when near the global minimum, which can be expressed as shown in Equation (3) below.
PNG
media_image3.png
95
511
media_image3.png
Greyscale
[0065]:
The optimization model 302, as part of the model fitting process, then takes the bias-corrected weight parameters {circumflex over (m)}.sub.t and î.sub.t to update parameters in the neural networks, which can be expressed as shown in Equation (4) below.
PNG
media_image4.png
88
485
media_image4.png
Greyscale
where lr is the learning rate, which can have a default of 0.001, and ε is a small positive constant, such as 10.sup.−8, for numerical stability,
[0067]:
the third and fourth layers, the loss layer 308 and the metric layer 310, are custom parameter-free layers used to record the loss function value and any relevant metrics, respectively. Incidentally, these custom layers are included in the optimization model 302 to allow for flexibility in defining arbitrary tensor-based loss functions and metrics operating on weights (such as portfolio weights) or differences in weights corresponding to the optimization objective and constraints without changing the architecture of the neural network.
[0066]:
The optimization model 302 executes backpropagation coupled with the above optimization process in order to find the neural network trainable weights that minimize its loss function. The loss function of the optimization model 304 is set to the objective function {tilde over (f)}. The output of the neural network is the set of portfolio weights W*=argmin.sub.w {tilde over (f)}(W). In some embodiments, input information, such as market input information, is already embedded in {tilde over (f)} and, therefore, the optimization model 302 fits a neural network specific to the objective.
In regard to claim 19: (Previously Presented)
ROSE, LI, Narkhede do not explicitly disclose:
- wherein computing the square root over the difference between one and the square of the first exponential decay rate comprises: determining a value of the first exponential decay rate based on a predetermined schedule
However, Popescu discloses:
- determining a value of the first exponential decay rate based on a predetermined schedule
[0064]:
The parameters β.sub.1 and β.sub.2 can control the decay rates of both moving averages. In some embodiments, default values can be used, such as β.sub.1=0.9 and β.sub.1=0.999. Since, m.sub.t and v.sub.t can both be initialized as 0, they can be biased towards 0 as both β.sub.1 and β.sub.2 close to 1. The optimization model 302 addresses this problem by computing bias-corrected m.sub.t and v.sub.t. This bias-correction is also performed to control the weights while reaching the global minimum to prevent high oscillations,
[0115]:
in some embodiments, performing the model fitting includes adapting parameter learning rates in real time based on an average of first and second moments by calculating an exponential moving average of a gradient and a moving average of a squared gradient, controlling decay rates of the exponential moving average of the gradient and the moving average of the squared gradient, bias correcting one or more weight parameters, and updating the initial weights using the bias corrected one or more weight parameters. As just one example of this disclosure, performing the training and optimization process can include performing a multi-period portfolio optimization, wherein the plurality of inputs includes returns data, risks data, volumes data, and initial weights from one or more feeder models, and wherein the goal of the optimization model is to output multi-period weights for a defined time period that maximize returns for a given forecasting based on market impact predictions.
[0066]:
let this input x.sub.0 vary per optimization problem with the number of periods in the multi-period optimization, so the capacity of the network can organically increase with the difficulty of the problem (more time periods can use a more complicated network). Therefore, the input is defined to be the time vector for the problem corresponding to the T periods in the multi-period optimization, thus x.sub.0=(1, . . . , T). This is akin to inputting a vector of states to the optimization problem, where the only state information is the time period. As optimization problems increase in difficulty, additional information can be incorporated into these states.
Claims 4-6, and 14-16 are rejected under 35 U.S.C. 103 unpatentable over
James Christopher ROSEWARNE et.al. (hereinafter ROSE) US 2026/0075209 A1,
in view of Xinlin LI et.al. (hereinafter LI) US 2020/0097818 A1.
in view of Meenal V. Narkhede et.al. (hereinafter Narkhede), Artificial Intelligence Review (2022) 55:291–322.
further in view Zeev Abraham et.al. (hereinafter Abraham) US 2020/0175678 A1.
In regard to claim 4: (Previously Presented)
ROSE discloses:
- wherein training the neural network comprises, for the weight tensor that includes the weights of the neural network:
[0070]:
Although the modules 170 and 174 receive tensors one at a time, incoming tensors may be grouped into batches of a size greater than one. Increasing the batch size can improve training as each weight update step is influenced by a variety of inputs. [BRI: In a CNN, the tensors do represent weight tensors that are learnable parameters inside each later where the weight acts as a filter to extract feature from the input data]
ROSE , LI and Narkhede do not explicitly disclose:
- determining a fan-in value of the weight tensor, the fan-in value representing a total number of input values on which the weights in the weight tensor are to be applied;
- and determining initial values of the weights in the weight tensor based on the fan-in value and a predetermined distribution.
However, Abraham discloses:
- determining a fan-in value of the weight tensor, the fan-in value representing a total number of input values on which the weights in the weight tensor are to be applied;
[0047]:
For a CNN, the convolution is performed on the input data by means of a filter or kernel. The present system and method can employ 4D kernels of a volume equal to the shape (depth× height × width × num_channels) where num_channels is the number of channels in the output of a previous layer (last tensor dimension), and the depth-width-height are obtained from the convolution specification used in the method. The convolution method can sample all input channels, and can be computed for each spatial location or voxel in an image volume, or only at voxels where kernel samples are valid. In this method, activation is a function applied element-wise to an input tensor.
[0178]:
The input volume is scaled to have a specified voxel size and is normalized to range [0, 1.0] before next steps. All trainable layers use weights obtained during the training. The model outputs 2 floating point numbers for each input voxel. For an input shape D×H×W output shape will be D×H×W×2.
[BRI: Perhaps as known to a POSITA, W x H x C represents the fan-in and the tensor shape is represented by tensor Shape=(N, W, H, D) where N is the num_channels)
- and determining initial values of the weights in the weight tensor based on the fan-in value and a predetermined distribution.
[0286]:
1. The weights in the trained neural network are initialized. An exemplary algorithm uses “Glorot uniform” initialization, in which each weight is initialized with a small Gaussian value with mean=0.0 and variance based on the fan-in and fan-out of the weight
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine ROSE, LI , Narkhede and Abraham.
ROSE teaches weights of the layer in a weight tensor, generating gradients and performing machine learning tasks and receiving a layer input by the layer and processing the layer that have the scaled updated values to generate a layer output.
LI teaches applying the respective gradient-based updates without applying any weight decay updates to the weights of the layer in the weight tensor and weight regularization and training with the initial scaled updated value as a startup weight.
Narkhede teaches scaled updated values of the weights of the layers different from the initial updated values.
Abraham teaches fan-in for weights.
One of ordinary skill would have motivation to combine ROSE, LI, Narkhede and Abraham that can provide enhanced efficiency and accuracy of the ML results (Abraham [0046]).
In regard to claim 5: (Previously Presented)
ROSE , LI and Narkhede do not explicitly disclose:
- wherein training the neural network comprises: determining a target norm based on the fan-in value of the weight tensor.
However, Abraham discloses in:
- wherein training the neural network comprises: determining a target norm based on the fan-in value of the weight tensor.
[0232]:
The jaws segmentation algorithm accepts volumetric data (3-dimensional tensor) of any size but expects specific resolution. Voxel size should be 0.5 mm. Input volume is scaled to have a specified voxel size and is normalized to range [0, 1.0] before next steps. All trainable layers use weights obtained during the training. [BRI: Perhaps known to a POSITA that a target nom on the fan-in of weight tensor is a “scaling factor for initializing or normalizing the weights]
In regard to claim 6: (Previously Presented)
ROSE , LI and Narkhede do not explicitly disclose:
- scaling the initial updated values of the weights in the weight tensor to generate scaled updated values to have the target norm.
However, Abraham discloses in:
- scaling the initial updated values of the weights in the weight tensor to generate scaled updated values to have the target norm.
[0050]:
Batch normalization is a method of initializing neural networks by explicitly forcing the activations throughout the network to take on a unit gaussian distribution at the beginning of the training. This operation improves network convergence.
[0286]:
1. The weights in the trained neural network are initialized. An exemplary algorithm uses “Glorot uniform” initialization, in which each weight is initialized with a small Gaussian value with mean=0.0 and variance based on the fan-in and fan-out of the weight.
[0178]:
The input volume is scaled to have a specified voxel size and is normalized to range [0, 1.0] before next steps. All trainable layers use weights obtained during the training.
[0003]:
One method of imaging a patient's dental region is using a dental cone beam CT (CBCT), which can be used to generate multi-dimensional images (e.g., 3-dimensional or 3D images) of the region
126] Uses a Geodesic Active Contours algorithm to refine segmentation of each tooth using the original CBCT data.
[0124] Images of teeth roots often have low intensity, and this step takes the low intensity into account by applying different processing parameters to the roots/crown area [BRI: intensity does represent the target norm where the image of the teeth roots are the target)
In regard to claim 14: (Previously Presented)
ROSE discloses:
- wherein training the neural network comprises, for the weight tensor that includes the weights of the neural network:
[0070]:
Although the modules 170 and 174 receive tensors one at a time, incoming tensors may be grouped into batches of a size greater than one. Increasing the batch size can improve training as each weight update step is influenced by a variety of inputs. [BRI: In a CNN, the tensors do represent weight tensors that are learnable parameters inside each later where the weight acts as a filter to extract feature from the input data]
ROSE , LI and Narkhede do not explicitly disclose:
- determining a fan-in value of the weight tensor, the fan-in value representing a total number of input values on which the weights in the weight tensor are to be applied;
- and determining initial values of the weights in the weight tensor based on the fan-in value and a predetermined distribution.
However, Abraham discloses:
- determining a fan-in value of the weight tensor, the fan-in value representing a total number of input values on which the weights in the weight tensor are to be applied;
[0047]:
For a CNN, the convolution is performed on the input data by means of a filter or kernel. The present system and method can employ 4D kernels of a volume equal to the shape (depth× height × width × num_channels) where num_channels is the number of channels in the output of a previous layer (last tensor dimension), and the depth-width-height are obtained from the convolution specification used in the method. The convolution method can sample all input channels, and can be computed for each spatial location or voxel in an image volume, or only at voxels where kernel samples are valid. In this method, activation is a function applied element-wise to an input tensor.
[0178]:
The input volume is scaled to have a specified voxel size and is normalized to range [0, 1.0] before next steps. All trainable layers use weights obtained during the training. The model outputs 2 floating point numbers for each input voxel. For an input shape D×H×W output shape will be D×H×W×2.
[BRI: Perhaps as known to a POSITA, W x H x C represents the fan-in and the tensor shape is represented by tensor Shape=(N, W, H, D) where N is the num_channels)
- and determining initial values of the weights in the weight tensor based on the fan-in value and a predetermined distribution.
[0286]:
1. The weights in the trained neural network are initialized. An exemplary algorithm uses “Glorot uniform” initialization, in which each weight is initialized with a small Gaussian value with mean=0.0 and variance based on the fan-in and fan-out of the weight
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine ROSE, LI , Narkhede and Abraham.
ROSE teaches weights of the layer in a weight tensor, generating gradients and performing machine learning tasks and receiving a layer input by the layer and processing the layer that have the scaled updated values to generate a layer output.
LI teaches applying the respective gradient-based updates without applying any weight decay updates to the weights of the layer in the weight tensor and weight regularization and training with the initial scaled updated value as a startup weight.
Narkhede teaches scaled updated values of the weights of the layers different from the initial updated values.
Abraham teaches fan-in for weights.
One of ordinary skill would have motivation to combine ROSE, LI, Narkhede and Abraham that can provide enhanced efficiency and accuracy of the ML results (Abraham [0046]).
In regard to claim 15: (Previously Presented)
ROSE , LI and Narkhede do not explicitly disclose:
- wherein training the neural network comprises: determining a target norm based on the fan-in value of the weight tensor.
However, Abraham discloses in:
- wherein training the neural network comprises: determining a target norm based on the fan-in value of the weight tensor.
[0232]:
The jaws segmentation algorithm accepts volumetric data (3-dimensional tensor) of any size but expects specific resolution. Voxel size should be 0.5 mm. Input volume is scaled to have a specified voxel size and is normalized to range [0, 1.0] before next steps. All trainable layers use weights obtained during the training. [BRI: Perhaps known to a POSITA that a target nom on the fan-in of weight tensor is a “scaling factor for initializing or normalizing the weights]
In regard to claim 16: (Previously Presented)
ROSE , LI and Narkhede do not explicitly disclose:
- scaling the initial updated values of the weights in the weight tensor to generate scaled updated values to have the target norm.
However, Abraham discloses in:
- scaling the initial updated values of the weights in the weight tensor to generate scaled updated values to have the target norm.
[0050]:
Batch normalization is a method of initializing neural networks by explicitly forcing the activations throughout the network to take on a unit gaussian distribution at the beginning of the training. This operation improves network convergence.
[0286]:
1. The weights in the trained neural network are initialized. An exemplary algorithm uses “Glorot uniform” initialization, in which each weight is initialized with a small Gaussian value with mean=0.0 and variance based on the fan-in and fan-out of the weight.
[0178]:
The input volume is scaled to have a specified voxel size and is normalized to range [0, 1.0] before next steps. All trainable layers use weights obtained during the training.
[0003]:
One method of imaging a patient's dental region is using a dental cone beam CT (CBCT), which can be used to generate multi-dimensional images (e.g., 3-dimensional or 3D images) of the region
126] Uses a Geodesic Active Contours algorithm to refine segmentation of each tooth using the original CBCT data.
[0124] Images of teeth roots often have low intensity, and this step takes the low intensity into account by applying different processing parameters to the roots/crown area [BRI: intensity does represent the target norm where the image of the teeth roots are the target)
Claims 10 and 20 are rejected under 35 U.S.C. 103 unpatentable over
James Christopher ROSEWARNE et.al. (hereinafter ROSE) US 2026/0075209 A1,
in view of Xinlin LI et.al. (hereinafter LI) US 2020/0097818 A1,
further in view of Yongxiong REN et.al. (hereinafter REN) US 2022/0310068 A1.
In regard to claim 10: (Previously Presented)
ROSE , LI and Narkhede do not explicitly disclose:
- wherein the neural network comprises a Transformer neural network and the weights in the weight tensor are associated with an attention layer of the Transformer neural network.
However, REN discloses in:
- wherein the neural network comprises a Transformer neural network and the weights in the weight tensor are associated with an attention layer of the Transformer neural network.
[0039]:
the neural network may use a transformer architecture, as shown in FIG. 9. FIG. 9 illustrates a speech transformer model
In [0039]:
The transformer may include an encoder and a decoder. The encoder may include a plurality of encoder layers (301-1, 301-2, . . . , 301-N) and the decoder may include a plurality of decoder layers 401-j, where j may be a positive integer between 1 and M, and M and N are positive integers.
In [0039]:
Each encoder layer may also include multiple add & norm layers. Each decoder layer may also include a multi-head cross attention module, a multi-head self-attention module, a feed forward module, and a plurality of add & norm layers.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine ROSE, LI , Narkhede and REN.
ROSE teaches weights of the layer in a weight tensor, generating gradients and performing machine learning tasks and receiving a layer input by the layer and processing the layer that have the scaled updated values to generate a layer output.
LI teaches applying the respective gradient-based updates without applying any weight decay updates to the weights of the layer in the weight tensor and weight regularization and training with the initial scaled updated value as a startup weight.
Narkhede teaches scaled updated values of the weights of the layers different from the initial updated values.
REN teaches transformer neural network.
One of ordinary skill would have motivation to combine ROSE, LI, Narkhede and REN that can provide increased accuracy using matrix set up for fan in (REN [0063])
In regard to claim 20: (Previously Presented)
ROSE , LI and Narkhede do not explicitly disclose:
- wherein the neural network comprises a Transformer neural network and the weights in the weight tensor are associated with an attention layer of the Transformer neural network.
However, REN discloses:
- wherein the neural network comprises a Transformer neural network and the weights in the weight tensor are associated with an attention layer of the Transformer neural network.
[0039]:
the neural network may use a transformer architecture, as shown in FIG. 9. FIG. 9 illustrates a speech transformer model
[0039]:
The transformer may include an encoder and a decoder. The encoder may include a plurality of encoder layers (301-1, 301-2, . . . , 301-N) and the decoder may include a plurality of decoder layers 401-j, where j may be a positive integer between 1 and M, and M and N are positive integers.
In [0039]:
Each encoder layer may also include multiple add & norm layers. Each decoder layer may also include a multi-head cross attention module, a multi-head self-attention module, a feed forward module, and a plurality of add & norm layers.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine ROSE, LI , Narkhede and REN.
ROSE teaches weights of the layer in a weight tensor, generating gradients and performing machine learning tasks and receiving a layer input by the layer and processing the layer that have the scaled updated values to generate a layer output.
LI teaches applying the respective gradient-based updates without applying any weight decay updates to the weights of the layer in the weight tensor and weight regularization and training with the initial scaled updated value as a startup weight.
Narkhede teaches scaled updated values of the weights of the layers different from the initial updated values.
REN teaches transformer neural network.
One of ordinary skill would have motivation to combine ROSE, LI, Narkhede and REN that can provide increased accuracy using matrix set up for fan in (REN [0063])
Conclusion
Any inquiry concerning this communication or earlier communications from the
examiner should be directed to TIRUMALE KRISHNASWAMY RAMESH whose telephone number is (571)272-4605. The examiner can normally be reached by phone.
Examiner interviews are available via telephone, in-person, and video conferencing
using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at
http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached on phone (571-272-3768). The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be
obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit:
https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for
information about filing in DOCX format.
For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TIRUMALE K RAMESH/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights