Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
The arguments filed 06/17/2025 have been entered. Claims 1-21 remain pending in the application.
Applicant’s arguments, with respect to claim rejections of claim(s) 1-21 under 35 U.S.C 103 filed 03/26/2025 have been considered and are not persuasive. Therefore, the previous rejections as set forth in the previous office action will be maintain.
The applicant argues that Hinton does not disclose obtaining a simple neural network while compromising a prediction accuracy to a certain degree. Hinton does not state at all that an accuracy can be decreased.
Moreover, although the Office Action asserted "... it would have been understood ... that such regularization term may as well be applied for the Teacher network ..." (page 9 of the Office Action), one of ordinary skill in the art who reviews Hegde, which discloses a distillation technique, would not have been motivated to arrive at such an idea. Hegde has absolutely no concept of training (updating) Teacher NN while Student NN is being generated. In this point, the Office Action's assertion is unreasonable and has no technical or factual basis. As such, Applicant submits that there is no motivation nor suggestion to combine Hegde and Hinton.
Further, there is no motivation or suggestion to further combine Brothers with Hilton and Hegde. Hegde discloses a distillation technique as to how Student NN is generated from Teacher NN. Teacher NN and Student NN in Hegde is in a relation that Student NN is obtained from Teacher NN through disclosed training (e.g., using a regularization term). Although the Office Action asserted that the relation between Teacher NN and Student NN becomes line the first neural network and the second neural network shown in Brothers, such a statement means nothing but to deny the entire disclosure of Hegde and obtain Student NN. As such, the Office Action's assertion has no technical or factual basis.
Further, Applicant submits that in the field of machine learning, the term "abstraction layer" is not a common term, and neither Hegde nor Hinton discloses or even suggest "an abstraction layer." Accordingly, it is inappropriate to refer to the abstraction layer in rejecting the claims based in Hilton and Hegde. Moreover, Applicant submits that Hedge cannot view an abstraction layer as a portion of a network including many layers. Although the Office Action asserted "the abstraction layer comprises the feature to derive a Student network from a Teacher network by Hegde" (page 5 of the Office Action), a layer that is a portion of a neural network cannot derive a network. A layer of a neural network merely outputs data obtained by processing input data, and has no feature of deriving a network. Furthermore, even if an abstraction layer may possibly correspond to Student NN, a portion of an entire network (Teacher NN) is a result of distillation, and thus Student NN makes a logical leap.
Even if, arguendo, with regard to Hegde, training of Student NN were considered as corresponding to retraining using a regularization term, in such a case, Hegde fails to teach a step equivalent to the generating of the second neural network recited in the claims of the present application.
The examiner respectfully disagrees. While Hinton does not explicitly state that accuracy may be compromised or decreased, this argument is not persuasive. Hinton explicitly teaches training a classification NN with multiple loss terms (classification and regularization soft nearest neighbor loss). The teaching is directed toward improving generalization and efficiency of training. Hinton discloses iteratively training at paragraph 76, suggesting the retraining of the same network to balance classification accuracy against additional representational objectives. The claim at issues do not recite “compromising accuracy” or “decreasing accuracy”. Applicant’s reliance on this point is misplaced because obviousness analysis does not require that prior art disclosed disadvantages, but only that it teaches the claimed features or provides reason to combine. The motivation to combine with Hegde remains appropriate. Hinton teaches retraining with additional loss functions to improve accuracy. Hegde teaches knowledge distillation to generate a simple student NN from a teacher NN. One of ordinary skilled in the art would have recognized that combining Hinton’s NN training techniques with Hegde achieves the well-known goals of both training accuracy and compressed/sparser neural network model.
Further, applicant’s argument improperly requires Hegde alone to discloses retraining of the teacher NN. The rejections rely on the combination of Hinton and Hegde, wherein Hinton already teaches iterative training, while Hegde teaches generating a student NN from a teacher NN at paragraph 44. One of ordinary skilled in the art would look to Hinton to supply the retraining aspect and Hegde to supply the teacher/student NN (first and second NN). One of ordinary skilled in the art would seek to improve the accuracy of teacher/student NN would have been motivated to apply Hinton’s retraining methods in the context of Hegde’s distillation framework. The claim also does not recite simultaneously update the first NN during the generation of the second NN. This argument mischaracterizes the claim. Rather, the claim expressly recites sequential steps: (1) training a first NN, (2) generating a second NN from the first NN, and (3) re-training the first NN using a second loss function after step (2). Therefore, applicant’s reliance on the absence of “updating while generating” is misplaced, as the claims require no such concept.
Further, applicant’s assertion that combining Brothers with Hinton/Hegde would deny Hegde’s disclosure is not persuasive. Brother discloses parameter variation and pruning techniques to optimize NN. These techniques are complementary to further optimize performance of training a NN. One of ordinary skilled in the art would have been able to recognize that these techniques may be applied sequentially to achieve improved efficiency and accuracy in training any NNs. Applicant has not identified any explicit teaching away in Hegde or Brothers. Neither reference suggests that retraining, distillation and pruning must be used in isolation. Optimization and compression of NNs using multiple techniques were well known to one of ordinary skilled in the art.
Further, while applicant is correct that the phrase “abstraction layer” is not used by Hinton or Hegde, the obviousness inquiry does not require verbatim terminology. Hegde discloses transferring relevant information from a teacher NN to a student NN. This inherently involves using internal feature representation (neural network layers) of the teacher NN – which correspond to the abstraction concept. Applicant argues a layer “merely outputs data” and cannot derive a network. However, Hegde’s disclosure of using features of a teacher NN to generate a student NN shows exactly that. The KD framework generated a compact and sparse student network based on the teacher network. To one of ordinary skilled in the art, such technique of the framework constitutes employing an abstraction layer to perform the technique. In other words, Hegde’s teaching of distilling a student NN from the teacher NN based on the teacher NN’s internal feature inherently performs the function of an abstraction layer. The term “abstraction layer” is also not used in the rejections but to demonstrate how one of ordinary skilled in the art may combine the teaching in view of each other.
Finally, as mentioned above, the rejection is based on the combination of Hinton in view of Hegde. Hinton clearly discloses retraining the NN using an additional loss as a regularization term. Hinton provides the framework in which a first loss (cross entropy loss/classification loss) is added with a second term (soft nearest neighbor loss) to obtain a second loss function. This satisfies the “retraining with a second loss function” aspect of the claim. Hegde then discloses deriving a student NN from the teacher NN through knowledge distillation, which satisfy the “generating a second NN from the first NN” and further provide the block sparse regularization technique as the regularization term to the first loss function (cross entropy loss) to obtain the final loss and also provide the benefit of the block sparse regularization technique (para 50). One of ordinary skilled in the art would have been able to substitute or further include this block sparse regularization term technique (Hegde) with the soft nearest neighbor loss technique (Hinton) to obtain such benefit, which lead to an improvement in training NN with relate to weights parameter manipulation.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5 are rejected under 35 U.S.C. 103 as being unpatentable in view of Hinton et.al (US 20220101624 A1), further in view of Hegde et.al (US 20200387782 A1), further in view of Brothers et.al (US 20160358070 A1)
Regarding claim 1,
Hinton teaches the 1st limitation “(1) training a first neural network having a first parameter, using a first loss function for optimization” (paragraph 0039, where Hinton discloses “training system that trains a classification neural network”, paragraph 0013 “In some implementations, the method further includes determining a classification loss ... classification neural network parameters” and paragraph 0014 “In some implementations, the classification loss includes a cross-entropy loss”. Hinton discloses first training a classification neural network having classification neural network parameters and determining a classification loss, which includes a cross-entropy loss as the first loss function for optimization.)
Hinton teaches the 3rd limitation “(3) re-training the first neural network using a second loss function for optimization, after (2), the second loss function being obtained by adding a regularization term to the first loss function” (paragraph 64 “The discriminative training system 200 trains the classification neural network 202 over multiple training iterations on a set of training data that includes multiple training examples.”, paragraph 54 “The soft nearest neighbor loss may regularize the classification neural network by encouraging intermediate representations of network inputs to characterize class-independent features capturing information that improves classification accuracy. The soft nearest neighbor loss may be added as an additional term to a classification loss” and paragraph 65 “After processing the current batch of network inputs 204 from the training data, the discriminative training system 200 updates the current parameter values of the classification neural network using one or both of: (i) a soft nearest neighbor loss 212, and (ii) a classification loss 214.” Hinton discloses the system perform training the classification neural network over multiple training iterations, thus suggest the retraining of the classification neural network in many iterations. A person ordinary skilled in the art would have been able to configure such that the first training iteration of the classification neural network comprise a cross-entropy loss, then in the next iteration of training the classification neural network, which suggest the claimed process of retraining of the first neural network, the classification neural network can be trained with a soft nearest neighbor loss added as an additional term to a classification loss to regularize the classification neural network, wherein the soft nearest neighbor loss is used to update the current parameter values of the classification neural network, which suggest the claimed process to optimize the first neural network using a second loss function. The whole process may occur after the process to obtain another neural network as disclosed by another reference by Hegde in combination with the teaching by Hinton disclosed below.)
Hinton does not teach the 2nd limitation “(2) performing at least one of generating, configuring or deriving a second neural network from the first neural network after (1)”. However, Hegde teaches this limitation (paragraph 44 “in the Knowledge distillation (KD) framework, relevant information is transferred from a complex deeper network or ensemble of networks, called teacher network(s), to a simpler shallow network, called student network.” Hegde discloses method and system for building a Student network using the knowledge learnt by a Teacher network while employing a Block Sparse Regularizer. Hegde discloses the Teacher network may be a pre-trained neural network and the framework may transfer relevant information from the Teacher network to create a simpler Student network. The Teacher network as disclosed by Hegde may corresponds to the classification neural network as disclosed in Hinton based on the teaching combination below.)
Hinton does not teach the 5th limitation “the regularization term is determined based on a correlation between, when input data to the first neural network and the second neural network are the same, a latent feature of the first neural network and a latent feature of the second neural network or a correlation between an inferred value of the first neural network and an inferred value of the second neural network”. However, Hegde teaches this limitation (Figure 2 and paragraph 0050 “At step 210, ... utilizing block sparse regularization and a variational dropout techniques, on the plurality of weights of the first neural network with reference to a set of weights of the second neural network to determine one or more weights to be dropped or retained, from or in, the plurality of weights of the first neural network”. Hegde discloses building a student network using the knowledge learnt by a pre-trained Teacher network (deriving a second neural network from the first neural network), and employ a Block Sparse Regularizer on a concatenated tensor of Teacher and Student network weights, wherein the Teacher network (first NN) may correspond to the classification neural network disclosed by Hinton based on the teaching combination below. The teaching by Hegde suggested the capability of utilizing a regularization term such as block sparse regularization based on weights parameter from the Student NN (second NN) and the Teacher NN (first NN). Figure 2 within the disclosure by Hegde showcase the process involved two neural networks, wherein both networks are using the same input and the block sparse regularization technique is applied based on the first and second network’s weights, which represent a latent feature of each network.
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the teaching of training a neural network with a loss function using a soft nearest neighbor loss as a new loss function and a regularization term by Hinton with the teaching of building a student network using the knowledge learnt by a pre-trained teacher network while employing a Block Sparse Regularizer by Hegde. The motivation to do so is referred to in Hegde’s disclosure (paragraph 0003 “Despite the active interest in deep learning, miniaturization of devices (smartphones, drones, head-mounts etc.) and significant progress in augmented/virtual reality devices, pose constraints on CPU/GPU, memory and battery life, is thus making it harder to deploy these models on resource constrained portable devices. To address these requirements, compressing DNN and accelerating their performance in such constrained environments is considered inevitable to the acceptable criteria”, paragraph 0011 “In an embodiment, the first difference in an output and the corresponding ground truth information of the subset of an input data is estimated using a cross-entropy loss function”, and paragraph 0061 “The steps 204 till 210 are iteratively performed until the final loss function depicted in equation 12 converges a predefined threshold so as to obtain a trained compressed and sparser neural network”. Hegde discloses the requirement to deploy deep learning models on resource constrained portable devices, thus require the process of compressing deep neural network and accelerating their performance in such constrained environments, and obtaining a trained compressed and sparser neural network using various techniques such as knowledge distillation and block sparse regularization. Hegde also suggests a cross-entropy loss similar to the cross-entropy loss for the classification network by Hinton (para 68 “The classification loss 214 may be, e.g., a cross entropy loss.”). Therefore, teaching by Hinton can further incorporate the teaching by Hegde to obtain a more compressed and sparser classification neural network, thus improve the overall system. Based on the teaching combination, a person ordinary skilled in the art may substitute or further incorporate the soft nearest neighbor loss regularization term in Hinton with the block spares regularization term in Hegde as they can both act as the regularization term on the cross-entropy loss. Hegde also discloses a final loss function (12) is obtained through combining the cross-entropy loss function with the block sparse regularization as disclosed in paragraph 57 “BSR is defined as Rg(.) as a function of WT:S ... The present disclosure incorporates Rg(WT:S) as a regularization term in (8) to arrive at a final loss function (also referred as ‘optimization function’) as given below: ... function (12)”.)
Hinton/Hegde does not teach the 4th limitation “wherein the second neural network has a second parameter obtained by adding a variation to the first parameter”. However, Brothers teaches this limitation (paragraph 0036, where Brothers discloses “Neural network analyzer 104 may be configured to perform the operations described herein on a first neural network to generate a second neural network, e.g., a modified version of the first neural network”, and paragraph 0076 “In one aspect, pruning, or at least one or more operations that may be included in pruning to the extent that weights or weight representations are changed, may be an example of a modification that adjusts one or more weights of a portion of the neural network”. Brother discloses a method and system for modifying or optimizing neural networks that automatically tune neural network parameters to achieve required performance. The system comprises of a neural network analyzer component that is configured to perform the operations of tuning network parameter to obtain a modified second neural network. For instance, weights or weight parameter of the first neural network may be changed or adjusted such as adding a variation to the weight parameter, thus obtaining an optimized second neural network with the adjusted weight parameters in accordance with the first neural network weight parameter.)
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the teaching of training a neural network with a loss function using a soft nearest neighbor loss as a new loss function and a regularization term by Hinton , and the teaching of building a student network using the knowledge learnt by a pre-trained teacher network while employing a Block Sparse Regularizer by Hegde with a method and system for modifying or optimizing neural networks that automatically tune neural network parameters by Brothers. The motivation to do so is referred to in Brothers’ disclosure (paragraph 0028, where Brothers discloses “the example embodiments described within this disclosure facilitate improving and/or optimizing the computational efficiency of the trained neural network while substantially maintaining the same input-output relationship of the trained neural network at least with respect to established performance requirements.”. Brother discloses the system facilitate improving and optimizing the computational efficiency of the trained neural network while substantially maintaining the same input-output relationship of the trained neural network with respect to established performance requirements. Hegde also discloses an approach to model compression such as parameter pruning in paragraph 0033 “Several approaches have been implemented in the past to model compression such as parameter pruning and sharing”. By utilizing the system, the user may obtain an optimized neural network that achieve an improved performance, wherein the performance can be the accuracy, runtime, computational efficiency throughput, or power consumption of the neural network. Therefore, the teaching by Hinton/Hegde can further incorporate the teaching by Brothers to obtain a system to modify one neural network’s parameter to obtain an optimized neural network with better performance. Base on the teaching combination, during the process of knowledge distillation as disclosed by Hegde to obtain a student neural network from the teacher neural network, the system can further incorporate the process of adjusting or changing the parameter of a neural network by adding a variation to obtain the student network with adjusted weight parameter to improve the training result of the NN.)
Regarding claim 2 depends on claim 1, thus the rejection of claim 1 is incorporated.
Brothers teaches the limitation “The neural network derivation method according to claim 1, wherein the first parameter includes a first weight parameter and the second parameter includes a second weight parameter” (paragraph 0036 “Neural network analyzer 104 may be configured to perform the operations described herein on a first neural network to generate a second neural network, e.g., a modified version of the first neural network”, paragraph 0076 “In one aspect, pruning, or at least one or more operations that may be included in pruning to the extent that weights or weight representations are changed, may be an example of a modification that adjusts one or more weights of a portion of the neural network”, and paragraph 0089 “In addition, other network parameters such as the weights” Brothers discloses a first and second neural network, wherein both network may comprise of weight parameters.)
Regarding claim 3,
Hegde teaches the limitation “the second neural network is configured based on the first neural network and further includes a configuration in which an input of at least one layer is obtained by adding a variation to a feature that is an output of a preceding layer.” (Figure 2, Paragraph 33 “The present disclosure focuses on KD, wherein the systems associated thereof implement method(s) to distil knowledge from a large, complex and neural network (e.g., possibly pre-trained teacher model) to another neural network (e.g., a small student network)”, and paragraph 57 “Parameters in the one or more layers of the neural network can be, for example, but are not limited to, gradient of a function, a mathematical function(s), and the like.” Hegde discloses using knowledge distillation to configure a student neural network from the teacher neural network. As demonstrated by figure 2, the student neural network comprises of one or more layers in-between the input and output layer, which is known to one of ordinary skilled in the art as hidden layer, in which output from one layer can be passed as input onto another layer. The output/input results may be parameters based on gradient of a function at each layer or a mathematical function, which suggest the teaching of adding a variation to a feature within the claim.
The applicant is further directed to the rejections of claim 1 as set forth above, as they are rejected based on the same rationale, because the claim recites similar limitations and processing steps.
Regarding claim 5, which depends on claim 1, thus the rejection of claim 1 is incorporated.
Hinton teaches the limitation “wherein the regularization term is determined based on a similarity between (i) a feature of an output of at least one layer, other than a last layer, of the first neural network and (ii) a feature of an output of a layer of the second neural network corresponding to the at least one layer” (paragraph 0064, where Hinton discloses “The intermediate representation 210 of the network input 204 refers to an output generated by one or more hidden layers 208 of the classification neural network by processing the network input”, and paragraph 0092 “The system determines the soft nearest neighbor loss based on, for each of multiple pairs of network inputs that include a first network input and a second network input, a respective measure of similarity between: (i) an intermediate representation of the first network input, and (ii) an intermediate representation of the second network input”. Hinton discloses the intermediate representation as an output generated by one or more layers of the neural network and the soft nearest neighbor loss, which is used as a regularization term is determined based on a measure of similarity between an intermediate representation of the first network input and the second network input, wherein the first and second neural network input and the first and second network intermediate representation are from two neural networks based on the teaching combination of Hinton/Hegde as disclosed above.)
Claims 4, 18 are rejected under 35 U.S.C. 103 as being unpatentable in view of Hinton et.al (US 20220101624 A1), further in view of Hegde et.al (US 20200387782 A1), further in view of Brothers et.al (US 20160358070 A1), further in view of Bordaweker et.al (US 20190294953 A1),
Regarding claim 4, which depends on claim 1, thus the rejection of claim 1 is incorporated.
Hinton teaches a part of the limitation “wherein the regularization term is determined based on ... similarity between a first inferred value of the first neural network and a second inferred value of the second neural network” (paragraph 0092, where Hinton discloses “The system determines the soft nearest neighbor loss based on, for each of multiple pairs of network inputs that include a first network input and a second network input, a respective measure of similarity between: (i) an intermediate representation of the first network input, and (ii) an intermediate representation of the second network input”. Within the teaching combination of Hinton/Hegde/Brothers, Hinton discloses the soft nearest neighbor loss, which is used as a regularization term is determined based on a measure of similarity between an intermediate representation of the first network input, and the second network input, wherein a person ordinary skilled in the art would have been able to recognize that the first and second neural network input and the first and second network intermediate representation can be from two different neural networks of Student network and Teacher network based on the teaching combination of Hinton/Hegde as disclosed above.)
Hinton/Hegde/Brothers does not teach “... a time- series variation ...”. However, Bordaweker teaches this limitation (paragraph 0055 “determines a window similarity between the first observation window and the second observation window based at least in part on the generated preceding similarity measure.”. Bordaweker discloses the training of one or more machine learning model to compare how similar data between two time series are.)
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the teaching of training a neural network with a loss function and soft nearest neighbor loss is used as a new loss function and a regularization term to further optimize the neural network, the teaching of building a student network using the knowledge learnt by a pre-trained teacher network while employing a Block Sparse Regularizer on a concatenated tensor of teacher and student network weights, and quantizing weights of first network to obtain weights of second networks by Hinton/Hegde/Brothers with the teaching of training machine learning models based in part of similarity determination between time series of input data by Bordaweker. The motivation to do so is referred to in Bordaweker’s disclosure (paragraph 0023, where Bordaweker discloses “the Comparison Component 145 computes the cosine similarity for the feature vectors, in order to generate a similarity measure indicating how similar the associated time series are”. Bordaweker discloses the similarity comparison of feature vectors of different time series using Cosine similarity. Hinton also discloses in paragraph 0052 “The similarity between pairs of data points may be measured using a numerical similarity measure, e.g., a Euclidean similarity measure or cosine similarity measure”. Hinton discloses using Cosine similarity to measure the similarity between pairs of data points of first and second neural network intermediate representation. Due to both references apply the same concept of similarity measure between two data points using Cosine similarity, thus the teaching of Hinton can incorporate the teaching of Bordaweker to include a time series variation in similarity comparison between its first and second intermediate representation of neural network input.)
Regarding claim 18,
Brothers teaches the limitation “wherein second input data obtained by adding a variation to the first input data is inputted to the second neural network” (paragraph 0036 “Neural network analyzer 104 may be configured to perform the operations described herein on a first neural network to generate a second neural network, e.g., a modified version of the first neural network”, and paragraph 0076 “In one aspect, pruning, or at least one or more operations that may be included in pruning to the extent that weights or weight representations are changed, may be an example of a modification that adjusts one or more weights of a portion of the neural network”. Brother discloses the tuning process comprises of pruning the first neural network by modifying the neural network parameters to obtain an optimized second neural network. A person ordinary skilled in the art would have been able to configure such that the modification process of changing weight parameters may be adjusted into changing or modifying or adding more into the input of the first neural network such that the second neural network may obtain a new input dataset to help account for different variation and prevent overfitting of training data.)
The applicant is further directed to the rejections of claim 1 with claim 4 as set forth above, as they are rejected based on the same rationale, because the claim recites similar limitation and processing steps.
Claims 6, 10-12, 16, 17 are rejected under 35 U.S.C. 103 as being unpatentable in view of Hinton et.al (US 20220101624 A1), further in view of Hegde et.al (US 20200387782 A1), further in view of Brothers et.al (US 20160358070 A1), further in view of Bordaweker et.al (US 20190294953 A1), further in view of Kang et.al (US 20190122100 A1)
Regarding claim 6 depends on claim 4, thus the rejection of claim 4 is incorporated.
Hinton teaches the limitation “(4) training a discriminator for determining the regularization term, using the second neural network before (3)” (paragraph 0025, where Hinton discloses “for each data element in a combined set of data elements including the set of synthetic data elements and the set of genuine data elements, a discriminator neural network is used to generate an embedding of the data element”. Hinton discloses including a discriminator neural network to generate an embedding of the data element for the combined set of data element, wherein the soft nearest neighbor loss, which is the regularization term, is determined for each multiple pairs of data elements from the combined set of synthetic and genuine data elements. However, a person ordinary skilled in the art would have been able to configure such that the discriminator may be applied for determining the block sparse regularization term disclosed by Hegde, as the block sparse regularization term replaced the soft nearest neighbor loss regularization term as explained above, the discriminator compute embeddings for data elements of synthetic and genuine data elements, such that the block sparse regularization term may be implemented with synthetic and genuine data elements.)
Hinton teaches the limitation “the discriminator is trained using a first expected value calculated from the first inferred value and the second inferred value” (paragraph 0081, where Hinton discloses “The discriminative neural network 310 enables the generative training system 300 to evaluate the entanglement of synthetic and genuine data elements in a learned embedding space”, and paragraph 0092 “The system determines ... for each of multiple pairs of network inputs that include a first network input and a second network input, a respective measure of similarity between: (i) an intermediate representation of the first network input, and (ii) an intermediate representation of the second network input”. Hinton discloses the mention system as a discriminative neural network, which is a discriminator, wherein the discriminator network system include multiple pairs of network inputs comprise of first and second network inputs from a first and second neural network, wherein the first and second network can be the Student and Teacher network according to the teaching combination as disclosed above, thus output an intermediate representation for both first and second network input, wherein these outputs are implication of an expected value calculated from the first inferred value of the first neural network and an expected value calculated from the second inferred value of the second neural network.)
Hinton/Hegde/Brothers/Bordaweker does not teach “the first parameter and the second parameter are inputted to the discriminator”. However, Kang teaches this limitation (paragraph 0015, where Kang discloses “According to an example, the parameters of the neural network may include various types of data that is input to the neural network, or data that is output from the neural network, for example, input activations, output activations, weights, biases, etc. of the neural network”. Kang discloses parameters may include data that is input to the neural network such as weight value, biases, etc. to the neural network model.)
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the teaching of training a neural network with a loss function and soft nearest neighbor loss is used as a new loss function and a regularization term to further optimize the neural network, the teaching of building a student network using the knowledge learnt by a pre-trained teacher network while employing a Block Sparse Regularizer on a concatenated tensor of teacher and student network weights, and quantizing weights of first network to obtain weights of second networks, and the incorporating of a time-series variation in similarity comparison of data points by Hinton/Hegde/Brothers/Bordaweker with the teaching of parameters may include input data to neural network by Kang. The motivation to do so is referred to in Kang’s disclosure (paragraph 0044, where Kang discloses “In order to analyze in real time a large amount of input data and extract desired information by implementing a neural network, it is found herein that technology to efficiently process neural network operations may be desirable, for example, such as in devices implemented with low power and low performance” and paragraph 0047 “As the training of the neural network is repeated, the floating-point parameters of the neural network may be tuned to generate more accurate output for a given input until an accuracy threshold is reached”. Kang discloses the quantization process with the desire for an efficiently process neural network in devices implemented with low power and low performance to analyze in real time a large amount of input data and extract desired information. The quantization technique referred within the application of Kang is one of the methods to achieve such goal as the parameters of the neural network may be tuned to generate more accurate output for a given input. Since Hinton/Hegde/Brothers/Bordaweker mentioned a first and second network inputs as disclosed above, and Brothers also mentioned the two networks wherein weights of one network can be quantize to generate weights for the second network. Therefore, the teaching of Hinton/Hegde/Brothers/Bordaweker can further incorporate the teaching of Kang to obtain a more accurate output given the parameters input as both Brothers and Kang mentioned the same technique of quantization.)
Regarding claim 10 depends on claim 4, thus the rejection of claim 4 is incorporated.
Kang teaches the limitation “the first parameter is expressed by a first numeric representation, and the second parameter is obtained by converting the first parameter into a second numeric representation” (paragraph 0084, where Kang discloses “The quantization refers to or means a conversion of a high-precision floating point real number-type parameter value to a lower-precision fixed-point integer-type parameter value”. Kang discloses the quantization to perform on a set of parameter value, wherein the set of parameter value can be parameter of first neural network as disclosed above and it is a number-type value thus indicate that the parameter is express by a numeric representation. The quantization process converses the parameter value to obtain a new parameter value, which is also a number type value, and can be identify as parameter value of the second parameter as disclosed above.)
Regarding claim 11 depends on claim 10, thus the rejection of claim 10 is incorporated.
Kang teaches the limitation “the first numeric representation is a real number consisting of a float value, and the second numeric representation is an integer, and the second parameter is obtained by quantizing the first parameter” (paragraph 0084, where Kang discloses “The quantization refers to or means a conversion of a high-precision floating point real number-type parameter value to a lower-precision fixed-point integer-type parameter value”. Kang discloses the quantization process, wherein the first number-type parameter is converse from a floating-point real number, which imply on the first numeric representation is a real number consisting of float value, and the conversion obtain a second integer type parameter value, which is the second numeric representation.)
Regarding claim 12 depends on claim 4, thus the rejection of claim 4 is incorporated. The applicant is further directed to the rejections of claim 6 with claim 4 as set forth above, as they are rejected based on the same rationale, because the claim recites similar limitations and processing steps.
Regarding claim 16 depends on claim 12, thus the rejection of claim 12 is incorporated. The applicant is further directed to the rejections of claim 10 as set forth above, as they are rejected based on the same rationale, because the claim recites similar limitations and processing steps.
Regarding claim 17 depends on claim 16, thus the rejection of claim 12 is incorporated. The applicant is further directed to the rejections of claim 11 as set forth above, as they are rejected based on the same rationale, because the claim recites similar limitations and processing steps.
Claims 7, 13 are rejected under 35 U.S.C. 103 as being unpatentable in view of Hinton et.al (US 20220101624 A1), further in view of Hegde et.al (US 20200387782 A1), further in view of Brothers et.al (US 20160358070 A1), further in view of Bordaweker et.al (US 20190294953 A1), further in view of Kang et.al (US 20190122100 A1), further in view of Bolt et.al (US 20050149463 A1)
Regarding claim 7 depends on claim 6, thus the rejection of claim 6 is incorporated.
Bordaweker teaches a part of the 1st limitation “... , the first expected value is -S when a first similarity between current first and second inferred values is high compared to a second similarity between a preceding first and second inferred values” (paragraph 0023 “the Comparison Component 145 computes the cosine similarity for the feature vectors, in order to generate a similarity measure indicating how similar the associated time series are” and paragraph 0044 “the Analysis Application 130 identifies similar Time Series 150, or specific windows or segments within a Time Series 150 ... In some embodiments, the Analysis Application 130 identifies and provides any matches that exceed a predefined threshold. For example, in one embodiment, the similarity measure ranges from zero to one, and all matches exceeding a threshold are provided. In some embodiments, the threshold is user-defined”. Bordaweker discloses using Cosine similarity to generate a similarity measure indicating how similar between the associated time series or between time segments within a time series are. Based on the teaching combination, at each time segments within the time series, a similarity measure between the intermediate representation of the first network input and the second network input is obtained as disclosed above by Hinton/Hegde/Brothers. The teaching of Bordaweker is then incorporated for a system to perform similarity comparison between similarity measures at different time window or segments. In some embodiment the threshold of similarity is user-defined and in one example the similarity is range from zero to one, thus indicating that a user can define the similarity range from -1 to a user-defined value, wherein -1 yield a high similarity according to a specific user-defined condition of similarity.)
Bordaweker teaches the limitation “and the first expected value is 0 when the similarity is not high compared to the preceding second inferred value in the time-series view of the second inferred value” (paragraph 0023, where Bordaweker discloses “the Comparison Component 145 computes the cosine similarity for the feature vectors, in order to generate a similarity measure”. Bordaweker discloses using Cosine similarity to compute the similarity measure between two data, wherein within Cosine similarity, zero value indicate that the similarity is not high.)
Hinton/Hegde/Brothers/Bordaweker/Kang does not teach a part of the 1st limitation “wherein for the first inferred value, the first expected value is always S, S being greater than 0 ...”. However, Bolt teaches this limitation (paragraph 0069, where Bolt discloses “Whether these weights (between the input and hidden neurons) are positive or negative determines whether the network output is positively or negatively monotonic with respect to each input”. Bolt discloses the determination of a positive output is with respect to an input and positive weights.)
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the teaching of training a neural network with a loss function and soft nearest neighbor loss is used as a new loss function and a regularization term to further optimize the neural network, the teaching of building a student network using the knowledge learnt by a pre-trained teacher network while employing a Block Sparse Regularizer on a concatenated tensor of teacher and student network weights, and quantizing weights of first network to obtain weights of second networks, and the incorporating of a time-series variation in similarity comparison of data points, and input parameters by Hinton/Hegde/Brothers/Bordaweker/Kang with the teaching of a positive output depend on input and positive weights by Bolt The motivation to do so is referred to in Bolt’s disclosure (paragraph 0073, where Bolt discloses “... let each weight, w, that needs to be constrained, can be redefined as a positive (or negative) function of a dummy weight, w*. (Positive functions are positive for all values of their arguments, and can be used to constrain weights to have positive values...) Once this has been done, the network can be trained by applying one of the standard unconstrained optimisation techniques that are used for training simultaneously all weights that do not need to be constrained and the dummy weights. It will be appreciated that other suitable functions could also be used. This method of producing monotonicity is particularly convenient, because the standard neural network training algorithms can be applied unmodified, making training fast and efficient”. Bolt discloses a method to create a positive function of dummy weight which produce only positive values, then the network can be trained by applying the original optimization technique with the original weights and dummy weights thus with the first neural network having its first inferred value, which are positive weights, the first neural network can then obtain the first expected value as a positive output. The above disclosure can help making training fast and efficient and it can be applied with no need to modify the current neural network. Therefore, the teaching combination of Hinton/Hegde/Brothers/Bordaweker/Kang can further incorporate the teaching of Bolt to obtain the first neural network that produce a first positive output based on its weights.)
Regarding claim 13 depends on claim 12, thus the rejection of claim 12 is incorporated. The applicant is further directed to the rejections of claim 7 as set forth above, as they are rejected based on the same rationale, because the claim recites similar limitations and processing steps.
Claims 8, 14 are rejected under 35 U.S.C. 103 as being unpatentable in view of Hinton et.al (US 20220101624 A1), further in view of Hegde et.al (US 20200387782 A1), further in view of Brothers et.al (US 20160358070 A1), further in view of Bordaweker et.al (US 20190294953 A1), further in view of Kang et.al (US 20190122100 A1), further in view of Cheng et.al (US 20210142210 A1)
Regarding claim 8 depends on claim 6, thus the rejection of claim 6 is incorporated.
Hinton teaches “wherein in (4), the discriminator is trained...” (paragraph 0079, where Hinton discloses “The generative training system 300 trains the discriminative neural network 310”. Hinton discloses the training of the discriminative neural network, which is otherwise understood as the discriminator.)
Hinton/Hegde/Brothers/Nordaweker/Kang does not teach “using a third loss function having, as inputs, a first feature calculated based on the first parameter and a second feature calculated based on the second parameter”. However, Cheng teaches this limitation (paragraph 0058, where Cheng discloses “Training of the learning model may, in part, be performed to train the learning model on at least the second loss function to learn a feature embedding of the heterogeneous features of the at least two tasks ... a triplet loss function, which, generally, is a function which takes, as parameters, a target data point x, a positive data point xp which matches the target data point with regard to a feature thereof, and a negative data point xn which does not match the target data point with regard to the feature”. Cheng discloses a training a model may further include a triplet loss function to learn a feature embedding, thus indicating that feature is used as input for the loss function, wherein the loss function having parameters with regard to the feature, thus indicating that feature is obtained based on parameters provided to the function.)
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the teaching of training a neural network with a loss function and soft nearest neighbor loss is used as a new loss function and a regularization term to further optimize the neural network, the teaching of building a student network using the knowledge learnt by a pre-trained teacher network while employing a Block Sparse Regularizer on a concatenated tensor of teacher and student network weights, and quantizing weights of first network to obtain weights of second networks, and the incorporating of a time-series variation in similarity comparison of data points, and input parameters by Hinton/Hegde/Brothers/Bordaweker/Kang with the teaching of another loss function with parameters in regard to the feature as input by Cheng. The motivation to do so is referred to in Cheng’s disclosure (paragraph 0058, where Cheng discloses “Therefore, training the learning model on the triplet loss function may generate a learned feature embedding which optimizes for minimizing the difference between the first distance and the second distance”. Cheng discloses using another loss function such as triplet loss function in addition to the first loss function on a neural network may help generate a learned feature embedding which optimizes for minimizing the difference between feature vector, thus further improve the training of the neural network. Therefore, the neural network system as disclosed by Hinton/Hegde/Brothers/Bordaweker/Kang can further incorporate the teaching of Cheng which is a triplet loss function to further improve the training of the system. By incorporating the teaching of Cheng, the discriminator as disclosed above can incorporate another loss function, wherein the loss function can receive feature input in accordance with the parameters as disclosed by Hinton in paragraph 0005 “processing the network input ..., in accordance with current values of classification neural network parameters”, thus obtaining a feature in accordance with the parameters of the first network input and a feature in accordance with the parameters of the second network input which can be used for the newly loss function.)
Regarding claim 14 depends on claim 13, thus the rejection of claim 13 is incorporated. The applicant is further directed to the rejections of claim 8 as set forth above, as they are rejected based on the same rationale, because the claim recites similar limitations and processing steps.
Claims 9, 15 are rejected under 35 U.S.C. 103 as being unpatentable in view of Hinton et.al (US 20220101624 A1), further in view of Hegde et.al (US 20200387782 A1), further in view of Brothers et.al (US 20160358070 A1), further in view of Bordaweker et.al (US 20190294953 A1), further in view of Kang et.al (US 20190122100 A1), further in view of Cheng (US 20210142210 A1), further in view of Poole (US 20200311911 A1)
Regarding claim 9 depends on claim 8, thus the rejection of claim 8 is incorporated.
Cheng teaches the limitation “wherein the third loss function is a triplet loss function” (paragraph 0058, where Cheng discloses “the second loss function may be a triplet loss function”. Cheng discloses the second loss function which after incorporating the teaching to the teaching combination, the second loss function serve as an additional loss function which is an implication of a third loss function beside from the two loss functions as disclosed above. This additional loss function is a triplet loss function.)
Cheng further teaches both limitations “the first feature obtained in Nth training is set as a positive feature” and “the second feature obtained in the Nth training is set as a negative feature” (paragraph 0058, where Cheng discloses “a triplet loss function, which, generally, is a function which takes, as parameters, a target data point x, a positive data point xp which matches the target data point with regard to a feature thereof, and a negative data point xn which does not match the target data point with regard to the feature”. Cheng discloses a positive data point with regard to a feature thereof, which indicate a positive feature of the feature in a training data set, and a negative data point with regard to a feature thereof, which indicate a negative feature of the feature in the training data set.)
Hinton teaches a part of the limitation “... (N-1)th training ..., N being greater than 1” (paragraph 0064, where Hinton discloses “The discriminative training system 200 trains the classification neural network 202 over multiple training iterations”. Hinton discloses the discriminative training system is trained over multiple training iterations, thus indicating that there are more than one iteration of training.)
Within the limitation “the first feature obtained in (N-1)th training is set as a reference feature, N being greater than 1”, Hinton/Hegde/Brothers/Bordaweker/Kang/Cheng does not teach “the first feature obtained ... is set as a reference feature”. However, Poole teaches this limitation (paragraph 0035, where Poole discloses “an identification of a first characteristic of the training data sets as a positive characteristic”, paragraph 0075 “selects a plurality of features 64 in the training data 60 that the domain expert 2 considers to be related to the pathology of interest. The features 64 may also be referred to as positive characteristics. The features 64 are chosen from the set of characteristics for which values are available in at least some of the training data sets”, and paragraph 0082 “the values for the selected features 64 may be computed from the training data. The values for the selected features 64 may be extracted from the training data. The values for the selected features 64 may be manually defined by the domain expert 2 or by a further expert.” Poole discloses selected features which are relevant to the classification for which the neural network is being trained, which is an implication of the reference feature. The selected features may be referred to as positive characteristics. The selected features may be computed or extracted from the training data or manually defined, thus indicating the first characteristic of the training data sets, which implies feature of training data can be set as selected features.)
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the teaching of training a neural network with a loss function and soft nearest neighbor loss is used as a new loss function and a regularization term to further optimize the neural network, the teaching of building a student network using the knowledge learnt by a pre-trained teacher network while employing a Block Sparse Regularizer on a concatenated tensor of teacher and student network weights, and quantizing weights of first network to obtain weights of second networks, and the incorporating of a time-series variation in similarity comparison of data points, input parameters, and another loss function by Hinton/Hegde/Brothers/Bordaweker/Kang/Cheng with the teaching of setting first characteristics of training data as selected relevant feature, wherein the first characteristics of training data is positive characteristics by Poole. The motivation to do so is referred to in Poole’s disclosure (paragraph 0060, where Poole discloses “Embodiments of training methods described below may deliver a pure neural network which may be efficiently run on a GPU.”. Poole discloses the benefit of its disclosure, including the process of determining selected relevant feature will help deliver a pure neural network which may be efficiently run on a GPU. Thus, by incorporating the teaching of Hinton/Hegde/Brothers/Bordaweker/Kang/Cheng with the teaching of Poole will help the teaching combination to obtain all benefits from the selected relevant feature as disclosed above.)
Regarding claim 15 depends on claim 14, thus the rejection of claim 14 is incorporated. The applicant is further directed to the rejections of claim 9 as set forth above, as they are rejected based on the same rationale, because the claim recites similar limitations and processing steps.
Claim 19-21 are rejected under 35 U.S.C. 103 as being unpatentable in view of Hinton et.al (US 20220101624 A1), further in view of Hegde et.al (US 20200387782 A1), further in view of Brothers et.al (US 20160358070 A1), further in view of Bordaweker et.al (US 20190294953 A1), further in view of Kang et.al (US 20190122100 A1), further in view of Cheng et.al (US 20210142210 A1), further in view of Okamoto et.al (US 20200125889 A1)
Regarding claim 19 depends on claim 18, thus the rejection of claim 18 is incorporated.
Hinton/Hegde/Brothers/Bordaweker/Kang/Cheng does not teach the limitation “the regularization term is determined to be smaller when the similarity is higher” and the limitation “the regularization term is determined to be larger when the similarity is lower”. However, Okamoto teaches this limitation (paragraph 0052, where Okamoto discloses “if the similarity is large, a value of the penalty term is decreased, and if the similarity is small, the value of the penalty term is increased”. Okamoto discloses a similarity comparison between two data with a penalty term, wherein if similarity is large the penalty term is decreased and if similarity is small the similarity is increased.)
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the teaching of training a neural network with a loss function and soft nearest neighbor loss is used as a new loss function and a regularization term to further optimize the neural network, the teaching of building a student network using the knowledge learnt by a pre-trained teacher network while employing a Block Sparse Regularizer on a concatenated tensor of teacher and student network weights, quantizing weights of first network to obtain weights of second networks, and the incorporating of a time-series variation in similarity comparison of data points, input parameters, and another loss function by Hinton/Hegde/Brothers/Bordaweker/Kang/Cheng with the teaching of an inverse relationship between similarity comparison and penalty term. The motivation to do so is referred to in Okamoto’s disclosure (paragraph 0052 “The accuracy rate is corrected depending on the penalty term in this way, whereby superiority or inferiority of a learning result can be accurately evaluated, and an image classification program having high classification accuracy can be generated”, and paragraph 0077 “Here, the similarity between the two indices can be obtained as a cosine similarity or a normalized correlation value by regarding the two indices as vectors”. Okamoto discloses performing the similarity comparison between two data points using Cosine similarity which is similar to what is disclosed by Hinton at paragraph 0052 “The similarity between pairs of data points may be measured using a numerical similarity measure, e.g., a Euclidean similarity measure or cosine similarity measure”. Okamoto also mentioned a penalty term to be adapted into its embodiment with a relationship between penalty term and similarity comparison, wherein the penalty term can be understood as another interpretation of a regularization term as it also performs the same function as disclosed of regularizing to determine accuracy of classification as disclosed by Hinton in paragraph 0069 “the soft nearest neighbor loss may regularize the classification neural network by encouraging intermediate representations of network inputs to characterize class-independent features capturing information that improves classification accuracy”. Therefore, the teaching of Hinton/Hegde/Brothers/Bordaweker/Kang/Cheng can further incorporate the teaching of Okamoto as they mentioned the same technique, thus the teaching combination can further inherit the characteristic of an inverse relationship between its similarity comparison result and regularization term.)
Regarding claim 20 depends on claim 19, thus the rejection of claim 19 is incorporated. The applicant is further directed to the rejections of claim 6 as set forth above, as they are rejected based on the same rationale, because the claim recites similar limitations and processing steps.
Regarding claim 21 depends on claim 20, thus the rejection of claim 20 is incorporated. The applicant is further directed to the rejections of claim 8 as set forth above, as they are rejected based on the same rationale, because the claim recites similar limitations and processing steps.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUY TU DIEP whose telephone number is (703)756-1738. The examiner can normally be reached M-F 8-4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached at (571) 270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUY T DIEP/Examiner, Art Unit 2123
/BEN M RIFKIN/Primary Examiner, Art Unit 2123
Prosecution Insights