DETAILED ACTION
This action is responsive to the Application filed on 05/16/2024. Claims 1-20 are pending in the case. Claims 1, 8, 19 and 18 are independent claims.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 8 is objected to because of the following informalities:
The claim appears to recite essentially duplicate limitations. The presence of both limitations does not further limit the claim. The duplicate limitation should be removed for clarity. The particular limitations are presented below with the underlined differences
processing, by the computing system, the one or more training inputs with at least the portion of the teacher machine learning model to generate the one or more layer representations at the layer of the teacher machine learning model
processing, by the computing system, the plurality of training inputs with at least the portion of the teacher machine learning model to respectively generate a plurality of layer representations at the layer of the teacher machine learning model;
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 6-7 and 12 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 6 recites “The computer-implemented method of any of claim 2…”
Claim 12 recites “The computer-implemented method of any of claim 1…”
The claim is not clear and should be corrected to “The computer-implemented method of claim…” and for purposes of examination is interpreted accordingly.
Claim 7 is rejected by virtue of dependency and failure to cure the deficiency of the base claim.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 8-13, 15-16, 18 and 20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Seddik “light weight neural networks from pca & lda based distilled dense neural networks.”
Claim 1/8/16
Seddik teaches, [claim 1] A computer-implemented method to generate machine learning models having improved computational efficiency, the method comprising [claim 8] The computer-implemented method of claim 1 [claim 16] One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors of a computing system, cause the computing system to perform operations, the operations comprising … obtaining, by a computing system comprising one or more computing devices, one or more training inputs; (pg 1 abstract “This paper presents two methods for building lightweight neural networks with similar accuracy than heavyweight ones with the advantage to be less greedy in memory and computing resources. So it can be implemented in edge and IoT devices.”, pg 4 Table 2 caption “the forward pass is applied (on a i7-7700HQ CPU @ 2.80GHz) to the train set of the respective datasets using a batch size of 50000 images”) processing, by the computing system, the one or more training inputs with at least a portion of a teacher machine learning model to generate one or more layer representations at a layer of the teacher machine learning model; [from claim 8] processing, by the computing system, the plurality of training inputs with at least the portion of the teacher machine learning model to respectively generate a plurality of layer representations at the layer of the teacher machine learning model; (pg 2 Section 2.1 “Consider a dense neural network, which we refer to as the teacher network (TN), composed of L layers and constructed in the following way, for ℓ
PNG
media_image1.png
57
333
media_image1.png
Greyscale
”, pg 2 Section 2.2 “Given a set of n training samples X” pg 2 algorithm 1
PNG
media_image2.png
39
378
media_image2.png
Greyscale
the teacher network is passed training samples X to return hidden representations H for each layer of the teacher model) performing, by the computing system, a principal components analysis technique on the one or more layer representations generated by the layer of the teacher machine learning model to generate ( pg 2 Section 2.2 “TheNet-PCAD method consists in perform ing a PCA at each hidden layer of TN” ) one or more sets of coefficient values respectively for the one or more layer representations and a plurality of principal directions, wherein the set of coefficient values for each layer representation comprises a plurality of coefficient values respectively associated with the plurality of principal directions; ( pg 2 Section 2.2 “Therefore, a PCA is performed at each layer ℓ in order to reduce the dimension of the hidden features pℓ, relying on the top kℓ largest eigenvectors of the sample covariance matrix:… the Homoscedastic loss…
PNG
media_image3.png
94
298
media_image3.png
Greyscale
As understood by one of ordinary and defined by the disclosure in para 0027, PCA involves finding the principal orthonormal directions of a set of points which is equivelent to the eignenvectors described in the art. Further projecting the eigenvector matrix U onto h(l) amounts to a set of coefficent values for a given layer representation on h(l)) and training, by the computing system, a student machine learning model to predict the one or more sets of coefficient values. (pg 2 Algorithm 1
PNG
media_image4.png
38
394
media_image4.png
Greyscale
Section 2.2 “the optimization problem for SN can be formulated as a multi-task problem….
PNG
media_image5.png
93
337
media_image5.png
Greyscale
here
PNG
media_image6.png
65
54
media_image6.png
Greyscale
are the learned layer representation values of the student network which are trained to be close to the projection of the PCA pricipal directions which is the claimed coefficents. The student network is therefore trained to predict the PCA coefficents.)
Claim 9
Seddik teaches claim 1
Seddik teaches, wherein training, by the computing system, the student machine learning model to predict the one or more sets of coefficient values comprises training, by the computing system, the student machine learning model to predict the one or more sets of coefficient values when given the one or more training inputs as input. (pg 2 “Consequently, the student network(SN) is composed of L layers and has the following structure:
PNG
media_image7.png
68
390
media_image7.png
Greyscale
… x corresponds to the input data features” the hidden layer outputs which as noted are a prediction of coefficent values are computed based on the input data features x, which are required for training)
Claim 10
Seddik teaches claim 1
Seddik teaches, wherein training, by the computing system, the student machine learning model to predict the one or more sets of coefficient values comprises training, by the computing system, the student machine learning model to predict the one or more sets of coefficient values when given a prior layer representation as an input, the prior layer representation generated by a second layer of the teacher machine learning model that is prior to the layer of the teacher machine learning model (pg 2 “Consequently, the student network(SN) is composed of L layers and has the following structure:
PNG
media_image7.png
68
390
media_image7.png
Greyscale
… x corresponds to the input data features” the hidden layer outputs which as noted are a prediction of coefficent values are computed based on the input data features x, which are required for training. Each layer based on a prior l-1 layer as input, which is further based on the given prior layer of the teacher network. Each middle layer of the teacher is prior to some other later layer in the teacher network)
Claim 11
Seddik teaches claim 1
Seddik teaches, wherein the layer of the teacher machine learning model comprises a hidden layer and the one or more layer representations comprise one or more embeddings. (Section 2.2 pg 2 “then training a SN to perform the same learning task as TN along with the task of mapping its hidden features at each layer with the reduced features of TN… where h(ℓ) i stands for the features of xi at layer ℓ” here h(l) are embeddings of either the input data or a prior layer of a neural network. A “mapping” is understood to be an embedding at each layer.)
Claim 12
Seddik teaches claim 1
Seddik teaches, wherein the layer of the teacher machine learning model comprises an output layer (pg 2 Section 2.2 Consider a dense neural network, which we refer to as the teacher network(TN), composed of L layers and constructed in the following way, for ℓ∈[L]:” the teacher network has several layers which are output layers with respect to prior layers serving as input.) and the one or more layer representations comprise one or more output probability representations. ( pg 3 Section 3 In particular, we consider three teacher networks composed of L =4 dense layers which are trained to perform a classification problem respectively on the datasets MNIST” the teacher network is trained to classify data. Thus, the layers at least include a layer which represents the output probability for a given classification.)
Claim 13
Seddik teaches claim 1
Seddik teaches, after training, by the computing system, the student machine learning model to predict the one or more sets of coefficient values: training, by the computing system, a prediction model to predict one or more ground truth labels associated with the one or more training inputs when given the one or more training inputs as input, the prediction model comprising the student machine learning model, a prediction head, and the plurality of principal directions. (Figure 3 caption pg 4 “Test accuracy of the student network in orange trained using Net-PCAD (left) and Net-LDAD (right), and test ac curacy of the student network in blue trained from scratch, in terms of k and across three different datasets” pg 2 Section 2.2 “The Net-PCAD method consists in performing a PCA…training a SN to perform the same learning task as TN” after training as claimed the model is trained to predict classifications or ground truth inputs associated with the training input. As described above, the Net-PCAD includes student model, plurality of principal directions encoded into the student teacher model, and a prediction head which is the component which generates classifications in the target task)
Claim 15
Seddik teaches claim 1
Seddik teaches, wherein the teacher machine learning model comprises a pre-trained model. (pg 1 Introduction “The proposed methods are based on knowledge distillation concept, where a large (teacher) pre-trained network is used to train a smaller (student) network”)
Claim 18
Seddik teaches, A computing system for generating machine learning predictions with improved efficiency, the computing system comprising: one or more processors; and one or more non-transitory computer-readable media that collectively store :a machine-learned prediction model,; (pg 1 abstract “This paper presents two methods for building lightweight neural networks with similar accuracy than heavyweight ones with the advantage to be less greedy in memory and computing resources. So it can be implemented in edge and IoT devices.”, pg 4 Table 2 caption “the forward pass is applied (on a i7-7700HQ CPU @ 2.80GHz) to the train set of the respective datasets using a batch size of 50000 images” the model is stored on a computer in order to perform a pass) a plurality of principal directions generated by performance of a principal components analysis on a plurality of layer representations generated at a layer of a teacher machine learning model; ( pg 2 Section 2.2 “TheNet-PCAD method consists in perform ing a PCA at each hidden layer of TN” , pg 2 Section 2.2 “Therefore, a PCA is performed at each layer ℓ in order to reduce the dimension of the hidden features pℓ, relying on the top kℓ largest eigenvectors of the sample covariance matrix:… the Homoscedastic loss…
PNG
media_image3.png
94
298
media_image3.png
Greyscale
As understood by one of ordinary and defined by the disclosure in para 0027, PCA involves finding the principal orthonormal directions of a set of points which is equivelent to the eignenvectors described in the art. Further projecting the eigenvector matrix U onto h(l) amounts to a set of coefficent values for a given layer representation on h(l)) a student machine learning model trained to predict a plurality of predicted coefficient values respectively associated with the plurality of principal directions when given a model input; (pg 2 Algorithm 1
PNG
media_image4.png
38
394
media_image4.png
Greyscale
Section 2.2 “the optimization problem for SN can be formulated as a multi-task problem….
PNG
media_image5.png
93
337
media_image5.png
Greyscale
here
PNG
media_image6.png
65
54
media_image6.png
Greyscale
are the learned layer representation values of the student network which are trained to be close to the projection of the PCA pricipal directions which is the claimed coefficents. The student network is therefore trained to predict the PCA coefficents.) and a prediction head configured to generate a model prediction based on the plurality of principal directions and predicted coefficient values pg 2 Section 2.2 “The Net-PCAD method consists in performing a PCA…training a SN to perform the same learning task as TN” the component of the SN which performs the learning task generate a prediction for the tasks based on it incorporated principal directions in its learned hidden state) and instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising: obtaining the model input; processing the model input with the student machine learning model to generate the plurality of predicted coefficient values; and processing the plurality of principal directions and predicted coefficient values with the prediction head to generate the model output. (pg 2 Section 2.1 “Consider a dense neural network, which we refer to as the teacher network (TN), composed of L layers and constructed in the following way, for ℓ
PNG
media_image1.png
57
333
media_image1.png
Greyscale
”, pg 2 Section 2.2 “Given a set of n training samples X” pg 2 algorithm 1
PNG
media_image2.png
39
378
media_image2.png
Greyscale
the teacher network is passed training samples X to return hidden representations H for each layer of the teacher model, ( pg 2 Section 2.2 “Therefore, a PCA is performed at each layer ℓ in order to reduce the dimension of the hidden features pℓ, relying on the top kℓ largest eigenvectors of the sample covariance matrix:… the Homoscedastic loss…
PNG
media_image3.png
94
298
media_image3.png
Greyscale
as noted above the algorithm 1 is performed on a computer thuse requiring claimed instructions including obtaining input x, processing via the student model to perform training according to the above loss, and processing the directions U in order to obtain a model ouput)
Claim 20
Seddik teaches, wherein the model input comprises an image and the model output comprises an image prediction. (pg 4 caption “to the train set of the respective datasets using a batch size of 50000 images.” The network operates on a dataset of images, whose subsequent predictions are therefore an image prediction)
Claim Rejections - 35 U.S.C. § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. §§ 102 and 103 (or as subject to pre-AIA 35 U.S.C. §§ 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2-5, 17 and 19 are rejected under 35 U.S.C. § 103 as being unpatentable over Seddik further in view of Avalos et al. “Representation Learning of Compositional Data”
Claim 2
Seddik teaches claim 1
Seddik does not explicitly teach, the principal components analysis technique comprises performing, by the computing system, a Bregman principal components analysis technique.
Avalos when addressing extending PCA analysis teaches, the principal components analysis technique comprises performing, by the computing system, a Bregman principal components analysis technique. (pg 1 Abstract “Our approach combines the benefits of the log-ratio transformation from compositional data analysis and exponential family PCA. A keytool in its derivation is a generalization of the scaled Bregman theorem, that relates the perspective transform of a Bregman divergence to the Bregman divergence of a perspective transform and a remainder conformal divergence. Our proposed approach includes a convenient surrogate (upper bound) loss of the exponential family PCA which has an easy to optimize form” A PCA algorithm which includes a Bregman divergence is a Bregman PCA technique)
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify PCA of Seddik to comprise a Bregman based PCA described by Avalos. One would have been motivated to make such a combination because Seddik and Avalos teaches PCA for dimensionality reduction. Further, Avalos describes the Bregman divergence as making PCA easy to optimize suitable for nonlinear autoencoders which are deep machine learning models “Our proposed approach includes a convenient surrogate (upper bound) loss of the exponential family PCA which has an easy to optimize form. We also derive the corresponding form for nonlinear autoencoders” (Abstract Avalos)
Claim 3
Seddik/Avalos teaches claim 2
Avalos teaches, performing, by the computing system, the Bregman principal components analysis technique comprises further generating (iii) a generalized mean vector. ( pg 2 Section 2 “Using Bregman divergences makes explicit a dual affine coordinate space… to first apply a log-ratio transformation to transpose the data into real Euclidean space…. The centered log-ratio transformation (clr) scales each subject vector by its geometric mean;” the geometric mean is a generalized mean vector)
Claim 4
Seddik/Avalos teaches claim 3
Avalos teaches, wherein the generalized mean vector comprises mean vector values that minimize a Bregman compression loss. (Section 3.2 pg 3 “As mentioned in the introduction, compositional data do not live in Euclidean space… From the Bayesian viewpoint, the PCA goal is to minimize a distance (L2 for usual PCA) which is equivalent to a Bregman divergence minimization…
PNG
media_image8.png
35
361
media_image8.png
Greyscale
” pg 4 “CoDA has found a workaround for the centering problem, centered log-ratio coordinates… the associated loss is the standard PCA loss on clr transformed data…
PNG
media_image9.png
43
512
media_image9.png
Greyscale
” pg 2
PNG
media_image10.png
53
448
media_image10.png
Greyscale
” as noted above C_KL is includes the geometric mean which is a value used to minimize the loss function claimed)
Claim 5
Seddik/Avalos teaches claim 3
Avalos when combined with Seddik teaches, the generalized mean vector comprises an inverse of an activation function of the layer of the teacher machine learning model applied to a mean of the one or more sets of coefficient values as an operator. (pg 2
PNG
media_image10.png
53
448
media_image10.png
Greyscale
” where g(x)=… the geometric mean of x” examiner notes here the C_KL term is based on an inverse of the mean vector g(x) which is an operator on the activation function because the data x which is being used as input to the PCA is the activation function of the layer of the teacher model as described in Seddik. Therefore, When Avalos is combined with Seddik the inverse mean vector is of the activation function values of the layers of the teacher maching learning model.)
Claim 17/19
Seddik/Avalos teach the limitations of claim 17/19 for the reasons set forth in the rejection of claim 2 in connection with the corresponding parent claims 16/18
Claim(s) 6 and 7 are rejected under 35 U.S.C. § 103 as being unpatentable over Seddik/Avalos further in view of Sato et al “Cholesky QR-based retraction on the generalized Stiefel manifold”
Claim 6
Seddik/Avalos teaches claim 2
Seddik/Avalos teaches, performing, by the computing system, the Bregman principal components analysis technique comprises (as noted in the rejection of claim 2)
Seddik/Avalos does not teach, enforcing an orthonormality constraint expressed with a Riemannian metric.
Sato however when addressing constraining optimization for matrix factorization teaches, enforcing an orthonormality constraint expressed with a Riemannian metric. (Introduction Section 1 “The Stiefel manifold with p<=n is the set (endowed with a manifold structure) of all nxp matrices whose columns are orthonormal… The generalized Stiefel manifold with an nxn symmetric positive definite matrix G is defined as
PNG
media_image11.png
76
411
media_image11.png
Greyscale
examiner notes that paragraph 0043 of the disclosure describes the very same equation for a generalized Stiefel manifold, where M in the disclosure or G in the art is defined as the Riemannian metric)
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify matrix decomposition of Seddik/Avalos to comprise the optimization under a generalized Stiefel manifold as described by Sato. One would have been motivated to make such a combination because Seddik/Avalos and Sato teach finding a solution to a generalized eigen value problem. Further, Sato notes. “Numerical experiments show that the proposed retraction is more efficient than the existing one based on the polar factorization” (abstract Sato)
Claim 7
Seddik/Avalos/Sato teaches claim 7
Seddik/Avalos teaches, performing, by the computing system, the Bregman principal components analysis technique (as noted in the rejection of claim 2)
Sato however when addressing constraining optimization for matrix factorization teaches, performing a QR decomposition technique such that a transpose of a first factor matrix times the Riemannian metric times the first factor matrix equals an identity matrix. (Introduction Section 1 “The Stiefel manifold with p<=n is the set (endowed with a manifold structure) of all nxp matrices whose columns are orthonormal… The generalized Stiefel manifold with an nxn symmetric positive definite matrix G is defined as
PNG
media_image11.png
76
411
media_image11.png
Greyscale
examiner notes that paragraph 0043 of the disclosure describes the very same equation for a generalized Stiefel manifold, where M in the disclosure or G in the art is defined as the Riemannian metric. Such a matrix factorization is a QR decomposition, as described by the disclosure.)
Claim(s) 14 are rejected under 35 U.S.C. § 103 as being unpatentable over Seddik further in view of Ye et al. “Lifelong Teacher-Student Network Learning”
Claim 14
Seddik teaches claim 1
Seddik teaches, training, by the computing system, the teacher machine learning model to predict one or more ground truth labels associated with the one or more training inputs when given the one or more training inputs as input; (pg 2 Section 2.2 “Given a set of n training samples X=[x1,...,xn]∈Rp0×n on which TN was initially trained to perform some arbitrary learning task.” Pg 3 Section 3 “we consider three teacher networks composed of L =4 dense layers which are trained to perform a classification problem respectively on the datasets MNIST” the teacher network is trained to predict ground truth classifications given training input) wherein performing, by the computing system, the principal components analysis technique comprises performing an online principal components analysis technique. (algorithm 1 pg 2
PNG
media_image12.png
237
482
media_image12.png
Greyscale
PCA is performed and simultaneously to the student training online because it is performed during training of a student network )
Seddik does not explicitly teach, simultaneous to training the student machine learning model to predict the one or more sets of coefficient values: [train the teacher machine learning model]
Ye however when addressing simultaneous training of student and teacher networks teaches, simultaneous to training the student machine learning model to predict the one or more sets of coefficient values: [train the teacher machine learning model] (pg 7 algorithm 1 “
PNG
media_image13.png
163
417
media_image13.png
Greyscale
” as shown the student and teacher network are trained simultaneously.)
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify training of Seddik to comprise simultaneous training of both student and teacher networks described by Ye. One would have been motivated to make such a combination because Seddik and Ye teaches learning a student network based on the context of a teacher network. Further, Ye notes that continuous joint learning allows for a lifelong model in which “experiments do not only focus on learning classification tasks but they also aim to learn disentangled data representations, under the lifelong learning setting” in which teacher and student mutually learn from each other in “an artificial symbiosis system of two networks”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHNATHAN R GERMICK whose telephone number is (571)272-8363. The examiner can normally be reached M-F 9:30-4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached on 571-272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.R.G./
Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122
