DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is responsive to the Application filed on 10/13/2023. Claims 1-17 are pending in the case. Claims 1, 16, and 17 are independent claims.
Claim Rejections - 35 U.S.C. § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. §§ 102 and 103 (or as subject to pre-AIA 35 U.S.C. §§ 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5, 10, and 12-17 are rejected under 35 U.S.C. § 103 as being unpatentable over Furlanello et al. (Furlanello, Tommaso, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. "Born again neural networks." In International conference on machine learning, pp. 1607-1616. PMLR, 2018, hereinafter Furlanello) in view of Hinton et al. (Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network." arXiv preprint arXiv:1503.02531 (2015), hereinafter Hinton) and Srivastava et al. (Srivastava, Nitish, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. "Dropout: a simple way to prevent neural networks from overfitting." The journal of machine learning research 15, no. 1 (2014): 1929-1958, hereinafter Srivastava).
As to independent claims 1, 16, and 17, Furlanello teaches a learning method comprising:
performing learning of a second learning model having an arrangement that is at least partially the same as an arrangement of a first learning model by distillation learning using an output of the first learning model (Abstract, "we train students parameterized identically to their teachers". The student reads on the second learning model and the teacher reads on the first learning model. Identical parameterization reads on at least partially the same. Page 1, "the student cannot match the teacher when trained directly on the data, the distillation process brings the student closer to matching the predictive power of the teacher");….
Furlanello does not appear to expressly teach dynamically changing, during the learning of the second learning model, at least one of a parameter of the first learning model, the arrangement of the first learning model, a parameter of the second learning model, and the arrangement of the second learning model.
Hinton teaches dynamically changing, during the learning of the second learning model, at least one of a parameter of the first learning model, the arrangement of the first learning model (Page 2, "general solution, called 'distillation', is to raise the temperature of the final softmax until the cumbersome model produces a suitably soft set of targets". Cumbersome model reads on the first learning model).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the student model of Furlanello to include the dynamic changing techniques of Hinton to improve generalization and reduce overfitting.
Srivastava teaches dynamically changing, during the learning of the second learning model,… the arrangement of the first learning model, a parameter of the second learning model, and the arrangement of the second learning model (Abstract, "randomly drop units (along with their connections) from the neural network during training").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the student model of Furlanello to include the dynamic changing techniques of Srivastava to improve generalization and reduce overfitting.
As to dependent claim 5, Srivastava further teaches during the learning of the second learning model, a connection between neurons in a fully-connected layer of the first learning model is dynamically changed (Abstract, "The key idea is to randomly drop units (along with their connections) from the neural network during training").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the student model of Furlanello to include the dynamic changing techniques of Srivastava to improve generalization and reduce overfitting.
As to dependent claim 10, Srivastava further teaches during the learning of the second learning model, a connection between neurons in a fully-connected layer of the second learning model is dynamically changed (Abstract, "The key idea is to randomly drop units (along with their connections) from the neural network during training").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the student model of Furlanello to include the dynamic changing techniques of Srivastava to improve generalization and reduce overfitting.
As to dependent claim 12, Furlanello further teaches the parameter of the first learning model is set as an initial value of the parameter of the second learning model (Page 1, "the distillation process brings the student closer to matching the predictive power of the teacher").
As to dependent claim 13, Hinton further teaches using teacher data used at the time of learning of the first learning model, learning of the second learning model learned by the distillation learning is performed (Page 3, "distillation, knowledge is transferred to the distilled model by training it on a transfer set and using a soft target distribution for each case in the transfer set that is produced by using the cumbersome model with a high temperature in its softmax").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the student model of Furlanello to include the dynamic changing techniques of Hinton to improve generalization and reduce overfitting.
As to dependent claim 14, Furlanello further teaches by the distillation learning using the output of the first learning model, learning of another second learning model set with the parameter of the second learning model learned by the learning is performed (Page 4, "apply BANs sequentially with multiple generations of knowledge transfer. In each case, the k-th model is trained, with knowledge transferred from the k - 1-th student").
As to dependent claim 15, Furlanello further teaches the first learning model is a learned model (Abstract, "Knowledge Distillation (KD) consists of transferring “knowledge” from one machine learning model (the teacher) to another (the student).").
Claims 2 and 7 are rejected under 35 U.S.C. § 103 as being unpatentable over Furlanello in view of Hinton, Srivastava, and Jang et al. (Jang, Eric, Shixiang Gu, and Ben Poole. "Categorical reparameterization with gumbel-softmax." arXiv preprint arXiv:1611.01144 (2016), hereinafter Jang).
As to dependent claim 2, the rejection of claim 1 is incorporated.
Hinton further teaches during the learning of the second learning model, a temperature of a softmax function with temperature as an activation function of a final output layer of the first learning model (Page 2, "using a “softmax” output layer". Page 3, "Using a higher value for T produces a softer probability distribution over classes").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the student model of Furlanello to include the dynamic changing techniques of Hinton to improve generalization and reduce overfitting.
Furlanello does not appear to expressly teach a temperature of a softmax function with temperature as an activation function of a final output layer of the first learning model is dynamically changed.
Jang teaches a temperature of a softmax function with temperature as an activation function of a final output layer of the first learning model is dynamically changed (Page 3, "the softmax temperature T can be annealed according to a variety of schedules and still perform well").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the student model of Furlanello to include the dynamic changing techniques of Jang to provide a simple, differentiable approximate sampling mechanism for categorical variables that can be integrated into neural networks and trained using standard backpropagation (see Jang at page 1).
As to dependent claim 7, the rejection of claim 1 is incorporated.
Hinton further teaches during the learning of the second learning model, a temperature of a softmax function with temperature as an activation function of a final output layer of the second learning model (Page 2, "using a “softmax” output layer". Page 3, "Using a higher value for T produces a softer probability distribution over classes").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the student model of Furlanello to include the dynamic changing techniques of Hinton to improve generalization and reduce overfitting.
Furlanello does not appear to expressly teach a temperature of a softmax function with temperature as an activation function of a final output layer of the second learning model is dynamically changed.
Jang teaches a temperature of a softmax function with temperature as an activation function of a final output layer of the second learning model is dynamically changed (Page 3, "the softmax temperature T can be annealed according to a variety of schedules and still perform well").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the student model of Furlanello to include the dynamic changing techniques of Jang to provide a simple, differentiable approximate sampling mechanism for categorical variables that can be integrated into neural networks and trained using standard backpropagation (see Jang at page 1).
Claims 6 and 11 are rejected under 35 U.S.C. § 103 as being unpatentable over Furlanello in view of Hinton, Srivastava, and DeVries et al. (DeVries, Terrance, and Graham W. Taylor. "Improved regularization of convolutional neural networks with cutout. arXiv 2017." arXiv preprint arXiv:1708.04552 10 (2017), hereinafter DeVries).
As to dependent claim 6, the rejection of claim 1 is incorporated.
Furlanello does not appear to expressly teach during the learning of the second learning model, pixel values of some or all of pixels in an image to be input to the first learning model are dynamically changed.
DeVries teaches during the learning of the second learning model, pixel values of some or all of pixels in an image to be input to the first learning model are dynamically changed (Abstract, "simple regularization technique of randomly masking out square regions of input during training, which we call cutout, can be used to improve the robustness and overall performance of convolutional neural networks").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the student model of Furlanello to include the image augmentation techniques of DeVries to improve the robustness and overall performance of convolutional neural networks (see DeVries at abstract).
As to dependent claim 11, the rejection of claim 1 is incorporated.
Furlanello does not appear to expressly teach during the learning of the second learning model, pixel values of some or all of pixels in an image to be input to the second learning model are dynamically changed.
DeVries teaches during the learning of the second learning model, pixel values of some or all of pixels in an image to be input to the second learning model are dynamically changed (Abstract, "simple regularization technique of randomly masking out square regions of input during training, which we call cutout, can be used to improve the robustness and overall performance of convolutional neural networks").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the student model of Furlanello to include the image augmentation techniques of DeVries to improve the robustness and overall performance of convolutional neural networks (see DeVries at abstract).
Allowable Subject Matter
Claims 3, 4, 8, and 9 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Citation of Pertinent Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Seki et al. (U.S. Pat. App. Pub. No. 2023/0054706) teaches a learning apparatus for training a student model with a teacher model includes a processor. The processor computes the performance difference between the teacher and student models. The processor makes at least one of a determination, based on the performance difference, of whether to use the teacher model and a determination, based on the performance difference, of whether to change the weight coefficient in calculating the loss in the student model.
Conclusion
The prior art made of record and not relied upon is considered pertinent to Applicant's disclosure. Applicant is required under 37 C.F.R. § 1.111(c) to consider these references fully when responding to this action.
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Casey R. Garner whose telephone number is 571-272-2467. The examiner can normally be reached Monday to Friday, 8am to 5pm, Eastern Time.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached on 571-270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center and the Private Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from Patent Center or Private PAIR. Status information for unpublished applications is available through Patent Center and Private PAIR to authorized users only. Should you have questions about access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/Casey R. Garner/Primary Examiner, Art Unit 2123
Prosecution Insights