Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 1 is/are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation "the base neural network" in lines 13, 16, 29, and 30 without reciting “a base neural network”. There is insufficient antecedent basis for this limitation in the claim. Dependent claims 2-7 and 9-14 incorporate by reference all the limitations of claim 1 and are rejected on the same grounds as above.
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claims 10 and 12 is/are rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends.
Claim 10 depends from claim 1 and recites “…wherein the system has access to a trained neural network and is configured to determine the base neural network by iteratively performing the training process comprising: forming sample data…”, “selecting, in dependence on the sample data…”, “forming a trained candidate base neural network by…”, “adopting the trained candidate base neural network…”, and “after multiple iterations of the training process, adopting a current candidate base neural network…”. However, claim 1 recites “wherein the base neural network is determined from a trained neural network by iteratively performing a further training process comprising: forming further sample data…”, “selecting, in dependence on the further sample data…”, “forming a trained candidate base neural network by…”, “adopting the trained candidate base neural network…”, and “after multiple iterations of the training process, adopting the current candidate base neural network…”. Thus, claim 10 does not further limit the subject matter of the claim from which it depends and fails to comply with 35 U.S.C. 112(d).
Claim 12 depends from claim 3 which depends from claim 1, and recites “wherein the base neural network is a teaching assistant network for facilitating formation of the simplified neural network”. However, claim 1 recites “wherein the base neural network is a teaching assistant network for facilitating formation of the simplified neural network”. Thus, claim 12 does not further limit the subject matter of the claim from which it depends and fails to comply with 35 U.S.C. 112(d).
Applicant may cancel the claim(s), amend the claim(s) to place the claim(s) in proper dependent form, rewrite the claim(s) in independent form, or present a sufficient showing that the dependent claim(s) complies with the statutory requirements.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 7, 9-13, and 15-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gu et al. (NPL from IDS: Search for Better Students to Learn Distilled Knowledge, published Jan. 2020, hereinafter “Gu”) in view of Xie et al. (NPL: Exploring Randomly Wired Neural Networks for Image Recognition, published April 2019, hereinafter “Xie”), and further in view of Mirzadeh et al. (NPL from IDS: Improved Knowledge Distillation via Teacher Assistant, published March 2020, hereinafter “Mirzadeh”).
Regarding claim 1, Gu teaches a machine learning system comprising one or more computers, each of the one or more computers comprising a processor and a memory storing instructions that, when executed by the processor, cause the processor to determine a simplified neural network by iteratively performing a training process comprising:
forming sample data by sampling an architecture of a current candidate neural network (Gu, Section 3 Paragraph 1 – “The search space is the topology graph of the teacher model. Each channel in the model is taken as a node, and weights connecting nodes as edges. By removing nodes (channels) and all edges directly connected to those nodes, we can obtain a subgraph, which corresponds to smaller neural network architecture” – teaches forming sample data by sampling an architecture of a current candidate neural network (search space is topology graph of teacher model, which is the current candidate neural network));
selecting, in dependence on the sample data, an architecture for a second candidate neural network (Gu, Section 3 Paragraph 1 – “The search space is the topology graph of the teacher model. Each channel in the model is taken as a node, and weights connecting nodes as edges. By removing nodes (channels) and all edges directly connected to those nodes, we can obtain a subgraph, which corresponds to smaller neural network architecture” – teaches selecting, in dependence on the sample data, an architecture for a second candidate neural network (search space is topology graph of teacher model, by removing nodes and edges a subgraph, or architecture, can be obtained for a second candidate neural network, or student));
forming a trained candidate neural network by training the second candidate neural network, wherein the training of the second candidate neural network comprises applying feedback to the second candidate neural network in dependence on a comparison of behaviours of the second candidate neural network and the base neural network (Gu, Section 3.1 Paragraph 2 – “In knowledge distillation process, the logits of teacher network at can offer more information to train the student network. One way to leverage such information is to match the softened outputs of student softmax(as/τ ) and teacher ft(xi) = softmax(at/τ ) via a KL-divergence loss” – teaches forming a trained candidate neural network by training the second candidate neural network (trains the student network), wherein the training of the second candidate neural network comprises applying feedback to the second candidate neural network based on a comparison of behaviors of the second candidate neural network and the base neural network (trains student network by matching softened outputs, or behaviors, of student and teacher via a KL-divergence loss, thus training the second candidate neural network in dependence on a comparison of behaviors of the second candidate and base neural networks)); and
adopting the trained candidate neural network as a current candidate neural network for a subsequent iteration of the training process (Gu, Section 3.2 Paragraph 1 – “In the last subsection, we introduce our distillation-ware loss function. The weights w and the scaling factors g are updated to minimize the loss function. The loss function is differentiable to the weights. The weights w can be updated by Stochastic Gradient Descent (SGD) with momentum or its variants.” and Fig. 1 description – “A new model is constructed on the teacher model by multiplying scaling factors. After an optimization process, the channels with zero scaling factors are removed. The remained small architecture is the selected student architecture.” – teaches adopting the trained candidate neural network as a current candidate neural network for a subsequent iteration of the training process (loss functions are minimized by updating weights w and scaling factors, new model is built on teacher model and remaining small architecture is selected as student architecture, thus adopting trained candidate neural network as current candidate neural network for subsequent iterations of training)).
wherein the base neural network is determined from a trained neural network by iteratively performing a further training process comprising:
forming sample data by sampling the architecture of a current candidate base neural network (Gu, Section 3 Paragraph 1 – “The search space is the topology graph of the teacher model. Each channel in the model is taken as a node, and weights connecting nodes as edges. By removing nodes (channels) and all edges directly connected to those nodes, we can obtain a subgraph, which corresponds to smaller neural network architecture” – teaches forming sample data by sampling an architecture of a current candidate neural network (search space is topology graph of teacher model, which is the current candidate base neural network));
selecting, in dependence on the sample data, an architecture for a second candidate base neural network (Gu, Section 3 Paragraph 1 – “The search space is the topology graph of the teacher model. Each channel in the model is taken as a node, and weights connecting nodes as edges. By removing nodes (channels) and all edges directly connected to those nodes, we can obtain a subgraph, which corresponds to smaller neural network architecture” – teaches selecting, in dependence on the sample data, an architecture for a second candidate base neural network (search space is topology graph of teacher model, by removing nodes and edges a subgraph, or architecture, can be obtained for a second candidate base neural network, or student));
forming a trained candidate base neural network by training the second candidate base neural network, wherein the training of the second candidate base neural network comprises applying feedback to the second candidate base neural network in dependence on a comparison of behaviours of the second candidate base neural network and the trained neural network (Gu, Section 3.1 Paragraph 2 – “In knowledge distillation process, the logits of teacher network at can offer more information to train the student network. One way to leverage such information is to match the softened outputs of student softmax(as/τ ) and teacher ft(xi) = softmax(at/τ ) via a KL-divergence loss” – teaches forming a trained candidate base neural network by training the second candidate base neural network (trains the student network), wherein the training of the second candidate base neural network comprises applying feedback to the second candidate base neural network based on a comparison of behaviors of the second candidate base neural network and the trained neural network (trains student network, or second candidate base neural network, by matching softened outputs, or behaviors, of student and teacher via a KL-divergence loss, thus training the second candidate base neural network in dependence on a comparison of behaviors of the second base candidate and trained neural networks));
adopting the trained candidate base neural network as a current candidate base neural network for a subsequent iteration of the further training process (Gu, Section 3.2 Paragraph 1 – “In the last subsection, we introduce our distillation-ware loss function. The weights w and the scaling factors g are updated to minimize the loss function. The loss function is differentiable to the weights. The weights w can be updated by Stochastic Gradient Descent (SGD) with momentum or its variants.”, Fig. 1 description – “A new model is constructed on the teacher model by multiplying scaling factors. After an optimization process, the channels with zero scaling factors are removed. The remained small architecture is the selected student architecture.”, and in Section 3.1 Last Paragraph – “At the end of the optimization, we remove all the channels with closed gated from the constructed model. The remaining small architecture is taken as the student architecture. Early stopping is applied to obtain an architecture with a certain number of FLOPs.” – teaches adopting the trained candidate neural network as a current candidate neural network for a subsequent iteration of the training process (new model is built on teacher model and remaining architecture, which may be obtained at a certain number of FLOPs with early stopping, is the selected architecture, thus adopting trained candidate base neural network as current base candidate neural network for subsequent iterations of training by updating weights and scaling factors to minimize a loss function)); and
after multiple iterations of the training process, adopting a current candidate base neural network as the base neural network (Gu, Fig. 1 description – “A new model is constructed on the teacher model by multiplying scaling factors. After an optimization process, the channels with zero scaling factors are removed. The remained small architecture is the selected student architecture.” and in Section 3.1 Last Paragraph – “At the end of the optimization, we remove all the channels with closed gated from the constructed model. The remaining small architecture is taken as the student architecture. Early stopping is applied to obtain an architecture with a certain number of FLOPs.” – teaches adopting a current candidate base neural network as the base neural network (remaining architecture, or the constructed new model, that was built on teacher model is the selected student architecture)),
Gu fails to explicitly wherein selecting is performed by optimization over a stochastic graph generator as a search space for network architecture generation.
However, analogous to the field of the claimed invention, Xie teaches:
selecting an architecture for a second candidate neural network, wherein the selecting is performed by optimization over a stochastic graph generator as a search space for network architecture generation (Xie, Pg. 3, Col. 1, Paragraph 4 – “The above network generator g(θ) performs a deterministic mapping: given the same θ, it always returns the same network architecture n. We can extend g to accept an additional argument s that is the seed of a pseudo-random number generator that is used internally by g. Given this seed, one can construct a (pseudo) random family of networks by calling g(θ,s) multiple times, keeping θ fixed but changing the value of s =1,2,3,.... For a fixed value of θ, a uniform probability distribution over all possible seed values induces a (likely non-uniform) probability distribution over N. We call generators of the form g(θ,s) stochastic network generators.”, Pg. 5, Col. 2, Paragraph 6 – “Our randomly wired neural networks are generated by a stochastic network generator g(θ,s). The random graph parameters, namely, P, M, (K,P) in ER, BA, WS respectively, are part of the parameters θ. The “optimization” of such a 1- or 2-parameter space is essentially done by trial and-error by human designers, e.g., by line/grid search. Conceptually, such “optimization” is not distinct from many other designs involved in our and other models (including NAS), e.g., the number of nodes, stages, and filters.”, and in Pg. 5, Col. 2, Paragraph 7 – “Optimization can also be done by scanning the random seed s, which is an implementation of random search. Random search is possible for any stochastic network generator, including ours and NAS.” – teaches selecting an architecture for a second candidate neural network (constructs family of networks by calling g(θ,s)), wherein selecting is performed by optimization over a stochastic graph generator (g(θ,s)) as a search space for network architecture generation (randomly wired neural networks are generated by stochastic network generator g(θ,s), θ includes random graph parameters, and performs optimization over the stochastic network generator as a search space for generation, or construction, of randomly wired neural networks));
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the stochastic graph generator and architecture selection methods of Xie to the base networks, candidate networks, and training processes of Gu in order to provide an architecture search space and select an architecture for a candidate base neural network. Doing so would produce multiple networks that outperform or are comparable to their fully manually designed counter parts and networks found by various neural architecture search methods, and optimize network generator design to provide an optimized probabilistic distribution over networks or architectures (Xie, Introduction).
The combination of Gu and Xie fails to explicitly teach wherein the base neural network is a teaching assistant network for facilitating the formation of the simplified neural network.
However, analogous to the field of the claimed invention, Mirzadeh teaches:
wherein the base neural network is a teaching assistant network for facilitating the formation of the simplified neural network (Mirzadeh, Page 3 Col. 2 Paragraph 4 – “In this paper, we propose to use intermediate-size networks to fill in the gap between them. The teacher assistant (TA) lies somewhere in between teacher and student in terms of size or capacity. First, the TA network is distilled from the teacher. Then, the TA plays the role of a teacher and trains the student via distillation. This strategy will alleviate factor 2 in the previous subsection by being closer to the student than the teacher. Therefore, the student is able to fit TA’s logit distribution more effectively than that of the teacher’s.” – teaches wherein the base neural network is a teaching assistant network for facilitating formation of the simplified network (teacher assistant lies between teacher and student, plays role of teacher and trains student via distillation, student is able to fit TA’s logit distribution more effectively, thus facilitating the formation of the simplified neural network)).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the teaching assistant network of Mirzadeh to the teachers, candidate neural networks, and systems of Gu and Xie in order to utilize a base neural network as a teaching assistant network distilled from the teacher for facilitating formation of a simplified neural network. Doing so would allow compression of large deep neural networks to be employed on edge devices while limiting the degradation of performance (Mirzadeh, Abstract) and fill the gap in size between teacher and student models (Mirzadeh, Introduction).
Claim 15 incorporates substantively all of the limitations of claim 1 in a computer-implemented method and is rejected on the same grounds as above.
Regarding claim 2, the combination of Gu, Xie, and Mirzadeh teaches a machine learning system as claimed in claim 1, the training process further comprises, after multiple iterations of the training process, outputting a current candidate neural network as the simplified neural network (Gu, Section 3.1 Paragraph 5 (Last Paragraph) – “At the end of the optimization, we remove all the channels with closed gated from the constructed model. The remaining small architecture is taken as the student architecture” – teaches outputting a current candidate neural network as the simplified network (at the end of optimization, the small architecture is taken as the student architecture)).
Claim 16 is similar to claim 2, hence similarly rejected.
Regarding claim 3, the combination of Gu, Xie, and Mirzadeh teaches a machine learning system as claimed in claim 1, wherein the simplified neural network has a smaller capacity and/or is less computationally intensive to implement than the base neural network (Gu, Abstract – “The architecture of the small student is often chosen to be similar to their teacher’s, with fewer layers or fewer channels, or both”, Section 3 Paragraph 1 – “The search space is the topology graph of the teacher model. Each channel in the model is taken as a node, and weights connecting nodes as edges. By removing nodes (channels) and all edges directly connected to those nodes, we can obtain a subgraph, which corresponds to smaller neural network architecture” and in Section 3.1 Paragraph 5 (Last Paragraph) – “At the end of the optimization, we remove all the channels with closed gated from the constructed model. The remaining small architecture is taken as the student architecture” – teaches wherein the simplified neural network has a smaller capacity and/or is less computationally intensive to implement than the base neural networks (selected student network is smaller than teacher network, thus smaller in capacity and/or less computationally intensive to implement than the teacher network)).
Claim 17 is similar to claim 3, hence similarly rejected.
Regarding claim 7, the combination of Gu, Xie, and Mirzadeh teaches a machine learning system as claimed in claim 1, wherein the sample data is formed by sampling the current candidate neural network according to a predetermined acquisition function (Gu, Section 3 Paragraph 1 – “The search space is the topology graph of the teacher model. Each channel in the model is taken as a node, and weights connecting nodes as edges”, Section 3 Paragraph 2 – “In our search space, channels are taken as individual units. Therefore, we apply a structured pruning method to get student architectures”, and in Section 3 Paragraph 3 – “More concretely, we specify a gate on each channel by multiplying the activation map of the channel by a scaling factor g. At the end of the optimization, the open gate (g 6= 0) means the corresponding channel is important to the distillation process, while the closed gate (g = 0) means the corresponding channel can be removed safely. For a layer with K channels in a teacher neural network, the corresponding g is a K-element vector. The number of channels of the obtained student architecture in this layer is identified by the number of non-zero elements in the vector g. A simple demonstration is shown in Figure 1” – teaches wherein the sample data (topology graph is the search space) is formed by sampling the current candidate neural network (topology graph of teacher model) according to a predetermined acquisition function (apply structured pruning method to get student architectures, by specifying a gate, thus an acquisition function, on each channel)).
Regarding claim 9, the combination of Gu, Xie, and Mirzadeh teaches a machine learning system as claimed in claim 1, wherein the forming the trained candidate neural network comprises causing the second candidate neural network to perform a plurality of tasks, causing the base neural network to perform the plurality of tasks, and modifying the second candidate neural network in dependence on a variance in performance between the second candidate neural network and the base neural network in performing the tasks (Gu, Section 3.1 Paragraph 2 – “In knowledge distillation process, the logits of teacher network at can offer more information to train the student network. One way to leverage such information is to match the softened outputs of student softmax(as/τ ) and teacher ft(xi) = softmax(at/τ ) via a KL-divergence loss… The overall loss to train the student network is Ls = LKD + λLCE where the hyperparameter λ is often set to a very small value, the second term works by regularizing the training process.” and in Section 3.1 Paragraph 3 – “Given an input xi, the softened output of a teacher model is ft(xi), and the softened output of the model constructed by adding gates is fs(xi, w, g), i.e., the constructed model in Figure 1. The weights and scaling factors therein are updated during the optimization. The loss function we propose is mathematically defined as follows. Eq (3)” – teaches wherein the forming the trained candidate neural network comprises causing the second candidate neural network to perform a plurality of tasks, causing the base neural network to perform the plurality of tasks, and modifying the second candidate neural network in dependence on a variance in performance between the second candidate neural network and the base neural network (in Eq. 3 the teacher ft and student fs perform over the same task xi- and the difference in performance between the teacher and student on the same task is used to modify the student)).
Regarding claim 10, the combination of Gu, Xie, and Mirzadeh teaches a machine learning system as claimed in claim 1, wherein the system has access to a trained neural network and is configured to determine the base neural network by iteratively performing a training process comprising:
forming sample data by sampling an architecture of a current candidate base neural network (Gu, Section 3 Paragraph 1 – “The search space is the topology graph of the teacher model. Each channel in the model is taken as a node, and weights connecting nodes as edges. By removing nodes (channels) and all edges directly connected to those nodes, we can obtain a subgraph, which corresponds to smaller neural network architecture” – teaches forming sample data by sampling an architecture of a current candidate neural network (search space is topology graph of teacher model, which is the current candidate base neural network));
selecting, in dependence on the sample data, an architecture for a second candidate base neural network (Gu, Section 3 Paragraph 1 – “The search space is the topology graph of the teacher model. Each channel in the model is taken as a node, and weights connecting nodes as edges. By removing nodes (channels) and all edges directly connected to those nodes, we can obtain a subgraph, which corresponds to smaller neural network architecture” – teaches selecting, in dependence on the sample data, an architecture for a second candidate neural network (search space is topology graph of teacher model, by removing nodes and edges a subgraph, or architecture, can be obtained for a second candidate base neural network));
forming a trained candidate base neural network by training the second candidate base neural network, wherein the training of the second candidate base neural network comprises applying feedback to the second candidate base neural network in dependence on a comparison of behaviours of the second candidate base neural network and the trained neural network (Gu, Section 3.1 Paragraph 2 – “In knowledge distillation process, the logits of teacher network at can offer more information to train the student network. One way to leverage such information is to match the softened outputs of student softmax(as/τ ) and teacher ft(xi) = softmax(at/τ ) via a KL-divergence loss” – teaches forming a trained candidate base neural network by training the second candidate base neural network (trains the student network), wherein the training of the second candidate neural network comprises applying feedback to the second candidate neural network based on a comparison of behaviors of the second candidate base neural network and the trained neural network (trains student network by matching softened outputs, or behaviors, of student and teacher via a KL-divergence loss, thus training the second candidate neural network in dependence on a comparison of behaviors of the second candidate and trained neural network)); and
adopting the trained candidate base neural network as a current candidate base neural network for a subsequent iteration of the training process (Gu, Section 3.2 Paragraph 1 – “In the last subsection, we introduce our distillation-ware loss function. The weights w and the scaling factors g are updated to minimize the loss function. The loss function is differentiable to the weights. The weights w can be updated by Stochastic Gradient Descent (SGD) with momentum or its variants.” and Fig. 1 description – “A new model is constructed on the teacher model by multiplying scaling factors. After an optimization process, the channels with zero scaling factors are removed. The remained small architecture is the selected student architecture.” – teaches adopting the trained candidate neural network as a current candidate neural network for a subsequent iteration of the training process (new model is built on teacher model and remaining small architecture is selected architecture, thus adopting trained candidate neural network as current candidate neural network for subsequent iterations of training by updating weights and scaling factors to minimize a loss function)); and
after multiple iterations of the training process, adopting a current candidate base neural network as the base neural network (Gu, Fig. 1 description – “A new model is constructed on the teacher model by multiplying scaling factors. After an optimization process, the channels with zero scaling factors are removed. The remained small architecture is the selected student architecture.” – teaches adopting a current candidate base neural network as the base neural network (remaining architecture that was built on teacher model is the selected architecture)).
Regarding claim 11, the combination of Gu, Xie, and Mirzadeh teaches a machine learning system as claimed in claim 10, wherein the base neural network has a smaller capacity and/or is less computationally intensive to implement than the trained neural network (Gu, Abstract – “The architecture of the small student is often chosen to be similar to their teacher’s, with fewer layers or fewer channels, or both”, Section 3 Paragraph 1 – “The search space is the topology graph of the teacher model. Each channel in the model is taken as a node, and weights connecting nodes as edges. By removing nodes (channels) and all edges directly connected to those nodes, we can obtain a subgraph, which corresponds to smaller neural network architecture” and in Section 3.1 Paragraph 5 (Last Paragraph) – “At the end of the optimization, we remove all the channels with closed gated from the constructed model. The remaining small architecture is taken as the student architecture” – teaches wherein the base neural network (selected architecture) has a smaller capacity and/or is less computationally intensive to implement than the trained neural networks (selected architecture is smaller than teacher network, thus smaller in capacity and/or less computationally intensive to implement than the teacher network)).
Regarding claim 12, the combination of Gu, Xie, and Mirzadeh teaches a machine learning system as claimed in claim 10.
The combination of Gu and Xie fails to explicitly teach wherein the base neural network is a teaching assistant network for facilitating formation of the simplified neural network
However, analogous to the field of the claimed invention, Mirzadeh teaches:
wherein the base neural network is a teaching assistant network for facilitating formation of the simplified neural network (Mirzadeh, Page 3 Col. 2 Paragraph 4 – “The teacher assistant (TA) lies somewhere in between teacher and student in terms of size or capacity. First, the TA network is distilled from the teacher. Then, the TA plays the role of a teacher and trains the student via distillation. This strategy will alleviate factor 2 in the previous subsection by being closer to the student than the teacher. Therefore, the student is able to fit TA’s logit distribution more effectively than that of the teacher’s.” – teaches wherein the base neural network is a teaching assistant network for facilitating formation of the simplified network (teacher assistant lies between teacher and student, plays role of teacher and trains student via distillation, student is able to fit TA’s logit distribution more effectively)).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the teaching assistant networks of Mirzadeh to further modify the base neural network and systems of Gu, Xie, and Mirzadeh in order to utilize teacher assistant networks for facilitating formation of the simplified network. Doing so would allow softer, less confident targets (Mirzadeh, Page 3 Col. 2 Paragraph 4) and fill the gap in size between teacher and student models (Mirzadeh, Introduction).
Regarding claim 13, the combination of Gu, Xie, and Mirzadeh teaches a machine learning system as claimed in claim 1, the system being configured to cause installation of the simplified neural network for execution on a device having lower computational complexity than the one or more computers (Gu, Introduction Paragraph 1 – “The computationally expensive inferences prevent the deploy of deep neural networks in small devices with limited memory size or latency-critical applications such as smartphones and self-driving cars.” and in Paragraph 3 – “The students trained under distillation are closer in performance to their larger teacher. The lower computational cost and memory footprint of the powerful student make its deployment much easier” – teaches the system configured to install the simplified network for execution on a device having lower computational complexity than the one or more computers (student network has lower computational cost and memory footprint thus easier to deploy on small device with limited memory size or latency-critical applications)).
Claim(s) 4-6 and 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gu, Xie, and Mirzadeh as applied to claims 1 and 15 above, and further in view of Shi et al. (NPL: Multi-objective Neural Architecture Search via Predictive Network Performance Optimization, published Nov. 2019, hereinafter “Shi”).
Regarding claim 4, the combination of Gu, Xie, and Mirzadeh teaches a machine learning system as claimed in claim 1.
The combination of Gu, Xie, and Mirzadeh fails to explicitly teach wherein the selecting an architecture for the second candidate neural network is performed by Bayesian optimization.
However, analogous to the field of the claimed invention, Shi teaches:
wherein the selecting an architecture for the second candidate neural network is performed by Bayesian optimization (Shi, Section 3.4 Paragraph 2 – “The algorithm of our proposed BOGCN-NAS is illustrated in Algorithm 1. Given the search space A, we initialize trained architecture sets U containing architectures (Ai , Xi) with their performance ti = {f1i , . . . , fmi}…Based on tˆj and multi-objective formulation (Section 3.1), we can generate a estimated Pareto Front and sample estimated Pareto optimal models as set S and fully-train them to obtain the true objective values tj .” – teaches selecting an architecture for the second candidate neural network performed by multi-objective Bayesian optimization (selects architecture from architecture sets based on multi-objective function)).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the architecture selection performed by Bayesian optimization of Shi to the search space and candidate architectures of Gu, Xie, and Mirzadeh in order to select an architecture by Bayesian optimization. As the search strategies of NAS include random search Bayesian optimization (Gu, Introduction) and doing so would enable NAS algorithms to consider many other objectives such as speed/accuracy trade-off (Shi, Introduction).
Claim 18 is similar to claim 4, hence similarly rejected.
Regarding claim 5, the combination of Gu, Xie, Mirzadeh, and Shi teaches a machine learning system as claimed in claim 4, wherein the selecting an architecture for the second candidate neural network is performed by multi- objective Bayesian optimization (Shi, Section 3.4 Paragraph 2 – “The algorithm of our proposed BOGCN-NAS is illustrated in Algorithm 1. Given the search space A, we initialize trained architecture sets U containing architectures (Ai , Xi) with their performance ti = {f1i , . . . , fmi}…Based on tˆj and multi-objective formulation (Section 3.1), we can generate a estimated Pareto Front and sample estimated Pareto optimal models as set S and fully-train them to obtain the true objective values tj .” – teaches selecting an architecture for the second candidate neural network performed by multi-objective Bayesian optimization (selects architecture from architecture sets based on multi-objective function)).
Claim 19 is similar to claim 5, hence similarly rejected.
Regarding claim 6, the combination of Gu, Xie, Mirzadeh, and Shi teaches a machine learning system as claimed in claim 4, wherein the selecting an architecture for the second candidate neural network is performed by Bayesian optimization having one or more objectives (Shi, Section 3.4 Paragraph 2 – “The algorithm of our proposed BOGCN-NAS is illustrated in Algorithm 1. Given the search space A, we initialize trained architecture sets U containing architectures (Ai , Xi) with their performance ti = {f1i , . . . , fmi}…Based on tˆj and multi-objective formulation (Section 3.1), we can generate a estimated Pareto Front and sample estimated Pareto optimal models as set S and fully-train them to obtain the true objective values tj .” – teaches selecting an architecture for the second candidate neural network performed by multi-objective Bayesian optimization (selects architecture from architecture sets based on multi-objective function)), wherein at least one of said objectives comprises (i) improved classification accuracy of the second candidate neural network and/or (ii) reduced computational intensiveness of the second candidate neural network (Shi, Section 3.1 Paragraph 1 – “We formulate NAS problem as a multi-objective optimization problem over the architecture search space A where objective functions can be accuracy, latency, number of parameters, etc. We aim to find architectures on the Pareto front of A. Specifically, when m = 1, it reduces to single-objective (usually accuracy) NAS” – teaches wherein at least on of said objectives comprises improved classification accuracy of the second candidate neural network (objective functions can be accuracy… single-objective function is usually accuracy) and/or reduced computational intensiveness of the second candidate neural network (objective functions can be latency or number of parameters, which would reduce the computational intensiveness of the candidate neural network)).
Claim 20 is similar to claim 6, hence similarly rejected.
Claim(s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gu, Xie, and Mirzadeh as applied to claims 1 and 15 above, and further in view of Xu et al. (US Pub. No. 2022/0130142, effective filing date of July 2019 (all citations found in Chinese Patent Application No. 201910627480.7), hereinafter “Xu”).
Regarding claim 14, the combination of Gu, Xie, and Mirzadeh teaches a machine learning system as claimed in claim 13.
The combination of Gu, Xie, and Mirzadeh fails to explicitly teach the stochastic graph have been predetermined in dependence on one or more capabilities of the device.
However, analogous to the field of the claimed invention, Xu teaches:
the stochastic graph have been predetermined in dependence on one or more capabilities of the device (Xu, [0217] – “Specifically, the types and the quantity of operations included in the search space may be first determined based on the application requirement of the target neural network, and then the types and the quantity of operations included in the search space are adjusted based on the condition of the video random access memory resource of the device performing neural architecture search,” – teaches the types and operations having been predetermined in dependence on one or more capabilities of the device (operations first determined by application requirement of the target neural network, and search space of NAS is based on one or more capabilities of the device, thus the search space is predetermined in dependence on one or more capabilities of the device)).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the operations predetermined based on device capabilities of Xu to the stochastic graph generator and search space of Gu, Xie, and Mirzadeh in order to optimize over a stochastic graph determined based on device capabilities. Doing so would enable the deployment of deep neural networks in small devices with limited memory size or latency-critical applications such as smartphones and self-driving cars (Gu, Introduction) and adjust the architecture search space based on conditions of the device (Xu, [0217])
Response to Arguments
Applicant’s arguments, see pp. 1 of Remarks, filed 13 April 2026, with respect to claims 1-14 have been fully considered and are persuasive. The 35 U.S.C. 101 rejection of claims 1-14 has been withdrawn. As noted in Remarks in pp. 1, paragraph 2 – “Claims 1 and 10 have been amended to recite structural components, e.g. ‘a machine learning system comprising one or more computers, each of the one or more computers comprising a processor and a memory…’ The originally filed specification describes computing devices comprising processors and memory that execute the machine learning models (see, e.g., paragraphs [0065] - [0067])”. Thus, claims 1 and 10, as amended, are directed to statutory subject matter and the 35 U.S.C. 101 rejection of claims 1-14 has been withdrawn.
Applicant’s arguments, see pp. 1-4, filed 13 April 2026, with respect to the rejection(s) of claim(s) 1-3, 7-11, 13, and 15-17 under 35 U.S.C. 102(a)(1) have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made under 35 U.S.C. 103 over Gu in view of Xie et al. (NPL: Exploring Randomly Wired Neural Networks for Image Recognition, published April 2019, hereinafter “Xie”), and further in view of Mirzadeh. Gu teaches the limitations of claim 1 regarding “forming sample data…”, “selecting, in dependence on the sample data…”, “forming a trained candidate neural network by training…”, “adopting the trained candidate neural network…”, “wherein the base neural network is determined from a trained neural network by…”, “forming further sample data…”, “selecting, in dependence on the further sample data…”, “forming a trained candidate base neural network by training…”, “adopting the trained candidate base neural network…”, and “after multiple iterations of the further training process, adopting the current candidate base neural network…”. Xie teaches the amended limitations of claim 1 regarding “wherein the selecting is performed by optimization over a stochastic graph generator…”. Mirzadeh teaches the amended limitations of claim 1 regarding “wherein the base neural network is a teaching assistant network…”
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Mazzawi et al. (US Pub. No. 2021/0019599, filed July 2020) teaches systems and methods for selecting candidate architectures by performing a search process using an architecture generation engine. Architectures may be selected by performing Bayesian optimization over the architecture engine to select a final architecture. Teaches wherein the selection is an iterative process.
Cao et al. (NPL: Learnable Embedding Space for Efficient Neural Architecture Compression, published April 2019) teaches a method for incrementally learning an embedding space over a domain of network architectures. Teaches where an optimized embedding space over the domain of compressed network architectures can be used to define a distribution on the architecture space that can generate a priority ordering of architectures for evaluation. Teaches performing architecture search with Bayesian optimization with a kernel function defined over the embedding space.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LOUIS C NYE whose telephone number is 571-272-0636. The examiner can normally be reached Monday - Friday 9:00AM - 5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MATT ELL can be reached at 571-270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LOUIS CHRISTOPHER NYE/Examiner, Art Unit 2141
/DANIEL T PELLETT/Primary Examiner, Art Unit 2121
Prosecution Insights