DETAILED ACTION
This action is responsive to the application filed on 09/30/2024. Claims 1-5, 7-20, and 22 are pending and have been examined. This action is Non-final.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C.120, 121, 365(c), or 386(c) is acknowledged.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 2, 4, 5, 7-10, 14-18, and 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1,
Step 1: The claim is directed to a method, which falls under the category of process. The claim satisfies step 1.
Step 2A Prong 1:
“identifying one or more sub-optimal combinations of operations in the user model;” – This limitation is directed to identifying the operations of a model (e.g., a drawn graph or a listing of the model’s operations) and judging which combinations of those operations are sub-optimal. The claim does not limit the user model to any particular size or form and does not require any particular technique for performing the identification. This limitation is a process that can be performed in the human mind using evaluation, observation, and judgment, and therefore is directed to a mental process.
“generating an optimal replacement for at least one identified sub-optimal combination of operations of the one or more sub-optimal combinations of operations;” – The limitation is directed to an alternative combination of operations to substitute for a combination judged to be sub-optimal. The claim recites only the desired result (an “optimal replacement”) without reciting how the replacement is generated. This limitation is a process that can be performed in the human mind using evaluation, observation, and judgment, and therefore is directed to a mental process.
“replacing the at least one identified sub-optimal combination of operations in the user model with the respective generated optimal replacement for the at least one identified sub-optimal combination of operations; and” -- Under its broadest reasonable interpretation, this limitation covers a person substituting, with the aid of pen and paper, the devised combination of operations for the sub-optimal combination in a representation of the model. A mental process performed with the aid of pen and paper remains a mental process. This limitation is a process that can be performed in the human mind using evaluation, observation, and judgment, and therefore is directed to a mental process.
Step 2A Prong 2 and Step 2B:
“receiving a user model;” -- This limitation amounts to mere data gathering, which is insignificant extra-solution activity that cannot be integrated to practical application (see MPEP 2106.05(g)). Furthermore, under Step 2B, receiving or transmitting data is well-understood, routine, and conventional activity, thus it does not provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
“returning the user model with each of the respective generated optimal replacements as a transformed model.” -- This limitation amounts to outputting the result of the judicial exception, which is insignificant post-solution activity that cannot be integrated to practical application (see MPEP 2106.05(g)). Furthermore, under Step 2B, receiving or transmitting data is well-understood, routine, and conventional activity, thus it does not provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
Therefore, claim 1 is non-patent eligible. Claim 17 is analogous to claim 1, aside from claim type and minute differences, and thus the same rejection applies as above. The main difference was that under Step 2A Prong 2 and Step 2B, the limitation “A system for optimizing a user model, the system comprising: a computer system comprising one or more processors and memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform a process comprising:” – This limitation recites a system that will optimize a user model and will comprise processors and memory configured to process the rest of the limitation. The limitation does not integrate to a practical application, nor does it provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Regarding claim 2,
Step 1: The claim is directed to a method, which falls under the category of process. The claim satisfies step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method of claim 1, wherein identifying the one or more sub-optimal combinations of operations in the user model comprises applying a learned neural network (NN) to the user model.” -- The limitation recites the identifying of the combinations of operations in the NN will comprise of applying a learning NN to the under model. The learned NN is recited at a high level of generality as a tool used to perform the identifying step of claim 1. The claim does not recite any particular architecture of the NN, how the NN is trained, or how the NN operates to identify the sub-optimal combinations, and therefore amounts to no more than mere instructions to apply the judicial exception using a generic machine learning model , which does not integrate to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(f)).
Therefore, claim 2 is non-patent eligible.
Regarding claim 4,
Step 1: The claim is directed to a method, which falls under the category of process. The claim satisfies step 1.
Step 2A Prong 1:
“The method of claim 2, wherein the optimal replacement comprises a subgraph, and” -- Under its broadest reasonable interpretation, a subgraph encompasses a group of connected operations in a representation of the model, which a person can devise mentally or with the aid of pen and paper. This limitation further specifies the mental generating step first recited in claim 1 and is a process that can be performed in the human mind using evaluation, observation, and judgment, and therefore is directed to a mental process.
Step 2A Prong 2 and Step 2B:
“wherein the learned NN model is trained to generate output comprising a label value for each node or subgraph combination of the user model, the label value indicating whether each node or subgraph combination of the user model is optimal or sub-optimal.” -- This limitation recites the learned NN solely in terms of its intended output, namely a classification of each node or subgraph combination as optimal or sub-optimal, which is the same judgment identified in claim 1. The claim does not recite how the NN is trained or how it produces the output, and therefore amounts to no more than mere instructions to apply the judicial exception using a generic machine learning model, which directed to mere instructions to apply and it does not integrate to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Therefore, claim 4 is non-patent eligible.
Regarding claim 5,
Step 1: The claim is directed to a method, which falls under the category of process. The claim satisfies step 1.
Step 2A Prong 1:
“The method of claim 1, wherein the optimal replacement comprises an optimal subgraph.” -- Under its broadest reasonable interpretation, a subgraph encompasses a group of connected operations in a representation of the model, which a person can devise mentally or with the aid of pen and paper. This limitation further specifies the mental generating step of claim 1 and is a process that can be performed in the human mind using evaluation, observation, and judgment, and therefore is directed to a mental process.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Therefore, claim 5 is non-patent eligible.
Regarding claim 7,
Step 1: The claim is directed to a method, which falls under the category of process. The claim satisfies step 1.
Step 2A Prong 1:
“The method of claim 1, wherein identifying the one or more sub-optimal combinations of operations in the user model comprises applying a heuristics-based algorithm to the user model.” -- Under its broadest reasonable interpretation, a heuristics-based algorithm encompasses a set of rules of thumb that a person can apply to a model to judge which combinations of operations are sub-optimal. This limitation is a process that can be performed in the human mind using evaluation, observation, and judgment, and therefore is directed to a mental process.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Therefore, claim 7 is non-patent eligible. Claim 20 is analogous to claim 7, aside from claim type and minute differences, and thus the same rejection applies as above.
Regarding claim 8,
Step 1: The claim is directed to a method, which falls under the category of process. The claim satisfies step 1.
Step 2A Prong 1:
“(ii) produce an expressivity estimate that is correlated with performance of the user model for each node in the IR, and” -- Under its broadest reasonable interpretation, this limitation covers a person evaluating each node of a representation of the model and assigning an estimate of that node’s expressivity. The claim does not recite how the estimate is produced. This limitation is a process that can be performed in the human mind using evaluation, observation, and judgment, and therefore is directed to a mental process.
“wherein the expressivity estimate is used to identify the one or more sub-optimal combinations of operations in the user model and optimal combinations of operations in the user model.” -- Under its broadest reasonable interpretation, this limitation covers a person judging, based on the estimates, which combinations of operations are sub-optimal and which are optimal. This limitation is a process that can be performed in the human mind using evaluation, observation, and judgment, and therefore is directed to a mental process.
Step 2A Prong 2 and Step 2B:
“The method of claim 7, wherein the heuristics-based algorithm is programmed to: (i) input an intermediate representation (IR) representing the user model with model parameters and other metadata; and” -- The recitation that the algorithm is “programmed” amounts to no more than mere instructions to implement the judicial exception on a generic computer, thus it does not integrate to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(f)). Also, inputting an IR of the user model amounts to mere data gathering, which is insignificant extra-solution activity that cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under Step 2B, receiving or transmitting data is well-understood, routine, and conventional activity, and thus it cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
Therefore, claim 8 is non-patent eligible.
Regarding claim 9,
Step 1: The claim is directed to a method, which falls under the category of process. The claim satisfies step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method of claim 1, wherein generating the optimal replacement for at least one of the identified sub-optimal combinations of operations comprises providing the identified one or more sub-optimal combinations of operations as inputs to a learned generative NN.” -- The learned generative NN is recited at a high level of generality as a tool used to perform the generating step first introduced in claim 1. The claim does not recite any particular architecture of the generative NN, how it is trained, or how it produces the replacement, and therefore amounts to no more than mere instructions to apply the judicial exception using a generic machine learning model, which is directed to mere instructions apply, and thus it does not integrate to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(f)).
Therefore, claim 9 is non-patent eligible.
Regarding claim 10,
Step 1: The claim is directed to a method, which falls under the category of process. The claim satisfies step 1.
Step 2A Prong 1:
“The method of claim 1, wherein generating the optimal replacement for at least one of the identified sub-optimal combinations of operations comprises conditioning the generation on the user model.” – The limitation is directed to conditioning the generation on the user model covers a person considering the existing model (e.g., its surrounding operations and overall structure) when devising the replacement. This limitation is a process that can be performed in the human mind using evaluation, observation, and judgment, and therefore is directed to a mental process.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Therefore, claim 10 is non-patent eligible.
Regarding claim 14,
Step 1: The claim is directed to a method, which falls under the category of process. The claim satisfies step 1.
Step 2A Prong 1:
“wherein the sub-optimal combinations of operations are identified in the IR of the user model.” – The limitation is directed to reviewing the drawn graph and judging which combinations of operations are sub-optimal. This limitation is a process that can be performed in the human mind using evaluation, observation, and judgment, and therefore is directed to a mental process.
Step 2A Prong 2 and Step 2B:
“The method of claim 1, wherein in response to receiving the user model, the method further comprises constructing an IR of the user model, and” -- The limitation recites in response to receiving the user model, the method further comprises constructing an IR of the user model. The limitation is directed to mere limiting to a particular environment/field of use, and thus it does not integrate to a practical application, nor does it provide significantly more than the judicial exception (see MPEP 2106.05(h)).
Therefore, claim 14 is non-patent eligible.
Regarding claim 15,
Step 1: The claim is directed to a method, which falls under the category of process. The claim satisfies step 1.
Step 2A Prong 1:
“The method of claim 1, wherein replacing the identified suboptimal combination of operations with the generated respective optimal replacement comprises iterating over nodes in the user model to remove and replace the identified one or more sub-optimal combinations of operations.” – The limitation is directed to stepping through each node of a representation of the model and, with the aid of pen and paper, crossing out the sub-optimal combinations and writing in their replacements. This limitation is a process that can be performed in the human mind using evaluation, observation, and judgment, and therefore is directed to a mental process.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Therefore, claim 15 is non-patent eligible.
Regarding claim 16,
Step 1: The claim is directed to a method, which falls under the category of process. The claim satisfies step 1.
There are no elements to be evaluated Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method of claim 1, wherein the method further comprises generating … to modify at least one of the identified one or more sub-optimal combinations of operations in the user model.” -- The limitation is directed to generating to modify combinations of operations in the user model. The limitation is directed to mere further limiting to a field of use/particular environment, and thus it does not integrate to a practical application, nor does it provide significantly more than the judicial exception (see MPEP 2106.05(h)).
“returning recommendations” -- This limitation amounts to outputting the result of the judicial exception, which is insignificant extra-solution activity that cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under step 2B, receiving or transmitting data is well-understood, routine, and conventional activity, and thus it cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
Therefore, claim 16 is non-patent eligible.
Regarding claim 18,
Step 1: The claim is directed to a system, which falls under the category of machine. The claim satisfies step 1.
Step 2A Prong 1:
“extracting ground truth labels from the training dataset;” -- The limitation is directed to reviewing known models and determining which labels to assign to them. This limitation is a process that can be performed in the human mind using evaluation, observation, and judgment, and therefore is directed to a mental process.
“annotating node and combinations of operations for the known machine learning models as optimal” -- The limitation is directed to judging nodes and combinations of operations of known models to be optimal and marking them as such with the aid of pen and paper. This limitation is a process that can be performed in the human mind using evaluation, observation, and judgment, and therefore is directed to a mental process.
“to predict an estimate for each node or combination of operations of a model indicating whether each node or combination of operations of the model is optimal or sub-optimal.” – The limitation is directed to predicting an estimate for each node or operation combo of a model indicated whether each node or operation combo is optimal or sub-optimal. A person can predict an estimate and determine if data pertaining nodes or operation combos is optimal or sub-optimal, and thus is a process that can be performed in the human mind using evaluation, observation, and judgment, and therefore is directed to a mental process.
Step 2A Prong 2 and Step 2B:
“The system of claim 17, wherein identifying the one or more sub-optimal combinations of operations in the user model comprises applying a learned neural network (NN) to the user model, wherein the learned NN was trained in a process that comprises:” -- The limitation recites the identifying of the combinations of operations in the NN will comprise of applying a learning NN to the under model. The learned NN is recited at a high level of generality as a tool used to perform the identifying step of claim 1. The claim does not recite any particular architecture of the NN, how the NN is trained, or how the NN operates to identify the sub-optimal combinations, and therefore amounts to no more than mere instructions to apply the judicial exception using a generic machine learning model , which does not integrate to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(f)).
“receiving a training dataset of known machine learning models for a particular task or domain;” -- This limitation recites receiving a training dataset of known machine learning models for a particular risk/domain. The limitation amounts to mere data gathering, which is insignificant extra-solution activity that cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under step to 2B, receiving or transmitting data is well-understood, routine, and conventional activity, and thus it does not provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
“training the learned NN based on at least one of the ground truth labels or the annotations” – The limitation is directed to training the learned NN based on ground truth tables or the annotations. The training is recited at a high level of generality, specifying only the data used and the intended output, without any particular training technique. This limitation amounts to no more than mere instructions to apply the judicial exception using generic machine learning training, and thus it does not integrate to a practical application, nor does it provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Therefore, claim 18 is non-patent eligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not
identically disclosed as set forth in section 102, if the differences between the claimed invention and the
prior art are such that the claimed invention as a whole would have been obvious before the effective filing
date of the claimed invention to a person having ordinary skill in the art to which the claimed invention
pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are
summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 5, 10, 14, 16, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over the NPL reference “NAT: Neural Architecture Transformer for Accurate and Compact Architectures” by Guo et al. (referred herein as Guo) in view of the NPL reference “TASO: Optimizing Deep Learning Computation with Automatic Generation of Graph Substitutions” by Jia et al. (referred herein as Jia).
Regarding claim 1, Guo teaches:
A method for optimizing a user model, the method comprising: receiving a user model; identifying one or more sub-optimal combinations of operations in the user model; ([Guo, page 2, sec 1] “Last, it is worth mentioning that our NAT model can be used as a general architecture optimizer which takes any architecture as the input and output an optimized one.” AND [Guo, Abstract] “However, even a well-searched architecture may still contain many non-significant or redundant modules or operations (e.g., convolution or pooling), which may not only incur substantial memory consumption and computation cost but also deteriorate the performance. Thus, it is necessary to optimize the operations inside an architecture to improve the performance without introducing extra computation cost.”, wherein the examiner interprets the general architecture optimizer that takes an architecture as input and recognizes non-significant or redundant operations that deteriorate performance to be the same as receiving a user model and identifying sub-optimal operations in the user model, because they are both taking an existing neural-network architecture as input and determining that operations within the architecture are undesirable, inefficient, or performance-degrading and therefore candidates for optimization.)
and returning the user model with each of the respective generated optimal replacements as a transformed model. ([Guo, page 2, sec 1] “Last, it is worth mentioning that our NAT model can be used as a general architecture optimizer which takes any architecture as the input and output an optimized one.” AND [Guo, Abstract] “the transformed architecture by NAT significantly outperforms both its original form and those architectures optimized by existing methods.”, wherein the examiner interprets outputting the optimized architecture after application of the architecture-optimization process to be the same as the claimed limitation because both provide the resulting modified neural-network architecture as the output of the optimization process.)
Guo does not teach generating an optimal replacement for at least one identified sub-optimal combination of operations of the one or more sub-optimal combinations of operations; replacing the at least one identified sub-optimal combination of operations in the user model with the respective generated optimal replacement for the at least one identified sub-optimal combination of operations;.
Jia teaches:
generating an optimal replacement for at least one identified sub-optimal combination of operations of the one or more sub-optimal combinations of operations; ([Jia, page 49, sec 2.1] “A graph substitution consists of three components: (1) a source graph that is matched to subgraphs in a computation graph; (2) a target graph that defines a functionally equivalent new subgraph to replace the matched subgraph; and (3) a mapping relation between input/output tensors in the source and target graphs.”, wherein the examiner interprets the use of a target graph defining a new subgraph in place of the matched source subgraph to be the same as the claimed limitation because both substitute a corresponding newly provided structure for a previously located portion of the existing computation graph.)
replacing the at least one identified sub-optimal combination of operations in the user model with the respective generated optimal replacement for the at least one identified sub-optimal combination of operations; ([Jia, page 49, sec 2.1] “A graph substitution consists of three components: (1) a source graph that is matched to subgraphs in a computation graph; (2) a target graph that defines a functionally equivalent new subgraph to replace the matched subgraph; and (3) a mapping relation between input/output tensors in the source and target graphs.” AND [Jia, page 55, sec 5] “For each dequeued graph G, TASO considers each verified substitution and possible layouts applicable to the substitution, and applies them to obtain functionally equivalent new graphs G′.”, wherein the examiner interprets the use of a target graph defining a functionally equivalent new subgraph for a matched source subgraph, followed by application of that verified substitution to produce a new computation graph, to be the same as the claimed limitation because both substitute a corresponding newly provided graph structure for a previously matched portion of an existing DNN computation graph as part of the optimization process.)
Guo, Jia, and the instant application are analogous art because they are all directed to techniques for optimizing neural-network architectures or computation graphs.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of identifying and receiving sub-optimal combinations of operations disclosed by Guo to include the graph-substitution technique disclosed by Jia. One would be motivated to do so to automatically generate and apply replacement graph structures for inefficient portions of the model in order to obtain an optimized computation graph, as suggested by Jia ([Jia, page 1], “TASO, the first DNN computation graph optimizer that automatically generates graph substitutions,” and “To optimize a given DNN computation graph, TASO performs a cost-based backtracking search, applying the substitutions to find an optimized graph.”).
Regarding claim 2, Guo and Jia teach The method of claim 1 (see rejection of claim 1).
Guo further teaches wherein identifying the one or more sub-optimal combinations of operations in the user model comprises applying a learned neural network (NN) to the user model. ([Guo, page 5, sec 3.3] “To learn the optimal policy π(·|β; θ) w.r.t. an arbitrary architecture β, we propose an effective learning method to optimize the operations inside the architecture. Specifically, we take an arbitrary architecture graph β as the input and output the optimization policy w.r.t β. Such a policy is used to optimize the operations of the given architecture. Since the choice of operation on an edge depends on the adjacent nodes and edges, we have to consider the attributes of both the current edge and its neighbors. For this reason, we employ a graph convolution networks (GCN) [21] to exploit the adjacency information of the operations in the architecture…We consider a two-layer GCN…Z refers to the probability distribution of different candidate operations on the edges, i.e., the learned policy π(·|β; θ).”, wherein the examiner interprets the two-layer GCN receiving an arbitrary architecture graph as input and producing a learned policy over candidate operations on the graph edges to be the same as the claimed limitation because the GCN is a trained neural-network model that processes the existing architecture and evaluates operation choices using information associated with the operations and their neighboring nodes and edges.)
Regarding claim 5, Guo and Jia teach The method of claim 1 (see rejection of claim 1).
Jia further teaches wherein the optimal replacement comprises an optimal subgraph. ([Jia, page 49, sec 2.1] “A graph substitution consists of three components: (1) a source graph that is matched to subgraphs in a computation graph; (2) a target graph that defines a functionally equivalent new subgraph to replace the matched subgraph” AND [Jia, page 54, sec 5] “The optimizer uses the MetaFlow [21] cost-based backtracking search algorithm to search for an optimized computation graph by applying verified substitutions.” AND [Jia, page 55, sec 5] “To search for an optimized graph, all candidate graphs are maintained in a priority queue P and are dequeued in increasing order of cost.” AND [Jia, Abstract] “To optimize a given DNN computation graph, TASO performs a cost-based backtracking search, applying the substitutions to find an optimized graph, which can be directly used by existing DNN frameworks.”, wherein the examiner interprets the functionally equivalent new graph structure supplied by a verified substitution and retained through a cost-based search for the improved computation graph to be the same as the claimed limitation because both provide a replacement graph structure selected as part of an optimization process to improve the existing computation graph.)
Guo, Jia, and the instant application are analogous art because they are all directed to techniques for optimizing neural-network architectures by replacing portions of an existing model's computation graph with improved subgraph structures.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 1 disclosed by Guo and Jia to include the “functionally equivalent new subgraph” selected through the “cost-based backtracking search” disclosed by Jia. One would be motivated to do so to efficiently obtain an optimized computation graph that remains directly usable by existing deep learning frameworks, as suggested by Jia ([Jia, Abstract] “To optimize a given DNN computation graph, TASO performs a cost-based backtracking search, applying the substitutions to find an optimized graph, which can be directly used by existing DNN frameworks.”).
Regarding claim 10, Guo and Jia teach The method of claim 1 (see rejection of claim 1).
Guo further teaches wherein generating the optimal replacement for at least one of the identified sub-optimal combinations of operations comprises conditioning the generation on the user model. ([Guo, page 4, sec 3.2] “given any architecture β, we instead sample α from some well learned policy, denoted by π(·|β; θ), namely α ∼ π(·|β; θ). In other words, NAT first learns the policy and then conducts sampling from it to obtain the optimized architecture.” AND [Guo, page 5, sec 3.3] “Specifically, we take an arbitrary architecture graph β as the input and output the optimization policy w.r.t β. Such a policy is used to optimize the operations of the given architecture.”, wherein the examiner interprets sampling an optimized architecture from a learned policy that is expressly dependent upon the particular input architecture β, and producing an optimization policy with respect to that input architecture, to be the same as the claimed limitation because the resulting optimization is generated based upon information from the particular neural-network architecture supplied to the optimization process.)
Regarding claim 14, Guo and Jia teach The method of claim 1 (see rejection of claim 1).
Jia further teaches wherein in response to receiving the user model, the method further comprises constructing an IR of the user model, and wherein the sub-optimal combinations of operations are identified in the IR of the user model. ([Jia, page 47, sec 1] “Deep neural network (DNN) frameworks represent a neural architecture as a computation graph, where each node is a mathematical tensor operator (e.g., matrix multiplication, convolution, etc.). To improve the runtime performance of a computation graph, the most common form of optimization is graph substitutions that replace a subgraph matching a specific pattern with a functionally equivalent subgraph with improved performance.”, wherein the examiner interprets representing a neural-network architecture as a computation graph and locating within that graph a subgraph matching a pattern selected for performance-improving substitution to be the same as the claimed limitation because the computation graph provides an intermediate graph-based representation of the received neural-network architecture, and portions of that representation are located for optimization based on their suitability for replacement with improved graph structures.)
Guo, Jia, and the instant application are analogous art because they are all directed to techniques for representing neural-network architectures as graph-based intermediate representations and optimizing operations identified within those representations.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 1 disclosed by Guo and Jia to include representing the neural architecture “as a computation graph, where each node is a mathematical tensor operator,” as disclosed by Jia. One would be motivated to do so to effectively improve the runtime performance of the model while reducing the human effort required for optimization, as suggested by Jia ([Jia, Abstract] “Our evaluation on five real-world DNN architectures shows that TASO outperforms existing DNN frameworks by up to 2.8×, while requiring significantly less human effort.” AND [Jia, page 47, sec 1] “Deep neural network (DNN) frameworks represent a neural architecture as a computation graph, where each node is a mathematical tensor operator (e.g., matrix multiplication, convolution, etc.). To improve the runtime performance of a computation graph, the most common form of optimization is graph substitutions that replace a subgraph matching a specific pattern with a functionally equivalent subgraph with improved performance.”,).
Regarding claim 16, Guo and Jia teach The method of claim 1 (see rejection of claim 1).
Guo further teaches wherein the method further comprises generating and returning recommendations to modify at least one of the identified one or more sub-optimal combinations of operations in the user model. ([Guo, page 5, sec 3.3] “Specifically, we take an arbitrary architecture graph β as the input and output the optimization policy w.r.t β. Such a policy is used to optimize the operations of the given architecture…Z refers to the probability distribution of different candidate operations on the edges, i.e., the learned policy π(·|β; θ).”, wherein the examiner interprets outputting an optimization policy containing probabilities for candidate operation choices associated with the architecture to be the same as the claimed limitation because both provide proposed changes for portions of an existing neural-network architecture determined to benefit from optimization.)
Regarding claim 17, Guo teaches:
receiving a user model; identifying one or more sub-optimal combinations of operations in the user model; ([Guo, page 2, sec 1] “Last, it is worth mentioning that our NAT model can be used as a general architecture optimizer which takes any architecture as the input and output an optimized one.” AND [Guo, Abstract] “However, even a well-searched architecture may still contain many non-significant or redundant modules or operations (e.g., convolution or pooling), which may not only incur substantial memory consumption and computation cost but also deteriorate the performance. Thus, it is necessary to optimize the operations inside an architecture to improve the performance without introducing extra computation cost.”, wherein the examiner interprets the general architecture optimizer that accepts an existing architecture and recognizes non-significant or redundant operations that deteriorate performance to be the same as the claimed limitation because both process an existing neural-network architecture and determine that portions thereof are undesirable, inefficient, or performance-degrading and therefore candidates for optimization.)
returning the user model with the generated modification as a transformed model. ([Guo, page 2, sec 1] “Last, it is worth mentioning that our NAT model can be used as a general architecture optimizer which takes any architecture as the input and output an optimized one.” and [Guo, Abstract] “the transformed architecture by NAT significantly outperforms both its original form and those architectures optimized by existing methods.”, wherein the examiner interprets outputting the optimized architecture after application of the architecture-optimization process to be the same as the claimed limitation because both provide the resulting modified neural-network architecture as the output of the optimization process.)
Guo does not teach A system for optimizing a user model, the system comprising: a computer system comprising one or more processors and memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform a process comprising:… that generating a modification for at least one identified sub-optimal combinations of operations; replacing the at least one identified sub-optimal combinations of operations in the user model with the generated modification;.
Jia teaches:
A system for optimizing a user model, the system comprising: a computer system comprising one or more processors and memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform a process comprising: ([Jia, page 55, sec 6], “our implementation of TASO contains around 8,000 lines of code for the core components (i.e., the substitution generator, verifier, and optimizer), and 1,400 lines of code for the operator reference implementations” AND [Jia, page 56, sec 7.1] “All experiments were performed on an Amazon p3.2xlarge instance [1] with an 8-core Intel E5-2600 CPU, 64 GB DRAM, and one NVIDIA Tesla V100 GPU.”, wherein the examiner interprets the disclosed software implementation executing on computing hardware having an eight-core CPU and memory to be the same as the claimed limitation because both provide stored executable instructions performed by processing hardware to carry out neural-network computation-graph optimization.)
generating a modification for at least one identified sub-optimal combinations of operations; ([Jia, page 3], “A graph substitution consists of three components: (1) a source graph that is matched to subgraphs in a computation graph; (2) a target graph that defines a functionally equivalent new subgraph to replace the matched subgraph; and (3) a mapping relation between input/output tensors in the source and target graphs.”, wherein the examiner interprets the target graph defining a newly provided structure corresponding to a matched portion of the computation graph to be the same as the claimed limitation because both provide a structural change for a previously located portion of an existing neural-network graph.)
replacing the at least one identified sub-optimal combinations of operations in the user model with the generated modification; ([Jia, page 49, sec 2.1] “A graph substitution consists of three components: (1) a source graph that is matched to subgraphs in a computation graph; (2) a target graph that defines a functionally equivalent new subgraph to replace the matched subgraph; and (3) a mapping relation between input/output tensors in the source and target graphs.” AND [Jia, page 55, sec 5] “For each dequeued graph G, TASO considers each verified substitution and possible layouts applicable to the substitution, and applies them to obtain functionally equivalent new graphs G′.”, wherein the examiner interprets using the target graph in place of a matched source portion and applying the verified substitution to produce a new computation graph to be the same as the claimed limitation because both substitute a newly provided graph structure for a previously matched portion of an existing DNN computation graph as part of the optimization process.)
Guo, Jia, and the instant application are analogous art because they are all directed to techniques for optimizing neural-network architectures or computation graphs.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of identifying and receiving sub-optimal combinations of operations disclosed by Guo to include the graph substitution technique disclosed by Jia. One would be motivated to do so to automatically generate and apply replacement graph structures for inefficient portions of the model in order to obtain an optimized computation graph, as suggested by Jia ([Jia, Abstract] “We propose TASO, the first DNN computation graph optimizer that automatically generates graph substitutions…To optimize a given DNN computation graph, TASO performs a cost-based backtracking search, applying the substitutions to find an optimized graph, which can be directly used by existing DNN frameworks.”).
Claims 3 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Guo in view of Jia, in view of the NPL reference “Neural Architecture Optimization with Graph VAE” by Li et al. (referred herein as Li), and further in view of the NPL reference “Zen-NAS: A Zero-Shot NAS for High-Performance Image Recognition” by Lin et al. (referred herein as Lin).
Regarding claim 3, Guo and Jia teach The method of claim 2 (see rejection of claim 2).
Guo and Jia does not teach wherein the learned NN was trained in a process that comprises: collecting intermediate representations (IRs) of deep learning (DL) models achieving a high performance for learning tasks as ground truth labels; perturbing each IR for the collected models to generate an input; and training the NN based on at least one of the ground truth labels or the input to: (i) map the input into an embedding space; (ii) reconstruct the embedding space; and (iii) penalize a difference of the reconstruction with the ground truth labels.
Li teaches wherein the learned NN was trained in a process that comprises: collecting intermediate representations (IRs) of deep learning (DL) models achieving a high performance for learning tasks as ground truth labels; … and training the NN based on at least one of the ground truth labels or the input to: (i) map the input into an embedding space; (ii) reconstruct the embedding space; and (iii) penalize a difference of the reconstruction with the ground truth labels. ([Li, Abstract] “The encoder and the decoder belong to a graph VAE, mapping architectures between continuous representations and network architectures. The predictors are two regression models, fitting the performance and computational complexity, respectively. Those predictors ensure the discovered architectures characterize both excellent performance and high computational efficiency.” AND [Li, page 2, sec 1] “The encoder is to map the architectures to a latent space, where we use graph neural networks (GNN) to obtain continuous representations of architectures…The decoder is to recover neural architectures from the continuous representations.” AND [Li, page 5, sec 3.3] “the maximization of this term urges the high similarity between the primal DAG and the decoded graph. We evaluate the reconstruction accuracy by summing up the similarity of nodes and edges between the primal DAG and the generated graph” AND [Li, page 7, sec 5] “Here, our training dataset (X, Y, Z) consists of 19,020 neural architectures from the ENAS software…We split the dataset into 90% training data and 10% testing data, where we use the training data to learn the VAE and the predictor”, wherein the examiner interprets the neural-architecture graphs and associated performance information used to learn the graph VAE and predictor, the encoder converting an architecture into a continuous latent representation, the decoder recovering the architecture therefrom, and the reconstruction objective measuring similarity between the original DAG and generated graph to be the same as the claimed limitation because both use representations of neural-network architectures during learning, convert architecture information into a continuous representation, recover architecture information from that representation, and impose a training objective based on differences between the recovered structure and the original structure.)
Guo, Jia, and Li do not teach perturbing each IR for the collected models to generate an input;.
Lin teaches perturbing each IR for the collected models to generate an input; ([Lin, page 8, sec 4.3] “In Algorithm 2, we randomly generate N structures. At each iteration step t, we randomly select a structure in the population P and mutate it. The mutation algorithm is presented in Algorithm 3. The width and depth of the selected layer is mutated in a given range.” AND [Lin, page 9, Algorithm 3] “Uniformly select a block h in Ft. Uniformly alternate the block type, kernel size, width and depth of h within some range. Return the mutated structure F̂t.”, wherein the examiner interprets changing the block type, kernel size, width, and depth of a selected network structure to produce a modified network structure to be the same as the claimed limitation because both alter an existing representation of a neural-network architecture to produce a modified representation for subsequent processing.)
Guo, Jia, Li, Lin, and the instant application are analogous art because they are all directed to techniques for learning representations of neural-network architectures and modifying architecture structures for use in neural-network optimization.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the learned NN of claim 2 disclosed by Guo and Jia to include the “graph VAE, mapping architectures between continuous representations and network architectures” disclosed by Li. One would be motivated to do so to effectively learn continuous representations of neural-network architectures that characterize both performance and computational efficiency, as suggested by Li ([Li, Abstract] “Those predictors ensure the discovered architectures characterize both excellent performance and high computational efficiency.”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to include the NN mutation technique disclosed by Lin. One would be motivated to do so to effectively generate varied neural-network architecture structures for subsequent evaluation and optimization within the learned architecture-representation framework, as suggested by Lin ([Lin, page 8, sec 4.3, “After T iterations, the network with the largest Zen-Score is returned as the output of Zen-NAS.”).
Claim 19 is analogous to claim 3, aside from claim type and minute differences, and thus the same rejection applies as above.
Claims 4, 15, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Guo in view of Jia and further in view of the NPL reference “Transferable Graph Optimizers for ML Compilers” by Zhou et al. (referred herein as Zhou).
Regarding claim 4, Guo and Jia teach The method of claim 2 (see rejection of claim 2).
Jia further teaches wherein the optimal replacement comprises a subgraph, ([Jia, page 49, sec 2.1] “A graph substitution consists of three components: (1) a source graph that is matched to subgraphs in a computation graph; (2) a target graph that defines a functionally equivalent new subgraph to replace the matched subgraph” AND [Jia, page 54, sec 5] “The optimizer uses the MetaFlow [21] cost-based backtracking search algorithm to search for an optimized computation graph by applying verified substitutions.”, wherein the examiner interprets the functionally equivalent new subgraph provided by the target graph and selected through a cost-based optimization process to be the same as the claimed limitation because both provide a graph-based substitute structure used as part of improving an existing computation graph.)
Guo and Jia do not teach and wherein the learned NN model is trained to generate output comprising a label value for each node or subgraph combination of the user model, the label value indicating whether each node or subgraph combination of the user model is optimal or sub-optimal.
Zhou teaches and wherein the learned NN model is trained to generate output comprising a label value for each node or subgraph combination of the user model, the label value indicating whether each node or subgraph combination of the user model is optimal or sub-optimal. ([Zhou, page 3, sec 3] “We first explain how each of these problems can be formulated as learning a policy for classification of nodes in the graph.” AND [Zhou, page 4, sec 4], “Our proposed policy network πθ consists a graph embedding network that learns the graphical representation hG of any computational graph, and a scalable decision network that learns a optimization strategy p(y|G) over the given graph representation. The two components are jointly trained in an end-to-end fashion. Let yi be the action for the i-th node.”; and [Zhou, page 3], “Each node is associated with a fusion priority score F ∈ F, where the size of F is a hyperparameter. The scores determine the order in which nodes are fused.”; “Simple strategies like considering ops in topological order can make inefficient fusion decisions and lead to suboptimal performance.”, wherein the examiner interprets the learned classification policy that produces an optimization decision or score associated with individual nodes of a computation graph to be the same as the claimed limitation because both use learned per-node output values to characterize portions of a graph for purposes of distinguishing graph-optimization decisions that improve performance from those associated with inefficient or suboptimal performance.)
Guo, Jia, Zhou, and the instant application are analogous art because they are all directed to techniques for analyzing and optimizing neural-network computation graphs by evaluating operations or nodes and determining modifications that improve graph performance.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 2 disclosed by Guo and Jia to include the node classification technique disclosed by Zhou. One would be motivated to do so to efficiently generate optimization decisions across the nodes of a computation graph while reducing the time required to determine appropriate graph modifications, as suggested by Zhou ([Zhou, page 2, sec 2] “In contrast, our approach generates decisions for the entire graph at once, significantly reducing the policy training time.”)
Regarding claim 15, Guo and Jia teach The method of claim 1 (see rejection of claim 1).
Guo further teaches wherein replacing the identified suboptimal combination of operations with the generated respective optimal replacement comprises … to remove and replace the identified one or more sub-optimal combinations of operations. ([Guo, page 2, sec 1] “Thus, the architecture optimization process is reduced to a series of decision making problems. Based on MDP, we seek to replace the expensive operations or redundant modules in the architecture with more computationally efficient ones. Specifically, NAT shall remove the redundant modules or replace these modules with skip connections.” AND [Guo, page 9, sec 4.3] “NAT replaces several redundant operations with the skip connections or directly removes the connection”, wherein the examiner interprets performing a series of architecture-transformation decisions that either substitute computationally efficient structures for redundant operations or eliminate such structures to be the same as the claimed limitation because both modify undesirable portions of an existing neural-network architecture through removal or substitution with improved structures.)
Guo and Jia do not teach iterating over nodes in the user model.
Zhou teaches iterating over nodes in the user model ([Zhou, page 7, sec 5] “For each of the computational graph optimization tasks, the end goal is to minimize the step time of the graph or a batch of dataflow graphs for a target system configuration (e.g. a 8-GPU cluster or a TPUcluster), by learning a policy that generates optimization decisions on a per-node basis.” AND [Zhou, page 4, sec 4] “Let y_i be the action for the i-th node. Ideally, we would like to compute the action distribution of the current node based on the actions of all previous nodes in an auto-regressive manner”, wherein the examiner interprets producing optimization decisions separately for individual graph elements and determining the action for a current element based on previously processed elements to be the same as the claimed limitation because both systematically process constituent elements of a computation graph when determining architecture modifications.)
Guo, Jia, Zhou, and the instant application are analogous art because they are all directed to techniques for optimizing neural-network computation graphs by evaluating and modifying operations or structural portions of an existing model.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 1 disclosed by Guo and Jia to include the optimization approach disclosed by Zhou. One would be motivated to do so to efficiently determine architecture-optimization actions across the constituent elements of a computation graph, as suggested by Zhou ([Zhou, page 2, sec 2] “In contrast, our approach generates decisions for the entire graph at once, significantly reducing the policy training time.”).
Regarding claim 18, Guo and Jia teach The system of claim 17 (see rejection of claim 17).
Guo further teaches wherein identifying the one or more sub-optimal combinations of operations in the user model comprises applying a learned neural network (NN) to the user model. ([Guo, page 5, sec 3.3] “To learn the optimal policy π(·|β; θ) w.r.t. an arbitrary architecture β, we propose an effective learning method to optimize the operations inside the architecture. Specifically, we take an arbitrary architecture graph β as the input and output the optimization policy w.r.t β. Such a policy is used to optimize the operations of the given architecture. Since the choice of operation on an edge depends on the adjacent nodes and edges, we have to consider the attributes of both the current edge and its neighbors. For this reason, we employ a graph convolution networks (GCN) [21] to exploit the adjacency information of the operations in the architecture…We consider a two-layer GCN…Z refers to the probability distribution of different candidate operations on the edges, i.e., the learned policy π(·|β; θ).”, wherein the examiner interprets the two-layer GCN receiving an arbitrary architecture graph as input and producing a learned policy over candidate operations on the graph edges to be the same as the claimed limitation because the GCN is a trained neural-network model that processes the existing architecture and evaluates operation choices using information associated with the operations and their neighboring nodes and edges.)
Guo and Jia do not teach wherein the learned NN was trained in a process that comprises: receiving a training dataset of known machine learning models for a particular task or domain; extracting ground truth labels from the training dataset; annotating node and combinations of operations for the known machine learning models as optimal training the learned NN based on at least one of the ground truth labels or the annotations to predict an estimate for each node or combination of operations of a model indicating whether each node or combination of operations of the model is optimal or sub-optimal.
Zhou teaches wherein the learned NN was trained in a process that comprises: receiving a training dataset of known machine learning models for a particular task or domain; extracting ground truth labels from the training dataset; annotating node and combinations of operations for the known machine learning models as optimal training the learned NN based on at least one of the ground truth labels or the annotations to predict an estimate for each node or combination of operations of a model indicating whether each node or combination of operations of the model is optimal or sub-optimal. ([Zhou, page 2, sec 1] “GO consists of an inductive graph-embedding network that encodes operation features and dependencies in a trainable graph representation, followed by a policy network of segmented recurrent attention layers. The policy network transforms the graph representations into an optimization decision with soft attention. These two networks can be jointly trained in an end-to-end fashion using a supervised reward. To generalize to arbitrary and held-out graphs, GO is trained jointly over a set of computation graphs (instead of one at a time) and then fine-tuned on new graphs.” AND [Zhou, page 3, sec 3] “ML computations are usually expressed as computation graphs, G(V, E), where nodes V represent computations and edges E represent data flows. We consider three related computational graph optimization problems from the ML compiler optimization stack. We first explain how each of these problems can be formulated as learning a policy for classification of nodes in the graph…Simple strategies like considering ops in topological order can make inefficient fusion decisions and lead to suboptimal performance…We can reformulate op fusion as a priority-based ordering problem: learning a policy π : G ↦ F that assigns a fusion priority F ∈ F for all ops in the graph to maximize a reward rG,F defined based on run time. Each node is associated with a fusion priority score F ∈ F, where the size of F is a hyperparameter. The scores determine the order in which nodes are fused.”, wherein the examiner interprets the set of computation graphs used to jointly train the graph-embedding and policy networks using a supervised reward to be the same as training using known machine-learning models and associated supervisory information, and further interprets the learned classification policy that associates optimization scores and decisions with the individual operations represented by graph nodes to be the same as predicting an estimate for portions of a model indicative of whether those portions should be treated as desirable or undesirable for optimization, because both techniques train a learned graph-processing model using performance-based supervision to evaluate constituent operations of computational graphs and distinguish graph decisions associated with improved performance from decisions producing sub-optimal performance.)
Guo, Jia, Zhou, and the instant application are analogous art because they are all directed to automated techniques for analyzing and optimizing operations within neural-network computation graphs.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system of claim 17 disclosed by Guo and Jia to include the node classification technique disclosed by Zhou. One would be motivated to do so to efficiently evaluate operations throughout a computation graph and generate graph-wide optimization decisions while reducing the time required to train the optimization policy, as suggested by Zhou ([Zhou, page 2, sec 2] “In contrast, our approach generates decisions for the entire graph at once, significantly reducing the policy training time.” AND [Zhou, page 3, sec 3] “Simple strategies like considering ops in topological order can make inefficient fusion decisions and lead to suboptimal performance.”)
Claims 7 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Guo in view of Jia further in view of Lin.
Regarding claim 7, Guo and Jia teach The method of claim 1 (see rejection of claim 1).
Guo and Jia do not teach wherein identifying the one or more sub-optimal combinations of operations in the user model comprises applying a heuristics-based algorithm to the user model.
Lin teaches wherein identifying the one or more sub-optimal combinations of operations in the user model comprises applying a heuristics-based algorithm to the user model. ([Lin, page 8, sec 4.3] “We design Zen-NAS algorithm to maximize the Zen-Score of the target network. The Zen-NAS uses Evolutionary Algorithm (EA) as architecture generator..At each iteration step t, we randomly select a structure in the population P and mutate it. The mutation algorithm is presented in Algorithm 3. The width and depth of the selected layer is mutated in a given range…Finally, we maintain the population size by removing networks with the smallest Zen-Scores. After T iterations, the network with the largest Zen-Score is returned as the output of Zen-NAS.”, wherein the examiner interprets the evolutionary search that iteratively selects, modifies, evaluates, and retains network structures according to a performance-related score to be the same as the claimed limitation because the disclosed procedure uses rule-based search and selection operations to evaluate structural portions of a neural-network architecture and determine which structures should be retained or modified.)
Guo, Jia, Lin, and the instant application are analogous art because they are all directed to techniques for analyzing and optimizing neural-network architectures by evaluating and modifying operations or structural portions of a model.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 1 disclosed by Guo and Jia to include the “Evolutionary Algorithm (EA) as architecture generator” disclosed by Lin. One would be motivated to do so to efficiently explore and select candidate architecture modifications using a simple automated search procedure, as suggested by Lin ([Lin, page 8, sec 4.3] “We design Zen-NAS algorithm to maximize the Zen-Score of the target network. The Zen-NAS uses Evolutionary Algorithm (EA) as architecture generator..At each iteration step t, we randomly select a structure in the population P and mutate it. The mutation algorithm is presented in Algorithm 3. The width and depth of the selected layer is mutated in a given range…Finally, we maintain the population size by removing networks with the smallest Zen-Scores. After T iterations, the network with the largest Zen-Score is returned as the output of Zen-NAS.”)
Claim 20 is analogous to claim 7, aside from claim type and minute differences, and thus the same rejection applies as above.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Guo in view of Jia, in view of Lin, and further in view of Zhou.
Regarding claim 8, Guo, Jia, and Lin teach The method of claim 7 (see rejection of claim 7).
Guo further teaches wherein the heuristics-based algorithm is programmed to: (i) input an intermediate representation (IR) representing the user model with model parameters and other metadata; ([Guo, page 5, sec 3.3] “Here, an architecture graph can be represented by a data pair (A, X), where A denotes the adjacency matrix of the graph and X denotes the attributes of the nodes together with their two input edges.” AND [Guo, page 4, sec 3.2] “Let wα and wβ be the well-learned model parameters of architectures α and β, respectively.”, wherein the examiner interprets the architecture graph containing an adjacency representation and attributes associated with its nodes and edges, together with parameters associated with the represented architecture, to be the same as the claimed limitation because both provide a graph-based representation of a neural-network architecture containing structural information, parameter information, and additional attributes used by an architecture-optimization process.)
Lin further teaches and (ii) produce an expressivity estimate that is correlated with performance of the user model…and wherein the expressivity estimate is used to identify the one or more sub-optimal combinations of operations in the user model and optimal combinations of operations in the user model. ([Lin, Abstract] “The Zen-Score represents the network expressivity and positively correlates with the model accuracy.” AND [Lin, page 8, sec 4.3] “Finally, we maintain the population size by removing networks with the smallest Zen-Scores. After T iterations, the network with the largest Zen-Score is returned as the output of Zen-NAS.”, wherein the examiner interprets the Zen-Score representing network expressivity and correlating with model accuracy, together with retaining higher-scoring structures and removing lower-scoring structures during architecture search, to be the same as the claimed limitation because both calculate a performance-correlated measure of expressivity and use that measure to distinguish less desirable architecture structures from more desirable architecture structures during neural-network optimization.)
Guo, Jia, and Lin do not teach for each node in the IR.
Zhou teaches for each node in the IR, ([Zhou, page 3, sec 3] “Each node is associated with a fusion priority score F ∈ F, where the size of F is a hyperparameter. The scores determine the order in which nodes are fused.” AND [Zhou, page 4, sec 4] “Let yi be the action for the i-th node.” AND [Zhou, page 5, sec 4.1] “In GO, nodes and edges in the dataflow graph are represented as the concatenation of their meta features (e.g., operation type, output shape) and are further encoded by the graph embedding network into a trainable representation.”, wherein the examiner interprets associating individual nodes of a graph representation with respective optimization scores or decisions based on node metadata to be the same as the claimed limitation because both perform graph-based evaluation at the individual-node level rather than assigning only a single value to the architecture as a whole.)
Guo, Jia, Lin, Zhou, and the instant application are analogous art because they are all directed to techniques for representing, evaluating, and optimizing neural-network architectures.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 7 disclosed by Guo, Jia, and Lin to include the node optimization decision making disclosed by Zhou. One would be motivated to do so to efficiently evaluate individual components of a computation graph when determining architecture-optimization decisions while improving the efficiency of the graph-optimization process, as suggested by Zhou ([Zhou, page 3, sec 3] “Each node is associated with a fusion priority score F ∈ F, where the size of F is a hyperparameter. The scores determine the order in which nodes are fused.” AND [Zhou, page 4, sec 4] “Let yi be the action for the i-th node.” AND [Zhou, page 5, sec 4.1] “In GO, nodes and edges in the dataflow graph are represented as the concatenation of their meta features (e.g., operation type, output shape) and are further encoded by the graph embedding network into a trainable representation.”)
Claims 9 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Guo in view of Jia further in view of Li.
Regarding claim 9, Guo and Jia teach The method of claim 1 (see rejection of claim 1).
Guo and Jia do not teach wherein generating the optimal replacement for at least one of the identified sub-optimal combinations of operations comprises providing the identified one or more sub-optimal combinations of operations as inputs to a learned generative NN.
Li teaches wherein generating the optimal replacement for at least one of the identified sub-optimal combinations of operations comprises providing the identified one or more sub-optimal combinations of operations as inputs to a learned generative NN. ([Li, Abstract] “The framework jointly learns four components: the encoder, the performance predictor, the complexity predictor and the decoder in an end-to-end manner. The encoder and the decoder belong to a graph VAE, mapping architectures between continuous representations and network architectures.” AND [Li, page 4, sec 3.2-3.3] “In the inference process, we directly optimize the neural architectures in the latent space based on the performance predictor fperf and the complexity predictor fcomp, where the objective is to maximize the predicted performance and minimize the predicted complexity. Then, we use the decoder D to decode the network architecture from the latent space S to the search space X…We utilize the generative model of the graph autoencoder D : S → X to decode latent representations to DAGs (the network architectures)”, wherein the examiner interprets supplying a performance-optimized representation of neural-architecture information to the learned graph-autoencoder decoder, which operates as a generative model to produce a resulting network architecture, to be the same as the claimed limitation because both provide architecture information selected for improvement to a trained generative neural-network component that produces an improved architectural structure.)
Guo, Jia, Li, and the instant application are analogous art because they are all directed to techniques for optimizing neural-network architectures by representing, modifying, or generating architectural structures to improve model performance.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 1 disclosed by Guo and Jia to include the “generative model of the graph autoencoder” disclosed by Li. One would be motivated to do so to efficiently generate improved neural-network architectures from representations selected according to predicted performance and computational complexity, as suggested by Li ([Li, page 4, sec 3.2-3.3] “In the inference process, we directly optimize the neural architectures in the latent space based on the performance predictor fperf and the complexity predictor fcomp, where the objective is to maximize the predicted performance and minimize the predicted complexity. Then, we use the decoder D to decode the network architecture from the latent space S to the search space X…We utilize the generative model of the graph autoencoder D : S → X to decode latent representations to DAGs (the network architectures)”).
Regarding claim 12, Guo and Jia teach The method of claim 10 (see rejection of claim 10).
Guo and Jia do not teach wherein conditioning the generation on the user model comprises using an intermediate representation (IR) of the user model as input to a generative model that is trained to generate optimal replacements for the user model.
Li teaches wherein conditioning the generation on the user model comprises using an intermediate representation (IR) of the user model as input to a generative model that is trained to generate optimal replacements for the user model. ([Li, page 2, sec 1, Fig 1] “The neural architecture is mapped into a continuous representation by the encoder network. Using stochastic gradient ascent, the continuous representation is optimized via maximizing the predictor, combining performance and complexity predictor. The optimized continuous representation is decoded out to a network architecture using the decoder.” AND [Li, page 3, sec 3] “The encoder E : X → S and the decoder D : S → X are used to convert representations between networks search space X and a continuous space S. Based on the latent space S, the performance predictor fperf : S → Y is used to fit the performance of network architectures, while the complexity predictor fcomp : S → Z is used to fit the computational complexity of network architectures. We first train the encoder, two predictors and the decoder in an end-to-end manner, and then use them to infer new architectures.” AND [Li, page 4, sec 3.3] “We utilize the generative model of the graph autoencoder D : S → X to decode latent representations to DAGs (the network architectures)” AND [Li, page 6, sec 4] “After the inference, we use the decoder D : S → X to construct a valid neural architecture from the optimized representation s′ according to the generative model pθ(x|s).”, wherein the examiner interprets the encoded continuous representation derived from an existing neural architecture and supplied to a trained graph-VAE decoder that constructs a new architecture from an optimized representation to be the same as the claimed limitation because both condition neural-architecture generation on a representation derived from an existing model and use a trained generative network to produce an improved architecture based on that representation.)
Guo, Jia, Li, and the instant application are analogous art because they are all directed to techniques for optimizing neural-network architectures by representing existing architectures and generating improved architecture structures.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 10 disclosed by Guo and Jia to include the prediction techniques disclosed by Li. One would be motivated to do so to effectively generate improved neural-network structures from optimized representations of existing architectures while accounting for performance and computational efficiency, as suggested by Li [Li, page 3, sec 3] “The encoder E : X → S and the decoder D : S → X are used to convert representations between networks search space X and a continuous space S. Based on the latent space S, the performance predictor fperf : S → Y is used to fit the performance of network architectures, while the complexity predictor fcomp : S → Z is used to fit the computational complexity of network architectures. We first train the encoder, two predictors and the decoder in an end-to-end manner, and then use them to infer new architectures.” AND [Li, page 4, sec 3.3] “We utilize the generative model of the graph autoencoder D : S → X to decode latent representations to DAGs (the network architectures)” AND [Li, page 6, sec 4] “After the inference, we use the decoder D : S → X to construct a valid neural architecture from the optimized representation s′ according to the generative model pθ(x|s).”)
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Guo in view of Jia further in view of the NPL reference “Network Morphism” by Wei et al. (referred herein as Wei).
Regarding claim 11, Guo and Jia teach The method of claim 10 (see rejection of claim 10).
Jia further teaches replacing the identified one or more sub-optimal combinations of operations with the respective generated optimal replacement. ([Jia, page 49, sec 2.1] “A graph substitution consists of three components: (1) a source graph that is matched to subgraphs in a computation graph; (2) a target graph that defines a functionally equivalent new subgraph to replace the matched subgraph; and (3) a mapping relation between input/output tensors in the source and target graphs.” AND [Jia, page 55, sec 5] “For each dequeued graph G, TASO considers each verified substitution and possible layouts applicable to the substitution, and applies them to obtain functionally equivalent new graphs G′.”, wherein the examiner interprets the use of a target graph defining a functionally equivalent new subgraph for a matched source subgraph, followed by application of the verified substitution to produce a new computation graph, to be the same as the claimed limitation because both substitute a corresponding newly provided graph structure for a previously identified portion of an existing DNN computation graph as part of the optimization process.)
Guo and Jia do not teach wherein conditioning the generation on the user model comprises preserving original information and data associated with the user model while…
Wei teaches wherein conditioning the generation on the user model comprises preserving original information and data associated with the user model while… ([Wei, Abstract] “We present a systematic study on how to morph a well-trained neural network to a new one so that its network function can be completely preserved…After morphing a parent network, the child network is expected to inherit the knowledge from its parent network” AND [Wei, page 2, sec 1] “In the context of neural networks, network morphism refers to a parameter-transferring map from a parent network to a child network that preserves its function and outputs.”, wherein the examiner interprets generating a modified child architecture from a parent architecture while maintaining the parent architecture’s function, outputs, and inherited knowledge to be the same as the claimed limitation because both use the existing neural-network model as the basis for producing a modified architecture while retaining information represented by the original model.)
Guo, Jia, Wei, and the instant application are analogous art because they are all directed to techniques for modifying or optimizing neural-network architectures while using information associated with an existing neural-network model.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 10 disclosed by Guo and Jia to include the network function preservation technique disclosed by Wei. One would be motivated to do so to efficiently retain learned knowledge from the existing neural-network model while modifying its architecture and thereby reduce the amount of retraining required after the modification, as suggested by Wei ([Wei, Abstract] “the child network shall not only inherit the entire knowledge from the parent network, but also be capable of growing into a more powerful one in much shortened training time”).
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Guo in view of Jia, in view of Li, in view of Lin, and further in view of Zhou.
Regarding claim 13, Guo and Jia teach The method of claim 1 (see rejection of claim 1).
Guo further teaches wherein identifying the one or more sub-optimal combinations of operations in the user model comprises applying a learned NN to the user model, the learned NN being trained in a process that comprises: ([Guo, page 5], “we employ a graph convolution networks (GCN) [21] to exploit the adjacency information of the operations in the architecture”; “we randomly sample architectures from the whole architecture space and use them to train our model.”; and “We apply the policy gradient [46] to train our model.”, wherein the examiner interprets the trained GCN that receives architecture-graph information and learns an optimization policy from sampled architectures to be the same as the claimed limitation because both employ a trained neural-network model that processes an existing neural-network architecture for purposes of determining architectural optimization decisions.)
Guo and Jia do not teach collecting intermediate representations (IRs) of deep learning (DL) models achieving a high performance for learning tasks to generate ground truth labels; perturbing each IR for the collected models to generate an input; annotating nodes of each IR based on at least one of: adding labels indicating suboptimal node organization for each node and edge resulting from adding the perturbation; or adding labels indicating optimal organization of nodes to remaining nodes of the IR; and training the NN based on at least one of the ground truth labels or the input to predict the ground truth labels for each node in the IR given the input.
Li teaches collecting intermediate representations (IRs) of deep learning (DL) models achieving a high performance for learning tasks to generate ground truth labels; ([Li, page 6, sec 4, Algorithm 1], “Input: The training data (X, Y) with the architectures and their performance…Output: The neural architecture x′ with the optimal performance.” AND [Li, page 6, sec 4] “Generated by the encoder, hout and s are the discrete representation and the latent continuous representation for the architecture x, respectively.”, wherein the examiner interprets the architecture representations together with their corresponding measured performance values used as training data to be the same as the claimed limitation because both associate representations of neural-network architectures with performance information used to train an architecture-optimization model.)
Guo, Jia, and Li do not teach perturbing each IR for the collected models to generate an input;.
Lin teaches perturbing each IR for the collected models to generate an input; ([Lin, page 8, sec 4.3] “At each iteration step t, we randomly select a structure in the population P and mutate it.” AND [Lin, page 9, sec 4.3, Algorithm 3], “Uniformly select a block h in Ft. Uniformly alternate the block type, kernel size, width and depth of h within some range. Return the mutated structure F̂t.”, wherein the examiner interprets modifying the block type, kernel size, width, or depth of an existing neural-network structure to produce a changed structure to be the same as the claimed limitation because both deliberately alter an existing representation of a neural-network architecture to produce a modified version for subsequent processing.)
Guo, Jia, Li, and Lin do not teach annotating nodes of each IR based on at least one of: adding labels indicating suboptimal node organization for each node and edge resulting from adding the perturbation; or adding labels indicating optimal organization of nodes to remaining nodes of the IR; and training the NN based on at least one of the ground truth labels or the input to predict the ground truth labels for each node in the IR given the input.
Zhou teaches:
annotating nodes of each IR based on at least one of: adding labels indicating suboptimal node organization for each node and edge resulting from adding the perturbation; or adding labels indicating optimal organization of nodes to remaining nodes of the IR; ([Zhou, page 3, sec 3] “We first explain how each of these problems can be formulated as learning a policy for classification of nodes in the graph…Simple strategies like considering ops in topological order can make inefficient fusion decisions and lead to suboptimal performance…Each node is associated with a fusion priority score F ∈ F, where the size of F is a hyperparameter. The scores determine the order in which nodes are fused.” and [Zhou, page 5, sec 4.2] “In graph optimization tasks, the optimal actions for a node are often influenced by actions for other nodes in the graph.”, wherein the examiner interprets classifying individual graph elements and associating each element with an optimization-related score used to distinguish decisions associated with improved or suboptimal graph performance to be the same as the claimed limitation because both assign node-specific information reflecting whether the organization or treatment of portions of a computation graph is favorable or unfavorable for optimization.)
and training the NN based on at least one of the ground truth labels or the input to predict the ground truth labels for each node in the IR given the input. ([Zhou, page 3, sec 3] “ML computations are usually expressed as computation graphs, G(V, E), where nodes V represent computations and edges E represent data flows. We consider three related computational graph optimization problems from the ML compiler optimization stack. We first explain how each of these problems can be formulated as learning a policy for classification of nodes in the graph.” AND [Zhou, page 7, sec 5] “For each of the computational graph optimization tasks, the end goal is to minimize the step time of the graph or a batch of dataflow graphs for a target system configuration (e.g. a 8-GPU cluster or a TPU cluster), by learning a policy that generates optimization decisions on a per-node basis.”, wherein the examiner interprets training a learned graph policy to classify graph nodes and produce corresponding optimization decisions for individual nodes to be the same as the claimed limitation because both train a neural-network-based graph model to produce node-specific prediction outputs from an input computation graph.)
Guo, Jia, Li, Lin, Zhou, and the instant application are analogous art because they are all directed to automated techniques for representing, evaluating, modifying, or optimizing neural-network architectures.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 1 disclosed by Guo and Jia to include the architecture-representation and performance-training technique disclosed by Li. One would be motivated to do so to effectively learn representations that permit neural-network architectures to be evaluated and optimized according to predicted performance, as suggested by Li ([Li, page 6, sec 4] “To find the architecture with optimal performance and high computational efficiency, we maximize the performance predictor and minimize the complexity predictor.”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to include the mutation technique disclosed by Lin. One would be motivated to do so to efficiently produce alternative architecture structures for evaluation during optimization, as suggested by Lin ([Lin, page 8, sec 4.3] “The choice of EA is due to its simplicity.”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to include the node classification technique disclosed by Zhou. One would be motivated to do so to efficiently produce optimization decisions for graph elements without sequentially processing individual nodes, as suggested by Zhou ([Zhou, page 3, sec 3] “ML computations are usually expressed as computation graphs, G(V, E), where nodes V represent computations and edges E represent data flows. We consider three related computational graph optimization problems from the ML compiler optimization stack. We first explain how each of these problems can be formulated as learning a policy for classification of nodes in the graph.” AND [Zhou, page 7, sec 5] “For each of the computational graph optimization tasks, the end goal is to minimize the step time of the graph or a batch of dataflow graphs for a target system configuration (e.g. a 8-GPU cluster or a TPU cluster), by learning a policy that generates optimization decisions on a per-node basis.”)
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Guo in view of Jia, and further in view of Wei.
Regarding claim 22, Guo and Jia teach The system of claim 17 (see rejection of claim 17).
Guo further teaches wherein generating the modification for at least one of the identified sub-optimal combinations of operations comprises conditioning the generation on the user model, ([Guo, page 4, sec 3.2] “given any architecture β, we instead sample α from some well learned policy, denoted by π(·|β; θ), namely α ∼ π(·|β; θ). In other words, NAT first learns the policy and then conducts sampling from it to obtain the optimized architecture.” AND [Guo, page 5, sec 3.3] “Specifically, we take an arbitrary architecture graph β as the input and output the optimization policy w.r.t β. Such a policy is used to optimize the operations of the given architecture.”, wherein the examiner interprets sampling an optimized architecture from a learned policy that is expressly dependent upon the particular input architecture β, and producing an optimization policy with respect to that input architecture, to be the same as the claimed limitation because the resulting optimization is generated based upon information from the particular neural-network architecture supplied to the optimization process.)
Jia further teaches replacing the identified one or more sub-optimal combinations of operations with the generated modification. ([Jia, page 49, sec 2.1], “A graph substitution consists of three components: (1) a source graph that is matched to subgraphs in a computation graph; (2) a target graph that defines a functionally equivalent new subgraph to replace the matched subgraph; and (3) a mapping relation between input/output tensors in the source and target graphs.” AND [Jia, page 55, sec 5] “For each dequeued graph G, TASO considers each verified substitution and possible layouts applicable to the substitution, and applies them to obtain functionally equivalent new graphs G′.”, wherein the examiner interprets the use of a target graph defining a functionally equivalent new subgraph for a matched source subgraph, followed by application of the verified substitution to produce a new computation graph, to be the same as the claimed limitation because both substitute a corresponding newly provided graph structure for a previously identified portion of an existing DNN computation graph as part of the optimization process.)
Guo and Jia do not teach wherein conditioning the generation on the user model comprises preserving original information and data associated with the user model while …
Wei teaches wherein conditioning the generation on the user model comprises preserving original information and data associated with the user model while … ([Wei, Abstract] “We present a systematic study on how to morph a well-trained neural network to a new one so that its network function can be completely preserved…After morphing a parent network, the child network is expected to inherit the knowledge from its parent network” AND [Wei, page 2, sec 1] “ In the context of neural networks, network morphism refers to a parameter-transferring map from a parent network to a child network that preserves its function and outputs.”, wherein the examiner interprets generating a modified child architecture from a parent architecture while maintaining the parent architecture’s function, outputs, and inherited knowledge to be the same as the claimed limitation because both use the existing neural-network model as the basis for producing a modified architecture while retaining information represented by the original model.)
Guo, Jia, Wei, and the instant application are analogous art because they are all directed to techniques for modifying or optimizing neural-network architectures.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system of claim 17 disclosed by Guo and Jia to include the network function preservation technique disclosed by Wei. One would be motivated to do so to efficiently retain learned knowledge from the existing neural-network model while modifying its architecture and thereby reduce the amount of retraining required after the modification, as suggested by Wei ([Wei, Abstract] “the child network shall not only inherit the entire knowledge from the parent network, but also be capable of growing into a more powerful one in much shortened training time”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DEVAN KAPOOR whose telephone number is (703)756-1434. The examiner can normally be reached Monday - Friday: 9:00AM - 5:00 PM EST (times may vary).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DEVAN KAPOOR/Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
