Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 25 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 25 recites the limitation "the metric value" at the end of the claim. There is insufficient antecedent basis for this limitation in the claim.
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claim 25 is rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends. Claim 25 depends from claim 5. Claim 5 recites “wherein the configuration settings are stored in a table or are generated by a machine learning model”. Claim 25 recites “wherein the table comprises…”. The use of the term “or” in the context of claim 5 denotes that the claim is satisfied by either storing configuration settings in a table or generating configuration settings by a machine learning model (See MPEP 2111). Claim 25 does not further limit the full scope of claim 5 and fails to include all the limitations of claim 5. Applicant may cancel the claim(s), amend the claim(s) to place the claim(s) in proper dependent form, rewrite the claim(s) in independent form, or present a sufficient showing that the dependent claim(s) complies with the statutory requirements.
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 1-2, 5, 7-8, 12, 14, 16-17, 19-20, and 23-24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang J. et al. (US Pub. No. 2021/0406677, filed June 2020, hereinafter “Wang J.”) in view of Xu et al. (NPL: ReForm: Static and Dynamic Resource-aware DNN Reconfiguration Framework for Mobile Device, published Aug. 2019, hereinafter “Xu”), and further in view of Tann et al. (NPL: Runtime Configurable Deep Neural Networks for Energy -Accuracy Trade-off, published Oct. 2016, hereinafter “Tann”).
Regarding claim 1, Wang J. teaches a computer implemented method, comprising:
receiving a single augmented neural network model, wherein the single augmented neural network model is produced by inserting configurable modules into an original neural network model to modify an execution graph of the original network model, and the configurable modules are controlled by configuration settings that provide alternate paths within the original neural network model that includes processing layers (Wang J., [0022] – “the network entity dynamically reconfigures the E2E DNN based on various factors, such as a change in signal and/or link quality, a change in participating devices in the UECS, or a change in a coordinating UE within the UECS”, [0035] – “Accordingly, the neural network table 216 includes any combination of neural network formation configuration elements (NN formation configuration elements), such as architecture and/or parameter configurations, that can be used to create a neural network formation configuration (NN formation configuration) that includes a combination of one or more NN formation configuration elements that define and/or form a DNN. In some aspects, a single index value of the neural network table 216 maps to a single NN formation configuration element (e.g., a 1:1 correspondence). Alternatively, or additionally, a single index value of the neural network table 216 maps to a NN formation configuration (e.g., a combination of NN formation configuration elements)”, and in [0042] – “In implementations, the training module 270 extracts learned parameter configurations from the DNN to identify the NN formation configuration elements and/or NN formation configuration, and then adds and/or updates the NN formation configuration elements and/or NN formation configuration in the neural network table 272. The extracted parameter configurations include any combination of information that defines the behavior of a neural network, such as node connections, coefficients, active layers, weights, biases, pooling, etc.” – teaches receiving a single augmented neural network model (NN formation configuration), wherein the single augmented neural network model is produced by inserting configurable modules into an original neural network model (training module 270 may add or update NN configuration elements, thus inserting configurable modules to the original model) to modify an execution graph of the original network model (module may add or update NN configuration elements, thus modifying an execution graph of the original network), wherein the configurable modules are controlled by configuration settings (the module 270 dynamically reconfigures the E2E DNN based on various factors) that provide alternate paths within the original neural network model that includes a path through processing layers (extracted parameter configurations include any combination of information that defines the behavior of a neural network, such as node connections, coefficients, active layers, weights, biases, pooling, etc.));
Wang J. fails to explicitly teach dynamically configuring, during inference, the single augmented neural network model responsive to real-time changes in performance constraints and a condition of an environment within which the single augmented neural network model is deployed by selectively enabling or disabling at least one of the configurable modules intervening training or modifying weights used by the original neural network model, wherein the configuration settings are correlated with the performance constraints
However, analogous to the field of the claimed invention, Xu teaches:
dynamically configuring, during inference, the single augmented neural network model responsive to real-time changes in performance constraints and a condition of an environment within which the single augmented neural network model is deployed by selectively enabling or disabling at least one of the configurable modules without intervening training or modifying weights used by the original neural network model, wherein the configuration settings are correlated with the performance constraints (Xu, Fig. 3 and Section 4 Paragraphs 1-2 – “Therefore, in this section, we propose a dynamic DNN model reconfiguration scheme to adapt DNN model to dynamic computation resource by selectively computing filters in the network” and “Fig. 3 shows the overview of proposed dynamic DNN model reconfiguration scheme. Firstly, we determine the filter computing priority by identifying a filter selection priority indicator. This indicator can be derived by conducting filter resource mapping and filter accuracy impact analysis. Then, with selection priority indicator obtained, we further propose our dynamic selective computing paradigm to dynamically reconfigure the DNN model generated from the static reconfiguration” and in Section 4.2 Paragraph 1 – “Since the resource constraints are dynamic during the DNN reconfiguration, we further propose the DNN dynamic selective computing algorithm to optimize the network without retraining. The algorithm is shown in Algorithm. 1. During DNN-based applications executing, the system consistently obtains the available resource br that can be allocated to DNN. Once any DNN computation costs Ctotalm exceeds the available budget br , the filter with least PI j i value in current status will be masked for computing in a filter pruning manner. Then the DNN total computation cost Ctotalm is updated and the filter with sub-least PIji value will be updated as the least one in next masking status. The system iteratively executes the masking process until Ctotalm below br . By applying this algorithm, a DNN can be dynamically reconfigured to meet any resource constraints introduced by real-time applications. Since all PIji values are determined by pre-analysis, no further computation cost will be introduced. Also, to ensure the real-time performance, no model retraining is utilized.” – teaches dynamically configuring (dynamically reconfigure), during inference (during DNN-based applications executing), the single augmented neural network model (generated DNN model) response to real-time changes in performance constraints (DNN can be dynamically reconfigured to meet any resource constraints introduced by real-time applications) by selective enabling or disabling at least one of the configurable augmentations (DNN model to dynamic computation resource by selectively computing filters in the network) without intervening training (during DNN-based applications executing) or modifying weights (DNN dynamic selective computing algorithm to optimize the network without retraining) used by the original neural network model, wherein the configuration settings are correlated with the performance constraints (we can formulate the resource consumption for any filter in the ith layer with regard to memory Mi , energy Ei , and latency Li). In addition to the previously cited passages, Xu further teaches in Section 4 Paragraph 1 – “Although the static DNN reconfiguration scheme can customize DNN models for static platform requirements, dynamic computation resource constraints might still be introduced by various real-time mobile applications. Therefore, in this section, we propose a dynamic DNN model reconfiguration scheme to adapt DNN model to dynamic computation resource by selectively computing filters in the network.” – teaches dynamically configuring, during inference, the single augmented neural network response to a condition of an environment within which the single augmented neural network is deployed (configures neural network responsive to dynamic computation resource constraints introduced by various real-time mobile applications, thus configures neural network responsive to a condition of an environment within which the model is deployed, as the dynamic resource constraints of real-time mobile applications are the conditions of the application environment));
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the dynamic neural network configuration, during inference, according to real time changes in performance constraints without intervening training of Xu to the production of a single augmented neural network by inserting configurable augmentations controlled by configuration settings of Wang J. in order to dynamically configure, during inference, the single augmented neural network by selectively enabling or disabling at least one of the configurable augmentations. Doing so would provide the ability to dynamically adapt the DNN and provide a flexible solution to responding to changing factors and improve the overall performance of data transfer and/or recovery (Wang J., [0022]) and the DNN model can be optimized for all dynamic computation resource constraints (Xu, Section 4 Paragraph 1).
The combination of Wang J. and Xu fails to explicitly teach executing the configured single augmented neural network model for an input to produce an output.
However, analogous to the field of the claimed invention, Tann teaches:
executing the configured single augmented neural network model for an input to produce an output (Tann, Section 4.2 Paragraph 1 — “As shown in Figure 5, our technique consists of a runtime configurable DNN of choice and a score margin classifier. Score margin is defined as the absolute difference between the two largest neuron outputs (scores) in the final layer of a DNN.” — teaches executing the configured single augmented neural network model (a DNN of choice) for an input to produce an output).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the executing of Tann to the dynamic configuration of the single augmented neural network of Wang J. and Xu in order to receive and execute the configured single augmented neural network model. Doing so would enable the system to evaluate the output results for each input (Tann, Section 2).
Claims 16 and 19 incorporate substantively all the limitations of claim 1 in a system and a non-transitory computer-readable media, and are rejected on similar grounds as above.
Regarding claim 2, the combination of Wang J., Xu, and Tann teach the computer- implemented method of claim 1, wherein the weights are applied at one or more processing layers of the configured single augmented neural network model that are enabled according to the configuration settings to produce the output (Wang J., [0035] – “In aspects, the CRM 212 includes a neural network table 216 that stores various architecture and/or parameter configurations that form a neural network, such as, by way of example and not of limitation, parameters that specify a fully connected layer neural network architecture, a convolutional layer neural network architecture, a recurrent neural network layer, a number of connected hidden neural network layers, an input layer architecture, an output layer architecture, a number of nodes utilized by the neural network, coefficients (e.g., weights and biases) utilized by the neural network, kernel parameters, a number of filters utilized by the neural network, strides/pooling configurations utilized by the neural network, an activation function of each neural network layer, interconnections between neural network layers, neural network layers to skip, and so forth. Accordingly, the neural network table 216 includes any combination of neural network formation configuration elements (NN formation configuration elements), such as architecture and/or parameter configurations, that can be used to create a neural network formation configuration (NN formation configuration) that includes a combination of one or more NN formation configuration elements that define and/or form a DNN.” – teaches the weights applied at one or more processing layers of the configured augmented neural network model (table 216 stores various architecture and/or parameter configurations that form a neural network such as layer architecture, number of nodes, coefficients that are applied during processing including weights, etc.) that are enabled according to configuration settings (table 216 contains any combination of neural network format configuration elements to form a DNN, including layers to skip), and in [0064] – “The nodes can use a variety of algorithms and/or analysis to generate output information based upon adaptive learning, such as single linear regression, multiple linear regression, logistic regression, step-wise regression, binary classification, multiclass classification, multi-variate adaptive regression splines, locally estimated scatterplot smoothing, and so forth. At times, the algorithm(s) include weights and/or coefficients that change based on adaptive learning.” – teaches producing an output (nodes of layers use algorithms including weights and/or coefficients to generate output information))
Regarding claim 5, the combination of Wang J., Xu, and Tann teach the computer-implemented method of claim 1, wherein the configuration settings are stored in a table or generated by a machine learning model (Wang J., [0043] – “In some implementations, a single index value of the neural network table 272 maps to a single NN formation configuration element (e.g., a 1:1 correspondence). Alternatively, or additionally, a single index value of the neural network table 272 maps to a NN formation configuration (e.g., a combination of NN formation configuration elements).” – teaches configuration settings stored in a table, and in [0041] – “The training module 270 teaches and/or trains DNNs using known input data. For instance, the training module 270 trains DNN(s) for different purposes, such as processing communications transmitted over a wireless communication system” and [0042] – “ In implementations, the training module 270 extracts learned parameter configurations from the DNN to identify the NN formation configuration elements and/or NN formation configuration, and then adds and/or updates the NN formation configuration elements and/or NN formation configuration in the neural network table 272.” – teaches generating the configuration settings by a machine learning model (module extracts, adds, or updates NN formation configuration elements, thus teaching generating configuration settings by a machine learning model) and wherein the configuration settings are stored in a table).
Regarding claim 7, the combination of Wang J., Xu, and Tann teach the computer-implemented method of claim 1, wherein the configurable modules selectively reduces a scale factor of activation height or width (Wang J., [0035] – “ In aspects, the CRM 212 includes a neural network table 216 that stores various architecture and/or parameter configurations that form a neural network, such as, by way of example and not of limitation, parameters that specify a fully connected layer neural network architecture, a convolutional layer neural network architecture, a recurrent neural network layer, a number of connected hidden neural network layers, an input layer architecture, an output layer architecture, a number of nodes utilized by the neural network, coefficients (e.g., weights and biases) utilized by the neural network, kernel parameters, a number of filters utilized by the neural network, strides/pooling configurations utilized by the neural network, an activation function of each neural network layer, interconnections between neural network layers, neural network layers to skip, and so forth.” – teaches wherein the configurable modules selectively reduce a scale factor of activation height or width (parameter configurations and updates that specify kernel parameters, filter parameters, and activation functions of each neural network layer. Selectively adjusting filter and activation function parameters encompasses selectively reducing, or increasing, a scale factor of activation height or width));
Claim 20 is similar to claim 7, hence similarly rejected.
Regarding claim 8, the combination of Wang J., Xu, and Tann teaches the computer-implemented method of claim 1, wherein the performance constraints comprise a metric and a value of the metric (Xu, Section 3.1 Paragraph 1 – “In mobile device, there are various computation resource constraints, which can be formulated into mathematical expression and be easily inserted into the optimization objective function. In our scheme, we focus on three typical constraints, including computation capacity, memory occupancy and energy consumption” and in Section 3.1.3 Paragraph 1 – “Therefore, the total energy consumption is: Eq. (3)” – teaches wherein the performance constraints comprise a metric (energy consumption) and a value of the metric (Eq. 3 shows the mathematical expression to determine values of the metric)).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the performance constraints comprising a metric and a value of the metric of Xu to further modify the method of Wang J., Xu, and Tann in order to establish metrics and values of metrics for performance constraints. Doing so would efficiently and effectively reconfigure a pre-trained DNN model for practical mobile deployment with regards to various static and dynamic computation resource constraints (Xu, Abstract).
Claim 17 is similar to claim 8, hence similarly rejected.
Regarding claim 12, the combination of Wang J., Xu, and Tann teaches the computer-implemented method of claim 1, wherein at least one of the steps of receiving, configuring, and executing are performed on a server or in a data center to generate the output and the input is streamed from a user device (Wang J., [0040] – “CRM 262 also includes a base station neural network manager 268 (BS neural network manager 268). Alternatively, or additionally, the BS neural network manager 268 may be implemented in whole or part as hardware logic or circuitry integrated with or separate from other components of the base station 120. In at least some aspects, the BS neural network manager 268 selects the NN formation configurations utilized by the base station 120 and/or UE 110 to configure deep neural networks for processing wireless communications, such as by selecting a combination of NN formation configuration elements to form a DNN for processing UECS communications. In some implementations, the BS neural network manager 268 receives feedback from the UE 110, and selects the NN formation configuration based on the feedback.” – teaches wherein at least one of the steps of receiving, configuring, and executing are performed on a server or in a data center (base station neural network manager select configurations utilized by base station and/or user equipment) to generate the output (to form a DNN) and the input is streamed from a user device (UE provides input such as feedback)).
Regarding claim 14, the combination of Wang J., Xu, and Tann teaches the computer-implemented method of claim 1, wherein at least one of the steps of receiving, configuring and executing are performed for training, testing, or certifying a neural network employed in a machine, robot, or autonomous vehicle (Tann, Section 4.1 Paragraph 1 — “This system can be deployed in either of the two following schemes to achieve energy savings... One possible scenario is that real-time constraints force a DNN to provide an answer within a smaller window of time as in the case of DNN accelerators deployed in autonomous vehicles, where a sudden, unexpected situation could force an on-chip DNN to make a decision within a tighter window of time.” — Tann teaches at least one of the steps of receiving, configuring, and executing are performed for training, testing, and certifying a neural network employed in a machine, robot, or autonomous vehicle).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the training, testing, or certifying a neural network employed in an autonomous vehicle of Tann to further modify the method of Wang J., Xu, and Tann in order to perform at least one of the steps for training, testing, or certifying a neural network employed in a machine, robot, or autonomous vehicle. Doing so would force a DNN to provide an answer within a smaller window of time as in autonomous vehicles (Tann, Section 4.1).
Regarding claim 23, the combination of Wang J., Xu, and Tann teach the computer-implemented method of claim 1, wherein the configuration settings are correlated with the performance constraints by:
measuring accuracy of outputs produced by the single augmented neural network model for a variety of the configuration settings and inputs compared with expected outputs corresponding to the inputs (Xu, Fig. 4 – shows resource mapping and layer accuracy impact & Section 4.1.2 Subsection 1 Paragraph 1 – “For each layer, the impact can be measured by the model’s accuracy drop when a certain portion of filters are gradually stop computing in this layer (empirically, we adopt 20% in each time). Larger accuracy drop indicates the layer’s bigger impact (denoted as LI) to the classification results. For example, Fig. 4 (d) shows the LI distribution of all layers in VGG-13. We can find that 2nd to 7th layers have relatively larger LI values, which indicate higher accuracy impact.” – teaches measuring accuracy of outputs produced by the single augmented neural network for a variety of the configuration settings and inputs compared with expected outputs corresponding to the inputs (Fig. 4 shows accuracy of outputs produced by the augmented neural network model for a variety of the configuration settings and inputs)); and
for each performance constraint, selecting the configuration setting of the configuration settings having a highest measured accuracy for the performance constraint (Xu, Section 4.1.3 Paragraph 1 – “Based on the analysis above, we can evaluate the consumption-accuracy trade-off for each filter, which can be used as the priority indicator for selective computing: Eq. (16) where LIi × norm(CIij) is the comprehensive accuracy impact for jth filter in ith layer, norm(Mi), norm(Ei), and norm(Li) are respectively the normalized memory, energy, and latency consumption. αM , αE , and αL are the consumption weights determined by practical constraints. The filters with higher PIij values are supposed to have higher accuracy impact and less resource consumption, which will be favored by selective computing” – teaches for each performance constraint (memory, energy, and latency consumption), selecting the configuration setting (PIij) having a highest measured accuracy for the performance constraint (higher PIij values have higher accuracy impact and less resource consumption, which will be favored by selective computing)).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the configuration settings correlated with performance constraints of Xu to further modify the method of Wang J., Xu, and Tann in order to select the configuration settings that provide the highest measured accuracy for each performance constraint. Doing so would allow the DNN model to be optimized for all dynamic computation resource constraints (Xu, Section 4).
Regarding claim 24, the combination of Wang J., Xu, and Tann teaches the method of claim 1, wherein the single augmented neural network is deployed within a vehicle and, responsive to detecting the vehicle is traveling on an expressway, execution resources for a pedestrian detection system are reduced and vehicle detection processing is increased (Tann, Section 4.1 Paragraph 1 — “This system can be deployed in either of the two following schemes to achieve energy savings... One possible scenario is that real-time constraints force a DNN to provide an answer within a smaller window of time as in the case of DNN accelerators deployed in autonomous vehicles, where a sudden, unexpected situation could force an on-chip DNN to make a decision within a tighter window of time.” and in Section 4.1 Paragraph 2 – “We develop Algorithm 2 for our feedback controller given in Figure 4 to regulate the number of channels or network capacity allowed at any given point, This algorithm first checks the energy and delay constraints and current system performance. If the energy and/or delay budget is not met, the network capacity is adjusted accordingly. At any point, the controller tries to adjust the capacity such that the system performance is close to, but does not exceed, the constraints.” – teaches wherein the single augmented neural network is deployed within a vehicle (DNN deployed in autonomous vehicles), and responsive to detecting the vehicle is traveling on an expressway (sudden, unexpected situation forces DNN to make decision within tighter window of time, thus detecting the vehicle is in a certain driving scenario which includes being on an expressway), execution resources for a pedestrian detection system are reduced and vehicle detection processing is increased (regulates network channels and/or capacity at any given point relative to delay constraints and system performance, thus teaching increases and reductions to networks in response to scenario constraints)).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the neural network employed in an autonomous vehicle of Tann to further modify the method of Wang J., Xu, and Tann. Doing so would adapt a DNN to make a decision within a smaller window of time in autonomous vehicles according to constraints (Tann, Section 4.1).
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang J., Xu, and Tann as applied to claim 1, 16, and 19 above, and further in view Thathachar et al. (US Pub. No. 2020/0090035, filed Sept. 2018, hereinafter “Thathachar”).
Regarding claim 11, the combination of Wang J., Xu, and Tann teach the computer-implemented method of claim 1.
The combination of Wang J., Xu, and Tann fails to explicitly teach wherein the neural network model is a transformer neural network model.
However, analogous to the field of the claimed invention, Thathachar teaches:
wherein the neural network model is a transformer neural network model (Thathachar, [0026] — “Memory Augmented Neural Networks (MANNS) provide opportunities to analyze the capabilities, generalization performance, and the limitations of those models.” — teaches augmented neural networks wherein the neural network is a transformer model, in [0028] — “the MANN architecture is referred to as an Encoder-Decoder NTM (ED-NTM). As set out below, different types of encoders are studied in systematic manner, showing an advantage of multi-task learning in obtaining the best possible encoder. This encoder enables transfer learning to solve a suite of working memory tasks.” — teaches using an encoder-decoder architecture, thus showing that the augmented neural network is a transformer).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the transformer neural network model of Thathachar to the method of Wang J., Xu, and Tann in order to dynamically configure a transformer neural network model. Doing so would extend neural network capabilities in solving diverse tasks, e.g., learning context-free grammars, remembering long sequences (long-term dependencies), learning to rapidly assimilate new data (e.g., one-shot learning) and visual question answering (Thathachar, [0025]).
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang J., Xu, and Tann as applied to claims 1, 16, and 19 above, and further in view of Rabinovich et al (US Pub. No. 2017/0262737, filed March 2017, hereinafter “Rabinovich”).
Regarding claim 13, the combination of Wang J., Xu, and Tann teach the computer-implemented method of claim 1.
The combination of Wang J., Xu, and Tann fails to explicitly teach wherein at least one of the steps of receiving, configuring, and executing are performed within a cloud computing environment.
However, analogous to the field of the claimed invention, Rabinovich teaches:
wherein at least one of the steps of receiving, configuring, and executing are performed within a cloud computing environment (Rabinovich, [0031] — “The system comprises any type of computing station that may be used to operate, interface with, or implement a neural network computing device 107 or user computing device 115. Examples of such computing systems include for example, servers, workstations, personal computers, or remote computing terminals connected to a networked or cloud-based computing platform.” — Rabinovich teaches at least one of the steps of receiving, configuring, or executing performed within a cloud computing environment).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the cloud computing environment of Rabinovich to the method of Wang J., Xu, and Tann in order to perform the augmented neural network within a cloud computing environment. Doing so would enable customizing the learning process with a target architecture in mind (Rabinovich, [0062]).
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang J., Xu, and Tann. as applied to claims 1, 16, and 19 above, and further in view of Yao et al. (US Pub. No. 2021/0201078, filed April 2017, hereinafter “Yao”).
Regarding claim 15, the combination of Wang J., Xu, and Tann teaches the computer-implemented method of claim 1.
The combination of Wang J., Xu, and Tann fails to explicitly teach wherein at least one of the steps of receiving, configuring, and executing is performed on a virtual machine comprising a portion of a graphics processing unit.
However, analogous to the field of dynamic neural network configuration, Yao teaches:
wherein at least one of the steps of receiving, configuring, and executing is performed on a virtual machine comprising a portion of a graphics processing unit (Yao, [0100] — “a virtualized graphics execution environment is presented in which the resources of the graphics processing engines 431-432, N are shared with multiple applications or virtual machines (VMs). The resources may be subdivided into “slices” which are allocated to different VMs and/or applications based on the processing requirements and priorities associated with the VMs and/or applications.” — Yao teaches at least one of the steps of receiving, configuring, and executing being performed on a virtual machine comprising a portion of a graphics processing unit).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the virtual machine of Yao to the method of Wang J., Xu, and Tann in order to perform the augmented neural network model on a virtual machine comprising a portion of a graphics processing unit. Doing so would provide virtualization facilities for the host processor to manage virtualization of the graphics processing engines, interrupts, and memory management (Yao, [0101]).
Claim(s) 21-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang J., Xu, and Tann as applied to claims 1, 16, and 19 above, and further in view of Wu et al. (NPL: FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Architecture Search, published May 2019, hereinafter “Wu”).
Regarding claim 21, the combination of Wang J., Xu, and Tann teach the computer-implemented method of claim 1.
The combination of Wang J., Xu, and Tann fails to explicitly teach for each configuration setting of the configuration settings, characterizing a performance of the single augmented neural network using the configuration setting and at least one input of the inputs to produce a performance estimate.
However, analogous to the field of the claimed invention, Wu teaches:
further comprising, for each configuration setting of the configuration settings, characterizing a performance of the single augmented neural network using the configuration setting and at least one input of the inputs to produce a performance estimate (Wu, Section 3.2 Paragraph 4 – “To solve this problem, we use a latency lookup table model to estimate the overall latency of a network based on the runtime of each operator. More formally, we assume Eq. (3) where bl(a) denotes the block at layer-l from architecture a. This assumes that on the target processor, the runtime of each operator is independent of other operators. The assumption is valid for many mobile CPUs and DSPs, where operators are computed sequentially one by one. This way, by benchmarking the latency of a few hundred operators used in the search space, we can easily estimate the actual runtime of the 1021 architectures in the entire search space.” – teaches, for each configuration setting (each operator, each architecture), characterizing a performance of the single augmented neural (LUT model to estimate overall latency of a network based on runtime, thus characterizing performance of a single augmented neural network) using the configuration setting (operator) and at least one input of the inputs (characterizes latency of a block within the network, which would utilize an input during runtime to produce an output) to produce a performance estimate (Eq. 3 describes producing the performance estimate)).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the computation of a performance estimate of Wu to the method of Wang J., Xu, and Tann in order to compute an estimate of the performance of the configuration settings. Doing so would enable quick and efficient computation of the runtime of an architecture across the entire architecture search space (Wu, Section 3.2).
Regarding claim 22, the combination of Wang J., Xu, Tann, and Wu teach the computer-implemented method of claim 21,
updating at least one performance estimate during inference (Xu, Section 4.2 Paragraph 1 – “During DNN-based applications executing, the system consistently obtains the available resource br that can be allocated to DNN. Once any DNN computation costs Ctotalm exceeds the available budget br , the filter with least PIij value in current status will be masked for computing in a filter pruning manner. Then the DNN total computation cost Ctotalm is updated and the filter with sub-least PIij value will be updated as the least one in next masking status.” – teaches updating at least one performance estimate (Ctotalm is updated) during inference (during DNN applications executing)).
The combination of Wang J., Xu, and Tann fails to explicitly teach storing the performance estimate of the single augmented neural network model for each configuration setting.
However, analogous to the field of the claimed invention, Yang teaches:
storing the performance estimate of the single augmented neural network model for each configuration setting (Wu, Section 3.2 Paragraph 4 – “To solve this problem, we use a latency lookup table model to estimate the overall latency of a network based on the runtime of each operator. More formally, we assume Eq. (3) where bl(a) denotes the block at layer-l from architecture a. This assumes that on the target processor, the runtime of each operator is independent of other operators. The assumption is valid for many mobile CPUs and DSPs, where operators are computed sequentially one by one. This way, by benchmarking the latency of a few hundred operators used in the search space, we can easily estimate the actual runtime of the 1021 architectures in the entire search space.” – teaches storing the performance estimate (latency) of the single augmented neural network model (estimates overall latency of a network, thus the performance estimate of the single augmented neural network model) for each configuration setting (latency lookup table that stores latency of each operator within each architecture))
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the storing of performance estimates for each configuration setting of Wu to the method of Wang J., Xu, and Tann in order to store the estimated performance for each configuration setting. Doing so would enable quick and efficient computation of the runtime of an architecture across the entire architecture search space (Wu, Section 3.2).
No Prior Art Rejection
Regarding claim 25, claim 25 is not examined on the merits with respect to prior art because of the identified 35 U.S.C. 112(b) and 112(d) rejections above. Due to these deficiencies, the metes and bounds of the claim are unclear and/or the claim is of improper dependent form.
Upon correction of the above-noted issues, claim 25 will be subject to further examination and consideration under 35 U.S.C. 102 and 103.
Response to Arguments
Applicant's arguments filed 21 January 2026 have been fully considered but they are not persuasive. Applicant argues on pp. 2 that Wang J. fails to teach or suggest “inserting configurable modules into the DNN…”. Examiner respectfully disagrees and points to Wang J. at [0042] – “In implementations, the training module 270 extracts learned parameter configurations from the DNN to identify the NN formation configuration elements and/or NN formation configuration, and then adds and/or updates the NN formation configuration elements and/or NN formation configuration in the neural network table 272” which teaches inserting a module into a DNN to identify, extract, add, and update NN formation configuration elements, which modifies an execution graph of a neural network.
Applicant further argues on pp. 3 that Xu fails to teach or suggest “configuration responsive to an ‘condition of an environment’ (e.g. context such as roadway type, sensor set, or application environment)…”. Examiner respectfully disagrees and points to Xu at Section 4 Paragraph 1 – “Although the static DNN reconfiguration scheme can customize DNN models for static platform requirements, dynamic computation resource constraints might still be introduced by various real-time mobile applications. Therefore, in this section, we propose a dynamic DNN model reconfiguration scheme to adapt DNN model to dynamic computation resource by selectively computing filters in the network.” – which teaches configuration responsive to a condition of an environment within which the single augmented neural network is deployed. Xu teaches configuring a neural network responsive to dynamic computation resource constraints introduced by various real-time mobile applications, thus configuring a neural network responsive to a condition of an environment within which the model is deployed, as the dynamic resource constraints of real-time mobile applications are the conditions of an application environment.
Applicant’s arguments, see pp. 4-5, filed 21 January 2026, with respect to the rejection(s) of claim(s) 21-22 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made over Wang J., Xu, and Tann further in view of Wu et al. (NPL: FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search, published May 2019). Wu teaches the limitations of claim 21 regarding “for each configuration setting of the configuration settings, characterizing a performance…” and Wu teaches the limitations of claim 22 regarding “storing the performance estimate of the single augmented neural network model for each configuration…”.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Gao et al. (NPL: Resource-Guided Configuration Space Reduction for Deep Learning Models, published May 2021) teaches deep learning models with a large number of configuration options and proposes DnnSAT, a resource-guided AutoML approach to reduce configuration space for deep learning models. Teaches metrics with upper and lower bounds utilized for guiding search in a configuration space.
Ying et al. (NPL: NAS-Bench-101: Towards Reproducible Neural Architecture Search, published 2019) teaches a dataset which is a table that maps neural network architectures to their training and evaluation metrics. Teaches a table of neural network configurations and their corresponding performance estimates for efficient look up of neural network configurations or architectures. Teaching characterizing performance estimates for each configuration setting.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LOUIS C NYE whose telephone number is (571)272-0636. The examiner can normally be reached Monday - Friday 9:00AM - 5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MATT ELL can be reached at 571-270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LOUIS CHRISTOPHER NYE/Examiner, Art Unit 2141
/MATTHEW ELL/Supervisory Patent Examiner, Art Unit 2141