Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-25 are pending. Claims 26-40 are canceled.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-25 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract ideas without significantly more.
Step 1: Claims 1-10 are directed to an apparatus, which is a machine, falling under a statutory category of invention. Claims 11-20 are directed to a non-transitory machine readable storage medium, which is a manufacture, falling under a statutory category of invention. Claims 21-25 are directed to a method, which is a process, falling under a statutory category of invention. Therefore, claims 1-25 are directed to patent eligible categories of invention.
Regarding claim 1:
Step 2A Prong 1: The following limitations recite abstract ideas:
The limitation “determine a first hardware architectural configuration of a hardware component based on a design constraint” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. For example, this covers a person observing the design constraint and making a mental judgment on a hardware architectural configuration based on the design constraint.
The limitation “generate an aggregate score by aggregating a plurality of design space performance indicators, ones of the plurality of design space performance indicators corresponding to respective ones of the plurality of objective design spaces” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. For example, this covers a person mentally observing the plurality of performance indicators, aggregate them mentally or with a pen and paper, and making a mental judgment on the score.
This also amounts to a mathematical concept in view of the following paragraphs from specification:
[0030]: “Based on a Pareto front of the design space(s) and one or more reference points, certain examples generate a performance indicator (e.g., metric) representing a multidimensional objective space that includes a set of solutions representing a tradeoff among the different objectives. The performance indicator can be, for example, a hypervolume indicator, an R2 indicator, etc. In examples in which two objectives are measured for each workload and/or modality of interest, the performance indicator may represent an area of an Pareto front.”
[0060]: “Examples discussed below assume that the design space performance indicator is a hypervolume indicator. However, the design space performance indicator can be an R2 indicator, a variance metric, and/or another goodness metric that considers a distribution of vectors in additional or alternative examples. In some examples, the design space performance indicator is a fitness function. When measuring two objectives (e.g., latency versus accuracy, etc.), a hypervolume represents an area of a Pareto front. For example, in a case of measuring latency versus accuracy objectives, the larger the hypervolume indicator value, the better the performance of the hardware architectural configuration.”
[0064]: “The graph 400 also includes an example reference point(s) 412 and an example hypervolume 414. The hypervolume 414 is a unary performance indicator that provides a measure to quantify a quality of the Pareto front 408. The hypervolume 414 is a measure of a size of an objective design space (e.g., an area in a bi-objective setting, a volume in a many-objective setting, etc.) enclosed by solutions on the Pareto front 408 and the reference point 412. That is, the reference point 412 is needed to calculate and define the hypervolume 414. In some examples, the hypervolume 414 for the Pareto front 408 is determined as an objective space that is dominated by the Pareto front 408 relative to some reference point 412.”
The limitation “search a design database based on the aggregate score to identify a second hardware architectural configuration” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. For example, this covers a person mentally observing the database and making a mental judgment on the appropriate second hardware architectural configuration.
This also amounts to a mathematical concept in view of the following paragraphs from specification:
[0028]: “Disclosed examples also utilize neural architecture search (NAS) techniques such as, but not limited to, evolutionary search, Bayesian optimization, and/or reinforcement learning to explore a search space of neural network architectures based on a workload to identify variations of the workload (e.g., sub-models, etc.).”
[0067]: “For example, the searcher circuitry 222 (discuss in detail below) may apply a search algorithm that uses a relative ranking of different scores (e.g., 0-1000, 0-1, etc.)”
Examiner notes that such optimization algorithms or search algorithms involve mathematical relationships, calculations, and/or equations/formulas.
The limitation “predict a performance of the second hardware architectural configuration to generate a performance metric by executing a proxy function corresponding to the second hardware architectural configuration” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. For example, this covers a person evaluating or observing a proxy function or an estimation function mentally or with a pen and paper and then making a mental judgment on the performance based on it.
This also amounts to a mathematical concept in view of the following paragraphs from specification:
[0034]: “Certain examples mitigate the high costs associated with measuring (e.g., evaluating) performance of a hardware architectural configuration by generating (e.g., building) an example low-fidelity performance estimator that leverages structural information (e.g., graph) about the hardware architectural configurations to build a machine learning (ML) based proxy function. … Thus, certain examples reduce design overhead associated with hardware accelerator design by utilizing a computationally cheaper (e.g., compared to a simulation) proxy function (e.g., using reinforcement learning, Bayesian optimization, etc.) to estimate a performance of the hardware architectural configuration typically determined during a simulation.”
[0076]: “The estimation function(s) 232 as executed by the performance estimator circuitry 230 of FIG. 2 applies a graph-based prediction algorithm to generate an example score(s) 234. For example, the estimation function 232 can generate a graph representation of a PE array considering that most hardware architectures (e.g., from a circuit design perspective) can be represented as a graph. In some examples, another general regressor (e.g., linear ridge, graph neural network regressor, etc.) can be used additionally or alternatively. … In some examples, the estimation function 232 represents a hardware architectural configuration as a vector.”
Step 2A Prong 2: The following limitations recite additional elements:
“at least one memory”
“machine readable instructions”
“processor circuitry to at least one of instantiate or execute the machine readable instructions”
“simulate an execution of the first hardware architectural configuration for a plurality of workloads to generate a respective plurality of objective design spaces, the objective design spaces based on one or more objectives”
These additional elements do not integrate the judicial exception into a practical application because they amount to no more than mere instructions to apply the judicial exception using a generic computer. A memory, machine readable instructions, and a processor are generic computer components. According to specification para [0029] and [0049]-[0050], simulating the execution of the first hardware architectural configuration for a plurality of workloads amounts to executing computer instructions using a processor. This amounts to using a generic computer. See MPEP 2106.05(f).
Even when viewed in combination, these additional elements do not integrate the judicial exception into a practical application.
Accordingly, the claim does not recite any additional elements that integrate the judicial exception into a practical application.
Step 2B: As previously discussed, the additional elements amount to no more than mere instructions to apply the exception using a generic computer. Such activities do not amount to significantly more than the judicial exception. See MPEP 2106.05(f).
Therefore, claim 1 is not eligible.
Regarding claim 2: The limitation “wherein the first hardware architectural configuration is a baseline design, and the second hardware architectural configuration is an adjusted first hardware architectural configuration” merely further limits the first hardware architectural configuration and the second hardware architectural configuration recited in claim 1. Therefore, the same analysis is applicable. Furthermore, adjusting the hardware architectural configuration amounts to a mental process as a person can mentally make a judgment on the appropriate modification to the configuration.
Regarding claim 3: The limitation “wherein the plurality of design space performance indicators are hypervolume indicators” merely further limits the plurality of design space performance indicators recited in claim 1. Therefore, the same analysis is applicable. Furthermore, hypervolume indicators amounts to a mathematical concept as explained in claim 1.
Regarding claim 4: The limitation “wherein the plurality of design space performance indicators are weighted, and wherein the aggregate score is a weighted aggregate score” merely further limits the plurality of design space performance indicators recited in claim 1. Therefore, the same analysis is applicable. Furthermore, weighting scores amounts to a mental process as a person can perform such weighting mentally or with a pen and paper. It also mounts to a mathematical concept as weighting values involves mathematical relationships, calculations, or formulas/equations.
Regarding claim 5: The limitation “wherein a first workload of the plurality of workloads is a deep neural network architecture” merely further limits the plurality of workloads recited in claim 1. Therefore, the same analysis is applicable.
This also amounts to generally linking the use of a judicial exception to a particular technological environment or field of use of merely using a neural network as one of the architectures. Such limitations do not integrate the judicial exception into a practical application or amount to significantly more than the judicial exception. See MPEP 2106.05(h).
Regarding claim 6: The limitation “wherein the plurality of workloads correspond to one or more modalities” merely further limits the plurality of workloads recited in claim 1. Therefore, the same analysis is applicable.
Regarding claim 7: The limitation “wherein the one or more objectives include at least one of minimize latency, maximize efficiency, or maximize accuracy” merely further limits the one or more objectives recited in claim 1. Therefore, the same analysis is applicable.
Regarding claim 8: The limitation “wherein executing the proxy function is utilizes fewer compute resources as compared to the simulation of the execution of the first hardware architectural configuration” merely further limits the proxy function recited in claim 1. Therefore, the same analysis is applicable.
Regarding claim 9:
The limitation “wherein the performance metric is a first performance metric” merely further limits the performance metric recited in claim 1. Therefore, the same analysis is applicable.
The limitation “wherein the processor circuitry at least one of instantiates or executes the machine readable instructions to: execute a search of the design space for the hardware component based on the first performance metric to determine a third hardware architectural configuration” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. For example, this covers a person mentally observing the first performance metric and making a judgment on the appropriate third hardware architectural configuration.
This also amounts to a mathematical concept in view of the following paragraphs from specification:
[0028]: “Disclosed examples also utilize neural architecture search (NAS) techniques such as, but not limited to, evolutionary search, Bayesian optimization, and/or reinforcement learning to explore a search space of neural network architectures based on a workload to identify variations of the workload (e.g., sub-models, etc.).”
[0067]: “For example, the searcher circuitry 222 (discuss in detail below) may apply a search algorithm that uses a relative ranking of different scores (e.g., 0-1000, 0-1, etc.)”
Examiner notes that such optimization algorithms or search algorithms involve mathematical relationships, calculations, and/or equations/formulas.
The limitation “predict a performance of the third hardware architectural configuration to generate a second performance metric by executing a proxy function corresponding to the third hardware architectural configuration” amounts to a mental process and a mathematical concepts for the similar reason as explained in claim 1.
Regarding claim 10:
The limitation “wherein, in response to the second performance metric exceeding a defined value, the processor circuitry least one of instantiates or executes the machine readable instructions to: simulate an execution of the third hardware architectural configuration for a plurality of workloads to generate a respective plurality of objective design spaces, the objective design spaces based on one or more objectives” amounts to mere instructions to apply the judicial exception using a generic computer as simulating an execution of a architectural configuration amounts to mere instructions to apply the judicial exception using a generic computer in view of para [0029] and [0049]-[0050] as explained in claim 1. See MPEP 2106.05(f).
The limitation “generate a design score for the third hardware architectural configuration by aggregating a plurality of design space performance indicators, the plurality of design space performance indicators to correspond to the plurality of objective design spaces” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. For example, a person can aggregate or combine scores mentally or with a pen and paper. It also mounts to a mathematical concept as aggregating values involves mathematical relationships, calculations, or formulas/equations.
The limitation “execute another search of the design space for the hardware component based on the design score for the third hardware architectural configuration to determine a fourth hardware architectural configuration” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. For example, this covers a person mentally observing the design score and making a judgment on the appropriate fourth hardware architectural configuration.
This also amounts to a mathematical concept in view of the following paragraphs from specification:
[0028]: “Disclosed examples also utilize neural architecture search (NAS) techniques such as, but not limited to, evolutionary search, Bayesian optimization, and/or reinforcement learning to explore a search space of neural network architectures based on a workload to identify variations of the workload (e.g., sub-models, etc.).”
[0067]: “For example, the searcher circuitry 222 (discuss in detail below) may apply a search algorithm that uses a relative ranking of different scores (e.g., 0-1000, 0-1, etc.)”
Examiner notes that such optimization algorithms or search algorithms involve mathematical relationships, calculations, and/or equations/formulas.
Regarding claim 11: Claim 11 is substantially similar to claim 1. Therefore, the similar analysis is applicable.
Furthermore, the limitation “A non-transitory machine readable storage medium comprising instructions that, when executed, cause processor circuitry to at least” does not integrate the judicial exception into a practical application or amount to significantly more than the judicial exception because it amounts to mere instructions to apply the judicial exception using a generic computer. See MPEP 2106.05(f).
Claims 12-25 are substantially similar to claims 1-10. Therefore, the similar analysis is applicable.
Accordingly, claims 1-25 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e. an abstract idea) without anything significantly more.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-25 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Sekanina (“Neural Architecture Search and Hardware Accelerator Co-Search: A Survey”).
Regarding claim 1, Sekanina discloses
at least one memory; machine readable instructions; and processor circuitry to at least one of instantiate or execute the machine readable instructions (Pg. 151350, Left column: “Latency can be estimated using: a surrogate model, e.g., [6], [126], [128]-[131]; a suitable hardware simulator executing a candidate CNN, e.g. [51], [100], [132]-[135]; a formula or model derived after analyzing the search space of possible CNN architectures, e.g., [14], [47], [49], [101], [136]-[138]; a LUT-based model [105], [139], [140].”) (Pg. 151355, Right column: “Focusing on Google's EdgeTPU, NAHAS [129] performs a joint search in the space of CNN architectures and hardware accelerator configurations, where hardware resources and latency are constraints.”) (Pg. 151355, Right column: “Focusing on Google's EdgeTPU, NAHAS [129] performs a joint search in the space of CNN architectures and hardware accelerator configurations, where hardware resources and latency are constraints. … An in-house simulator is used to estimate latency and other hardware-related parameters.”) to:
Examiner notes that such hardware simulators are performed on a computer including a processor, a memory, and machine readable instructions.
determine a first hardware architectural configuration of a hardware component based on a design constraint (Pg. 151354, Left column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105]. In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”) (Pg. 151353, Right column: “In another co-search strategy used by, e.g., in [47], [51], the hardware optimization algorithm receives a CNN as the input and optimizes the hardware accelerator concerning desired objectives (Fig. 13).”) (Pg. 151355, Left column: “An RL controller generates parameters describing the CNN architecture (based on ProxylessNAS) as well as the hardware parameters such as the number of PEs, buffer size, and dataflow pattern. … DANCE thus introduces a novel differentiable evaluator, which takes the architecture parameters from the RL controller, searches for the optimal hardware accelerator design, and evaluates its cost metrics.”) (Pg. 151343, Right column: “Using cost models (such as MAESTRO [71]), the run time, resources utilization, delay, and power can be estimated for a given accelerator, DNN, and data set.”);
simulate an execution of the first hardware architectural configuration for a plurality of workloads to generate a respective plurality of objective design spaces, the objective design spaces based on one or more objectives (Pg. 151348, Left column: “Typical objectives to be optimized for a given hardware platform are latency, throughput, energy efficiency, and memory usage.”) (Pg. 151354, Left column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105]. In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”) (Pg. 151350, Left column: “Latency can be estimated using: a surrogate model, e.g., [6], [126], [128]-[131]; a suitable hardware simulator executing a candidate CNN, e.g. [51], [100], [132]-[135]; a formula or model derived after analyzing the search space of possible CNN architectures, e.g., [14], [47], [49], [101], [136]-[138]; a LUT-based model [105], [139], [140].”) (Pg. 151355, Right column: “Focusing on Google's EdgeTPU, NAHAS [129] performs a joint search in the space of CNN architectures and hardware accelerator configurations, where hardware resources and latency are constraints. The architecture search space is based on a new fused inverted bottleneck layer with tunable parameters. The accelerator search space is defined by seven parameters (e.g., the PE array size, the number of SIMD units, register file capacity). An in-house simulator is used to estimate latency and other hardware-related parameters.”);
generate an aggregate score by aggregating a plurality of design space performance indicators, ones of the plurality of design space performance indicators corresponding to respective ones of the plurality of objective design spaces (Pg. 151353, Left column: “The Multi-objective strategy is based either on a Pareto front construction method (`Pareto'), aggregation method (`Agg'), or applying some constraints (`Constr').”) (Pg. 151349, Left column: “A common approach to solve the multi-objective NAS problem adopted by the NAS community is either (i) to transform it into a single-objective one (using suitable constraints, prioritization, or aggregation techniques)”) (Pg. 151349, Right column: “The aggregation methods introduce a suitable aggregation function (such as the weighted sum, weighted exponential sum, or weighted product) for the objective functions and optimize the composition.”);
search a design database based on the aggregate score to identify a second hardware architectural configuration (Pg. 151353, Left column: “In addition to the architecture search space and parameter search space (weights), there is an additional search space, called the hardware search space, containing all possible hardware configurations. … The space of hardware configurations can be searched together with the space of DNN architectures using the same search algorithm … In the Multi-objective: strategy column, the `co-search' means that there are two independent search algorithms, i.e., a co-search is conducted; one search algorithm operates in the network architecture space and the other in the hardware search space. The Multi-objective strategy is based either on a Pareto front construction method (`Pareto'), aggregation method (`Agg'), or applying some constraints (`Constr').”) (Pg. 151353, Right column: “A straightforward approach to organizing the co-search is to generate a CNN-accelerator pair, which is evaluated by training the network to obtain its accuracy and measuring the hardware parameters (Fig. 12). Based on this evaluation, the next candidate pairs are generated until the desired solution is not obtained.”) (Pg. 151354, Left column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105]. In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”) (Figure 13 shows the iterative search procedure for identifying a hardware architectural configuration.) (Pg. 151354, Right column: “In AutoDNN [14], each candidate CNN consists of several hardware-aware parameterizable cells called Bundles. By means of these cells, specialized software can map any candidate CNN generated by NAS to an FPGA accelerator based on a fine-grained tile-based pipeline architecture whose components are pre-designed and stored in a component library. Latency and resources are estimated and used back in theNAS algorithm.”) (Pg. 151349, Right column: “The aggregation methods introduce a suitable aggregation function (such as the weighted sum, weighted exponential sum, or weighted product) for the objective functions and optimize the composition.”) (Figure 13 shows the iterative search procedure for identifying a hardware architectural configuration.) (Pg. 151354, Left column: “In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS.”); and
predict a performance of the second hardware architectural configuration to generate a performance metric by executing a proxy function corresponding to the second hardware architectural configuration (Pg. 151343, Right column: “Using cost models (such as MAESTRO [71]), the run time, resources utilization, delay, and power can be estimated for a given accelerator, DNN, and data set.”) (Pg. 151350, Left column: “Latency can be estimated using: a surrogate model, e.g., [6], [126], [128]-[131]; a suitable hardware simulator executing a candidate CNN, e.g. [51], [100], [132]-[135]; a formula or model derived after analyzing the search space of possible CNN architectures, e.g., [14], [47], [49], [101], [136]-[138]; a LUT-based model [105], [139], [140].”) (Pg. 151353, Right column: “In another co-search strategy used by, e.g., in [47], [51], the hardware optimization algorithm receives a CNN as the input and optimizes the hardware accelerator concerning desired objectives (Fig. 13).”) (Pg. 151354, Right column: “The RL controller samples parameters of a candidate CNN architecture and its possible quantization. For the sampled network, the hardware builder searches the hardware space to find a suitable hardware model. Each candidate hardware model is validated against the specification (latency constraint) during the search, and the result is sent back to the controller. … In Codesign-NAS [139], RL controller selects a CNN architecture from a CNN search space and a hardware architecture from an accelerator design space. Both are sent to the evaluator that implements the CNN on the proposed accelerator to find accuracy and efficiency metrics, such as latency, area, and power (based on pre-computed models).”) (Pg. 151355, Left column: “An RL controller generates parameters describing the CNN architecture (based on ProxylessNAS) as well as the hardware parameters such as the number of PEs, buffer size, and dataflow pattern. At the heart of DANCE is the modeling of the accelerator evaluation software using a neural network (the evaluator) that can be used as a differentiable loss function. DANCE thus introduces a novel differentiable evaluator, which takes the architecture parameters from the RL controller, searches for the optimal hardware accelerator design, and evaluates its cost metrics.”) (Pg. 151356, Left column: “After that, a reward based on both the yielded accuracy and pipeline efficiency is generated, which is used to update the RNN controller.”).
Regarding claim 2, Sekanina discloses
wherein the first hardware architectural configuration is a baseline design, and the second hardware architectural configuration is an adjusted first hardware architectural configuration (Pg. 151354, Left column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105]. In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”) (Pg. 151353, Right column: “A straightforward approach to organizing the co-search is to generate a CNN-accelerator pair, which is evaluated by training the network to obtain its accuracy and measuring the hardware parameters (Fig. 12). Based on this evaluation, the next candidate pairs are generated until the desired solution is not obtained.”) (Figure 13 shows the iterative search procedure for identifying a hardware architectural configuration.).
Regarding claim 3, Sekanina discloses
wherein the plurality of design space performance indicators are hypervolume indicators (Pg. 151349, Left column: “For example, NSGANetV1 [94] employs the hypervolume performance metric, which calculates the dominated area (hypervolume, in the general case) from the set of solutions to a reference point which is usually an estimate of the nadir point – a vector concatenating worst objective values of the Pareto front.”).
Regarding claim 4, Sekanina discloses
wherein the plurality of design space performance indicators are weighted, and wherein the aggregate score is a weighted aggregate score (Pg. 151349, Right column: “The aggregation methods introduce a suitable aggregation function (such as the weighted sum, weighted exponential sum, or weighted product) for the objective functions and optimize the composition.”).
Regarding claim 5, Sekanina discloses
wherein a first workload of the plurality of workloads is a deep neural network architecture (Pg. 151354, Left column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105].”) (Pg. 151353, Right column: “In another co-search strategy used by, e.g., in [47], [51], the hardware optimization algorithm receives a CNN as the input and optimizes the hardware accelerator concerning desired objectives (Fig. 13).”) (Pg. 151343, Right column: “Using cost models (such as MAESTRO [71]), the run time, resources utilization, delay, and power can be estimated for a given accelerator, DNN, and data set.”) (Pg. 151355, Left column: “An RL controller generates parameters describing the CNN architecture (based on ProxylessNAS) as well as the hardware parameters such as the number of PEs, buffer size, and dataflow pattern. … DANCE thus introduces a novel differentiable evaluator, which takes the architecture parameters from the RL controller, searches for the optimal hardware accelerator design, and evaluates its cost metrics.”) (Pg. 151343, Right column: “Using cost models (such as MAESTRO [71]), the run time, resources utilization, delay, and power can be estimated for a given accelerator, DNN, and data set.”).
Regarding claim 6, Sekanina discloses
wherein the plurality of workloads correspond to one or more modalities (Pg. 151340, Right column, Fig. 4: “FIGURE 4. Examples of hand-crafted CNNs and their blocks: (a) AlexNet for CIFAR-10 classification; (b) Residual block from ResNet [31]; (c) Inception block from GoogleNet [30]”) (Pg. 151341, Left column: “Please note that some of the following networks exist in several versions (e.g., ResNet-8, ResNet-14 etc.), which differ in the number of layers, the structure of building blocks, and some other parameters.”) (Pg. 151344, Right column: “In the case of the macro search space (Fig. 7a), the search space is determined by a set of possible operations for each node, hyperparameters of the network architecture, and a network template. … Another option is to parameterize a well-known CNN (such as ResNet [31] or MobileNetV3 [33]) and use it as a template (e.g., in [7], [51], [79][81]) for building and constraining the search space.”).
Regarding claim 7, Sekanina discloses
wherein the one or more objectives include at least one of minimize latency, maximize efficiency, or maximize accuracy (Pg. 151338, Left column: “Hence, the newest NAS methods co-optimize NN architectures and hardware configuration to further improve latency and other parameters that are important in many applications such as IoT or mobile phones. These methods work in three search spaces (weights, NN architectures, and hardware configurations) and must innovatively orchestrate several search algorithms to produce the best trade-offs between the accuracy and various hardware-relevant metrics [8], [14], [15].”) (Pgs. 151354-151355: “In Codesign-NAS [139], RL controller selects a CNN architecture from a CNN search space and a hardware architecture from an accelerator design space. Both are sent to the evaluator that implements the CNN on the proposed accelerator to find accuracy and efficiency metrics, such as latency, area, and power (based on pre-computed models).”) (Pg. 151343, Right column: “Using cost models (such as MAESTRO [71]), the run time, resources utilization, delay, and power can be estimated for a given accelerator, DNN, and data set.”) (Pg. 151355, Left column: “An RL controller generates parameters describing the CNN architecture (based on ProxylessNAS) as well as the hardware parameters such as the number of PEs, buffer size, and dataflow pattern. At the heart of DANCE is the modeling of the accelerator evaluation software using a neural network (the evaluator) that can be used as a differentiable loss function. DANCE thus introduces a novel differentiable evaluator, which takes the architecture parameters from the RL controller, searches for the optimal hardware accelerator design, and evaluates its cost metrics.”).
Regarding claim 8, Sekanina discloses
wherein executing the proxy function is utilizes fewer compute resources as compared to the simulation of the execution of the first hardware architectural configuration (Pg. 151354, Left column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105]. In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”) (Figure 13 shows the iterative search procedure for identifying a hardware architectural configuration.) (Pg. 151343, Right column: “Using cost models (such as MAESTRO [71]), the run time, resources utilization, delay, and power can be estimated for a given accelerator, DNN, and data set.”).
Regarding claim 9, Sekanina discloses
wherein the performance metric is a first performance metric (Pgs. 151354-151355: “In Codesign-NAS [139], RL controller selects a CNN architecture from a CNN search space and a hardware architecture from an accelerator design space. Both are sent to the evaluator that implements the CNN on the proposed accelerator to find accuracy and efficiency metrics, such as latency, area, and power (based on pre-computed models).”) (Pg. 151354, Left column: “In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”), and
wherein the processor circuitry at least one of instantiates or executes the machine readable instructions to: execute a search of the design space for the hardware component based on the first performance metric to determine a third hardware architectural configuration (Pg. 151353, Left column: “In addition to the architecture search space and parameter search space (weights), there is an additional search space, called the hardware search space, containing all possible hardware configurations. … The space of hardware configurations can be searched together with the space of DNN architectures using the same search algorithm … In the Multi-objective: strategy column, the `co-search' means that there are two independent search algorithms, i.e., a co-search is conducted; one search algorithm operates in the network architecture space and the other in the hardware search space. The Multi-objective strategy is based either on a Pareto front construction method (`Pareto'), aggregation method (`Agg'), or applying some constraints (`Constr').”) (Pg. 151353, Right column: “A straightforward approach to organizing the co-search is to generate a CNN-accelerator pair, which is evaluated by training the network to obtain its accuracy and measuring the hardware parameters (Fig. 12). Based on this evaluation, the next candidate pairs are generated until the desired solution is not obtained.”) (Pg. 151354, Left column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105]. In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”) (Figure 13 shows the iterative search procedure for identifying a hardware architectural configuration.) (Pg. 151354, Right column: “In AutoDNN [14], each candidate CNN consists of several hardware-aware parameterizable cells called Bundles. By means of these cells, specialized software can map any candidate CNN generated by NAS to an FPGA accelerator based on a fine-grained tile-based pipeline architecture whose components are pre-designed and stored in a component library. Latency and resources are estimated and used back in theNAS algorithm.”) (Pg. 151349, Right column: “The aggregation methods introduce a suitable aggregation function (such as the weighted sum, weighted exponential sum, or weighted product) for the objective functions and optimize the composition.”) (Figure 13 shows the iterative search procedure for identifying a hardware architectural configuration.) (Pg. 151354, Left column: “In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS.”); and
predict a performance of the third hardware architectural configuration to generate a second performance metric by executing a proxy function corresponding to the third hardware architectural configuration (Pg. 151343, Right column: “Using cost models (such as MAESTRO [71]), the run time, resources utilization, delay, and power can be estimated for a given accelerator, DNN, and data set.”) (Pg. 151350, Left column: “Latency can be estimated using: a surrogate model, e.g., [6], [126], [128]-[131]; a suitable hardware simulator executing a candidate CNN, e.g. [51], [100], [132]-[135]; a formula or model derived after analyzing the search space of possible CNN architectures, e.g., [14], [47], [49], [101], [136]-[138]; a LUT-based model [105], [139], [140].”) (Pg. 151353, Right column: “In another co-search strategy used by, e.g., in [47], [51], the hardware optimization algorithm receives a CNN as the input and optimizes the hardware accelerator concerning desired objectives (Fig. 13).”) (Pg. 151354, Right column: “The RL controller samples parameters of a candidate CNN architecture and its possible quantization. For the sampled network, the hardware builder searches the hardware space to find a suitable hardware model. Each candidate hardware model is validated against the specification (latency constraint) during the search, and the result is sent back to the controller. … In Codesign-NAS [139], RL controller selects a CNN architecture from a CNN search space and a hardware architecture from an accelerator design space. Both are sent to the evaluator that implements the CNN on the proposed accelerator to find accuracy and efficiency metrics, such as latency, area, and power (based on pre-computed models).”) (Pg. 151355, Left column: “An RL controller generates parameters describing the CNN architecture (based on ProxylessNAS) as well as the hardware parameters such as the number of PEs, buffer size, and dataflow pattern. At the heart of DANCE is the modeling of the accelerator evaluation software using a neural network (the evaluator) that can be used as a differentiable loss function. DANCE thus introduces a novel differentiable evaluator, which takes the architecture parameters from the RL controller, searches for the optimal hardware accelerator design, and evaluates its cost metrics.”) (Pg. 151356, Left column: “After that, a reward based on both the yielded accuracy and pipeline efficiency is generated, which is used to update the RNN controller.”).
Regarding claim 10, Sekanina discloses
wherein, in response to the second performance metric exceeding a defined value, the processor circuitry least one of instantiates or executes the machine readable instructions to: simulate an execution of the third hardware architectural configuration for a plurality of workloads to generate a respective plurality of objective design spaces, the objective design spaces based on one or more objectives (Pg. 151353, Right column: “In another co-search strategy used by, e.g., in [47], [51], the hardware optimization algorithm receives a CNN as the input and optimizes the hardware accelerator concerning desired objectives (Fig. 13).”) (Figure 13 shows checking whether or not the constraint has been satisfied.);
generate a design score for the third hardware architectural configuration by aggregating a plurality of design space performance indicators, the plurality of design space performance indicators to correspond to the plurality of objective design spaces (Pg. 151353, Left column: “The Multi-objective strategy is based either on a Pareto front construction method (`Pareto'), aggregation method (`Agg'), or applying some constraints (`Constr').”) (Pg. 151349, Left column: “A common approach to solve the multi-objective NAS problem adopted by the NAS community is either (i) to transform it into a single-objective one (using suitable constraints, prioritization, or aggregation techniques)”) (Pg. 151349, Right column: “The aggregation methods introduce a suitable aggregation function (such as the weighted sum, weighted exponential sum, or weighted product) for the objective functions and optimize the composition.”) (Pg. 151354, Left column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105]. In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”) (Pg. 151354, Right column: “The RL controller samples parameters of a candidate CNN architecture and its possible quantization. For the sampled network, the hardware builder searches the hardware space to find a suitable hardware model. Each candidate hardware model is validated against the specification (latency constraint) during the search, and the result is sent back to the controller. … In Codesign-NAS [139], RL controller selects a CNN architecture from a CNN search space and a hardware architecture from an accelerator design space. Both are sent to the evaluator that implements the CNN on the proposed accelerator to find accuracy and efficiency metrics, such as latency, area, and power (based on pre-computed models).”) (Pg. 151355, Left column: “An RL controller generates parameters describing the CNN architecture (based on ProxylessNAS) as well as the hardware parameters such as the number of PEs, buffer size, and dataflow pattern. At the heart of DANCE is the modeling of the accelerator evaluation software using a neural network (the evaluator) that can be used as a differentiable loss function. DANCE thus introduces a novel differentiable evaluator, which takes the architecture parameters from the RL controller, searches for the optimal hardware accelerator design, and evaluates its cost metrics.”) (Pg. 151356, Left column: “After that, a reward based on both the yielded accuracy and pipeline efficiency is generated, which is used to update the RNN controller.”)