Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is in response to submission of application on 11/29/2023.
Claims 1-20 are presented for examination.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-3, 7, 9-11, 15, and 17-19 are rejected under 35 U.S.C. 102(a)(l) as being anticipated by Learned Hardware/Software Co-Design of Neural Accelerators to Shi et al. (hereinafter Shi).
Per claim 1, Shi discloses A method comprising [Shi, pg. 1, Abstract, “This paper instead casts the problem as hardware/software co-design, with the goal of automatically identifying desirable points in the joint design space. The key to our solution is a new constrained Bayesian optimization framework that avoids invalid solutions by exploiting the highly constrained features of this design space…”. (note: this shows the sequential steps of the method)]:
receiving a set of hardware parameters and a set of software parameters for configuring a device [Shi, pg. 2, 2.1 Parameterizing The Design Space, “Software design points can be parameterized by the loop ordering, loop tiling, and computational parallelism of the seven-level loop nest used to compute a convolutional layer (see appendix), as has been noted by recent work (Parashar et al., 2019; Yang et al., 2020). These software parameters are subject to hardware constraints, such as the quantity and layout of processing elements (PEs) and the size of storage elements. Hardware parameters can be broken down into a two broad categories: Resource configurations represent the physical aspects of hardware, such as buffer sizes, tile sizes, and the cluster size of global buffers, as well as the layouts of the PE array and of the global buffer. Dataflow configurations represent the usage of the PE array that are implemented in hardware, such as the blocking factors and degree of parallelism at the PE level, which also determines the communication patterns among PEs.”. (note: Shi discloses receiving both hardware and software parameters for configuring a neural network accelerator device.)];
determining a first device configuration for the device using a first set of hardware parameters from the set of hardware parameters and a first set of software parameters from the set of software parameters [Shi, pg. 1, 1 Introduction, “…this hardware/software co-design can be framed as a joint search of the space of all of the valid mappings and hardware architectures that can correctly execute the model. We formally parameterize this space based on prior work (Parashar et al., 2019), and we find that standard optimization techniques, including off-the-shelf Bayesian optimization, perform poorly because the design space is semi-discrete and the vast majority of the points in the space are infeasible”. (note: this describes determining device configurations (hardware/software design points) from the parameterized design space); pg. 2, Figure 1: Overview of BO-based nested search for hardware/software co-design, “hardware BO optimizer generates hardware hi”. (note: this explicitly describes determining a hardware configuration (hardware hi) from hardware parameters during each iteration of the optimization process); pg. 5, 4.1 Overview of Nested Hardware/Software Optimization “For the chosen hardware design, our framework performs the software search for each individual neural layer in its constrained software mapping space Ss | h, NNj to optimize the mapping parameters,”. (note: Shi then determines the corresponding software configuration (mapping parameters) for the determined hardware configuration, confirming that a complete device configuration (hardware and software) is determined)];
applying the first set of hardware parameters and the first set of software parameters to a machine learning process, wherein a first output from the machine learning process comprises a first software model accuracy evaluation value for the first set of hardware parameters from the set of hardware parameters and a first hardware cost estimation value for the first set of software parameters from the set of software parameters [Shi, pg. 5, 4.1 Overview of Nested Hardware/Software Optimization “In the nested search process, we first use the hardware optimizer to generate a design of hardware. In particular, we perform the hardware search in the space of possible hardware Sh to optimize all hardware parameters, where the objective is to minimize f (xh | NN) which we define as the energy-delay product (EDP) of running the neural network (NN) model on the given hardware, assuming the optimal software mapping for each individual layer.”. (note: this shows the application of the first set of hardware parameters (xh, the generated hardware design) to the machine learning process (Bayesian optimization). The output f (xh | NN), the energy-delay product (EDP) of running the neural network on that hardware, is a software model performance evaluation value for the hardware parameters. It evaluates the performance of the software (the neural network model) given that particular set of hardware parameters.); “For the chosen hardware design, our framework performs the software search for each individual neural layer in its constrained software mapping space SS|h, NNj to optimize the mapping parameters, where NNj denotes the jth layer in the neural network model, and the objective becomes f (xs | xh, NNj), which is defined as the EDP of running the layer j on the fixed hardware.”. (note: this shows the application of the first set of software parameters (xs, the software mapping) to the same Bayesian optimization machine learning process. The output f (xs | xh, NNj), which is the EDP cost of executing the software mapping on the hardware, is a hardware cost estimation value for the software parameters. It measures what hardware cost (energy-delay product (EDP)) the software parameter set incurs on the device.); pg. 2, Figure 1: Overview of BO-based nested search for hardware/software co-design, “software BO optimizer generates a mapping sijkevaluate mapping sijk on hardware hi as Layer_EDP and feed back to layerwise software BO optimizer
// compute Model_EDP on hi
compute Model_EDP on hi as sunm of best layerwise EDP
∑
j
m
i
n
k
(
s
i
j
k
)
and feed back to hardware BO optimizer”. (note: this shows the overall structure. The software parameters (mapping sijk) are applied with the hardware parameters (hardware hi) to the machine learning process. The process produces two distinct outputs, the Layer_EDP (a hardware cost estimation value associated with the software mapping parameters) and the Model_EDP (a software model performance evaluation value associated with the hardware parameters). Both outputs are a result from the joint application of both parameter sets to the Bayesian optimization machine learning process.); pg. 6, 5.1 Methodology “Timeloop takes two inputs: 1) the hardware configuration, which consists of the hardware-related parameters, and 2) the software mapping, which consists of the software parameters that describe the mapping.”. (note: the Bayesian optimization machine learning process drives Timeloop, which takes both the hardware parameters and the software parameters as its inputs and produces performance evaluation outputs from them. This shows the application of both parameter sets to the machine learning process results in the composite output comprising both the software model accuracy evaluation value and the hardware cost estimation value.)],
wherein the first output from the machine learning process simultaneously determines the first software model accuracy evaluation value and the first hardware cost estimation value for the first device configuration [Shi, pg. 1, Abstract, “This paper instead casts the problem as hardware/software co-design, with the goal of automatically identifying desirable points in the joint design space. The key to our solution is a new constrained Bayesian optimization framework that avoids invalid solutions by exploiting the highly constrained features of this design space, which are semicontinuous/semi-discrete.”. (note: this applies both hardware and software parameters jointly to a machine learning process (Bayesian optimization)); pg. 2, 1 Introduction, “We therefore propose a nested, constrained Bayesian optimization (BO) formulation that uses Bayesian models of hardware and software performance to guide the search towards promising regions of the design space.”. (note: this shows the simultaneous output. The nested BO formulation generates outputs for both hardware cost and software performance within the same overall ML process.); pg. 2, Figure 1: Overview of BO-based nested search for hardware/software co-design, “evaluate mapping sijk on hardware hi as Layer_EDP and feed back to layerwise software BO optimizer
// compute Model_EDP on hi
compute Model_EDP on hi as sunm of best layerwise EDP
∑
j
m
i
n
k
(
s
i
j
k
)
and feed back to hardware BO optimizer”. (note: the machine learning process simultaneously produces a hardware cost estimation value (EDP is a measure of hardware cost, which is comparable to latency/area/throughput) and a software model accuracy and performance evaluation value (layerwise EDP shows software mapping quality) )]; and
sequentially applying a second set of hardware parameters and a second set of software parameters to the machine learning process to generate second output from the machine learning process [Shi, pg. 5, 4.1 Overview of Nested Hardware/Software Optimization, “The iterative search between hardware and software will repeat for a user-defined number of trials. In this work, we set 50 for hardware search and 250 for software search.”. (note: this shows sequentially applying multiple sets of hardware and software parameters to the Bayesian optimization machine learning process over multiple iterations, generating outputs at each iteration. This directly shows the sequential application of a second set of parameters.); pg. 2, Figure 1: Overview of BO-based nested search for hardware/software co-design., “for i = 1 : number of hardware trials”. (note: this for loop structure confirms sequential iteration, applying sequential sets of hardware and software parameters to generate multiple outputs from the machine learning process)].
Per claim 2, Shi discloses claim 1, further disclosing training a machine learning (ML) model during a first level of training using the first device configuration based on the first set of hardware parameters, the first set of software parameters, and the first output from applying the first set of hardware parameters and the first set of software parameters to the machine learning process;
training the ML model during a second level of training using a second device configuration, the second set of hardware parameters, the second set of software parameters, and the second output [Shi, pg. 5, 4.1 Overview of Nested Hardware/Software Optimization, “A random sample is used in the first iteration of both the hardware and software search.”. (note: the first iteration of the BO loop constitutes the first level of training. The first device configuration (a randomly sampled hardware-software pair) is used along with its evaluated output (the EDP from Timeloop) to initialize and begin training the Bayesian optimization model (the GP). This shows that the first iteration uses the first set of hardware parameters, first set of software parameters, and the first output.); “The iterative search between hardware and software will repeat for a user-defined number of trials. In this work, we set 50 for hardware search and 250 for software search.”. (note: after the first iteration completes and the GP model has been trained on the first configuration’s output, the BO process proceeds to a second iteration, generating a new (second) set of hardware and software parameters, evaluating them to produce a second output, and then using that second configuration and its output to further train the GP model. Shi’s 50 hardware trials and 250 software trials show that this is an iterative process of successive distinct training updates across many levels, with the first and second levels being the first two distinct iterations.); “In our Bayesian optimization (BO) framework, we use separate BO models to search in the hardware and software space.”; pg. 4, 3.2 Gaussian Processes “All kernel and mean hyperparameters are learned by maximizing the marginal likelihood of the GP on the current dataset.”. (note: this shows that after each iteration, the GP model’s hyperparameters are updated by maximizing the marginal likelihood on the accumulated dataset, which increases by one new data point (the new configuration and its output) with each iteration. The first level of training uses the first configuration and its output, the second level of training uses the first and second configuration and outputs. This sequential, iteration-based updating of the machine learning model on successive device configurations and their outputs shows the first level and second level training structure, where each level is a distinct training step using that iteration’s device configuration and output.]; and
using the trained ML model to predict a third device configuration that maximizes output for corresponding with hardware parameters and software parameters [Shi, pg. 5, 4.1 Overview of Nested Hardware/Software Optimization, “The combination of hardware and software that achieves the best EDP during the optimization process becomes the final model-specific hardware structure and layer-specific software mappings”. (note: the trained Bayesian optimization models are used to predict the third (optimal) device configuration that maximizes the objective (EDP))].
Per claim 3, Shi discloses claim 1, further disclosing the machine learning process is a Bayesian Optimization process with a Gaussian Process Regression [Shi, pg. 3, 3.2 Gaussian Processes, “A common surrogate model is a Gaussian process (GP) (Rasmussen & Williams, 2006) due to its simplicity and flexibility.”. (note: Gaussian Process Regression is used as the surrogate model within the Bayesian Optimization process); pg. 3, 3.1 Overview, “Bayesian optimization (Jones et al., 1998; Brochu et al., 2010; Shahriari et al., 2015) is an effective approach for the optimization of expensive, possibly noisy black-box functions.”. (note: the machine learning process is explicitly Bayesian Optimization)].
Per claim 7, Shi discloses claim 1, further disclosing the first set of hardware parameters and the first set of software parameters are provided back to the machine learning process to sequentially determine optimization values of different configuration settings [Shi, pg. 2, Figure 1: Overview of BO-based nested search for hardware/software co-design, “evaluate mapping sijk on hardware hi as Layer_EDP and feed back to layerwise software BO optimizer
// compute Model_EDP on hi
compute Model_EDP on hi as sunm of best layerwise EDP
∑
j
m
i
n
k
(
s
i
j
k
)
and feed back to hardware BO optimizer”. (note: this feeds back evaluation results to both the hardware and software Bayesian optimization processes, enabling sequential determination of optimization values); pg. 2 “The outer loop optimizes over hardware architectures, while the inner loop optimizes over software mappings for a given architecture.”. (note: this shows a nested optimization sequence)].
Claims 9-11 and 15 are directed towards the apparatus performed by the method of claims 1-3 and 7 respectively. Therefore, the rejections applied to claims 1-3 and 7 also apply to claims 9-11 and 15 [Shi, pg. 6, 5.1 Methodology, “We conduct our evaluation on Timeloop (Parashar et al., 2019), which is an open-source infrastructure for evaluating the hardware design and software optimization of DNN accelerators.” (note: Shi uses a computer system running Timeloop)].
Claims 17-19 are directed towards the article of manufacture performed by the method of claims 1-3, respectively. Therefore, the rejections applied to claims 1-3 also apply to claims 17-19 [Shi, pg. 2, 1 Introduction, “Our approach is extensible to a variety of different neural network architectures, and we make the code publicly available.”].
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 4, 5, 12, 13, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Shi in view of Understanding Reuse, Performance, and Hardware Cost of DNN Dataflows: A Data-Centric Approach Using MAESTRO to Kwon et al. (hereinafter Kwon).
Per claim 4, Shi discloses claim 1.
Shi does not fully disclose, but with Kwon does teach: the first output comprises a latency, an area, and a throughput of the first device configuration that are measured in a simulated environment that implements the first device configuration with the first hardware parameters and the first software parameters [Kwon, pg. 1, Abstract “We codify this analysis into an analytical cost model, MAESTRO (Modeling Accelerator Efficiency via Spatio-Temporal Reuse and Occupancy), that estimates various cost-benefit tradeoffs of a dataflow including execution time and energy efficiency for a DNN model and hardware configuration… identify 2.5M valid designs…including Pareto-optimal throughput- and energy-optimized design points”. (note: execution time is the definition of latency, demonstrating that MAESTRO generates latency and throughput as outputs); pg. 2, 1 Introduction, “We demonstrate MAESTRO’s abstract hardware model and analytic model to be within 90-95% accuracy of actual open-source RTL [24] while being 1029-4116× faster (10ms to run MAESTRO versus 7.2-28.8 hours for an equivalent RTL simulation on a workstation with Xeon E5-2699 processor and 64GB memory).”. (note: MAESTRO measures performance metrics in a simulated environment (abstract hardware model and analytic model): pg. 2, 1 Introduction “…we demonstrate how the MAESTRO cost model can be used by accelerator designers to determine Pareto-optimal parameters for an accelerator with a given area, energy, or throughput budget.”. (note: MAESTRO outputs area, throughput, and latency metrics from its simulated environment)].
Shi and Kwon are analogous art because they are from similar problem solving area in machine language based hardware and software co-design optimization for neural network accelerators.
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate the MAESTRO backend as taught by Kwon into the Bayesian optimization framework.
The suggestion/motivation for doing so would be improved latency and performance, and faster optimization convergence. [pg. 2, 1 Introduction “…we demonstrate how the MAESTRO cost model can be used by accelerator designers to determine Pareto-optimal parameters for an accelerator with a given area, energy, or throughput budget.”; pg. 2, 1 Introduction, “We demonstrate MAESTRO’s abstract hardware model and analytic model to be within 90-95% accuracy of actual open-source RTL [24] while being 1029-4116× faster (10ms to run MAESTRO versus 7.2-28.8 hours for an equivalent RTL simulation…).”.]
Per claim 5, Shi discloses claim 1.
Shi does not fully disclose, but with Kwon does teach: the first output is generated using a closed-form hardware cost model of the machine learning process [Kwon, pg. 1, Abstract “We codify this analysis into an analytical cost model, MAESTRO (Modeling Accelerator Efficiency via Spatio-Temporal Reuse and Occupancy), that estimates various cost-benefit tradeoffs of a dataflow including execution time and energy efficiency for a DNN model and hardware configuration”. (note: the analytical cost model of MAESTRO is a closed-form hardware cost model, it uses mathematical analytical expressions to compute hardware performance outputs); pg. 2, 1 Introduction, “MAESTRO takes as input 1) a DNN model with a set of layers, 2) a dataflow description for each layer specified using our proposed directives, and 3) the hardware configuration. Based on these inputs, MAESTRO outputs estimates of end-to-end execution time, energy (including all compute, buffer, and interconnect activities), NoC costs, and so on.”. (note: MAESTRO analytical cost model takes hardware and software parameters as inputs and produces hardware cost outputs in closed form)].
Shi and Kwon are analogous art because they are from similar problem solving area in machine language based hardware and software co-design optimization for neural network accelerators.
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate the MAESTRO backend as taught by Kwon into the Bayesian optimization framework.
The suggestion/motivation for doing so would be improved latency and performance, and faster optimization convergence. [pg. 2, 1 Introduction “…we demonstrate how the MAESTRO cost model can be used by accelerator designers to determine Pareto-optimal parameters for an accelerator with a given area, energy, or throughput budget.”; pg. 2, 1 Introduction, “We demonstrate MAESTRO’s abstract hardware model and analytic model to be within 90-95% accuracy of actual open-source RTL [24] while being 1029-4116× faster (10ms to run MAESTRO versus 7.2-28.8 hours for an equivalent RTL simulation…).”.]
Claim 12 and 20 are substantially similar in scope and spirit to claim 4, and Claim 13 is substantially similar in scope and spirit to claim 5. Therefore, the rejection of claim 4 and 5 is applied accordingly. Kwon further shows the method being implemented by an apparatus [Kwon, pg. 3, 2.2 DNN Accelerators, “As described in Figure 2, most DNN accelerators employ hundreds of processing elements (PEs) to exploit inherent parallelism in DNN applications. PEs typically include scratchpad memories (L1) and ALUs that perform multiply-accumulate operations (MACs).”]. Kwon also shows the method being implemented by an article of manufacture [Kwon, pg. 7, 4 Quantitative Dataflow Analysis, “Based on the approach, we implement an analysis framework, MAESTRO, which consists of five engines: tensor, cluster, reuse, performance analysis, and cost analysis…For details, we present them in our open-source repository [2].”].
Claims 6 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Shi in view of CODEBench: A Neural Architecture and Hardware Accelerator Co-Design Framework to Tuli et al. (hereinafter Tuli).
Per claim 6, Shi discloses claim 1.
Shi does not fully disclose, but with Tuli does teach: the first set of hardware parameters, the first set of software parameters, the second set of hardware parameters, and the second set of software parameters are selected using an active learning process [Tuli, pg. 4, “…an active learning pipeline that makes the search more efficient. Here, by gradients to the input, we mean the gradients to the CNN–accelerator pair simulated in the next iteration of the active learning loop.”. (note: the active learning pipeline is used to select which hardware and software parameters (CNN-acceleartor pairs) to evaluate next)].
Shi and Tuli are analogous art because they are from the same field of endeavor, automated hardware and software co-design using machine learning based optimization.
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the bilevel optimization loops as taught by Shi with the two-level surrogate training methodology as taught by Tuli.
The suggestion/motivation for doing so would be active learning improves search efficiency [Tuli, pg. 4, “…an active learning pipeline that makes the search more efficient. Here, by gradients to the input, we mean the gradients to the CNN–accelerator pair simulated in the next iteration of the active learning loop. BOSHCODE is a fundamental pillar for the joint exploration of vast hardware-software design spaces.”].
Claim 14 is substantially similar in scope and spirit to claim 6. Therefore, the rejection of claim 6 is applied accordingly. Tuli further shows the method being implemented by an apparatus [Tuli, pg. 22, “All models are trained on NVIDIA A100 GPUs and 2.6-GHz AMD EPYC Rome processors.”].
Claims 8 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Shi in view of FlexiBO: A Decoupled Cost-Aware Multi-Objective Optimization Approach for Deep Neural Networks to Iqbal et al. (hereinafter Iqbal).
Per claim 8, Shi discloses claim 1.
Shi does not fully disclose, but with Iqbal does teach stopping the determining of device configurations when the output corresponding with the hardware parameters and the software parameters exceeds a pre-determined threshold value [Iqbal, pg. 654 “Given the same budget θT, the cost-awareness of the acquisition function enables FlexiBO to sample the search space more efficiently compared to other state-of-the-art approaches.”. (note: FlexiBO uses a total budget θT as a predetermined threshold, stopping configuration determination when this budget is exhausted)]
Shi and Iqbal are analogous art because they are from the same field of endeavor encompassing Bayesian optimization for software and hardware parameter optimization of DNN systems.
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate the budget-based stopping criterion θT as taught by Iqbal into the optimization loop of Shi, specifically terminating the determination of device configurations when the evaluation output satisfies a predetermined performance threshold.
The suggestion/motivation for doing so would have been stopping when the gain from additional evaluations falls below a threshold, when the output value exceeds a predetermined threshold performance level, indicating convergence. This would avoid unnecessary computational expense once the optimization has converged [Iqbal, pg. 654 “Given the same budget θT, the cost-awareness of the acquisition function enables FlexiBO to sample the search space more...”; pg. 645 “…preventing FlexiBO from performing expensive measurements for little to no gain.”].
Claim 16 is substantially similar in scope and spirit to claim 8. Therefore, the rejection of claim 8 is applied accordingly. Iqbal further shows the method being implemented by an apparatus [Iqbal, pg. 646, “One of the key challenges in designing an optimal DNN system is to efficiently explore the vast design space, with non-trivial interactions of options from different components across the system stack, for example, CPU frequency, GPU frequency, number of epochs, etc.”].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Sayed M Shah whose telephone number is (571)272-9406. The examiner can normally be reached Monday-Friday 6:00 am - 2:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached at (571) 270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SAYED MUNEER SHAH/Examiner, Art Unit 2124
/Kevin W Figueroa/Primary Examiner, Art Unit 2124
Prosecution Insights