DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Responsive to the communication dated 03/27/2026
Claims 1-25 are pending. Claims 26-40 are canceled.
Examiner’s Note
The examination of this application has been taken over by a new examiner; while an attempt has been made to adhere to the previous examiner’s style and formatting as much as possible for the sake of consistency, minor differences in presentation may exist. Nevertheless, while they may be presented slightly differently in the following action, the previous rejections are maintained except where necessitated by amendment.
Finality
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Response to Arguments- 35 USC § 101
Applicant's arguments filed 03/27/2026 have been fully considered but they are not persuasive.
Applicant argues that a person could not perform the amended processing pipeline mentally.
Examiner responds by explaining that, as amended, the pipeline can indeed be performed mentally or as a combination of mental processes and mathematic concepts.
To break down the various amended limitations:
simulate an execution of the first hardware architectural configuration for a plurality of workloads to generate performance metrics for the plurality of workloads based on a plurality of objectives;
Applying a computer to perform a generic simulation at a high level of generality is simply the act of instructing a computer to perform generic functions to perform that simulation, which is merely an instruction to apply a computer to the judicial exception. The claim only recites the idea of a solution or outcome, i.e. that the execution is “simulated” without reciting how this simulation is actually accomplished. Further, the computer elements claimed are cited as merely generic tools to perform the operations; for additional clarity see ([Par 50] “In some examples, the simulator circuitry 208 may be instantiated by hardware logic circuitry, which may be implemented by an ASIC, XPU, or the FPGA circuitry 1100 of FIG. 11 structured to perform operations corresponding to the machine readable instructions. Additionally or alternatively, the simulator circuitry 208 may be instantiated by any other combination of hardware, software, and/or firmware. For example, the simulator circuitry 208 may be implemented by at least one or more hardware circuits (e.g., processor circuitry, discrete and/or integrated analog and/or digital circuitry, an FPGA, an ASIC, an XPU, a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to execute some or all of the machine readable instructions and/or to perform some or all of the operations corresponding to the machine readable instructions without executing software or firmware, but other structures are likewise appropriate.” [Par 52] “he example workload database 212 is structured to store workload(s) that can be used during simulation of a hardware architectural configuration. In some examples, the workload database 212 can store example Al model(s) (e.g., trained and/or pre-trained), DNN workload(s), other workload types, datasets, etc. across different modalities and/or domains. Domains can include text problem domains (e.g., embedding, language model, classification, retrieval question answering, etc.), image problem domains (e.g., classification, detection, etc.), video problem domains (e.g., detection, recognition, etc.), and/or audio problem domains (e.g., embeddings, etc.). For example, the workload database 212 can include workloads such as, but not limited to ResNet models, AlexNet models, transformers, and/or any other workload that can execute on a given hardware architectural component. In some examples, the workload database 212 stores workloads that are compatible with the performance evaluator circuitry 210. However, the workload database 212 stores other workloads in additional or alternative examples. In some examples, the workload database 212 includes an interface that enables new workloads (e.g., workloads not already in the workload database 212) to be added to the workload database 212. For example, the interface can allow the new workloads to be added automatically during a design process (e.g., from the performance evaluator circuitry 210), manually by a user, etc.”)
The courts have found that such mere instructions to apply are not indicative of integration into a practical application nor recitation of significantly more than the judicial exception (MPEP 2106.05(f) “Another consideration when determining whether a claim integrates a judicial exception into a practical application in Step 2A Prong Two or recites significantly more than a judicial exception in Step 2B is whether the additional elements amount to more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer. As explained by the Supreme Court, in order to make a claim directed to a judicial exception patent-eligible, the additional element or combination of elements must do "‘more than simply stat[e] the [judicial exception] while adding the words ‘apply it’". Alice Corp. v. CLS Bank, 573 U.S. 208, 221, 110 USPQ2d 1976, 1982-83 (2014) (quoting Mayo Collaborative Servs. V. Prometheus Labs., Inc., 566 U.S. 66, 72, 101 USPQ2d 1961, 1965). Thus, for example, claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible. Alice Corp., 573 U.S. at 223, 110 USPQ2d at 1983”)
The limitation “identify, for respective ones of the plurality of workloads, a Pareto front from the performance metrics for the respective workload;” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. Identifying a Pareto front from the performance metrics is a mental process equivalent to determining the configurations that best optimize these performance metrics on a drawn graph. For example, if the compared metrics are latency and accuracy, determining the Pareto front amounts to identifying the points corresponding to configurations that, for a particular latency, maximize accuracy. Determining the pareto front from these data points consists of merely determining a curve or piecewise function that approximates these points.
The limitation “determine, for the respective ones of the plurality of workloads, a respective objective design space based on a region of an objective space enclosed by the Pareto front for the respective workload and a reference point;” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. Given the identified Pareto front and the graph it was drawn on, the graph itself representing the entire possible design space, as well as a reference point, determining the respective objective design space is a mental process equivalent to limiting further considered configurations to those that correspond to points that would be enclosed within the Pareto front relative to the reference point. As this is merely the recognition of certain areas of a graph bounded by certain points, this is merely a mental process.
Further, the limitation “generate an aggregate score by aggregating a plurality of design space performance indicators, ones of the plurality of design space performance indicators corresponding to respective ones of the respective plurality of objective design spaces determined for the respective ones of the plurality of workloads;” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. For example, this covers a person mentally observing the plurality of performance indicators, aggregate them mentally or with a pen and paper, and making a mental judgment on the score.
This also amounts to a mathematical concept in view of the following paragraphs from specification:
[0030]: “Based on a Pareto front of the design space(s) and one or more reference points, certain examples generate a performance indicator (e.g., metric) representing a multidimensional objective space that includes a set of solutions representing a tradeoff among the different objectives. The performance indicator can be, for example, a hypervolume indicator, an R2 indicator, etc. In examples in which two objectives are measured for each workload and/or modality of interest, the performance indicator may represent an area of an Pareto front.”
[0060]: “Examples discussed below assume that the design space performance indicator is a hypervolume indicator. However, the design space performance indicator can be an R2 indicator, a variance metric, and/or another goodness metric that considers a distribution of vectors in additional or alternative examples. In some examples, the design space performance indicator is a fitness function. When measuring two objectives (e.g., latency versus accuracy, etc.), a hypervolume represents an area of a Pareto front. For example, in a case of measuring latency versus accuracy objectives, the larger the hypervolume indicator value, the better the performance of the hardware architectural configuration.”
[0064]: “The graph 400 also includes an example reference point(s) 412 and an example hypervolume 414. The hypervolume 414 is a unary performance indicator that provides a measure to quantify a quality of the Pareto front 408. The hypervolume 414 is a measure of a size of an objective design space (e.g., an area in a bi-objective setting, a volume in a many-objective setting, etc.) enclosed by solutions on the Pareto front 408 and the reference point 412. That is, the reference point 412 is needed to calculate and define the hypervolume 414. In some examples, the hypervolume 414 for the Pareto front 408 is determined as an objective space that is dominated by the Pareto front 408 relative to some reference point 412.”
Applicant argues that the claims are integrated into a practical application because the process reduces the computational cost of performing hardware design operations.
Examiner responds by explaining that this reduction of computational cost is merely the result of limiting the size of the design space that must be searched through the mental processes described above and running an approximate model on designs from that space. It is important to note that the inventive concept is not in actually constructing this proxy model, of which little detail is recited, but merely in the selection of what data to use. As this improvement is solely furnished by the mental process of data selection itself, it cannot form the basis of integration into a practical application. (MPEP 2106.05(a)(I): An inventive concept "cannot be furnished by the unpatentable law of nature (or natural phenomenon or abstract idea) itself." Genetic Techs. Ltd. v. Merial LLC, 818 F.3d 1369, 1376, 118 USPQ2d 1541, 1546 (Fed. Cir. 2016))
Response to Arguments- 35 USC § 103
Applicant's arguments filed 03/27/2026 have been fully considered but they are not persuasive.
Applicant argues that no prior art teaches the newly amended claim limitations of “simulate an execution of the first hardware architectural configuration for a plurality of workloads to generate performance metrics for the plurality of workloads based on a plurality of objectives;
identify, for respective ones of the plurality of workloads, a Pareto front from the performance metrics for the respective workload;
determine, for the respective ones of the plurality of workloads, a respective objective design space based on a region of an objective space enclosed by the Pareto front for the respective workload and a reference point;
generate an aggregate score by aggregating a plurality of design space performance indicators, ones of the plurality of design space performance indicators corresponding to respective ones of the respective plurality of objective design spaces determined for the respective ones of the plurality of workloads;”
Examiner responds by explaining that these features are taught by the previously cited Sekanina (“Neural Architecture Search and Hardware Accelerator Co-Search: A Survey”) in view of new reference Network Space Search for Pareto-Efficient Spaces (Hereinafter Min-Fong)
In particular, Sekanina makes obvious simulate an execution of the first hardware architectural configuration for a plurality of workloads to generate performance metrics for the plurality of workloads based on a plurality of objectives; (Pg. 151348, Left column: “Typical objectives to be optimized for a given hardware platform are latency, throughput, energy efficiency, and memory usage.”) (Pg. 151354, Left column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105]. In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”) (Pg. 151350, Left column: “Latency can be estimated using: a surrogate model, e.g., [6], [126], [128]-[131]; a suitable hardware simulator executing a candidate CNN, e.g. [51], [100], [132]-[135]; a formula or model derived after analyzing the search space of possible CNN architectures, e.g., [14], [47], [49], [101], [136]-[138]; a LUT-based model [105], [139], [140].”) (Pg. 151355, Right column: “Focusing on Google's EdgeTPU, NAHAS [129] performs a joint search in the space of CNN architectures and hardware accelerator configurations, where hardware resources and latency are constraints. The architecture search space is based on a new fused inverted bottleneck layer with tunable parameters. The accelerator search space is defined by seven parameters (e.g., the PE array size, the number of SIMD units, register file capacity). An in-house simulator is used to estimate latency and other hardware-related parameters.”);
identify, for respective ones of the plurality of workloads, a Pareto front from the performance metrics for the respective workload; (Pg. 151354, Right Column – Page 15135, Left Column: “In Codesign-NAS [139], RL controller selects a CNN architecture from a CNN search space and a hardware architecture from an accelerator design space. Both are sent to the evaluator that implements the CNN on the proposed accelerator to find accuracy and efficiency metrics, such as latency, area, and power (based on pre-computed models). The authors enumerated 4 billion model-accelerator pairs to study the Pareto-front in a representative co-design search space. They proposed three different search strategies to navigate the co-design search space under one or two constraints. The CNN search space is based on NASBench [174] and the accelerator design space utilizes CHaiDNN — a library for the acceleration of CNNs on System-on chip FPGAs.”)
It should be noted that, contrary to the applicant’s arguments, claims 1 and 11 do not actually require comparison between different pareto fronts, and therefore whether or not this passage is directed to comparing competing NAS methods, to which the examiner does not concede, it still reads on the claim language.
(Pg. 151349, Left Column: “As it is not known ‘‘how far’’ the obtained solutions are from the truly Pareto optimal solutions, a common practice is to introduce a quality metric capable of measuring the distance between two sets of solutions obtained with two multi-objective optimization methods (see, for example, [122]) and compare them under this metric. For example, NSGANetV1 [94] employs the hypervolume performance metric, which calculates the dominated area (hypervolume, in the general case) from the set of solutions to a reference point which is usually an estimate of the nadir point– a vector concatenating worst objective values of the Pareto front.”)
Min-Fong makes obvious determine, for the respective ones of the plurality of workloads, a respective objective design space based on a region of an objective space enclosed by the Pareto front for the respective workload (Fig. 1: “Network Space Search (NSS) and Elite Spaces. We propose Network Space Search, a new paradigm for Auto-ML aiming at no manual effort, to automatically search for Pareto-efficient network spaces in Expanded Search Space, where minimal human expertise is imposed. The discovered network spaces, named Elite Spaces, are sustainably aligned with the Pareto front across several complexity constraints. Moreover, Elite Spaces can be further utilized for designing satisfactory networks and served as NAS search spaces to find satisfactory architectures.”) (Pg. 2, Left column: “Moreover, Elite Spaces can be further served as NAS search spaces for benefiting current NAS approaches to improve performance (e.g. In CIFAR-100, an averagely 2.3% lower error rate and 3.7% closer to target complexity than the baseline with around 90% fewer samples required to find satisfactory networks).”) (Pg. 4, Right column: “After searching by optimizing Eq. 5, we can obtain the network spaces satisfying multi-objectives, where the searched spaces A∗ are named Elite Spaces. More concretely, Elite Spaces are derived from the optimized probability distribution PΘ after the searching process. We sample n spaces from PΘ and identify the one closest to the FLOPs constraint as our Elite Space. Unlike hand crafted network spaces or commonly adopted NAS search spaces [21, 27], Elite Spaces are obtained without prior knowledge or human expertise on network designs. Moreover, Elite Spaces can be further exploited in designing promising networks and served as search spaces, benefiting NAS approaches to improve performance”) (Pg. 6, Right column: “constraints in both CIFAR-10 and CIFAR-100 datasets. Elite Spaces achieve satisfactory trade-offs between the error rates and meeting the FLOPs constraints, and are aligned with the Pareto front of Expanded Search Space. Since Elite Spaces discovered by our NSS method are guaranteed to consist of superior networks provided in various FLOPs regimes, they can be utilized for designing promising networks. More importantly, Elite Spaces are searched by NSS automatically, therefore the human effort involved in network designs is significantly reduced”)
Further, in regard to claim 21 in particular:
In particular, Sekanina makes obvious simulating, by executing the instructions with the at least one processor, an execution of the first hardware architectural configuration for first workloads and second workloads to generate performance metrics based on one or more performance objectives, the first workloads including a first one of the first workloads and variations of the first one of the first workloads, the second workloads including a first one of the second workloads and variations of the first one of the second workloads, (Pg. 151348, Left column: “Typical objectives to be optimized for a given hardware platform are latency, throughput, energy efficiency, and memory usage. Hence, in specific steps of the NAS algorithm, all relevant objectives have to be evaluated, either by direct measurement on real hardware or estimated using software models (see Fig. 6).”) (Pg. 151354, Left Column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105]. In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”) (Pg. 151350, Left column: “Latency can be estimated using: a surrogate model, e.g., [6], [126], [128]-[131]; a suitable hardware simulator executing a candidate CNN, e.g. [51], [100], [132]-[135]; a formula or model derived after analyzing the search space of possible CNN architectures, e.g., [14], [47], [49], [101], [136]-[138]; a LUT-based model [105], [139], [140].”) (Pg. 151355, Right column: “Focusing on Google’s EdgeTPU, NAHAS [129] performs a joint search in the space of CNN architectures and hard ware accelerator configurations, where hardware resources and latency are constraints. The architecture search space is based on a new fused inverted bottleneck layer with tunable parameters. The accelerator search space is defined by seven parameters (e.g., the PE array size, the number of SIMD units, register file capacity). An in-house simulator is used to estimate latency and other hardware-related parameters.”) (Table 3: Shows a comparison of various different models, each model corresponds to a workload. Note that several are variations of each other)
identifying, by executing the instructions with the at least one processor, a first Pareto front from the performance metrics for the first workloads and a second Pareto front from the performance metrics for the second workloads; (Pg. 151349, Right column: “Truly multi-objective optimization methods iteratively build the Pareto front in the course of optimization by com paring candidate solutions using the non-dominance relation, promoting reasonable solutions, and trying to cover the expected Pareto front. This approach has significantly been developed within the evolutionary computation community. One of the most popular methods is NSGA-II [124]. It is based on sorting individuals in a population according to the dominance relation into multiple fronts. The first front contains all non-dominated solutions. Each subsequent front is constructed by removing all the preceding fronts from the population and finding a new Pareto front in the remaining individuals. The solutions within the individual fronts are then sorted according to the crowding distance metric. This metric helps to preserve the diversity of the population along the fronts. Best individuals then serve as parents for the new population. NSGA-II thus always produces a set of non-dominated solutions (i.e., a Pareto front) when it is terminated.”) (Table 3: Shows a comparison of various different models, each model corresponds to a workload. Note that several are variations of each other)
Further, this iterative approach to optimization and pareto front development generates and compares multiple pareto fronts on a per-model basis; although comparison of different NAS methods is discussed elsewhere in the reference, this clearly describes the comparison of the models themselves.
(Pg. 151349, Right column: “Truly multi-objective optimization methods iteratively build the Pareto front in the course of optimization by com paring candidate solutions using the non-dominance relation, promoting reasonable solutions, and trying to cover the expected Pareto front. This approach has significantly been developed within the evolutionary computation community. One of the most popular methods is NSGA-II [124]. It is based on sorting individuals in a population according to the dominance relation into multiple fronts. The first front contains all non-dominated solutions. Each subsequent front is constructed by removing all the preceding fronts from the population and finding a new Pareto front in the remaining individuals. The solutions within the individual fronts are then sorted according to the crowding distance metric. This metric helps to preserve the diversity of the population along the fronts. Best individuals then serve as parents for the new population. NSGA-II thus always produces a set of non-dominated solutions (i.e., a Pareto front) when it is terminated.”) (Pg. 151349, Left column: “As it is not known ‘‘how far’’ the obtained solutions are from the truly Pareto optimal solutions, a common practice is to introduce a quality metric capable of measuring the distance between two sets of solutions obtained with two multi-objective optimization methods (see, for example, [122]) and compare them under this metric. For example, NSGANetV1 [94] employs the hypervolume performance metric, which calculates the dominated area (hypervolume, in the general case) from the set of solutions to a reference point which is usually an estimate of the nadir point– a vector concatenating worst objective values of the Pareto front.” [Examiner’s note: as the reference point is based on the values of the pareto front, generating a new pareto front in place of an old one also means the reference point is updated to a new reference point])
Min-Fong makes obvious determining, by executing the instructions with the at least one processor, a first objective design space based on a region of an objective space enclosed by the pareto front and a second objective design space based on a region of the objective space enclosed by other data (Fig. 1: “Network Space Search (NSS) and Elite Spaces. We propose Network Space Search, a new paradigm for Auto-ML aiming at no manual effort, to automatically search for Pareto-efficient network spaces in Expanded Search Space, where minimal human expertise is imposed. The discovered network spaces, named Elite Spaces, are sustainably aligned with the Pareto front across several complexity constraints. Moreover, Elite Spaces can be further utilized for designing satisfactory networks and served as NAS search spaces to find satisfactory architectures.”) (Pg. 2, Left column: “Moreover, Elite Spaces can be further served as NAS search spaces for benefiting current NAS approaches to improve performance (e.g. In CIFAR-100, an averagely 2.3% lower error rate and 3.7% closer to target complexity than the baseline with around 90% fewer samples required to find satisfactory networks).”) (Pg. 4, Right column: “After searching by optimizing Eq. 5, we can obtain the network spaces satisfying multi-objectives, where the searched spaces A∗ are named Elite Spaces. More concretely, Elite Spaces are derived from the optimized probability distribution PΘ after the searching process. We sample n spaces from PΘ and identify the one closest to the FLOPs constraint as our Elite Space. Unlike hand crafted network spaces or commonly adopted NAS search spaces [21, 27], Elite Spaces are obtained without prior knowledge or human expertise on network designs. Moreover, Elite Spaces can be further exploited in designing promising networks and served as search spaces, benefiting NAS approaches to improve performance”) (Pg 6, Right column: “constraints in both CIFAR-10 and CIFAR-100 datasets. Elite Spaces achieve satisfactory trade-offs between the error rates and meeting the FLOPs constraints, and are aligned with the Pareto front of Expanded Search Space. Since Elite Spaces discovered by our NSS method are guaranteed to consist of superior networks provided in various FLOPs regimes, they can be utilized for designing promising networks. More importantly, Elite Spaces are searched by NSS automatically, therefore the human effort involved in network designs is significantly reduced” [Examiner’s note: the reference point is a point that defines the outer edge of the design space. As the elite space and expanded search space have different outer edges, they clearly also have different reference points bounding that space])
PNG
media_image1.png
405
662
media_image1.png
Greyscale
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-25 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract ideas without significantly more.
Step 1: Claims 1-10 are directed to an apparatus, which is a machine, falling under a statutory category of invention. Claims 11-20 are directed to a non-transitory machine readable storage medium, which is a manufacture, falling under a statutory category of invention. Claims 21-25 are directed to a method, which is a process, falling under a statutory category of invention. Therefore, claims 1-25 are directed to patent eligible categories of invention.
Regarding claim 1:
Step 2A Prong 1: The following limitations recite abstract ideas:
The limitation “determine a first hardware architectural configuration of a hardware component based on a design constraint” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. For example, this covers a person observing the design constraint and making a mental judgment on a hardware architectural configuration based on the design constraint.
The limitation “identify, for respective ones of the plurality of workloads, a Pareto front from the performance metrics for the respective workload;” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. Identifying a Pareto front from the performance metrics is a mental process equivalent to determining the configurations that best optimize these performance metrics on a drawn graph. For example, if the compared metrics are latency and accuracy, determining the Pareto front amounts to identifying the points corresponding to configurations that, for a particular latency, maximize accuracy. Determining the pareto front from these data points consists of merely determining a curve or piecewise function that approximates these points.
The limitation “determine, for the respective ones of the plurality of workloads, a respective objective design space based on a region of an objective space enclosed by the Pareto front for the respective workload and a reference point;” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. Given the identified Pareto front and the graph it was drawn on, the graph itself representing the entire possible design space, as well as a reference point, determining the respective objective design space is a mental process equivalent to limiting further considered configurations to those that correspond to points that would be enclosed within the Pareto front relative to the reference point. As this is merely the recognition of certain areas of a graph bounded by certain points, this is merely a mental process.
The limitation “generate an aggregate score by aggregating a plurality of design space performance indicators, ones of the plurality of design space performance indicators corresponding to respective ones of the respective plurality of objective design spaces determined for the respective ones of the plurality of workloads;” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. For example, this covers a person mentally observing the plurality of performance indicators, aggregate them mentally or with a pen and paper, and making a mental judgment on the score.
This also amounts to a mathematical concept in view of the following paragraphs from specification:
[0030]: “Based on a Pareto front of the design space(s) and one or more reference points, certain examples generate a performance indicator (e.g., metric) representing a multidimensional objective space that includes a set of solutions representing a tradeoff among the different objectives. The performance indicator can be, for example, a hypervolume indicator, an R2 indicator, etc. In examples in which two objectives are measured for each workload and/or modality of interest, the performance indicator may represent an area of an Pareto front.”
[0060]: “Examples discussed below assume that the design space performance indicator is a hypervolume indicator. However, the design space performance indicator can be an R2 indicator, a variance metric, and/or another goodness metric that considers a distribution of vectors in additional or alternative examples. In some examples, the design space performance indicator is a fitness function. When measuring two objectives (e.g., latency versus accuracy, etc.), a hypervolume represents an area of a Pareto front. For example, in a case of measuring latency versus accuracy objectives, the larger the hypervolume indicator value, the better the performance of the hardware architectural configuration.”
[0064]: “The graph 400 also includes an example reference point(s) 412 and an example hypervolume 414. The hypervolume 414 is a unary performance indicator that provides a measure to quantify a quality of the Pareto front 408. The hypervolume 414 is a measure of a size of an objective design space (e.g., an area in a bi-objective setting, a volume in a many-objective setting, etc.) enclosed by solutions on the Pareto front 408 and the reference point 412. That is, the reference point 412 is needed to calculate and define the hypervolume 414. In some examples, the hypervolume 414 for the Pareto front 408 is determined as an objective space that is dominated by the Pareto front 408 relative to some reference point 412.”
The limitation “search a design database based on the aggregate score to identify a second hardware architectural configuration” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. For example, this covers a person mentally observing the database and making a mental judgment on the appropriate second hardware architectural configuration.
This also amounts to a mathematical concept in view of the following paragraphs from specification:
[0028]: “Disclosed examples also utilize neural architecture search (NAS) techniques such as, but not limited to, evolutionary search, Bayesian optimization, and/or reinforcement learning to explore a search space of neural network architectures based on a workload to identify variations of the workload (e.g., sub-models, etc.).”
[0067]: “For example, the searcher circuitry 222 (discuss in detail below) may apply a search algorithm that uses a relative ranking of different scores (e.g., 0-1000, 0-1, etc.)”
Examiner notes that such optimization algorithms or search algorithms involve mathematical relationships, calculations, and/or equations/formulas.
The limitation “predict a performance of the second hardware architectural configuration to generate a performance metric by executing a proxy function corresponding to the second hardware architectural configuration” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. For example, this covers a person evaluating or observing a proxy function or an estimation function mentally or with a pen and paper and then making a mental judgment on the performance based on it.
This also amounts to a mathematical concept in view of the following paragraphs from specification:
[0034]: “Certain examples mitigate the high costs associated with measuring (e.g., evaluating) performance of a hardware architectural configuration by generating (e.g., building) an example low-fidelity performance estimator that leverages structural information (e.g., graph) about the hardware architectural configurations to build a machine learning (ML) based proxy function. … Thus, certain examples reduce design overhead associated with hardware accelerator design by utilizing a computationally cheaper (e.g., compared to a simulation) proxy function (e.g., using reinforcement learning, Bayesian optimization, etc.) to estimate a performance of the hardware architectural configuration typically determined during a simulation.”
[0076]: “The estimation function(s) 232 as executed by the performance estimator circuitry 230 of FIG. 2 applies a graph-based prediction algorithm to generate an example score(s) 234. For example, the estimation function 232 can generate a graph representation of a PE array considering that most hardware architectures (e.g., from a circuit design perspective) can be represented as a graph. In some examples, another general regressor (e.g., linear ridge, graph neural network regressor, etc.) can be used additionally or alternatively. … In some examples, the estimation function 232 represents a hardware architectural configuration as a vector.”
Step 2A Prong 2: The following limitations recite additional elements:
“at least one memory”
“machine readable instructions”
“processor circuitry to at least one of instantiate or execute the machine readable instructions”
“simulate an execution of the first hardware architectural configuration for a plurality of workloads to generate performance metrics for the plurality of workloads based on a plurality of objectives;”
These additional elements do not integrate the judicial exception into a practical application because they amount to no more than mere instructions to apply the judicial exception using a generic computer. A memory, machine readable instructions, and a processor are generic computer components. According to specification para [0029] and [0049]-[0050], simulating the execution of the first hardware architectural configuration for a plurality of workloads amounts to executing computer instructions using a processor. This amounts to using a generic computer. See MPEP 2106.05(f).
Even when viewed in combination, these additional elements do not integrate the judicial exception into a practical application.
Accordingly, the claim does not recite any additional elements that integrate the judicial exception into a practical application.
Step 2B: As previously discussed, the additional elements amount to no more than mere instructions to apply the exception using a generic computer. Such activities do not amount to significantly more than the judicial exception. See MPEP 2106.05(f).
Therefore, claim 1 is not eligible.
Regarding claim 2: The limitation “wherein the first hardware architectural configuration is a baseline design, and the second hardware architectural configuration is an adjusted first hardware architectural configuration” merely further limits the first hardware architectural configuration and the second hardware architectural configuration recited in claim 1. Therefore, the same analysis is applicable. Furthermore, adjusting the hardware architectural configuration amounts to a mental process as a person can mentally make a judgment on the appropriate modification to the configuration.
Regarding claim 3: The limitation “wherein the plurality of design space performance indicators are hypervolume indicators” merely further limits the plurality of design space performance indicators recited in claim 1. Therefore, the same analysis is applicable. Furthermore, hypervolume indicators amounts to a mathematical concept as explained in claim 1.
Regarding claim 4: The limitation “wherein the plurality of design space performance indicators are weighted, and wherein the aggregate score is a weighted aggregate score” merely further limits the plurality of design space performance indicators recited in claim 1. Therefore, the same analysis is applicable. Furthermore, weighting scores amounts to a mental process as a person can perform such weighting mentally or with a pen and paper. It also mounts to a mathematical concept as weighting values involves mathematical relationships, calculations, or formulas/equations.
Regarding claim 5: The limitation “wherein a first workload of the plurality of workloads is a deep neural network architecture” merely further limits the plurality of workloads recited in claim 1. Therefore, the same analysis is applicable.
This also amounts to generally linking the use of a judicial exception to a particular technological environment or field of use of merely using a neural network as one of the architectures. Such limitations do not integrate the judicial exception into a practical application or amount to significantly more than the judicial exception. See MPEP 2106.05(h).
Regarding claim 6: The limitation “wherein the plurality of workloads correspond to one or more modalities” merely further limits the plurality of workloads recited in claim 1. Therefore, the same analysis is applicable.
Regarding claim 7: The limitation “wherein the plurality of objectives include at least one of minimize latency, maximize efficiency, or maximize accuracy” merely further limits the one or more objectives recited in claim 1. Therefore, the same analysis is applicable.
Regarding claim 8: The limitation “wherein executing the proxy function is utilizes fewer compute resources as compared to the simulation of the execution of the first hardware architectural configuration” merely further limits the proxy function recited in claim 1. Therefore, the same analysis is applicable.
Regarding claim 9:
The limitation “wherein the performance metric is a first performance metric” merely further limits the performance metric recited in claim 1. Therefore, the same analysis is applicable.
The limitation “wherein the processor circuitry at least one of instantiates or executes the machine readable instructions to: execute a search of the design space for the hardware component based on the first performance metric to determine a third hardware architectural configuration” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. For example, this covers a person mentally observing the first performance metric and making a judgment on the appropriate third hardware architectural configuration.
This also amounts to a mathematical concept in view of the following paragraphs from specification:
[0028]: “Disclosed examples also utilize neural architecture search (NAS) techniques such as, but not limited to, evolutionary search, Bayesian optimization, and/or reinforcement learning to explore a search space of neural network architectures based on a workload to identify variations of the workload (e.g., sub-models, etc.).”
[0067]: “For example, the searcher circuitry 222 (discuss in detail below) may apply a search algorithm that uses a relative ranking of different scores (e.g., 0-1000, 0-1, etc.)”
Examiner notes that such optimization algorithms or search algorithms involve mathematical relationships, calculations, and/or equations/formulas.
The limitation “predict a performance of the third hardware architectural configuration to generate a second performance metric by executing a proxy function corresponding to the third hardware architectural configuration” amounts to a mental process and a mathematical concepts for the similar reason as explained in claim 1.
Regarding claim 10:
The limitation “wherein, in response to the second performance metric exceeding a defined value, the processor circuitry least one of instantiates or executes the machine readable instructions to: simulate an execution of the third hardware architectural configuration for a plurality of workloads to generate a respective plurality of objective design spaces, the objective design spaces based on one or more objectives” amounts to mere instructions to apply the judicial exception using a generic computer as simulating an execution of a architectural configuration amounts to mere instructions to apply the judicial exception using a generic computer in view of para [0029] and [0049]-[0050] as explained in claim 1. See MPEP 2106.05(f).
The limitation “generate a design score for the third hardware architectural configuration by aggregating a plurality of design space performance indicators, the plurality of design space performance indicators to correspond to the plurality of objective design spaces” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. For example, a person can aggregate or combine scores mentally or with a pen and paper. It also mounts to a mathematical concept as aggregating values involves mathematical relationships, calculations, or formulas/equations.
The limitation “execute another search of the design space for the hardware component based on the design score for the third hardware architectural configuration to determine a fourth hardware architectural configuration” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. For example, this covers a person mentally observing the design score and making a judgment on the appropriate fourth hardware architectural configuration.
This also amounts to a mathematical concept in view of the following paragraphs from specification:
[0028]: “Disclosed examples also utilize neural architecture search (NAS) techniques such as, but not limited to, evolutionary search, Bayesian optimization, and/or reinforcement learning to explore a search space of neural network architectures based on a workload to identify variations of the workload (e.g., sub-models, etc.).”
[0067]: “For example, the searcher circuitry 222 (discuss in detail below) may apply a search algorithm that uses a relative ranking of different scores (e.g., 0-1000, 0-1, etc.)”
Examiner notes that such optimization algorithms or search algorithms involve mathematical relationships, calculations, and/or equations/formulas.
Regarding claim 11: Claim 11 is substantially similar to claim 1. Therefore, the similar analysis is applicable.
Furthermore, the limitation “A non-transitory machine readable storage medium comprising instructions that, when executed, cause processor circuitry to at least” does not integrate the judicial exception into a practical application or amount to significantly more than the judicial exception because it amounts to mere instructions to apply the judicial exception using a generic computer. See MPEP 2106.05(f).
Regarding claim 21:
Step 2A Prong 1: The following limitations recite abstract ideas:
The limitation “determining, by executing instructions with at least one processor, a first hardware architectural configuration of a hardware component based on a design constraint;” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. For example, this covers a person observing the design constraint and making a mental judgment on a hardware architectural configuration based on the design constraint.
The limitation “identifying, by executing the instructions with the at least one processor, a first Pareto front from the performance metrics for the first workloads and a second Pareto front from the performance metrics for the second workloads;” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. Identifying Pareto fronts from the performance metrics is a mental process equivalent to determining the configurations that best optimize these performance metrics on a drawn graph. For example, if the compared metrics are latency and accuracy, determining the Pareto fronts amounts to identifying the points corresponding to configurations that, for a particular latency, maximize accuracy. Determining the pareto fronts from these data points consists of merely determining a curve or piecewise function that approximates these points.
The limitation “determining, by executing the instructions with the at least one processor, a first objective design space based on a region of an objective space enclosed by the first Pareto front and a first reference point, and a second objective design space based on a region of the objective space enclosed by the second Pareto front and a second reference point;” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. Given the identified Pareto fronts and the graphs they were drawn on, the graphs themselves representing each entire possible design space, as well as the reference points, determining the respective objective design spaces is a mental process equivalent to limiting further considered configurations to those that correspond to points that would be enclosed within the Pareto fronts relative to the respective reference points. As this is merely the recognition of certain areas of graphs bounded by certain points, this is merely a mental process.
The limitation “generating, by executing the instructions with the at least one processor, an aggregate score by combining a plurality of design space performance indicators, ones of the plurality of design space performance indicators corresponding to the first objective design space and the second objective design space” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. For example, this covers a person mentally observing the plurality of performance indicators, aggregate them mentally or with a pen and paper, and making a mental judgment on the score.
This also amounts to a mathematical concept in view of the following paragraphs from specification:
[0030]: “Based on a Pareto front of the design space(s) and one or more reference points, certain examples generate a performance indicator (e.g., metric) representing a multidimensional objective space that includes a set of solutions representing a tradeoff among the different objectives. The performance indicator can be, for example, a hypervolume indicator, an R2 indicator, etc. In examples in which two objectives are measured for each workload and/or modality of interest, the performance indicator may represent an area of an Pareto front.”
[0060]: “Examples discussed below assume that the design space performance indicator is a hypervolume indicator. However, the design space performance indicator can be an R2 indicator, a variance metric, and/or another goodness metric that considers a distribution of vectors in additional or alternative examples. In some examples, the design space performance indicator is a fitness function. When measuring two objectives (e.g., latency versus accuracy, etc.), a hypervolume represents an area of a Pareto front. For example, in a case of measuring latency versus accuracy objectives, the larger the hypervolume indicator value, the better the performance of the hardware architectural configuration.”
[0064]: “The graph 400 also includes an example reference point(s) 412 and an example hypervolume 414. The hypervolume 414 is a unary performance indicator that provides a measure to quantify a quality of the Pareto front 408. The hypervolume 414 is a measure of a size of an objective design space (e.g., an area in a bi-objective setting, a volume in a many-objective setting, etc.) enclosed by solutions on the Pareto front 408 and the reference point 412. That is, the reference point 412 is needed to calculate and define the hypervolume 414. In some examples, the hypervolume 414 for the Pareto front 408 is determined as an objective space that is dominated by the Pareto front 408 relative to some reference point 412.”
The limitation “searching, by executing the instructions with the at least one processor, a design datastore based on the aggregate score to identify a second hardware architectural configuration; and” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. For example, this covers a person mentally observing the database and making a mental judgment on the appropriate second hardware architectural configuration.
This also amounts to a mathematical concept in view of the following paragraphs from specification:
[0028]: “Disclosed examples also utilize neural architecture search (NAS) techniques such as, but not limited to, evolutionary search, Bayesian optimization, and/or reinforcement learning to explore a search space of neural network architectures based on a workload to identify variations of the workload (e.g., sub-models, etc.).”
[0067]: “For example, the searcher circuitry 222 (discuss in detail below) may apply a search algorithm that uses a relative ranking of different scores (e.g., 0-1000, 0-1, etc.)”
Examiner notes that such optimization algorithms or search algorithms involve mathematical relationships, calculations, and/or equations/formulas.
The limitation “estimating, by executing the instructions with the at least one processor, a performance of the second hardware architectural configuration to generate a performance score by executing an estimation function corresponding to the second hardware architectural configuration.” under broadest reasonable interpretation covers a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, but for the recitation of a computer. For example, this covers a person evaluating or observing an estimation function mentally or with a pen and paper and then making a mental judgment on the performance based on it.
This also amounts to a mathematical concept in view of the following paragraphs from specification:
[0034]: “Certain examples mitigate the high costs associated with measuring (e.g., evaluating) performance of a hardware architectural configuration by generating (e.g., building) an example low-fidelity performance estimator that leverages structural information (e.g., graph) about the hardware architectural configurations to build a machine learning (ML) based proxy function. … Thus, certain examples reduce design overhead associated with hardware accelerator design by utilizing a computationally cheaper (e.g., compared to a simulation) proxy function (e.g., using reinforcement learning, Bayesian optimization, etc.) to estimate a performance of the hardware architectural configuration typically determined during a simulation.”
[0076]: “The estimation function(s) 232 as executed by the performance estimator circuitry 230 of FIG. 2 applies a graph-based prediction algorithm to generate an example score(s) 234. For example, the estimation function 232 can generate a graph representation of a PE array considering that most hardware architectures (e.g., from a circuit design perspective) can be represented as a graph. In some examples, another general regressor (e.g., linear ridge, graph neural network regressor, etc.) can be used additionally or alternatively. … In some examples, the estimation function 232 represents a hardware architectural configuration as a vector.”
Step 2A Prong 2: The following limitations recite additional elements:
“executing instructions with at least one processor”
“simulating, by executing the instructions with the at least one processor, an execution of the first hardware architectural configuration for first workloads and second workloads to generate performance metrics based on one or more performance objectives, the first workloads including a first one of the first workloads and variations of the first one of the first workloads, the second workloads including a first one of the second workloads and variations of the first one of the second workloads,”
These additional elements do not integrate the judicial exception into a practical application because they amount to no more than mere instructions to apply the judicial exception using a generic computer. A memory, machine readable instructions, and a processor are generic computer components. According to specification para [0029] and [0049]-[0050], simulating the execution of the first hardware architectural configuration for a plurality of workloads amounts to executing computer instructions using a processor. This amounts to using a generic computer. See MPEP 2106.05(f).
Even when viewed in combination, these additional elements do not integrate the judicial exception into a practical application.
Accordingly, the claim does not recite any additional elements that integrate the judicial exception into a practical application.
Step 2B: As previously discussed, the additional elements amount to no more than mere instructions to apply the exception using a generic computer. Such activities do not amount to significantly more than the judicial exception. See MPEP 2106.05(f).
Therefore, claim 21 is not eligible.
Claims 12-20 and 22-25 are substantially similar to claims 1-10. Therefore, the similar analysis is applicable.
Accordingly, claims 1-25 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e. an abstract idea) without anything significantly more.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
(1) Claims 1-25 are rejected under 35 U.S.C. 103 as being unpatentable over Neural Architecture Search and Hardware Accelerator Co-Search: A Survey (Hereinafter Sekanina) in view of Network Space Search for Pareto-Efficient Spaces (Hereinafter Min-Fong)
Regarding claim 1, Sekanina discloses at least one memory; machine readable instructions; and processor circuitry to at least one of instantiate or execute the machine readable instructions ([Pg. 151350, Left column: “Latency can be estimated using: a surrogate model, e.g., [6], [126], [128]-[131]; a suitable hardware simulator executing a candidate CNN, e.g. [51], [100], [132]-[135]; a formula or model derived after analyzing the search space of possible CNN architectures, e.g., [14], [47], [49], [101], [136]-[138]; a LUT-based model [105], [139], [140].”) (Pg. 151355, Right column: “Focusing on Google's EdgeTPU, NAHAS [129] performs a joint search in the space of CNN architectures and hardware accelerator configurations, where hardware resources and latency are constraints.”) (Pg. 151355, Right column: “Focusing on Google's EdgeTPU, NAHAS [129] performs a joint search in the space of CNN architectures and hardware accelerator configurations, where hardware resources and latency are constraints. … An in-house simulator is used to estimate latency and other hardware-related parameters.” [Examiner’s note: such hardware simulators are performed on a computer including a processor, a memory, and machine readable instructions.])
determine a first hardware architectural configuration of a hardware component based on a design constraint (Pg. 151354, Left column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105]. In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”) (Pg. 151353, Right column: “In another co-search strategy used by, e.g., in [47], [51], the hardware optimization algorithm receives a CNN as the input and optimizes the hardware accelerator concerning desired objectives (Fig. 13).”) (Pg. 151355, Left column: “An RL controller generates parameters describing the CNN architecture (based on ProxylessNAS) as well as the hardware parameters such as the number of PEs, buffer size, and dataflow pattern. … DANCE thus introduces a novel differentiable evaluator, which takes the architecture parameters from the RL controller, searches for the optimal hardware accelerator design, and evaluates its cost metrics.”) (Pg. 151343, Right column: “Using cost models (such as MAESTRO [71]), the run time, resources utilization, delay, and power can be estimated for a given accelerator, DNN, and data set.”);
simulate an execution of the first hardware architectural configuration for a plurality of workloads to generate performance metrics for the plurality of workloads based on a plurality of objectives; (Pg. 151348, Left column: “Typical objectives to be optimized for a given hardware platform are latency, throughput, energy efficiency, and memory usage.”) (Pg. 151354, Left column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105]. In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”) (Pg. 151350, Left column: “Latency can be estimated using: a surrogate model, e.g., [6], [126], [128]-[131]; a suitable hardware simulator executing a candidate CNN, e.g. [51], [100], [132]-[135]; a formula or model derived after analyzing the search space of possible CNN architectures, e.g., [14], [47], [49], [101], [136]-[138]; a LUT-based model [105], [139], [140].”) (Pg. 151355, Right column: “Focusing on Google's EdgeTPU, NAHAS [129] performs a joint search in the space of CNN architectures and hardware accelerator configurations, where hardware resources and latency are constraints. The architecture search space is based on a new fused inverted bottleneck layer with tunable parameters. The accelerator search space is defined by seven parameters (e.g., the PE array size, the number of SIMD units, register file capacity). An in-house simulator is used to estimate latency and other hardware-related parameters.”);
identify, for respective ones of the plurality of workloads, a Pareto front from the performance metrics for the respective workload; (Pg. 151354, Right Column – Page 15135, Left Column: “In Codesign-NAS [139], RL controller selects a CNN architecture from a CNN search space and a hardware archi tecture from an accelerator design space. Both are sent to the evaluator that implements the CNN on the proposed accelerator to find accuracy and efficiency metrics, such as latency, area, and power (based on pre-computed models). The authors enumerated 4 billion model-accelerator pairs to study the Pareto-front in a representative co-design search space. They proposed three different search strategies to navigate the co-design search space under one or two constraints. The CNN search space is based on NASBench [174] and the accelerator design space utilizes CHaiDNN — a library for the acceleration of CNNs on System-on chip FPGAs.”)
(Pg. 151349, Left Column: “As it is not known ‘‘how far’’ the obtained solutions are from the truly Pareto optimal solutions, a common practice is to introduce a quality metric capable of measuring the distance between two sets of solutions obtained with two multi-objective optimization methods (see, for example, [122]) and compare them under this metric. For example, NSGANetV1 [94] employs the hypervolume performance metric, which calculates the dominated area (hypervolume, in the general case) from the set of solutions to a reference point which is usually an estimate of the nadir point– a vector concatenating worst objective values of the Pareto front.”)
generate an aggregate score by aggregating a plurality of design space performance indicators, ones of the plurality of design space performance indicators corresponding to respective ones of the respective plurality of objective design spaces determined for the respective ones of the plurality of workloads (Pg. 151353, Left column: “The Multi-objective strategy is based either on a Pareto front construction method (`Pareto'), aggregation method (`Agg'), or applying some constraints (`Constr').”) (Pg. 151349, Left column: “A common approach to solve the multi-objective NAS problem adopted by the NAS community is either (i) to transform it into a single-objective one (using suitable constraints, prioritization, or aggregation techniques)”) (Pg. 151349, Right column: “The aggregation methods introduce a suitable aggregation function (such as the weighted sum, weighted exponential sum, or weighted product) for the objective functions and optimize the composition.”);
search a design database based on the aggregate score to identify a second hardware architectural configuration (Pg. 151353, Left column: “In addition to the architecture search space and parameter search space (weights), there is an additional search space, called the hardware search space, containing all possible hardware configurations. … The space of hardware configurations can be searched together with the space of DNN architectures using the same search algorithm … In the Multi-objective: strategy column, the `co-search' means that there are two independent search algorithms, i.e., a co-search is conducted; one search algorithm operates in the network architecture space and the other in the hardware search space. The Multi-objective strategy is based either on a Pareto front construction method (`Pareto'), aggregation method (`Agg'), or applying some constraints (`Constr').”) (Pg. 151353, Right column: “A straightforward approach to organizing the co-search is to generate a CNN-accelerator pair, which is evaluated by training the network to obtain its accuracy and measuring the hardware parameters (Fig. 12). Based on this evaluation, the next candidate pairs are generated until the desired solution is not obtained.”) (Pg. 151354, Left column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105]. In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”) (Figure 13 shows the iterative search procedure for identifying a hardware architectural configuration.) (Pg. 151354, Right column: “In AutoDNN [14], each candidate CNN consists of several hardware-aware parameterizable cells called Bundles. By means of these cells, specialized software can map any candidate CNN generated by NAS to an FPGA accelerator based on a fine-grained tile-based pipeline architecture whose components are pre-designed and stored in a component library. Latency and resources are estimated and used back in theNAS algorithm.”) (Pg. 151349, Right column: “The aggregation methods introduce a suitable aggregation function (such as the weighted sum, weighted exponential sum, or weighted product) for the objective functions and optimize the composition.”) (Figure 13 shows the iterative search procedure for identifying a hardware architectural configuration.) (Pg. 151354, Left column: “In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS.”); and
predict a performance of the second hardware architectural configuration to generate a performance metric by executing a proxy function corresponding to the second hardware architectural configuration (Pg. 151343, Right column: “Using cost models (such as MAESTRO [71]), the run time, resources utilization, delay, and power can be estimated for a given accelerator, DNN, and data set.”) (Pg. 151350, Left column: “Latency can be estimated using: a surrogate model, e.g., [6], [126], [128]-[131]; a suitable hardware simulator executing a candidate CNN, e.g. [51], [100], [132]-[135]; a formula or model derived after analyzing the search space of possible CNN architectures, e.g., [14], [47], [49], [101], [136]-[138]; a LUT-based model [105], [139], [140].”) (Pg. 151353, Right column: “In another co-search strategy used by, e.g., in [47], [51], the hardware optimization algorithm receives a CNN as the input and optimizes the hardware accelerator concerning desired objectives (Fig. 13).”) (Pg. 151354, Right column: “The RL controller samples parameters of a candidate CNN architecture and its possible quantization. For the sampled network, the hardware builder searches the hardware space to find a suitable hardware model. Each candidate hardware model is validated against the specification (latency constraint) during the search, and the result is sent back to the controller. … In Codesign-NAS [139], RL controller selects a CNN architecture from a CNN search space and a hardware architecture from an accelerator design space. Both are sent to the evaluator that implements the CNN on the proposed accelerator to find accuracy and efficiency metrics, such as latency, area, and power (based on pre-computed models).”) (Pg. 151355, Left column: “An RL controller generates parameters describing the CNN architecture (based on ProxylessNAS) as well as the hardware parameters such as the number of PEs, buffer size, and dataflow pattern. At the heart of DANCE is the modeling of the accelerator evaluation software using a neural network (the evaluator) that can be used as a differentiable loss function. DANCE thus introduces a novel differentiable evaluator, which takes the architecture parameters from the RL controller, searches for the optimal hardware accelerator design, and evaluates its cost metrics.”) (Pg. 151356, Left column: “After that, a reward based on both the yielded accuracy and pipeline efficiency is generated, which is used to update the RNN controller.”).
Sekanina does not explicitly teach determine, for the respective ones of the plurality of workloads, a respective objective design space based on a region of an objective space enclosed by the Pareto front for the respective workload
Min-Fong makes obvious determine, for the respective ones of the plurality of workloads, a respective objective design space based on a region of an objective space enclosed by the Pareto front for the respective workload (Fig. 1: “Network Space Search (NSS) and Elite Spaces. We propose Network Space Search, a new paradigm for Auto-ML aiming at no manual effort, to automatically search for Pareto-efficient network spaces in Expanded Search Space, where minimal human expertise is imposed. The discovered network spaces, named Elite Spaces, are sustainably aligned with the Pareto front across several complexity constraints. Moreover, Elite Spaces can be further utilized for designing satisfactory networks and served as NAS search spaces to find satisfactory architectures.”) (Pg. 2, Left column: “Moreover, Elite Spaces can be further served as NAS search spaces for benefiting current NAS approaches to improve performance (e.g. In CIFAR-100, an averagely 2.3% lower error rate and 3.7% closer to target complexity than the baseline with around 90% fewer samples required to find satisfactory networks).”) (Pg. 4, Right column: “After searching by optimizing Eq. 5, we can obtain the network spaces satisfying multi-objectives, where the searched spaces A∗ are named Elite Spaces. More concretely, Elite Spaces are derived from the optimized probability distribution PΘ after the searching process. We sample n spaces from PΘ and identify the one closest to the FLOPs constraint as our Elite Space. Unlike hand crafted network spaces or commonly adopted NAS search spaces [21, 27], Elite Spaces are obtained without prior knowledge or human expertise on network designs. Moreover, Elite Spaces can be further exploited in designing promising networks and served as search spaces, benefiting NAS approaches to improve performance”) (Pg. 6, Right column: “constraints in both CIFAR-10 and CIFAR-100 datasets. Elite Spaces achieve satisfactory trade-offs between the error rates and meeting the FLOPs constraints, and are aligned with the Pareto front of Expanded Search Space. Since Elite Spaces discovered by our NSS method are guaranteed to consist of superior networks provided in various FLOPs regimes, they can be utilized for designing promising networks. More importantly, Elite Spaces are searched by NSS automatically, therefore the human effort involved in network designs is significantly reduced”)
Min-Fong is analogous art because it is within the field of neural network architecture improvement. It would have been obvious to one of ordinary skill in the art to combine Min-Fong with Sekanina before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to better optimize neural network setups. Min-Fong explains the issues faced with current design-space exploration techniques, such as limited exploration and the sheer scale of data that must be considered (Pg. 1, Right column -Pg. 2, Left column: “Design spaces [23, 24] have lately been demonstrated to be a decisive factor for designing networks. Accordingly, several design principles are proposed to deliver promising networks [24]. However, these design principles are based on human expertise and require extensive experiments for validation. In contrast to handcrafted designs, NAS automatically searches for favorable architectures within a predefined search space, which has shown to be a critical fac tor affecting the performance and efficiency of NAS approaches [18]. Recently, it is common to reuse the tailored search spaces (e.g. DARTS [21] and MobileNet [10] search spaces) in previous works [21, 31, 34, 33]. However, these approaches ignore the potential of exploring untailored spaces, which may cause a gap between tailored NAS search spaces and untailored ones. Furthermore, defining a new search space requires tremendous prior knowledge to best include superior networks. Hence, it is beneficial to discover superior network spaces automatically, improving the performance of NAS by bridging the gap and reducing human expertise for manually defining NAS search spaces.”) To this end, Min-Fong presents a method for better define design exploration spaces and enable consideration of more varied and optimal designs that may have been overlooked or skipped using previous methods (Pg. 2, Left column: “In this paper, we propose to Search for Network Spaces Automatically. In order to relax the tremendous prior knowledge imposed in the search spaces, we exclude the prevalently used DARTS or MobileNet search spaces and instead search for network spaces on Expanded Search Space, which contains scalability with minimal assumptions in network designs. We then define a new problem, Network Space Search (NSS), as searching for favorable network spaces instead of a single architecture. To obtain industry-favorable network spaces, efficiency and practical computation trade-offs are essential factors. Therefore, we propose our NSS method upon differentiable approaches and incorporate multi-objectives into the searching process to search for network spaces under various complexity constraints. The network spaces obtained by our NSS method, named Elite Spaces, are Pareto-efficient spaces aligned with the Pareto front [5] with respect to performance and complexity. Moreover, Elite Spaces can be further served as NAS search spaces for benefiting current NAS approaches to improve performance (e.g. In CIFAR-100, an averagely 2.3% lower error rate and 3.7% closer to target complexity than the baseline with around 90% fewer samples required to find satisfactory networks). Finally, our NSS method is capable of searching for superior spaces from various search spaces with different complexity, showing the applicability in unexplored and untailored spaces. Our contributions are summarized below: • We propose a whole new Auto-ML framework, Net work Space Search (NSS), to automatically search for favorable network spaces instead of a single architecture, reducing the human expertise involved in both designing network designs and defining NAS search spaces. To facilitate the NSS framework, we also de fine Expanded Search Space as a large-scale search space to search for favorable network spaces. • We further incorporate multi-objectives into NSS to search for network spaces under various complexity constraints, and the discovered network spaces, named Elite Spaces, deliver satisfactory performance and are aligned with the Pareto front of Expanded Search Space. Elite Spaces can further be served as NAS search spaces to improve the effectiveness of differentiable NAS methods. • Our NSS approach is capable of being exploited in unexplored network spaces with various complexity, demonstrating considerable potential in searching for network spaces in an automatic fashion.”) Overall, one of ordinary skill in the art would have recognized that combining Min-Fong with Sekanina would allow for more effective design space exploration, ultimately enabling higher performance neural network setups to be developed.
Regarding claim 2, Sekanina discloses
wherein the first hardware architectural configuration is a baseline design, and the second hardware architectural configuration is an adjusted first hardware architectural configuration (Pg. 151354, Left column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105]. In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”) (Pg. 151353, Right column: “A straightforward approach to organizing the co-search is to generate a CNN-accelerator pair, which is evaluated by training the network to obtain its accuracy and measuring the hardware parameters (Fig. 12). Based on this evaluation, the next candidate pairs are generated until the desired solution is not obtained.”) (Figure 13 shows the iterative search procedure for identifying a hardware architectural configuration.).
Regarding claim 3, Sekanina discloses
wherein the plurality of design space performance indicators are hypervolume indicators (Pg. 151349, Left column: “For example, NSGANetV1 [94] employs the hypervolume performance metric, which calculates the dominated area (hypervolume, in the general case) from the set of solutions to a reference point which is usually an estimate of the nadir point – a vector concatenating worst objective values of the Pareto front.”).
Regarding claim 4, Sekanina discloses
wherein the plurality of design space performance indicators are weighted, and wherein the aggregate score is a weighted aggregate score (Pg. 151349, Right column: “The aggregation methods introduce a suitable aggregation function (such as the weighted sum, weighted exponential sum, or weighted product) for the objective functions and optimize the composition.”).
Regarding claim 5, Sekanina discloses
wherein a first workload of the plurality of workloads is a deep neural network architecture (Pg. 151354, Left column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105].”) (Pg. 151353, Right column: “In another co-search strategy used by, e.g., in [47], [51], the hardware optimization algorithm receives a CNN as the input and optimizes the hardware accelerator concerning desired objectives (Fig. 13).”) (Pg. 151343, Right column: “Using cost models (such as MAESTRO [71]), the run time, resources utilization, delay, and power can be estimated for a given accelerator, DNN, and data set.”) (Pg. 151355, Left column: “An RL controller generates parameters describing the CNN architecture (based on ProxylessNAS) as well as the hardware parameters such as the number of PEs, buffer size, and dataflow pattern. … DANCE thus introduces a novel differentiable evaluator, which takes the architecture parameters from the RL controller, searches for the optimal hardware accelerator design, and evaluates its cost metrics.”) (Pg. 151343, Right column: “Using cost models (such as MAESTRO [71]), the run time, resources utilization, delay, and power can be estimated for a given accelerator, DNN, and data set.”).
Regarding claim 6, Sekanina discloses
wherein the plurality of workloads correspond to one or more modalities (Pg. 151340, Right column, Fig. 4: “FIGURE 4. Examples of hand-crafted CNNs and their blocks: (a) AlexNet for CIFAR-10 classification; (b) Residual block from ResNet [31]; (c) Inception block from GoogleNet [30]”) (Pg. 151341, Left column: “Please note that some of the following networks exist in several versions (e.g., ResNet-8, ResNet-14 etc.), which differ in the number of layers, the structure of building blocks, and some other parameters.”) (Pg. 151344, Right column: “In the case of the macro search space (Fig. 7a), the search space is determined by a set of possible operations for each node, hyperparameters of the network architecture, and a network template. … Another option is to parameterize a well-known CNN (such as ResNet [31] or MobileNetV3 [33]) and use it as a template (e.g., in [7], [51], [79][81]) for building and constraining the search space.”).
Regarding claim 7, Sekanina discloses
wherein the plurality of objectives include at least one of minimize latency, maximize efficiency, or maximize accuracy (Pg. 151338, Left column: “Hence, the newest NAS methods co-optimize NN architectures and hardware configuration to further improve latency and other parameters that are important in many applications such as IoT or mobile phones. These methods work in three search spaces (weights, NN architectures, and hardware configurations) and must innovatively orchestrate several search algorithms to produce the best trade-offs between the accuracy and various hardware-relevant metrics [8], [14], [15].”) (Pgs. 151354-151355: “In Codesign-NAS [139], RL controller selects a CNN architecture from a CNN search space and a hardware architecture from an accelerator design space. Both are sent to the evaluator that implements the CNN on the proposed accelerator to find accuracy and efficiency metrics, such as latency, area, and power (based on pre-computed models).”) (Pg. 151343, Right column: “Using cost models (such as MAESTRO [71]), the run time, resources utilization, delay, and power can be estimated for a given accelerator, DNN, and data set.”) (Pg. 151355, Left column: “An RL controller generates parameters describing the CNN architecture (based on ProxylessNAS) as well as the hardware parameters such as the number of PEs, buffer size, and dataflow pattern. At the heart of DANCE is the modeling of the accelerator evaluation software using a neural network (the evaluator) that can be used as a differentiable loss function. DANCE thus introduces a novel differentiable evaluator, which takes the architecture parameters from the RL controller, searches for the optimal hardware accelerator design, and evaluates its cost metrics.”).
Regarding claim 8, Sekanina discloses
wherein executing the proxy function is utilizes fewer compute resources as compared to the simulation of the execution of the first hardware architectural configuration (Pg. 151354, Left column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105]. In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”) (Figure 13 shows the iterative search procedure for identifying a hardware architectural configuration.) (Pg. 151343, Right column: “Using cost models (such as MAESTRO [71]), the run time, resources utilization, delay, and power can be estimated for a given accelerator, DNN, and data set.”).
Regarding claim 9, Sekanina discloses
wherein the performance metric is a first performance metric (Pgs. 151354-151355: “In Codesign-NAS [139], RL controller selects a CNN architecture from a CNN search space and a hardware architecture from an accelerator design space. Both are sent to the evaluator that implements the CNN on the proposed accelerator to find accuracy and efficiency metrics, such as latency, area, and power (based on pre-computed models).”) (Pg. 151354, Left column: “In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”), and
wherein the processor circuitry at least one of instantiates or executes the machine readable instructions to: execute a search of the design space for the hardware component based on the first performance metric to determine a third hardware architectural configuration (Pg. 151353, Left column: “In addition to the architecture search space and parameter search space (weights), there is an additional search space, called the hardware search space, containing all possible hardware configurations. … The space of hardware configurations can be searched together with the space of DNN architectures using the same search algorithm … In the Multi-objective: strategy column, the `co-search' means that there are two independent search algorithms, i.e., a co-search is conducted; one search algorithm operates in the network architecture space and the other in the hardware search space. The Multi-objective strategy is based either on a Pareto front construction method (`Pareto'), aggregation method (`Agg'), or applying some constraints (`Constr').”) (Pg. 151353, Right column: “A straightforward approach to organizing the co-search is to generate a CNN-accelerator pair, which is evaluated by training the network to obtain its accuracy and measuring the hardware parameters (Fig. 12). Based on this evaluation, the next candidate pairs are generated until the desired solution is not obtained.”) (Pg. 151354, Left column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105]. In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”) (Figure 13 shows the iterative search procedure for identifying a hardware architectural configuration.) (Pg. 151354, Right column: “In AutoDNN [14], each candidate CNN consists of several hardware-aware parameterizable cells called Bundles. By means of these cells, specialized software can map any candidate CNN generated by NAS to an FPGA accelerator based on a fine-grained tile-based pipeline architecture whose components are pre-designed and stored in a component library. Latency and resources are estimated and used back in theNAS algorithm.”) (Pg. 151349, Right column: “The aggregation methods introduce a suitable aggregation function (such as the weighted sum, weighted exponential sum, or weighted product) for the objective functions and optimize the composition.”) (Figure 13 shows the iterative search procedure for identifying a hardware architectural configuration.) (Pg. 151354, Left column: “In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS.”); and
predict a performance of the third hardware architectural configuration to generate a second performance metric by executing a proxy function corresponding to the third hardware architectural configuration (Pg. 151343, Right column: “Using cost models (such as MAESTRO [71]), the run time, resources utilization, delay, and power can be estimated for a given accelerator, DNN, and data set.”) (Pg. 151350, Left column: “Latency can be estimated using: a surrogate model, e.g., [6], [126], [128]-[131]; a suitable hardware simulator executing a candidate CNN, e.g. [51], [100], [132]-[135]; a formula or model derived after analyzing the search space of possible CNN architectures, e.g., [14], [47], [49], [101], [136]-[138]; a LUT-based model [105], [139], [140].”) (Pg. 151353, Right column: “In another co-search strategy used by, e.g., in [47], [51], the hardware optimization algorithm receives a CNN as the input and optimizes the hardware accelerator concerning desired objectives (Fig. 13).”) (Pg. 151354, Right column: “The RL controller samples parameters of a candidate CNN architecture and its possible quantization. For the sampled network, the hardware builder searches the hardware space to find a suitable hardware model. Each candidate hardware model is validated against the specification (latency constraint) during the search, and the result is sent back to the controller. … In Codesign-NAS [139], RL controller selects a CNN architecture from a CNN search space and a hardware architecture from an accelerator design space. Both are sent to the evaluator that implements the CNN on the proposed accelerator to find accuracy and efficiency metrics, such as latency, area, and power (based on pre-computed models).”) (Pg. 151355, Left column: “An RL controller generates parameters describing the CNN architecture (based on ProxylessNAS) as well as the hardware parameters such as the number of PEs, buffer size, and dataflow pattern. At the heart of DANCE is the modeling of the accelerator evaluation software using a neural network (the evaluator) that can be used as a differentiable loss function. DANCE thus introduces a novel differentiable evaluator, which takes the architecture parameters from the RL controller, searches for the optimal hardware accelerator design, and evaluates its cost metrics.”) (Pg. 151356, Left column: “After that, a reward based on both the yielded accuracy and pipeline efficiency is generated, which is used to update the RNN controller.”).
Regarding claim 10, Sekanina discloses
wherein, in response to the second performance metric exceeding a defined value, the processor circuitry least one of instantiates or executes the machine readable instructions to: simulate an execution of the third hardware architectural configuration for a plurality of workloads to generate a respective plurality of objective design spaces, the objective design spaces based on one or more objectives (Pg. 151353, Right column: “In another co-search strategy used by, e.g., in [47], [51], the hardware optimization algorithm receives a CNN as the input and optimizes the hardware accelerator concerning desired objectives (Fig. 13).”) (Figure 13 shows checking whether or not the constraint has been satisfied.);
generate a design score for the third hardware architectural configuration by aggregating a plurality of design space performance indicators, the plurality of design space performance indicators to correspond to the plurality of objective design spaces (Pg. 151353, Left column: “The Multi-objective strategy is based either on a Pareto front construction method (`Pareto'), aggregation method (`Agg'), or applying some constraints (`Constr').”) (Pg. 151349, Left column: “A common approach to solve the multi-objective NAS problem adopted by the NAS community is either (i) to transform it into a single-objective one (using suitable constraints, prioritization, or aggregation techniques)”) (Pg. 151349, Right column: “The aggregation methods introduce a suitable aggregation function (such as the weighted sum, weighted exponential sum, or weighted product) for the objective functions and optimize the composition.”) (Pg. 151354, Left column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105]. In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”) (Pg. 151354, Right column: “The RL controller samples parameters of a candidate CNN architecture and its possible quantization. For the sampled network, the hardware builder searches the hardware space to find a suitable hardware model. Each candidate hardware model is validated against the specification (latency constraint) during the search, and the result is sent back to the controller. … In Codesign-NAS [139], RL controller selects a CNN architecture from a CNN search space and a hardware architecture from an accelerator design space. Both are sent to the evaluator that implements the CNN on the proposed accelerator to find accuracy and efficiency metrics, such as latency, area, and power (based on pre-computed models).”) (Pg. 151355, Left column: “An RL controller generates parameters describing the CNN architecture (based on ProxylessNAS) as well as the hardware parameters such as the number of PEs, buffer size, and dataflow pattern. At the heart of DANCE is the modeling of the accelerator evaluation software using a neural network (the evaluator) that can be used as a differentiable loss function. DANCE thus introduces a novel differentiable evaluator, which takes the architecture parameters from the RL controller, searches for the optimal hardware accelerator design, and evaluates its cost metrics.”) (Pg. 151356, Left column: “After that, a reward based on both the yielded accuracy and pipeline efficiency is generated, which is used to update the RNN controller.”); and
execute another search of the design space for the hardware component based on the design score for the third hardware architectural configuration to determine a fourth hardware architectural configuration (Pg. 151353, Left column: “In addition to the architecture search space and parameter search space (weights), there is an additional search space, called the hardware search space, containing all possible hardware configurations. … The space of hardware configurations can be searched together with the space of DNN architectures using the same search algorithm … In the Multi-objective: strategy column, the `co-search' means that there are two independent search algorithms, i.e., a co-search is conducted; one search algorithm operates in the network architecture space and the other in the hardware search space. The Multi-objective strategy is based either on a Pareto front construction method (`Pareto'), aggregation method (`Agg'), or applying some constraints (`Constr').”) (Pg. 151353, Right column: “A straightforward approach to organizing the co-search is to generate a CNN-accelerator pair, which is evaluated by training the network to obtain its accuracy and measuring the hardware parameters (Fig. 12). Based on this evaluation, the next candidate pairs are generated until the desired solution is not obtained.”) (Pg. 151354, Left column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105]. In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”) (Figure 13 shows the iterative search procedure for identifying a hardware architectural configuration.) (Pg. 151354, Right column: “In AutoDNN [14], each candidate CNN consists of several hardware-aware parameterizable cells called Bundles. By means of these cells, specialized software can map any candidate CNN generated by NAS to an FPGA accelerator based on a fine-grained tile-based pipeline architecture whose components are pre-designed and stored in a component library. Latency and resources are estimated and used back in theNAS algorithm.”) (Pg. 151349, Right column: “The aggregation methods introduce a suitable aggregation function (such as the weighted sum, weighted exponential sum, or weighted product) for the objective functions and optimize the composition.”) (Figure 13 shows the iterative search procedure for identifying a hardware architectural configuration.) (Pg. 151354, Left column: “In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS.”).
Regarding claim 11, claim 11 is substantially similar to claim 1. Therefore, the similar analysis is applicable.
Furthermore, Sekanina discloses A non-transitory machine readable storage medium comprising instructions that, when executed, cause processor circuitry (Pg. 151350, Left column: “Latency can be estimated using: a surrogate model, e.g., [6], [126], [128]-[131]; a suitable hardware simulator executing a candidate CNN, e.g. [51], [100], [132]-[135]; a formula or model derived after analyzing the search space of possible CNN architectures, e.g., [14], [47], [49], [101], [136]-[138]; a LUT-based model [105], [139], [140].”) (Pg. 151355, Right column: “Focusing on Google's EdgeTPU, NAHAS [129] performs a joint search in the space of CNN architectures and hardware accelerator configurations, where hardware resources and latency are constraints.”) (Pg. 151355, Right column: “Focusing on Google's EdgeTPU, NAHAS [129] performs a joint search in the space of CNN architectures and hardware accelerator configurations, where hardware resources and latency are constraints. … An in-house simulator is used to estimate latency and other hardware-related parameters.”).
Claims 12-20 and 22-25 are substantially similar to claims 1-10. Therefore, the similar analysis is applicable.
Regarding claim 21, Sekanina discloses
determining, by executing instructions with at least one processor, a first hardware architectural configuration of a hardware component based on a design constraint; (Pg. 151354, Left column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105]. In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”) (Pg. 151353, Right column: “In another co-search strategy used by, e.g., in [47], [51], the hardware optimization algorithm receives a CNN as the input and optimizes the hardware accelerator concerning desired objectives (Fig. 13).”) (Pg. 151355, Left column: “An RL controller generates parameters describing the CNN architecture (based on ProxylessNAS) as well as the hardware parameters such as the number of PEs, buffer size, and dataflow pattern. … DANCE thus introduces a novel differentiable evaluator, which takes the architecture parameters from the RL controller, searches for the optimal hardware accelerator design, and evaluates its cost metrics.”) (Pg. 151343, Right column: “Using cost models (such as MAESTRO [71]), the run time, resources utilization, delay, and power can be estimated for a given accelerator, DNN, and data set.”); ([Pg. 151350, Left column: “Latency can be estimated using: a surrogate model, e.g., [6], [126], [128]-[131]; a suitable hardware simulator executing a candidate CNN, e.g. [51], [100], [132]-[135]; a formula or model derived after analyzing the search space of possible CNN architectures, e.g., [14], [47], [49], [101], [136]-[138]; a LUT-based model [105], [139], [140].”) (Pg. 151355, Right column: “Focusing on Google's EdgeTPU, NAHAS [129] performs a joint search in the space of CNN architectures and hardware accelerator configurations, where hardware resources and latency are constraints.”) (Pg. 151355, Right column: “Focusing on Google's EdgeTPU, NAHAS [129] performs a joint search in the space of CNN architectures and hardware accelerator configurations, where hardware resources and latency are constraints. … An in-house simulator is used to estimate latency and other hardware-related parameters.” [Examiner’s note: such hardware simulators are performed on a computer including a processor, a memory, and machine readable instructions.])
simulating, by executing the instructions with the at least one processor, an execution of the first hardware architectural configuration for first workloads and second workloads to generate performance metrics based on one or more performance objectives, the first workloads including a first one of the first workloads and variations of the first one of the first workloads, the second workloads including a first one of the second workloads and variations of the first one of the second workloads, (Pg. 151348, Left column: “Typical objectives to be optimized for a given hardware platform are latency, throughput, energy efficiency, and memory usage. Hence, in specific steps of the NAS algorithm, all relevant objectives have to be evaluated, either by direct measurement on real hardware or estimated using software models (see Fig. 6).”) (Pg. 151354, Left Column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105]. In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”) (Pg. 151350, Left column: “Latency can be estimated using: a surrogate model, e.g., [6], [126], [128]-[131]; a suitable hardware simulator executing a candidate CNN, e.g. [51], [100], [132]-[135]; a formula or model derived after analyzing the search space of possible CNN architectures, e.g., [14], [47], [49], [101], [136]-[138]; a LUT-based model [105], [139], [140].”) (Pg. 151355, Right column: “Focusing on Google’s EdgeTPU, NAHAS [129] performs a joint search in the space of CNN architectures and hard ware accelerator configurations, where hardware resources and latency are constraints. The architecture search space is based on a new fused inverted bottleneck layer with tunable parameters. The accelerator search space is defined by seven parameters (e.g., the PE array size, the number of SIMD units, register file capacity). An in-house simulator is used to estimate latency and other hardware-related parameters.”) (Table 3: Shows a comparison of various different models, each model corresponds to a workload. Note that several are variations of each other)
identifying, by executing the instructions with the at least one processor, a first Pareto front from the performance metrics for the first workloads and a second Pareto front from the performance metrics for the second workloads; (Pg. 151349, Right column: “Truly multi-objective optimization methods iteratively build the Pareto front in the course of optimization by com paring candidate solutions using the non-dominance relation, promoting reasonable solutions, and trying to cover the expected Pareto front. This approach has significantly been developed within the evolutionary computation community. One of the most popular methods is NSGA-II [124]. It is based on sorting individuals in a population according to the dominance relation into multiple fronts. The first front contains all non-dominated solutions. Each subsequent front is constructed by removing all the preceding fronts from the population and finding a new Pareto front in the remaining individuals. The solutions within the individual fronts are then sorted according to the crowding distance metric. This metric helps to preserve the diversity of the population along the fronts. Best individuals then serve as parents for the new population. NSGA-II thus always produces a set of non-dominated solutions (i.e., a Pareto front) when it is terminated.”) (Table 3: Shows a comparison of various different models, each model corresponds to a workload. Note that several are variations of each other)
(Pg. 151349, Right column: “Truly multi-objective optimization methods iteratively build the Pareto front in the course of optimization by com paring candidate solutions using the non-dominance relation, promoting reasonable solutions, and trying to cover the expected Pareto front. This approach has significantly been developed within the evolutionary computation community. One of the most popular methods is NSGA-II [124]. It is based on sorting individuals in a population according to the dominance relation into multiple fronts. The first front contains all non-dominated solutions. Each subsequent front is constructed by removing all the preceding fronts from the population and finding a new Pareto front in the remaining individuals. The solutions within the individual fronts are then sorted according to the crowding distance metric. This metric helps to preserve the diversity of the population along the fronts. Best individuals then serve as parents for the new population. NSGA-II thus always produces a set of non-dominated solutions (i.e., a Pareto front) when it is terminated.”) (Pg. 151349, Left column: “As it is not known ‘‘how far’’ the obtained solutions are from the truly Pareto optimal solutions, a common practice is to introduce a quality metric capable of measuring the distance between two sets of solutions obtained with two multi-objective optimization methods (see, for example, [122]) and compare them under this metric. For example, NSGANetV1 [94] employs the hypervolume performance metric, which calculates the dominated area (hypervolume, in the general case) from the set of solutions to a reference point which is usually an estimate of the nadir point– a vector concatenating worst objective values of the Pareto front.”) [Examiner’s note: as the reference point is based on the values of the pareto front, generating a new pareto front in place of an old one also means the reference point is updated to a new reference point])
generating, by executing the instructions with the at least one processor, an aggregate score by combining a plurality of design space performance indicators, ones of the plurality of design space performance indicators corresponding to the first objective design space and the second objective design space (Pg. 151353, Left column: “The Multi-objective strategy is based either on a Pareto front construction method (`Pareto'), aggregation method (`Agg'), or applying some constraints (`Constr').”) (Pg. 151349, Left column: “A common approach to solve the multi-objective NAS problem adopted by the NAS community is either (i) to transform it into a single-objective one (using suitable constraints, prioritization, or aggregation techniques)”) (Pg. 151349, Right column: “The aggregation methods introduce a suitable aggregation function (such as the weighted sum, weighted exponential sum, or weighted product) for the objective functions and optimize the composition.”) (Pg. 151349, Right column: “Truly multi-objective optimization methods iteratively build the Pareto front in the course of optimization by com paring candidate solutions using the non-dominance relation, promoting reasonable solutions, and trying to cover the expected Pareto front. This approach has significantly been developed within the evolutionary computation community. One of the most popular methods is NSGA-II [124]. It is based on sorting individuals in a population according to the dominance relation into multiple fronts. The first front contains all non-dominated solutions. Each subsequent front is constructed by removing all the preceding fronts from the population and finding a new Pareto front in the remaining individuals. The solutions within the individual fronts are then sorted according to the crowding distance metric. This metric helps to preserve the diversity of the population along the fronts. Best individuals then serve as parents for the new population. NSGA-II thus always produces a set of non-dominated solutions (i.e., a Pareto front) when it is terminated.”) (Table 3: Shows a comparison of various different models, each model corresponds to a workload. Note that several are variations of each other)
searching, by executing the instructions with the at least one processor, a design datastore based on the aggregate score to identify a second hardware architectural configuration; and(Pg. 151353, Left column: “In addition to the architecture search space and parameter search space (weights), there is an additional search space, called the hardware search space, containing all possible hardware configurations. … The space of hardware configurations can be searched together with the space of DNN architectures using the same search algorithm … In the Multi-objective: strategy column, the `co-search' means that there are two independent search algorithms, i.e., a co-search is conducted; one search algorithm operates in the network architecture space and the other in the hardware search space. The Multi-objective strategy is based either on a Pareto front construction method (`Pareto'), aggregation method (`Agg'), or applying some constraints (`Constr').”) (Pg. 151353, Right column: “A straightforward approach to organizing the co-search is to generate a CNN-accelerator pair, which is evaluated by training the network to obtain its accuracy and measuring the hardware parameters (Fig. 12). Based on this evaluation, the next candidate pairs are generated until the desired solution is not obtained.”) (Pg. 151354, Left column: “DNA enables co-searching for the CNN architecture together with the accelerators' configuration (e.g., the PE array size, the local and global buffer sizes, dataflow) and the mapping method (e.g., loop tiling strategy and loop size/order). DNA consists of two search algorithms: (1) the Differentiable Accelerator Search (DAS) in a generic accelerator design space, and (2) the Differentiable Network Search (DNS) based on FBNet [105]. In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS. In order to continue the search in the CNN architecture space by DNS, the hardware cost loss is needed. It is obtained as an average hardware cost for each operator on the M optimized accelerators generated from the previous step.”) (Figure 13 shows the iterative search procedure for identifying a hardware architectural configuration.) (Pg. 151354, Right column: “In AutoDNN [14], each candidate CNN consists of several hardware-aware parameterizable cells called Bundles. By means of these cells, specialized software can map any candidate CNN generated by NAS to an FPGA accelerator based on a fine-grained tile-based pipeline architecture whose components are pre-designed and stored in a component library. Latency and resources are estimated and used back in theNAS algorithm.”) (Pg. 151349, Right column: “The aggregation methods introduce a suitable aggregation function (such as the weighted sum, weighted exponential sum, or weighted product) for the objective functions and optimize the composition.”) (Figure 13 shows the iterative search procedure for identifying a hardware architectural configuration.) (Pg. 151354, Left column: “In each iteration, the global co-search algorithm samples M networks from the current network distribution NET(a) and obtains the optimal accelerator for each of them using DAS.”);
estimating, by executing the instructions with the at least one processor, a performance of the second hardware architectural configuration to generate a performance score by executing an estimation function corresponding to the second hardware architectural configuration. (Pg. 151343, Right column: “Using cost models (such as MAESTRO [71]), the run time, resources utilization, delay, and power can be estimated for a given accelerator, DNN, and data set.”) (Pg. 151350, Left column: “Latency can be estimated using: a surrogate model, e.g., [6], [126], [128]-[131]; a suitable hardware simulator executing a candidate CNN, e.g. [51], [100], [132]-[135]; a formula or model derived after analyzing the search space of possible CNN architectures, e.g., [14], [47], [49], [101], [136]-[138]; a LUT-based model [105], [139], [140].”) (Pg. 151353, Right column: “In another co-search strategy used by, e.g., in [47], [51], the hardware optimization algorithm receives a CNN as the input and optimizes the hardware accelerator concerning desired objectives (Fig. 13).”) (Pg. 151354, Right column: “The RL controller samples parameters of a candidate CNN architecture and its possible quantization. For the sampled network, the hardware builder searches the hardware space to find a suitable hardware model. Each candidate hardware model is validated against the specification (latency constraint) during the search, and the result is sent back to the controller. … In Codesign-NAS [139], RL controller selects a CNN architecture from a CNN search space and a hardware architecture from an accelerator design space. Both are sent to the evaluator that implements the CNN on the proposed accelerator to find accuracy and efficiency metrics, such as latency, area, and power (based on pre-computed models).”) (Pg. 151355, Left column: “An RL controller generates parameters describing the CNN architecture (based on ProxylessNAS) as well as the hardware parameters such as the number of PEs, buffer size, and dataflow pattern. At the heart of DANCE is the modeling of the accelerator evaluation software using a neural network (the evaluator) that can be used as a differentiable loss function. DANCE thus introduces a novel differentiable evaluator, which takes the architecture parameters from the RL controller, searches for the optimal hardware accelerator design, and evaluates its cost metrics.”) (Pg. 151356, Left column: “After that, a reward based on both the yielded accuracy and pipeline efficiency is generated, which is used to update the RNN controller.”).
Sekanina does not explicitly teach determining, by executing the instructions with the at least one processor, a first objective design space based on a region of an objective space enclosed by the pareto front and a second objective design space based on a region of the objective space enclosed by other data.
Min-Fong makes obvious determining, by executing the instructions with the at least one processor, a first objective design space based on a region of an objective space enclosed by the pareto front and a second objective design space based on a region of the objective space enclosed by other data (Fig. 1: “Network Space Search (NSS) and Elite Spaces. We propose Network Space Search, a new paradigm for Auto-ML aiming at no manual effort, to automatically search for Pareto-efficient network spaces in Expanded Search Space, where minimal human expertise is imposed. The discovered network spaces, named Elite Spaces, are sustainably aligned with the Pareto front across several complexity constraints. Moreover, Elite Spaces can be further utilized for designing satisfactory networks and served as NAS search spaces to find satisfactory architectures.”) (Pg. 2, Left column: “Moreover, Elite Spaces can be further served as NAS search spaces for benefiting current NAS approaches to improve performance (e.g. In CIFAR-100, an averagely 2.3% lower error rate and 3.7% closer to target complexity than the baseline with around 90% fewer samples required to find satisfactory networks).”) (Pg. 4, Right column: “After searching by optimizing Eq. 5, we can obtain the network spaces satisfying multi-objectives, where the searched spaces A∗ are named Elite Spaces. More concretely, Elite Spaces are derived from the optimized probability distribution PΘ after the searching process. We sample n spaces from PΘ and identify the one closest to the FLOPs constraint as our Elite Space. Unlike hand crafted network spaces or commonly adopted NAS search spaces [21, 27], Elite Spaces are obtained without prior knowledge or human expertise on network designs. Moreover, Elite Spaces can be further exploited in designing promising networks and served as search spaces, benefiting NAS approaches to improve performance”) (Pg 6, Right column: “constraints in both CIFAR-10 and CIFAR-100 datasets. Elite Spaces achieve satisfactory trade-offs between the error rates and meeting the FLOPs constraints, and are aligned with the Pareto front of Expanded Search Space. Since Elite Spaces discovered by our NSS method are guaranteed to consist of superior networks provided in various FLOPs regimes, they can be utilized for designing promising networks. More importantly, Elite Spaces are searched by NSS automatically, therefore the human effort involved in network designs is significantly reduced” [Examiner’s note: the reference point is a point that defines the outer edge of the design space. As the elite space and expanded search space have different outer edges, they clearly also have different reference points bounding that space])
Min-Fong is analogous art because it is within the field of neural network architecture improvement. It would have been obvious to one of ordinary skill in the art to combine Min-Fong with Sekanina before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to better optimize neural network setups. Min-Fong explains the issues faced with current design-space exploration techniques, such as limited exploration and the sheer scale of data that must be considered (Pg. 1, Right column -Pg. 2, Left column: “Design spaces [23, 24] have lately been demonstrated to be a decisive factor for designing networks. Accordingly, several design principles are proposed to deliver promising networks [24]. However, these design principles are based on human expertise and require extensive experiments for validation. In contrast to handcrafted designs, NAS automatically searches for favorable architectures within a predefined search space, which has shown to be a critical fac tor affecting the performance and efficiency of NAS approaches [18]. Recently, it is common to reuse the tailored search spaces (e.g. DARTS [21] and MobileNet [10] search spaces) in previous works [21, 31, 34, 33]. However, these approaches ignore the potential of exploring untailored spaces, which may cause a gap between tailored NAS search spaces and untailored ones. Furthermore, defining a new search space requires tremendous prior knowledge to best include superior networks. Hence, it is beneficial to discover superior network spaces automatically, improving the performance of NAS by bridging the gap and reducing human expertise for manually defining NAS search spaces.”) To this end, Min-Fong presents a method for better define design exploration spaces and enable consideration of more varied and optimal designs that may have been overlooked or skipped using previous methods (Pg. 2, Left column: “In this paper, we propose to Search for Network Spaces Automatically. In order to relax the tremendous prior knowledge imposed in the search spaces, we exclude the prevalently used DARTS or MobileNet search spaces and instead search for network spaces on Expanded Search Space, which contains scalability with minimal assumptions in network designs. We then define a new problem, Network Space Search (NSS), as searching for favorable network spaces instead of a single architecture. To obtain industry-favorable network spaces, efficiency and practical computation trade-offs are essential factors. Therefore, we propose our NSS method upon differentiable approaches and incorporate multi-objectives into the searching process to search for network spaces under various complexity constraints. The network spaces obtained by our NSS method, named Elite Spaces, are Pareto-efficient spaces aligned with the Pareto front [5] with respect to performance and complexity. Moreover, Elite Spaces can be further served as NAS search spaces for benefiting current NAS approaches to improve performance (e.g. In CIFAR-100, an averagely 2.3% lower error rate and 3.7% closer to target complexity than the baseline with around 90% fewer samples required to find satisfactory networks). Finally, our NSS method is capable of searching for superior spaces from various search spaces with different complexity, showing the applicability in unexplored and untailored spaces. Our contributions are summarized below: • We propose a whole new Auto-ML framework, Net work Space Search (NSS), to automatically search for favorable network spaces instead of a single architecture, reducing the human expertise involved in both designing network designs and defining NAS search spaces. To facilitate the NSS framework, we also de fine Expanded Search Space as a large-scale search space to search for favorable network spaces. • We further incorporate multi-objectives into NSS to search for network spaces under various complexity constraints, and the discovered network spaces, named Elite Spaces, deliver satisfactory performance and are aligned with the Pareto front of Expanded Search Space. Elite Spaces can further be served as NAS search spaces to improve the effectiveness of differentiable NAS methods. • Our NSS approach is capable of being exploited in unexplored network spaces with various complexity, demonstrating considerable potential in searching for network spaces in an automatic fashion.”) Overall, one of ordinary skill in the art would have recognized that combining Min-Fong with Sekanina would allow for more effective design space exploration, ultimately enabling higher performance neural network setups to be developed.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael P Mirabito whose telephone number is (703)756-1494. The examiner can normally be reached M-F 10:30 am - 6:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emerson Puente can be reached at (571) 272-3652. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.P.M./Examiner, Art Unit 2187
/EMERSON C PUENTE/Supervisory Patent Examiner, Art Unit 2187
