DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is responsive to the claims dated 5/31/2024.
Claims 12-22 are presented for examination.
Claim Objections
Claims 20, 21 and 22 are objected to because of the following informalities: In claim 20, the term "invlufrd" in the recitation "the size for an implementation of the neural network which invlufrd the same implementation for the same operator is determined depending on the stored influence" is a non-word that appears to be a typographical error and should read "includes," consistent with the earlier recitation in the same claim of a size "which includes the implementation of the operator." In claim 21, the recitation "on the computing device,and for each set of implementations" omits a space and should read "on the computing device, and for each set of implementations." In claim 22, the recitation "providing a computing devicewhich includes the neural network" omits a space and should read "providing a computing device which includes the neural network." Appropriate correction is required.
Claim Rejections - 35 U.S.C. 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 13, 16, 17, and 20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 13 recites the limitation the artificial neural network. There is insufficient antecedent basis for this limitation in the claim. See MPEP Section 2173.05(e).
Claim 13 depends from claim 12. Claim 12 introduces a neural network (a computer-implemented method for selecting implementations for operators for a neural network which includes a set of different operators of an operator type) but does not introduce an artificial neural network. Because the claims earlier recite a neural network and then separately recite the artificial neural network, it is unclear whether the artificial neural network refers to the previously recited neural network of claim 12 or to a further, separately claimed network, and one of ordinary skill in the art would not be reasonably apprised of the scope of the claim. For purposes of examination, the artificial neural network is interpreted under the broadest reasonable interpretation as the neural network introduced in claim 12, consistent with the specification, which uses the terms neural network and artificial neural network interchangeably and states that the implementation of the neural network is provided for executing the artificial neural network.
Claim 16 is indefinite because the label first implementation (and, correspondingly, second implementation) is used to denote two different things and its referent at the metric-determining and selecting steps cannot be ascertained with reasonable certainty. See MPEP Section 2173.05. Claim 16 first uses first implementation to denote an operator-level implementation found in a search space (a first implementation is determined in a first search space for the implementation for the first subset), and then uses the same label to denote a network level build (a first implementation of the neural network is determined in which the common operators of the first subset are implemented with the first implementation). The same duplication occurs for second implementation. The claim then recites that the at least one metric is determined for the implementations of the first implementation and for the implementations of the second implementation, and that either the first implementation or the second implementation is selected depending on a comparison of the at least one metric. Because the identical labels first implementation and second implementation are used at the metric-determining and selecting steps without indicating whether they refer to the operator-level implementation or to the neural-network-level implementation, and because the phrase the implementations of the first implementation (implementations of an implementation) has no clear referent, one of ordinary skill in the art cannot determine with reasonable certainty which implementation is measured by the metric and which implementation is selected. The claim therefore fails to inform, with reasonable certainty, a person having ordinary skill in the art about the scope of the invention. See Nautilus, Inc. v. Biosig Instruments, Inc., 572 U.S. 898, 901, 910 (2014); In re Packard, 751 F.3d 1307 (Fed. Cir. 2014). For purposes of examination, and consistent with the specification (steps 206-1, 206-2, and 206-3), the at least one metric is interpreted as being determined for the first implementation of the neural network and for the second implementation of the neural network, and the selecting step is interpreted as a selection between those two neural-network implementations for the first subset.
Claim 17 depends from claim 16 and incorporates the indefinite first implementation and second implementation recitations; it does not cure the defect and is rejected for the same reason. Claim 17 further recites that the first implementation is replaced by the second implementation depending on the comparison, which likewise leaves unclear whether an operator-level or a neural-network-level implementation is replaced. (The recitation of a better implementation is defined within claim 17 as an implementation with lower latency, or higher throughput, or lower energy consumption, and is not separately indefinite).
Claim Rejections - 35 U.S.C. 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 12-22 are rejected under 35 U.S.C. 103 as being unpatentable over Benmeziane et al. (hereinafter Benmeziane) "A Comprehensive Survey on Hardware-Aware Neural Architecture Search" (2021), in view of Azadbakht et al. (hereinafter Azadbakht) "Drastically Reducing the Number of Trainable Parameters in Deep CNNs by Inter-layer Kernel-sharing" (2022), and in further view of Liberis et al. (hereinafter Liberis) "uNAS: Constrained Neural Architecture Search for Microcontrollers" (2020).
Benmeziane was disclosed in an IDS dated 7/10/2025.
Regarding independent claim 12, Benmeziane teaches a computer-implemented method for selecting implementations for operators for a neural network which includes a set of different operators of an operator type, for providing a computing device including the neural network, the method comprising the following steps (Benmeziane: page 8, Section V.A Architecture Search Space, "it defines a set of basic network operators and how these operators can be connected to construct the computation graph of the model"; Benmeziane's hardware aware search selects, for the operators (operators) of a neural network (a neural network), implementations to be deployed for a target computing device (a computing device)):
for each subset of the subsets, determining a set of implementations for the operators of the subset (Benmeziane: page 18, Section X, 2) Microsoft NNI, "one can specify multiple operators for one single layer including depthwise convolution, dilated convolution, maxpooling"; Benmeziane defines, for the operators of the network (for each subset of the subsets), a pool of candidate operator choices (a set of implementations for the operators of the subset) from which the search draws, so that each operator is optimized over its own set of candidate implementations);
for each implementation of the set of implementations, determining at least one metric that characterizes an implementation of the neural network with the implementation on the computing device (Benmeziane: page 12, Section VII, "the hardware metrics need to be measured either by a real-time execution of each architecture on the targeted platform or an estimation method"; Benmeziane obtains, for each candidate implementation, a hardware cost metric (at least one metric) that characterizes the implementation of the neural network as executed on the target platform (the computing device)),
the at least one metric characterizing a latency, or a throughput, or an energy consumption of the implementation of the neural network with the implementation on the computing device (Benmeziane: page 7, Section IV, Hardware-aware search strategy, "a special evaluator that measures the hardware cost metric (e.g., latency, memory usage, energy consumption)"; the measured hardware cost metric (the at least one metric) characterizes at least the latency and the energy consumption of the implementation on the target hardware, which are members of the recited alternative group of which only one need be taught); and
for each set of implementations, selecting one implementation (Benmeziane: page 7, Section IV, Hardware-aware search strategy, "Both model accuracy and hardware cost guide the search and enable the NAS to find the most efficient architecture"; page 18, Section X, 2) Microsoft NNI "the NNI will automatically find the best candidate based on a gradient-based optimization"; Benmeziane selects, from the set of candidate implementations, the single best-performing implementation (one implementation) according to the measured hardware cost).
Benmeziane does not expressly teach determining subsets of the set of different operators.
However, Azadbakht teaches determining subsets of the set of different operators (Azadbakht: page 2, Section 2 Method, " We define a sharing group as a set of isomorphic layers sharing their kernels…one might partition a set of isomorphic layers into two or even more sharing groups"; Azadbakht partitions the isomorphic layers (the set of different operators) of a convolutional neural network into sharing groups (subsets), each group being implemented with one shared set of kernels).
Because Benmeziane and Azadbakht are analogous art within the same field of endeavor, specifically the hardware-efficient deployment of neural networks on resource-constrained computing devices, and are each reasonably pertinent to the problem of reducing the resource cost of a neural network implementation, accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to partition the operators searched by Benmeziane into the sharing groups taught by Azadbakht, with a reasonable expectation of success, so as to teach determining subsets of the set of different operators. This modification would have been motivated by the desire to reduce the number of trainable parameters and the memory footprint of the deployed model (Azadbakht: page 2, Section 2 Method).
Benmeziane and Azadbakht do not expressly teach wherein the implementations are selected that are Pareto-optimal with respect to the at least one metric and a size of an implementation of the neural network with the selected implementations.
However, Liberis teaches wherein the implementations are selected that are Pareto-optimal with respect to the at least one metric and a size of an implementation of the neural network with the selected implementations (Liberis: page 4, Section 3.2, "This turns NAS into a multiobjective optimisation problem…We include four objectives, three of which are resource constraints…(2) peak memory usage, (3) model size and (4) latency…It is common to consider a Pareto front as a set of potential solutions: a set of points on which one objective function cannot be improved without making another objective worse"; Liberis selects networks lying on a Pareto front (Pareto-optimal) that jointly trades the inference latency (the at least one metric) against the model size (a size of an implementation of the neural network)).
Because Benmeziane, in view of Azadbakht, and Liberis are analogous art within the same field of endeavor, specifically the hardware-aware optimization of neural networks for resource-constrained computing devices, and are each reasonably pertinent to the problem of balancing the performance and the resource cost of a deployed neural network, accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to select the implementations found by Benmeziane and Azadbakht according to the Pareto-optimal multiobjective criterion taught by Liberis, with a reasonable expectation of success, thereby teaching wherein the implementations are selected that are Pareto-optimal with respect to the at least one metric and a size of an implementation of the neural network with the selected implementations. This modification would have been motivated by the desire to obtain networks that keep low memory and storage usage and low inference latency on the target device (Liberis: page 1, Abstract).
Regarding dependent claim 13, Benmeziane, in view of Azadbakht and Liberis, teach the method of claim 12, wherein the implementation of the neural network is installed on the computing device or the implementation of the neural network is provided for executing the artificial neural network (interpreted as executing the neural network per the 35 U.S.C. 112(b) rejection set forth above) (Benmeziane: page 16, Section VIII, Table VI, "The sampled model is executed on the hardware target while searching"; Benmeziane provides the selected implementation of the neural network (the implementation of the neural network) for execution on the target hardware (the computing device), satisfying the recited alternative that the implementation is provided for executing the neural network, of which only one alternative need be taught).
Regarding dependent claim 14, Benmeziane, in view of Azadbakht and Liberis, teach the method of claim 12, wherein, for each subset, possible implementations of the operators in the subset are specified (Azadbakht: page 1, Abstract, "layers having the same kernel size, input and output channels"; because kernel-sharing is only possible among isomorphic operators, the shared kernels feasible for those operators are the possible implementations (possible implementations) specified for the operators in that subset), wherein a search space for each implementation for the subset includes an intersection of the possible implementations (Azadbakht: page 2, Section 2 Method, "During the forward pass, isomorphic layers in a sharing group use the same set of trainable parameters (i.e., shared kernels)"; because one shared set of kernels must be valid for every operator in the sharing group, the search space for the subset is limited to the kernels common to all of its operators, that is, an intersection of the possible implementations (an intersection of the possible implementations)).
Regarding dependent claim 15, Benmeziane, in view of Azadbakht and Liberis, teach the method of claim 12, wherein, at least one subset of the set of different operators is determined which includes operators that can be implemented with the same implementation (Azadbakht: page 2, Section 2 Method, "During the forward pass, isomorphic layers in a sharing group use the same set of trainable parameters (i.e., shared kernels)"; the isomorphic layers grouped into a sharing group (a subset of the set of different operators) are the operators that are implemented with one shared set of kernels (the same implementation)).
Regarding dependent claim 16, Benmeziane, in view of Azadbakht and Liberis, teach the method of claim 12, wherein a first subset of the set of different operators and a second subset of the set of different operators are determined, wherein the first subset and the second subset include common operators (Azadbakht: page 2, Section 2 Method, "one might partition a set of isomorphic layers into two or even more sharing groups"; Azadbakht determines two or more sharing groups (a first subset and a second subset) drawn from the same set of isomorphic layers, such that isomorphic layers belonging to both groups are common operators (common operators)), wherein a first implementation is determined in a first search space for the implementation for the first subset and a second implementation is determined for the common operators in a second search space for the implementation for the second subset (Benmeziane: page 8, Section V.A, "it defines a set of basic network operators and how these operators can be connected to construct the computation graph of the model"; Benmeziane searches each subset's own pool of candidate implementations (a first search space and a second search space) to determine a candidate implementation (a first implementation and a second implementation) for the operators of that subset), wherein a first implementation of the neural network is determined in which the common operators of the first subset are implemented with the first implementation, wherein a second implementation of the neural network is determined in which the common operators of the first subset are implemented with the second implementation (Benmeziane: page 8, Section V.A, "it defines a set of basic network operators and how these operators can be connected to construct the computation graph of the model"; page 16, Section VIII, Table VI, "The sampled model is executed on the hardware target while searching", Lookup Table Models, Analytical Estimation, Prediction Mdoel; Benmeziane constructs, for each candidate, a sampled model of the neural network (an implementation of the neural network) in which the common operators take a selected operator implementation, so that one sampled model has the common operators implemented with the first implementation and another sampled model has the common operators implemented with the second implementation), wherein the at least one metric is determined for the implementations of the first implementation (interpreted as first implementation of the neural network per the 35 U.S.C. 112(b) rejection set forth above), wherein the at least one metric is determined for the implementations of the second implementation (interpreted as second implementation of the neural network per the 35 U.S.C. 112(b) rejection set forth above), and wherein either the first implementation or the second implementation is selected depending on a comparison of the at least one metric determined for the first implementation and the second implementation (interpreted as and wherein either the first implementation of the neural network or the second implementation of the neural network is selected depending on a comparison of the at least one metric determined for the first implementation of the neural network and the second implementation of the neural network per the 35 U.S.C. 112(b) rejection set forth above) (Benmeziane: page 12, Section VII, "the hardware metrics need to be measured either by a real-time execution of each architecture on the targeted platform or an estimation method"; page 7, Section IV, "enable the NAS to find the most efficient architecture"; Benmeziane determines the hardware cost metric (the at least one metric) for each candidate implementation of the neural network and selects the candidate having the better measured metric, that is, either the first or the second implementation of the neural network depending on the comparison of their metrics).
Regarding dependent claim 17, Benmeziane, in view of Azadbakht and Liberis, teach the method of claim 16, wherein the first implementation (interpreted as the first implementation of the neural network per the 35 U.S.C. 112(b) rejection set forth above) is replaced by the second implementation (interpreted as the second implementation of the neural network per the 35 U.S.C. 112(b) rejection set forth above) depending on the comparison when the at least one metric determined for the second implementation (interpreted as the second implementation of the neural network per the 35 U.S.C. 112(b) rejection set forth above) indicates a better implementation of the neural network than the at least one metric determined for the first implementation (interpreted as the first implementation of the neural network per the 35 U.S.C. 112(b) rejection set forth above), the better implementation being an implementation with lower latency, or higher throughput, or lower energy consumption (Benmeziane: page 15, Section VIII, b) Latency "search for the trade-off between inference time and accuracy"; Benmeziane retains the candidate implementation of the neural network having the better measured metric, replacing a first implementation with a second implementation when the second exhibits lower inference time (lower latency)).
Regarding dependent claim 18, Benmeziane, in view of Azadbakht and Liberis, teach the method of claim 12, wherein at least two of the implementations for the subsets are searched for at least partly in parallel (Benmeziane: page 18, Section X, "The search runs in parallel on CPUs and GPUs, with an adaptive search strategy for different GPU memory limits"; Benmeziane conducts the search for candidate implementations at least partly in parallel across multiple processors, so that at least two of the implementations for the subsets (at least two of the implementations for the subsets) are searched concurrently).
Regarding dependent claim 19, Benmeziane, in view of Azadbakht and Liberis, teaches the method of claim 12, wherein an implementation of the neural network is determined in which the operators of a subset are implemented with the same implementation from a search space of the subset (Azadbakht: page 2, Section 2 Method, "During the forward pass, isomorphic layers in a sharing group use the same set of trainable parameters (i.e., shared kernels)"; the operators of a sharing group are implemented with one shared implementation drawn from the group's shared-kernel space (a search space of the subset)), wherein the at least one metric or a size is determined during an execution of the implementation on the computing device or in a simulation of the execution of the implementation on the computing device (Benmeziane: page 16, Section VIII, Table VI, "The sampled model is executed on the hardware target while searching", Lookup Table Models, Analytical Estimation, Prediction Model; Benmeziane determines the metric (the at least one metric) during an execution of the implementation on the target hardware (an execution of the implementation on the computing device)).
Regarding dependent claim 20, Benmeziane, in view of Azadbakht and Liberis, teach the method of claim 19, wherein an influence of an implementation of an operator on the size for the implementation of the neural network is determined which includes the implementation of the operator, wherein the influence is stored, and wherein the size for an implementation of the neural network which comprises the same implementation for the same operator is determined depending on the stored influence (Benmeziane: page 16, Section VIII, Table VI, "A lookup table is created beforehand and filled with each operator latency on the targeted hardware. Once the search starts, the system will calculate the overall cost from the lookup table"; Benmeziane precomputes and stores each operator's per-implementation cost contribution (an influence … is determined … the influence is stored) in a lookup table and, for any implementation of the neural network that reuses the same operator implementation, determines the overall cost by summing the stored per-operator contributions (determined depending on the stored influence). Benmeziane thus stores and reuses a per-operator contribution that is additively combined across the operators of the network, and applying this same precompute, store, and reuse technique to a per-operator contribution to the size of an implementation of the neural network, which is likewise determined additively across the operators of the network, would have been obvious to a person of ordinary skill in the art as a predictable use of the stored per-operator contribution technique taught by Benmeziane at page 16, Section VIII, Table VI, so that the influence of an operator implementation on the size is determined once, the influence is stored, and the size for an implementation of the neural network which comprises the same implementation for the same operator is determined depending on the stored influence).
Regarding dependent claim 21, Benmeziane, in view of Azadbakht and Liberis, teaches a device for selecting implementations for operators for a neural network which comprises a set of different operators of an operator type for providing a computing device which includes the neural network, the device comprising: at least one processor; and at least one memory; wherein the device is configured to execute instructions, upon execution of which by the at least one processor, causing the processor to perform the following the steps: verbatim steps recited in claim 12, wherein the at least one memory stores the instructions (Liberis: page 1, Introduction, "MCUs are ultra-small computers with a low-frequency processor, a persistent program memory (Flash) and volatile static RAM …"; Liberis teaches a computing device having a processor (at least one processor) and a persistent program memory and static RAM (at least one memory) that stores the program instructions (the instructions) executed to perform the search). The remaining limitations of claim 21 recite the device-form counterparts of the method steps of claim 12 and are taught for the same reasons set forth above for claim 12. The motivation to combine Benmeziane, Azadbakht, and Liberis is the same as set forth for claim 12.
Regarding dependent claim 22, Benmeziane, in view of Azadbakht and Liberis, teaches a non-transitory computer-readable medium on which is stored a program including instructions for selecting implementations for operators for a neural network which includes a set of different operators of an operator type for providing a computing device which includes the neural network, the instructions, when executed by a computer, causing the computer to perform the following steps (Liberis: page 1, Introduction, "a persistent program memory (Flash)"; Liberis teaches a non-transitory persistent program memory (a non-transitory computer-readable medium) on which the program including instructions (a program including instructions) is stored for execution by the microcontroller's processor (a computer)). The remaining limitations of claim 22 recite the medium-form counterparts of the method steps of claim 12 and are taught for the same reasons set forth above for claim 12. The motivation to combine Benmeziane, Azadbakht, and Liberis is the same as set forth for claim 12.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Houlsby et al., US 2022/0092416 A1 (Mar. 24, 2022) (ABSTRACT Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining neural network architectures. One of the methods includes receiving training data for training a task neural network to perform a particular machine learning task; and selecting, from a space of possible architectures, an architecture for the task neural network, wherein the space of possible architectures is represented as a graph of nodes connected by edges, each node in the graph representing a decision point in selecting the architecture and each edge in the graph representing an action).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KUANG FU CHEN whose telephone number is (571)272-1393. The examiner can normally be reached M-F 9:00-5:30pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached on (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KC CHEN/Primary Patent Examiner, Art Unit 2143
Prosecution Insights