DETAILED ACTION
This action is responsive to Applicant’s reply filed May 18th 2026. This action is made final.
Status of the Claims
Claims 1, 7, 9, 11, 18 and 21-22 are amended.
Claim status is currently pending and under examination for claims 1-7, 9-16 and 18-22 of which independent claims are 1, 11 and 18.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s amendments to the Claims have overcome each and every 101 rejections previously set forth in the Non-Final Office Action mailed January 23rd 2026.
Applicant’s arguments regarding the art rejections are moot in view of the new grounds of rejection necessitated by Applicant’s amendment.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The following are the references relied upon in the rejections below:
Kobayashi (US 20180365557 A1)
Bhardwaj (US 20230196093 A1)
Claims 1-7, 10 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Kobayashi in view of Bhardwaj.
With respect to claim 1, Kobayashi teaches:
A method performed on a computing device, the method comprising ([0006] “there is provided an information processing method including: generating, by a processor, another neural network”):
obtaining a first machine learning model having one or more first inference operations (Kobayashi discloses a seed network (‘a first machine learning model’), “FIG. 1 is a diagram for explaining generation of a neural network by mutation. Referring to FIG. 1, it can be seen that a seed network SN includes 10 layers including “Input” and “Output”. Further, as illustrated in an example in FIG. 1, the neural network according to the present disclosure may include a middle layer, an activating function, or the like, as well as the input and output layers. For example, in the example in FIG. 1. “Conv1” and “Conv2” indicate Convolution layers, and “Pool1” and “Pool2” indicate Max-Pooling” [0056-0057]. See Figure 1.);
identifying a plurality of second inference operations … (Kobayashi discloses “the neural network MN1 is another neural network generated by causing the seed network SN to mutate. Referring to the neural network MN1, it can be seen that part of a layer configuration changes from that in a network structure of the seed network SN. Specifically, in the neural network MN1, an activating function “relu1” relating to the seed network SN changes to another activating function “Tanh1”. In this manner, with the information processing method according to the present disclosure, by changing layer types of layers constituting a network structure, it is possible to generate another neural network with a different network structure” [0058]. See Figure 1.);
generating second machine learning models by replacing individual first inference operations of the first machine learning model with individual second inference operations … (Kobayashi discloses “the generating unit 310 randomly determines a generation method of another neural network to be applied to the original neural network (S1101). In this event, the original neural network may be the seed network designated by the user or may be a network randomly selected by the generating unit 310 from neural networks relating to Pareto optimal solutions updated by the evaluating unit 320. The generating unit 310 then generates another neural network with a different network structure from the original neural network on the basis of the generation method selected in step S1101. Referring to an example illustrated in FIG. 5, the generating unit 310 according to the present embodiment may generate the above-described another neural network by causing the original neural network to mutate (S1102)” [0085-0086].
Kobayashi discloses “mutation according to the present embodiment may include insertion of a layer, deletion of a layer, change of a layer type, change of a parameter, a graph branch and deletion of a graph branch. Referring to FIG. 6, first, the generating unit 310 randomly determines a method of mutation to be applied to the original neural network (S1201). Subsequently, the generating unit 310 changes a network structure of the original neural network on the basis of the method selected” [0093-0094]. See [0095-0098] discussing the process of performing mutations.);
executing the second machine learning models having the individual second inference operations (Kobayashi discloses “the evaluating unit 320 has a function of acquiring an evaluation result of the generated neural network. The evaluating unit 320 may acquire the above-described evaluation result by, for example, causing a computing resource on cloud to execute the generated neural network. Further, the evaluating unit 320 may acquire the evaluation result by causing an emulator or various kinds of devices connected via the network 20 to execute the neural network” [0079-0080].);
based at least on the executing, determining one or more metrics that characterize performance of the second machine learning models … (Kobayashi discloses “the evaluating unit 320 has a function of acquiring an evaluation result of the generated neural network. The evaluating unit 320 may acquire the above-described evaluation result by, for example, causing a computing resource on cloud to execute the generated neural network. Further, the evaluating unit 320 may acquire the evaluation result by causing an emulator or various kinds of devices connected via the network 20 to execute the neural network. Further, the evaluation result acquired by the evaluating unit 320 may include a calculation amount relating to the generated neural network and at least one of a training error and a validation error (hereinafter, the training error and the validation error may be collectively expressed as an error” [0079-0080].);
selecting a final machine learning model from the second machine learning models based at least on the one or more metrics (Kobayashi discloses Figure 9 (reproduced below) depicting Pareto optimal solutions (labeled P4 – P6). Pareto optimal solutions depict the tradeoff between error and calculation amount (performance) metrics.
PNG
media_image1.png
753
1084
media_image1.png
Greyscale
Kobayashi further discloses “trade-off information relating to an error and a calculation amount is presented to the user, the trade-off information according to the present embodiment is not limited to such an example. In the trade-off information according to the present embodiment, for example, memory usage, an amount of heat generation, power consumption, or the like, relating to hardware may be used as well as the calculation amount. Further, in the trade-off information, total cost of hardware calculated from the calculation amount, total service cost, or the like, including server cost, or the like, may be used” [0116].
Kobayashi discloses “it is possible to present a candidate selected from the neural networks relating to the Pareto optimal solutions to the user. Here, the above-described candidate may include a network relating to maximum performance, a network relating to an intermediate solution and a network relating to a minimum calculation amount. Further, with the information processing method according to the present embodiment, it is possible to allow the user to download a file relating to execution of a network. By this means, the user can easily select a network which satisfies conditions and acquire a file relating to execution of the network” [0125].);
and outputting the final machine learning model for subsequent execution on devices … (See [0125] describing a user can download and execute a file of a selected neural network (‘final machine learning model’) that satisfies conditions.).
However, Kobayashi does not teach identifying second inference operations that are supported by multiply and accumulate circuitry of an inference hardware architecture and replacing individual first inference operations with second inference operations, which is taught by Bhardwaj:
identifying a plurality of second inference operations that are supported by multiply and accumulate circuitry of an inference hardware architecture ([0057] “the second (reduced) format version of a block thus includes a set of one or more operations of a (or the) type that the hardware accelerator is configured to perform, such that the hardware accelerator is operable to execute the model including the blocks in the second (reduced) format in a more efficient or optimised manner, e.g. with higher utilisation of the hardware processing circuitry (e.g. higher utilisation of an NPU's MAC units) than if using (executing) the blocks in the first, non-reduced format.” [0016] “the NPU 306 may comprise a plurality of multiply-accumulate circuits 306A (otherwise known as a multiplier-accumulators, or “MAC units”) which are configured to perform MAC operations. The NPU 306 may be optimised for performing certain types of operation that are commonly used when processing a neural network, e.g. such that there is higher MAC unit utilisation when performing these types of operation.”
Reduced format blocks (‘second inference operations’) are blocks that execute a neural network model using a higher MAC unit utilization (‘multiply and accumulate circuitry’) of a NPU (‘inference hardware architecture’), therefore reduced format blocks are second inference operations that are supported by multiply and accumulate circuitry of an inference hardware architecture.);
generating [a] second machine learning [model] by replacing individual first inference operations of the first machine learning model with individual second inference operations that are supported by the multiply and accumulate circuitry of the inference hardware architecture ([0047] “replacing one or more of the first format blocks in the trained neural network with corresponding blocks in a second, different (reduced) format to generate a second version of the neural network for execution, the second version of the neural network thus including a respective one or more blocks in the second format, wherein the blocks in the second format when executed replicate the result of the sequence of processing operations for the corresponding first format blocks in the trained neural network”
A second version (‘second machine learning model’) of a trained neural network (‘first machine learning model’) is generated by replacing first format blocks (‘individual first inference operations’) of the trained neural network with reduced format blocks (‘individual second inference operations that are supported by the multiply and accumulate circuitry of the inference hardware architecture’).);
executing the second machine learning [model] having the individual second inference operations ([0099] “That is, once the neural network has been trained appropriately for the desired neural network processing task using the blocks in the first (full) format, rather than simply deploying the as-trained version of the neural network for execution, one or more (or all) of the ‘reducible’ inverted bottleneck blocks are then replaced with corresponding reduced format versions of the blocks for execution. The model including the reduced format version of the neural network can then be executed as desired in order to perform a neural network processing task.”);
based at least on the executing, determining one or more metrics that characterize performance of the second machine learning [model] when executed on the multiply and accumulate circuitry of the inference hardware architecture ([0067] “it is possible to then replace the full format version of the neural network with a reduced format version for execution, without loss of accuracy.”
[0127] “using the new types of blocks that can be reduced in the manner of the technology described herein, it is therefore possible to provide a more hardware-friendly model (that can be executed more efficiently by a dedicated hardware accelerator, e.g. NPU) whilst still achieving higher accuracy, and without increasing the size of the neural network.”);
and outputting the final machine learning model for subsequent execution on devices that have the multiply and accumulate circuitry of the inference hardware architecture ([0271] “The reduced format version of the neural network is then provided for output (for execution) (step 802). An executable model is thus generated that includes the reduced format version of the neural network (as well as suitable neural network parameters (weights) for the neural network). The model can then be executed as desired to perform neural network processing tasks (step 803). For example, the model may be trained to perform an image classification task. The trained model may then be deployed and executed as desired on an input image in order to perform the classification task.”
[0017] “A benefit of providing an NPU is therefore that at least these types of operations can then be performed in a more optimised manner, e.g. compared to using another processor (e.g. the CPU) to perform the calculations in a general purpose manner. This also then frees up other components (e.g. the host processor (CPU)) to perform other processing tasks, as desired, which may improve the overall processing efficiency. This can be particularly important for resource constrained devices, such as mobile devices, where the CPU resource may be limited.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to combine the method of Kobayashi with the technique disclosed by Bhardwaj to replace operations in a neural network with operations that enable higher utilization of MAC operations. By replacing operations in a neural network with operations that enable higher utilization of MAC operations, a neural network that can be executed more efficiently can be generated, thereby enabling a neural network to be deployed on resource-constrained devices.
With respect to claim 2, the combined method of Kobayashi/Bhardwaj teaches:
the method of claim 1, wherein the one or more metrics relate to losses or accuracy of the second machine learning models (Kobayashi discloses “the evaluation result acquired by the evaluating unit 320 may include a calculation amount relating to the generated neural network and at least one of a training error and a validation error (hereinafter, the training error and the validation error may be collectively expressed as an error). The evaluating unit 320 can acquire the above-described calculation amount on the basis of a network structure of the generated neural network. Further, the evaluating unit 320 has a function of updating a Pareto optimal solution relating to the evaluated neural network on the basis of the evaluation result of the generated neural network. That is, the evaluating unit 320 acquires the evaluation result of the neural network generated by the generating unit 310 and repeatedly executes updating of the Pareto optimal solution on the basis of the evaluation result” [0080-0081]. See Figure 9 (reproduced above) and explanation discussing Pareto optimal solutions.).
With respect to claim 3, the combined method of Kobayashi/Bhardwaj teaches:
the method of claim 1, wherein the one or more metrics relate to latencies, power consumption, or memory utilization of the second machine learning models (Kobayashi discloses “the evaluation result acquired by the evaluating unit 320 may include a calculation amount relating to the generated neural network and at least one of a training error and a validation error (hereinafter, the training error and the validation error may be collectively expressed as an error). The evaluating unit 320 can acquire the above-described calculation amount on the basis of a network structure of the generated neural network. Further, the evaluating unit 320 has a function of updating a Pareto optimal solution relating to the evaluated neural network on the basis of the evaluation result of the generated neural network. That is, the evaluating unit 320 acquires the evaluation result of the neural network generated by the generating unit 310 and repeatedly executes updating of the Pareto optimal solution on the basis of the evaluation result” [0080-0081].
Kobayashi further discloses “trade-off information relating to an error and a calculation amount is presented to the user, the trade-off information according to the present embodiment is not limited to such an example. In the trade-off information according to the present embodiment, for example, memory usage, an amount of heat generation, power consumption, or the like, relating to hardware may be used as well as the calculation amount. Further, in the trade-off information, total cost of hardware calculated from the calculation amount, total service cost, or the like, including server cost, or the like, may be used” [0116]. See Figure 9 (reproduced above) and explanation discussing how Pareto optimal solutions depict trade-offs.).
With respect to claim 4, the combined method of Kobayashi/Bhardwaj teaches:
the method of claim 1, further comprising: simulating execution of the second machine learning models on a central processing unit to determine the one or more metrics (Kobayashi discloses “the evaluating unit 320 has a function of acquiring an evaluation result of the generated neural network. The evaluating unit 320 may acquire the above-described evaluation result by, for example, causing a computing resource on cloud to execute the generated neural network. Further, the evaluating unit 320 may acquire the evaluation result by causing an emulator or various kinds of devices connected via the network 20 to execute the neural network. Further, the evaluation result acquired by the evaluating unit 320 may include a calculation amount relating to the generated neural network and at least one of a training error and a validation error (hereinafter, the training error and the validation error may be collectively expressed as an error” [0079-0080].).
With respect to claim 5, the combined method of Kobayashi/Bhardwaj teaches:
the method of claim 1, further comprising: determining a frontier of the second machine learning models with respect to multiple metrics (Kobayashi discloses Figure 9 (reproduced above) depicting Pareto optimal solutions on a boundary (‘frontier’). As explained above, Pareto optimal solutions represent the trade-off between error and calculation amount (‘multiple metrics’) of the generated neural networks.
Kobayashi further discloses “the boundary PL of the Pareto optimal solution illustrated in FIG. 8B is updated on the basis of the evaluation result of the neural network generated by the generating unit 310. In the example illustrated in FIG. 8B, validation errors P1 to P3 of neural networks relating to new Pareto optimal solutions are displayed on the boundary PL of the Pareto optimal solution” [0112].);
and selecting the final machine learning model from the frontier (Kobayashi discloses “it is possible to present a candidate selected from the neural networks relating to the Pareto optimal solutions to the user. Here, the above-described candidate may include a network relating to maximum performance, a network relating to an intermediate solution and a network relating to a minimum calculation amount. Further, with the information processing method according to the present embodiment, it is possible to allow the user to download a file relating to execution of a network. By this means, the user can easily select a network which satisfies conditions and acquire a file relating to execution of the network” [0125].).
With respect to claim 6, the combined method of Kobayashi/Bhardwaj teaches:
the method of claim 1, further comprising: performing two or more iterations of selecting a subset of the second machine learning models for further modification and generating further second machine learning models from the selected subset (Kobayashi discloses “the evaluating unit 320 can acquire an evaluation result of the generated neural network and update the Pareto optimal solution on the basis of the evaluation result. Further, the generating unit 310 may generate another network on the basis of the neural network randomly selected from the neural networks P1 to P3 relating to the Pareto optimal solutions updated by the evaluating unit 320. That is, with the information processing method according to the present embodiment, another neural network is generated from a neural network relating to a Pareto optimal solution, and updating of the Pareto optimal solution based on the evaluation of the other neural network is repeatedly executed” [0113].).
With respect to claim 7, the combined method of Kobayashi/Bhardwaj teaches:
the method of claim 1, wherein generating an individual second machine learning model comprises removing, from the first machine learning model, an individual first inference operation … and replacing the individual first inference operation with a particular second inference operation … (Kobayashi discloses “mutation according to the present embodiment may include insertion of a layer, deletion of a layer, change of a layer type, change of a parameter, a graph branch and deletion of a graph branch. Referring to FIG. 6, first, the generating unit 310 randomly determines a method of mutation to be applied to the original neural network (S1201). Subsequently, the generating unit 310 changes a network structure of the original neural network on the basis of the method selected” [0093-0094].).
However, Kobayashi does not teach replacing inference operations that are not supported by corresponding multiply and accumulate circuitry of the inference hardware architecture, which is taught by Bhardwaj:
wherein generating an individual second machine learning model comprises removing, from the first machine learning model, an individual first inference operation that is not supported by corresponding multiply and accumulate circuitry of the inference hardware architecture ([0063] “a set of first plural processing operations defined for the blocks in the first format (for training) is replaced in the second format version of the block with a corresponding set of second plural processing operations, wherein the number of processing operations in the second format version of the block is reduced compared to the first format version of the block.”
[0082] “the sequence of convolution operations represented by an inverted bottleneck block can be replaced in the output model (i.e. the model that is used at execution time) with a (reduced) set of one or more “regular” convolution operations of a type that can be more efficiently processed (e.g. regular 3×3 convolutions).”
[0057] “the second (reduced) format version of a block thus includes a set of one or more operations of a (or the) type that the hardware accelerator is configured to perform, such that the hardware accelerator is operable to execute the model including the blocks in the second (reduced) format in a more efficient or optimised manner, e.g. with higher utilisation of the hardware processing circuitry (e.g. higher utilisation of an NPU's MAC units) than if using (executing) the blocks in the first, non-reduced format.”
Reduced format blocks (‘second inference operations’) are blocks that execute a neural network model using a higher MAC unit utilization (‘multiply and accumulate circuitry’) of a NPU (‘inference hardware architecture’), therefore reduced format blocks are second inference operations that are supported by multiply and accumulate circuitry of an inference hardware architecture. First format blocks (‘individual first inference operations’) are blocks that are not efficiently executed on an NPU, therefore first format blocks are not supported by corresponding multiply and accumulate circuitry of the inference hardware architecture. An output model (‘individual second machine learning model’) is generated by replacing a first format block with a reduced format block, thereby removing from a first machine learning model an individual first inference operation.)
and replacing the individual first inference operation with a particular second inference operation that is supported by corresponding multiply and accumulate circuitry of the inference hardware architecture ([0098] Thus, according to the technology described herein, an inverted bottleneck block may be (and at least some of the inverted bottleneck blocks are) replaced in the version of the model that is provided for output (for execution) with a corresponding reduced format version of the block that similarly processes a set of k input features into a respective set of k′ output features, but using a set of one or more “regular” convolution operations (e.g. regular 3×3 convolutions) of a type than can be more efficiently executed in hardware, i.e. without expanding the features to a different dimension.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the combined method of Kobayashi/Bhardwaj with the technique disclosed by Bhardwaj to replace operations in a neural network with operations that enable higher utilization of MAC operations. By replacing operations in a neural network with operations that enable higher utilization of MAC operations, a neural network that can be executed more efficiently on target hardware can be generated, thereby enabling a neural network to be deployed on resource-constrained devices.
With respect to claim 10, the combined method of Kobayashi/Bhardwaj teaches:
the method of claim 1, further comprising: outputting multiple final machine learning models selected according to different metrics (Kobayashi discloses “FIG. 9 is a diagram illustrating a configuration example of a form to be presented to the user when search of a network structure is finished. Referring to FIG. 9, a form F1 in which a search result is displayed includes a region V1 for displaying a Pareto optimal solution and a region V2 for displaying outline of the evaluation result. Here, referring to region V1, in an example illustrated in FIG. 9, it can be seen that neural networks P4 to P6 relating to three Pareto optimal solutions are highlighted in addition to a state of the Pareto optimal solutions illustrated in FIG. 8C. Here, the neural networks P4 to P6 may be respectively a network relating to maximum performance, a network relating to an intermediate solution and a network relating to a minimum calculation amount. In this event, the neural network P4 may be a network with the least error among the found networks” [0117-0119]. See Figure 9 (reproduced above).).
With respect to claim 18, Kobayashi teaches:
a computing device comprising ([0069] “the information processing apparatus 10 according to the present disclosure includes a display unit 110, an input unit 120, a form control unit 130 and a server communication unit 140”):
a hardware processing unit configured to execute a plurality of … inference operations … ([0202] information processing apparatus includes CPU
[0217] “generation of a neural network and updating of a Pareto optimal solution may be realized by the information processing apparatus 10.”);
and a storage resource storing computer-readable instructions which, when executed by the hardware processing unit, cause the hardware processing unit to ([0205] “the ROM 872 is a device that stores programs read by the CPU 871, data used for operations, and the like. For example, a program read by the CPU 871, various kinds of parameters that appropriately change when the program is executed, and the like are temporarily or permanently stored in the RAM 873”):
determine a device context for the computing device ([0195-0196] “the user may be able to designate a target to be optimized by selecting the optimization target (Optimize for). For example, the user may be able to designate optimization of both the learning accuracy and the calculation amount or may designate optimization of one of the learning accuracy and the calculation amount. By designating the optimization target, the user can obtain a search result which matches application. … The user can prevent search of a network for which a calculation amount is too large or a network for which learning accuracy is too low by designating the above-described search range.”
[0201] “search according to the present disclosure may be controlled on the basis of, for example, the number of times of trial of search designated by the user, and limit information of memory usage, or the like, relating to hardware in which a neural network is implemented. Setting of search according to the present disclosure can be changed as appropriate in accordance with specifications and operation relating to a neural network”
See Figure 20 illustrating a screen a user can use to configure neural network search settings.);
based at least on the device context, select a particular machine learning model from a plurality of machine learning models available to the computing device, the plurality of machine learning models having different inference operations … (Kobayashi discloses “in search of a network structure according to the present disclosure, various settings by the user may be accepted. FIG. 20 is an example of a setting screen relating to search of the present disclosure. Here, the example illustrated in FIG. 20 may be an example of a screen displayed at the display unit 110 of the information processing apparatus 10. Referring to FIG. 20, the setting screen relating to search of the present disclosure may include, for example, setting items relating to a search method, an optimization target, a search range, early stopping and time limit” [0192-0193].
Kobayashi discloses Figure 9 (reproduced above) depicting a plurality of Pareto optimal solutions (labeled P4 – P6). Pareto optimal solutions depict the tradeoff between error and calculation amount (performance) metrics. Each Pareto optimal solution is derived from evaluating a generated neural network.
Kobayashi discloses “it is possible to present a candidate selected from the neural networks relating to the Pareto optimal solutions to the user. Here, the above-described candidate may include a network relating to maximum performance, a network relating to an intermediate solution and a network relating to a minimum calculation amount. Further, with the information processing method according to the present embodiment, it is possible to allow the user to download a file relating to execution of a network. By this means, the user can easily select a network which satisfies conditions and acquire a file relating to execution of the network” [0125].);
and execute the particular machine learning model … to perform a particular task ([0125] “it is possible to allow the user to download a file relating to execution of a network. By this means, the user can easily select a network which satisfies conditions and acquire a file relating to execution of the network”).
However, Kobayashi does not teach a hardware processing unit configured to execute a plurality of supported inference operations using multiply and accumulate circuitry and a machine learning model having different inference operations supported by multiply and accumulate circuitry, which is taught by Bhardwaj:
a hardware processing unit configured to execute a plurality of supported inference operations using multiply and accumulate circuitry on the hardware processing unit ([0057] “the second (reduced) format version of a block thus includes a set of one or more operations of a (or the) type that the hardware accelerator is configured to perform, such that the hardware accelerator is operable to execute the model including the blocks in the second (reduced) format in a more efficient or optimised manner, e.g. with higher utilisation of the hardware processing circuitry (e.g. higher utilisation of an NPU's MAC units) than if using (executing) the blocks in the first, non-reduced format.”
Reduced format blocks (‘supported inference operations’) are blocks that execute a neural network model using a higher MAC unit utilization (‘multiply and accumulate circuitry’) of a NPU (‘hardware processing unit’), therefore reduced format blocks are inference operations that are supported by multiply and accumulate circuitry of a hardware processing unit.);
based at least on the device context, select a particular machine learning model … available to the computing device ([0021] “One example of a convolutional neural network that is particularly efficient for mobile and other resource-constrained environments is the so-called “MobileNetV2” architecture”
[0082] “the overall neural network structure is similar in design to MobileNetV2 (which is based on a sequence of IBN blocks, as described above) but at least some of the IBN blocks in the MobileNetV2 model are in an embodiment replaced with ‘reducible’ inverted bottleneck block”
[0046] “training the neural network to perform a neural network processing task, wherein training the neural network comprises training a first version of the neural network, the first version of the neural network including one or more blocks in a first format, wherein each block in the first format comprises a respective sequence of processing operations;”
A first version of a neural network (‘particular machine learning model’) is trained and then has some of its IBN blocks replaced with reduced blocks (supported inference operations). The neural network is designed for mobile and resource-constrained environments (‘device context’), therefore when training a first version of a neural network, the neural network is designed (selected) based on the resource constraints (device context) available to mobile environments.),
the … machine learning [model] having different inference operations supported by the multiply and accumulate circuitry on the hardware processing unit ([0047] “after training the first version of the neural network: replacing one or more of the first format blocks in the trained neural network with corresponding blocks in a second, different (reduced) format to generate a second version of the neural network for execution, the second version of the neural network thus including a respective one or more blocks in the second format, wherein the blocks in the second format when executed replicate the result of the sequence of processing operations for the corresponding first format blocks in the trained neural network”);
and execute the particular machine learning model using the multiply and accumulate circuitry to perform a particular task ([0271] “The reduced format version of the neural network is then provided for output (for execution) (step 802). An executable model is thus generated that includes the reduced format version of the neural network (as well as suitable neural network parameters (weights) for the neural network). The model can then be executed as desired to perform neural network processing tasks (step 803). For example, the model may be trained to perform an image classification task. The trained model may then be deployed and executed as desired on an input image in order to perform the classification task.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to combine the method of Kobayashi with the technique disclosed by Bhardwaj to generate a neural network with operations supported by multiply and accumulate circuitry. By generating a neural network with operations supported by multiply and accumulate circuitry, a neural network can be comprised of operations that highly utilize the multiply and accumulate circuitry of an NPU, thereby allowing the neural network to execute more efficiently.
With respect to claim 19, the combined method of Kobayashi/Bhardwaj teaches:
the computing device of claim 18, the device context relating to availability of power or memory on the computing device (Kobayashi discloses “search according to the present disclosure may be controlled on the basis of, for example, the number of times of trial of search designated by the user, and limit information of memory usage, or the like, relating to hardware in which a neural network is implemented. Setting of search according to the present disclosure can be changed as appropriate in accordance with specifications and operation relating to a neural network” [0201]).
With respect to claim 20, the combined method of Kobayashi/Bhardwaj teaches:
the computing device of claim 19, wherein the computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to: in a first instance when availability of memory for the computing device is constrained, select a first machine learning model as the particular machine learning model to execute to perform the particular task, the first machine learning model having been generated based at least on a first metric relating to memory utilization (The Examiner interprets “in a first instance when availability of memory for the computing device is constrained” according to its broadest reasonable interpretation as encompassing limiting memory usage as disclosed by Kobayashi.
Kobayashi discloses “note that search of a network according to the present disclosure may be controlled through setting other than the above-described examples. Search according to the present disclosure may be controlled on the basis of, for example, the number of times of trial of search designated by the user, and limit information of memory usage, or the like, relating to hardware in which a neural network is implemented” [0125].
Kobayashi discloses “note that, while, in the above description, a case has been described as an example where trade-off information relating to an error and a calculation amount is presented to the user, the trade-off information according to the present embodiment is not limited to such an example. In the trade-off information according to the present embodiment, for example, memory usage, an amount of heat generation, power consumption, or the like, relating to hardware may be used as well as the calculation amount. Further, in the trade-off information, total cost of hardware calculated from the calculation amount, total service cost, or the like, including server cost, or the like, may be used. Still further, switching of the above-described items may be realized by user selection. The evaluating unit 320 can calculate the above-described values on the basis of information relating to hardware and service, which is stored in advance” [0116].
Kobayashi discloses Figure 9 (reproduced above) depicting a plurality of Pareto optimal solutions (labeled P4 – P6). Each Pareto optimal solution depicts the tradeoff between two evaluated metrics of a generated neural network.
See [0125] discussing how a user can select a neural network from Pareto optimal solutions based on whether a neural network satisfies given conditions.);
and in a second instance when availability of power to the computing device is constrained, select a second machine learning model as the particular machine learning model to execute to perform the particular task, the second machine learning model having been generated based at least on a second metric relating to power consumption (Kobayashi discloses power consumption can be a metric in a trade-off relationship, “note that, while, in the above description, a case has been described as an example where trade-off information relating to an error and a calculation amount is presented to the user, the trade-off information according to the present embodiment is not limited to such an example. In the trade-off information according to the present embodiment, for example, memory usage, an amount of heat generation, power consumption, or the like, relating to hardware may be used as well as the calculation amount. Further, in the trade-off information, total cost of hardware calculated from the calculation amount, total service cost, or the like, including server cost, or the like, may be used. Still further, switching of the above-described items may be realized by user selection. The evaluating unit 320 can calculate the above-described values on the basis of information relating to hardware and service, which is stored in advance” [0116].
The Examiner interprets “in a second instance when availability of power to the computing device is constrained” according to its broadest reasonable interpretation as suppressing power consumption as disclosed by Kobayashi.
Kobayashi discloses that a metric in a trade-off relationship is suppressed, which as previously discussed would include power consumption, “however, because a calculation amount largely affects memory usage and execution time of hardware in which a neural network is mounted, a neural network with high learning accuracy is not always the best neural network. In other words, in a neural network, a calculation amount and learning accuracy have, so-called, trade-off relationship. Therefore, a method for searching for a network structure with higher learning accuracy while suppressing a calculation amount has been desired” [0051].
Kobayashi discloses Figure 9 (reproduced above) depicting a plurality of Pareto optimal solutions (labeled P4 – P6). Each Pareto optimal solution depicts the tradeoff between two evaluated metrics of a generated neural network.
See [0125] discussing how a user can select a neural network from Pareto optimal solutions based on whether a neural network satisfies given conditions.).
The following are the references relied upon in the rejections below:
Stehle, Kevin, Günther Schindler, and Holger Fröning. "On the difficulty of designing processor arrays for deep neural networks." International Workshop on IoT, Edge, and Mobile for Embedded Machine Learning. Cham: Springer International Publishing, 2020.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Kobayashi in view of Bhardwaj, further in view of Stehle.
With respect to claim 9, the combined method of Kobayashi/Bhardwaj teaches:
the method of claim of 1, further comprising: obtaining respective [operation] metrics via the hardware emulation (Kobayashi discloses “the evaluating unit 320 may acquire the evaluation result by causing an emulator or various kinds of devices connected via the network 20 to execute the neural network. Further, the evaluation result acquired by the evaluating unit 320 may include a calculation amount relating to the generated neural network and at least one of a training error and a validation error (hereinafter, the training error and the validation error may be collectively expressed as an error” [0079-0080].);
and using the respective [operation] metrics to select individual second machine learning models as parent models for further modification or to select the final machine learning model (Kobayashi discloses “it is possible to acquire an evaluation result of a generated neural network and update a Pareto optimal solution relating to the evaluated neural network on the basis of the acquisition result.” [0061].
Kobayashi discloses “it is possible to present a candidate selected from the neural networks relating to the Pareto optimal solutions to the user. Here, the above-described candidate may include a network relating to maximum performance, a network relating to an intermediate solution and a network relating to a minimum calculation amount. Further, with the information processing method according to the present embodiment, it is possible to allow the user to download a file relating to execution of a network. By this means, the user can easily select a network which satisfies conditions and acquire a file relating to execution of the network” [0125].).
However, the combined method of Kobayashi/Bhardwaj does not teach obtaining per-operation metrics via hardware emulation of multiply and accumulate circuitry, which is taught by Stehle:
obtaining respective per-operation metrics via hardware emulation of the multiply and accumulate circuitry of the inference hardware architecture ((P. 232, Sec. 3, ¶1) “This work is primarily concerned with design space explorations, in order to quickly assess the suitability of a given architecture configuration for a particular DNN or DNN mix. For such an exploration it is sufficient to assess this suitability based on a set of abstract metrics, thus emulation is chosen as underlying method, which is in particular fast in comparison to simulations.”
(P. 233-234, Sec. 3, Last Paragraph) “To simplify the integration of the emulator into existing machine learning frameworks, we implemented a wrapper library that dynamically creates emulator instances of certain configurations (bit widths for weights, input and output activations, array dimensions, and accumulator array size).”
(P. 232, Sec. 3, Last Paragraph) “The core is a weight-stationary Systolic Array of parameterized width n and height m. …. Its PEs perform MAC operations using partial sums, weights, and activations”
(P. 234, Sec. 4.1, ¶1) “The emulator can guide accelerator developers while searching for optimal systolic array configurations for their respective applications. In this section, we exemplify the process of finding such optimal configurations using Pareto optimum on the example of a ResNet-152 model with input images. The Pareto optimum is calculated for data movement cost and utilization, both with respect to the total number of cycles required for inference”);
and using the respective per-operation metrics to select individual second machine learning models as parent models for further modification or to select the final machine learning model ((P. 232, Sec. 2, Last Paragraph) “The tool aims to assist deep learning experts to develop neural architectures that fit well onto a certain processor array and hardware experts to design processor arrays based on certain neural architectures”
(P. 235, Sec. 4.2, ¶1) “We conduct further performance analyses (as described in Subsect. 4.1) for a wide range of CNN models to show the impact of evolving neural architectures on parameter optimality of systolic array configurations”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the combined method of Kobayashi/Bhardwaj with the emulation technique disclosed by Stehle to use emulation to obtain metrics of each operation. By using emulation to obtain metrics of each operation, an optimal neural network architecture can be developed that runs efficiently on a target hardware, thereby resulting in a neural network that has low latency.
The following are the references relied upon in the rejections below:
Maiyuran et al. (US 20210073318 A1)
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Kobayashi in view of Bhardwaj, further in view of Maiyuran.
With respect to claim 21, the combined method of Kobayashi/Bhardwaj teaches:
the method of claim 1, the inference hardware architecture … configured to perform multiply and accumulate operations … (Kobayashi discloses “in FIG. 8A to FIG. 8C, a case will be described as an example where trade-off information relating to a calculation amount of an objective function and an error rate is presented. Therefore. FIG. 8A to FIG. 8C indicate an error rate on a vertical axis and indicate a calculation amount of an objective function on a horizontal axis. Further, in FIG. 8A to FIG. 8C, the number of times of multiply add is employed as an example relating to the calculation amount of the objective function” [0109].),
a first convolution operation with a first input tensor size, a first output tensor size, and a first kernel size (Kobayashi discloses convolution layer Conv1 (‘first convolution operation’), “a kernel shape relating to “Conv1” is changed from 5 (vertical)×5 (horizontal) of the seed network SN to 4 (vertical)×8 (horizontal). Further, in the neural network MN3 after search, a pool shape relating to “Pool2” is changed from 2 (vertical)×2 (horizontal) of the seed network SN to 2 (vertical)×4 (horizontal)” [0128-0129].
Kobayashi discloses Figure 10A (reproduced below) depicting a mutated neural network generated after replacing convolution layers of an original seed network. Convolution layer Conv1 with a kernel shape (‘first kernel size’) of 4x8 replaces the original convolution layer Conv1 of the seed network. Input and output tensors are implied by convolution layer Conv1 having output maps, which further implies input and output tensor sizes.
PNG
media_image2.png
688
675
media_image2.png
Greyscale
),
a second convolution operation with a second input tensor size, a second output tensor size, and a second kernel size (Kobayashi discloses Figure 10A (reproduced above) depicting a mutated neural network generated after replacing convolution layers of an original seed network. Convolution layer Conv2 (‘second convolution operation’) with a kernel shape (‘second kernel size’) of 3x3 replaces the original convolution layer Conv2 of the seed network. Input and output tensors are implied by convolution layer Conv2 having output maps, which further implies input and output tensor sizes. See Figure 1 depicting seed network SN with original Conv2 having a kernel shape of 5x5.),
wherein the first machine learning model includes a third convolution operation having a third input tensor size, a third output tensor size, and a third kernel size (Kobayashi discloses convolution layer Conv1 (‘third convolution operation’) of a seed network (‘first machine learning model’), “a seed network SN includes 10 layers including “Input” and “Output”. Further, as illustrated in an example in FIG. 1, the neural network according to the present disclosure may include a middle layer, an activating function, or the like, as well as the input and output layers. For example, in the example in FIG. 1. “Conv1” and “Conv2” indicate Convolution layers, and “Pool1” and “Pool2” indicate Max-Pooling. Therefore, in “Conv1” and “Conv2”, parameters such as kernel shapes and the number of output maps are displayed, and in “Pool1” and “Pool2”, parameters indicating pool shapes are displayed” [0056-0057].
Input and output tensors are implied by original convolution layer Conv1 (before mutation) of a seed model having output maps, which further implies input and output tensor sizes. See Figure 1 depicting Conv1 with a kernel shape (‘third kernel size’) and output maps.)
and the final machine learning model is generated by replacing the third convolution operation with at least one of the first convolution operation or the second convolution operation (Kobayashi discloses “Referring to FIG. 10A, it can be seen that, in the neural network MN3 after search, the number of parameters relating to “Conv1” and “Pool2” changes compared to that in the seed network SN. Specifically, in the neural network MN3 after search, a kernel shape relating to “Conv1” is changed from 5 (vertical)×5 (horizontal) of the seed network SN to 4 (vertical)×8 (horizontal). Further, in the neural network MN3 after search, a pool shape relating to “Pool2” is changed from 2 (vertical)×2 (horizontal) of the seed network SN to 2 (vertical)×4 (horizontal)” [0128-0129].
Kobayashi discloses Figure 10A (reproduced above) depicting a mutated neural network (‘final machine learning model’) generated after changing (‘replacing’) convolution layers of an original seed network. Convolution layer Conv1 (‘first convolution operation’) with a kernel shape of 4x8 replaces the original convolution layer Conv1 (‘third convolution operation’) of the seed network.).
However, Kobayashi does not teach using multiply and accumulate circuitry to perform convolution operations which is taught by Bhardwaj:
perform first multiply and accumulate operations using the multiply and accumulate circuitry ([0057] “the second (reduced) format version of a block thus includes a set of one or more operations of a (or the) type that the hardware accelerator is configured to perform, such that the hardware accelerator is operable to execute the model including the blocks in the second (reduced) format in a more efficient or optimised manner, e.g. with higher utilisation of the hardware processing circuitry (e.g. higher utilisation of an NPU's MAC units) than if using (executing) the blocks in the first, non-reduced format.”
Reduced format blocks are blocks that execute a neural network model using a higher MAC unit utilization (multiply and accumulate circuitry) of a NPU, therefore reduced format blocks are multiply and accumulate operations that are executed using multiply and accumulate circuitry.),
the first multiply and accumulate operations corresponding to a first convolution operation … ([0062] “In the second (reduced) format, the first (full) set of tensor multiplications or convolutions may be analytically collapsed into a second (reduced) set of tensor multiplications or convolutions”),
perform second multiply and accumulate operations using the multiply and accumulate circuitry ([0099] “That is, once the neural network has been trained appropriately for the desired neural network processing task using the blocks in the first (full) format, rather than simply deploying the as-trained version of the neural network for execution, one or more (or all) of the ‘reducible’ inverted bottleneck blocks are then replaced with corresponding reduced format versions of the blocks for execution. The model including the reduced format version of the neural network can then be executed as desired in order to perform a neural network processing task”),
the second multiply and accumulate operations corresponding to a second convolution operation … (Inverted bottleneck blocks (‘second convolution operation’) are replaced with reduced format blocks (multiply and accumulate operations that are executed using multiply and accumulate circuitry) for execution.),
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the combined method of Kobayashi/Bhardwaj with the technique disclosed by Bhardwaj to execute convolution operations on MAC circuitry. By executing convolution operations on MAC circuitry, a neural network can be executed more efficiently, thereby enabling machine learning tasks to be performed on resource-constrained devices.
Furthermore, the combination of Kobayashi/Bhardwaj does not each a systolic array configured to perform multiply and accumulate operations on input data using parallel nodes, which is taught by Maiyuran:
the inference hardware architecture comprising a systolic array configured to perform multiply and accumulate operations on input data using parallel nodes ([Abstract] “apparatus comprises a systolic array including matrix multiplication hardware to perform multiply-add operations on received matrix data comprising data from a plurality of input matrices and sparse matrix acceleration hardware to detect zero values in the matrix data and perform one or more optimizations on the matrix data to reduce multiply-add operations to be performed by the matrix multiplication hardware”
[0124] “the compute unit 610 can also include a systolic array 612, and a math unit 613. The systolic array 612 includes a W wide and D deep network of data processing units that can be used to perform vector or other data-parallel operations in a systolic manner. In one embodiment the systolic array 612 can be configured to perform matrix operations, such as matrix dot product operations”),
wherein the systolic array is configured to: in response to receiving a first opcode, perform first multiply and accumulate operations using the multiply and accumulate circuitry ([0129] “instruction opcode 712 defines the operation that the execution unit is to perform. The execution units execute each instruction in parallel across the multiple data elements of each operand. For example, in response to an add instruction the execution unit performs a simultaneous add operation across each color channel representing a texture element or picture element.”),
the first multiply and accumulate operations corresponding to a first convolution operation … ([0194] “the input to a convolution layer can be a multidimensional array of data that defines the various color components of an input image”
When an opcode (an add instruction) is received, an execution unit performs an add operation across each color channel (‘first convolution operation’).),
and in response to receiving a second opcode, perform second multiply and accumulate operations using the multiply and accumulate circuitry ([0124] “the execution unit 600 also includes a compute unit 610 that includes multiple different types of functional units. … The compute unit 610 can also include a systolic array 612, and a math unit 613. The systolic array 612 includes a W wide and D deep network of data processing units that can be used to perform vector or other data-parallel operations in a systolic manner.”
[0247] “accelerator 1910 includes an array 1913 (e.g., systolic array) having logic to perform machine learning matrix multiplication operations.”
[0193-0194] “embodiments described herein can be configured to perform the types of parallel processing that is particularly suited for training and deploying neural networks for machine learning. … there are a variety of types of neural network implementations used in machine learning. One exemplary type of neural network is the feedforward network, as previously described. A second exemplary type of neural network is the Convolutional Neural Network (CNN). A CNN is a specialized feedforward neural network for processing data having a known, grid-like topology, such as image data.”
A systolic array is used to perform matrix multiplication operations for a CNN in parallel. An execution unit includes a systolic array and receives an opcode to perform an add operation simultaneously (see [0129]).),
the second multiply and accumulate operations corresponding to a second convolution operation … (The systolic array of an execution unit is used to perform matrix multiplication operations (‘second convolution operation’) for a CNN.),
wherein the first machine learning model includes a third convolution operation … (The systolic array of an execution unit is used to perform matrix multiplication operations (‘third convolution operation’) for a CNN.)
and the systolic array does not have a corresponding opcode for the third convolution operation ([0134] “instructions are grouped based on opcode 712 bit-fields to simplify Opcode decode 740. For an 8-bit opcode, bits 4, 5, and 6 allow the execution unit to determine the type of opcode. … The illustrated opcode decode 740, in one embodiment, can be used to determine which portion of an execution unit will be used to execute a decoded instruction. For example, some instructions may be designated as systolic instructions that will be performed by a systolic array. Other instructions, such as ray-tracing instructions (not shown) can be routed to a ray-tracing core or ray-tracing logic within a slice or partition of execution logic.”
Opcodes (‘third convolution operation’) are decoded to determine the type of opcode it is. The type of an opcode is used to determine which portion of the execution unit will execute the opcode. Depending on its type, an opcode is determined to be a systolic instruction to be executed by the systolic array, otherwise the opcode is routed.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the combined method of Kobayashi/Bhardwaj with the systolic array disclosed by Maiyuran to perform multiply-add operations in a parallel manner. By performing multiply-add operations using a systolic array, processing speed can be increased due to a distributed, parallel workload, thereby increasing processing efficiency.
The following are the references relied upon in the rejections below:
Gupta, Suyog, and Berkin Akin. "Accelerator-aware neural network design using automl." arXiv preprint arXiv:2003.02838 (2020).
Seo (US 12488228 B2)
Claims 11-16 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Bhardwaj in view of Gupta, further in view of Seo.
With respect to claim 11, Bhardwaj teaches:
A system comprising ([0233] “a computer program product for use with a computer system”):
a hardware processing unit ([0230] data processor);
and a storage resource storing computer-readable instructions which, when executed by the hardware processing unit, cause the hardware processing unit to ([0233] “series of computer readable instructions fixed on a tangible, non-transitory medium”):
perform a search of a machine learning model search space having a plurality of inference operations that are supported by an inference hardware architecture ([0057] “the second (reduced) format version of a block thus includes a set of one or more operations of a (or the) type that the hardware accelerator is configured to perform, such that the hardware accelerator is operable to execute the model including the blocks in the second (reduced) format in a more efficient or optimised manner, e.g. with higher utilisation of the hardware processing circuitry (e.g. higher utilisation of an NPU's MAC units) than if using (executing) the blocks in the first, non-reduced format.”
[0272] “In order to determine which blocks can be replaced in this way, there is also provided an automatic lightweight, hardware-aware NAS algorithm that can be executed in order to determine where to place IBN blocks and where to use RIBN blocks within a model.”
[0276] “Therefore, from a NAS perspective, the replacements described above can serve as a part of a novel search space to design neural network structures that are better optimised for execution by the available processing resource.”
Reduced format blocks (‘supported inference operations’) are blocks that execute a neural network model using a higher MAC unit utilization (multiply and accumulate circuitry) of a NPU (‘inference hardware architecture’), therefore reduced format blocks are inference operations that are supported by multiply and accumulate circuitry of an inference hardware architecture. Reduced format blocks (replacements) serve as part of a search space to perform neural architecture search (NAS).),
the search involving … multiply and accumulate circuitry of the inference hardware architecture (Reduced format blocks (inference operations that are supported by multiply and accumulate circuitry of an inference hardware architecture) serve as part of a search space to perform neural architecture search (NAS), see [0057,0276].);
and output a final machine learning model selected from the machine learning model search space for subsequent execution … ([0160] “after the NAS block structure selection is performed to determine which blocks should comprise which type of inverted bottleneck block, the version of the neural network generated by the NAS block structure selection, i.e. including the determined arrangement of traditional IBN blocks and reducible inverted bottleneck blocks of the technology described herein is then subject to a further training (e.g. fine-tuning) process to determine the neural network parameters (weights) for that version of the neural network, with at least some of the reducible inverted bottleneck blocks in that version of the neural network then being replaced with reduced format blocks for execution.”).
However, Bhardwaj does not teach the search involving emulation of multiply and accumulate circuitry of the inference hardware architecture, which is taught by Gupta:
perform a search of a machine learning model search space having a plurality of inference operations that are supported by an inference hardware architecture ((P. 1, Sec. 1) “We extend these NAS frameworks to search for computer vision models customized for the different instantiations of Google’s Edge TPU neural network hardware accelerator architecture: Edge TPU in the USB/PCI-e attached Coral devices and in the Pixel 4 smartphone. We pay special attention to the design of the search space used for sampling the candidate neural network architectures. In particular, we augment the search space with building blocks known to achieve high overall utilization on the Edge TPU architecture. In addition, we prohibit the use of operations incompatible with the production software stack, thereby yielding models that are readily deployed on the target devices”),
the search involving emulation of multiply and accumulate circuitry of the inference hardware architecture (Gupta discloses “to address the challenges of real-device measurements, we used a cycle-accurate Edge TPU performance simulator to estimate the latencies of the candidate models. Our simulator faithfully models most of the key subsystems to evaluate full models under a few minutes while providing a very close proxy for the real device” (P. 2, Sec. 2.1).
(P. 3, Sec. 2.2, ¶3) “In Figure 4, on the top, 5x5 kernel size choice leads to 2.78x increase in the number of MACs and parameters compared to 3x3 kernel size which leads to 2.71x increase in the runtime (1122us vs. 414us). However, on the bottom we observe that the same increase in the kernel size, number of MACs and parameters lead to only a 35% increase in the runtime (27us vs 20us). For this case, it turns out that the combination of a shallow input tensor depth”
See Figure 4 on P. 3 depicting convolution blocks perform MAC (multiply-accumulate) operations.
Candidate models are generated from a search space comprised of convolution blocks known to achieve high overall utilization on an Edge TPU (‘inference hardware architecture’). The candidate models are able to perform MAC operations, therefore an Edge TPU must have multiply and accumulate circuitry in order to execute the MAC operations. An Edge TPU simulator is cycle-accurate and faithfully models key subsystems to evaluate candidate models that perform MAC operations, therefore emulating multiply and accumulate circuitry of Edge TPU.);
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to combine the neural architecture search method of Bhardwaj with the performance simulator of Gupta to simulate candidate model performance. By simulating candidate model performance, machine learning model performance can be evaluated without running the actual model on the target hardware, thereby mitigating real-world inference costs and accelerating the model design process.
Furthermore, the combined neural architecture search method of Bhardwaj/Gupta does not teach outputting a machine learning model on a first type of processing unit that does not have multiply and accumulate circuitry and on a second type of processing unit that does have multiply and accumulate circuitry, which is taught by Seo:
and output a final machine learning model … for subsequent execution on a first type of processing unit that does not have the multiply and accumulate circuitry of the inference hardware architecture and a second type of processing unit that does have the multiply and accumulate circuitry of the inference hardware architecture ((Col. 2, Lines 5-8) “a method for distributing computations of DNNs in an accelerator. The method includes mapping multiply-and-accumulate (MAC) operations to a plurality of IMC PEs and mapping non-MAC operations to an SIMD processor”
(Col. 4, Lines 63-66) “a flexible SIMD processor that supports a wide range of non-MAC operations such as average-/max-pooling, element-wise addition, residual operation, etc”
(Col. 10, Lines 42-44) “IMC PE 14 architecture also supports the fully-connected (FC) layers whose basic computation is also a MAC operation.”
The MAC and non-MAC operations of a deep neural network (DNN) are distributed to different processors. IMC PEs (‘second type of processing unit’) support MAC operations, therefore IMC PEs are a second type of processing unit that does have multiply and accumulate circuitry of the inference hardware architecture (‘accelerator’). A SIMD processor (‘first type of processing unit’) supports non-MAC operations, therefore a SIMD processor is a first type of processing unit that does not have multiply and accumulate circuitry of the inference hardware architecture (‘accelerator’).),
wherein the search considers placement of individual inference operations on the first type of processing unit and the second type of processing unit ((Col. 11, Line 63 to Col. 12, Line 2) “distribute various computations of DNNs onto a large number of instances of IMC macros and digital computation modules is provided. In DNNs, there are MAC operations (typically >90% of operations) and non-MAC operations. MAC operations are mapped to the IMC macros 12/IMC PEs 14, and non-MAC operations to the custom SIMD processor”),
and the final machine learning model is configured for certain inference operations that do not employ the multiply and accumulate circuitry to be performed on the first type of processing unit and other inference operations that employ the multiply and accumulate circuitry to be performed on the second type of processing unit ((Col. 12, Lines 16-19) “using the proposed custom ISA to effectively reduce instruction count and latency for deep learning workloads using the IMC-based programmable accelerator”
(Col. 7, Lines 49-52) “the IMC stage performs the MAC operations (e.g., VMM operations) in one of the size IMC PEs 14. Subsequently, the SIMD stage performs other non-MAC vector operations with the SIMD processor 22.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the combined neural architecture search method of Bhardwaj/Gupta with the technique disclosed by Seo to partition a neural network based on MAC operation compatibility. By partitioning a neural network based on MAC operation compatibility, a neural network’s MAC operations can be efficiently executed on a dedicated processor, thereby reducing latency and maximizing execution efficiency.
With respect to claim 12, the combined neural architecture search method of Bhardwaj/Gupta/Seo teaches:
the system of claim 11, wherein the inference operations include convolution operations, vector operations, or matrix operations having specified input and output data sizes (Gupta discloses “our search space includes several potentially useful blocks with varying kernel and tensor sizes … we introduce a fused inverted bottleneck convolution block that fuses the initial expansion convolution with the depthwise convolution into a single full convolution (Figure 3). Originally this block expands the depth of the input tensor and performs a “cheaper” depthwise convolution with a larger depth dimension. Although, the fused alternative performs a more “expensive” full convolution at a larger depth dimension, it can utilize the hardware resources better and provide more trainable parameters which can be a good latency-accuracy trade-off … In Figure 4, on the top, 5x5 kernel size choice leads to 2.78x increase in the number of MACs and parameters compared to 3x3 kernel size which leads to 2.71x increase in the runtime (1122us vs. 414us)” (P. 3, Sec. 2.2, Paragraphs 2-3). See Figures 3 and 4 on P. 3 depicting convolution blocks with fixed input and output tensor sizes.).
Before the effective filing date of the claimed invention, it would have been obvious to modify the combined neural architecture search method of Bhardwaj/Gupta/Seo with the search space disclosed by Gupta to create a search space comprised of convolution blocks with varying tensor sizes. By creating a search space comprised of convolution blocks with varying tensor sizes, machine learning engineers can design a model based on hardware constraints, thereby allowing engineers to balance the trade-off between hardware resource utilization and runtime.
With respect to claim 13, the combined neural architecture search method of Bhardwaj/Gupta/Seo teaches:
the system of claim 11, wherein the search is performed starting from a seed model that has been selected based on performance with respect to a particular task ([0064-0065] “The neural network in the technology described herein can thus be (and is) trained using blocks in the first (full) format, using the full sequence of operations that are defined for the blocks in the first format. This means that the model (neural network) can be trained to a higher degree of accuracy (e.g. compared to training a version of the neural network using the reduced format blocks). However, after the neural network has been suitably trained, and the trained neural network is to be output for execution as part of an executable model (that can subsequently be executed as desired to perform neural network processing tasks), at least some of the first (full) format blocks in the version of the neural network that was trained are replaced in the version of the neural network for execution by corresponding blocks in the second (reduced) format.”
A neural network trained using blocks of a first (full) format is a seed model that has been trained to a high degree of accuracy. The neural network’s blocks are then replaced with blocks of a reduced format.).
With respect to claim 14, the combined neural architecture search method of Bhardwaj/Gupta/Seo teaches:
the system of claim 13, wherein the seed model includes a particular inference operation that is not supported by the inference hardware architecture ([0057] “the second (reduced) format version of a block thus includes a set of one or more operations of a (or the) type that the hardware accelerator is configured to perform, such that the hardware accelerator is operable to execute the model including the blocks in the second (reduced) format in a more efficient or optimised manner, e.g. with higher utilisation of the hardware processing circuitry (e.g. higher utilisation of an NPU's MAC units) than if using (executing) the blocks in the first, non-reduced format.”
[0276] “Therefore, from a NAS perspective, the replacements described above can serve as a part of a novel search space to design neural network structures that are better optimised for execution by the available processing resource.”
Reduced format blocks (‘supported inference operations’) are blocks that execute a neural network model using a higher MAC unit utilization (multiply and accumulate circuitry) of a NPU (‘inference hardware architecture’), therefore reduced format blocks are inference operations that are supported by multiply and accumulate circuitry of an inference hardware architecture.
A neural network is trained using blocks of a first (full) format (‘seed model’) and its blocks are replaced with reduced format blocks (see [0064-0065]). Blocks of a first (full) format are not optimized for execution on a NPU, therefore the neural network (‘seed model’) has inference operations that are not supported by an NPU (‘inference hardware architecture’).).
With respect to claim 15, the combined neural architecture search method of Bhardwaj/Gupta/Seo teaches:
the system of claim 14, wherein the final machine learning model does not include the particular inference operation ([0160] “after the NAS block structure selection is performed to determine which blocks should comprise which type of inverted bottleneck block, the version of the neural network generated by the NAS block structure selection, i.e. including the determined arrangement of traditional IBN blocks and reducible inverted bottleneck blocks of the technology described herein is then subject to a further training (e.g. fine-tuning) process to determine the neural network parameters (weights) for that version of the neural network, with at least some of the reducible inverted bottleneck blocks in that version of the neural network then being replaced with reduced format blocks for execution.”
A neural network is trained using blocks of a first (full) format (‘seed model’) and its blocks are replaced with reduced format blocks (see [0064-0065]), thereby resulting in a final machine learning model that does have the first (full) format blocks (inference operations that are not supported).).
With respect to claim 16, the combined neural architecture search method of Bhardwaj/Gupta/Seo teaches:
the system of claim 11, wherein the search involves training multiple machine learning models having different inference operations supported by the inference hardware architecture (Gupta discloses “a typical neural architecture search framework consists of the following modules: a controller that samples from search space of all possible architectures, a trainer that trains the models on some dataset to arrive at an accuracy metric, an objective function that scores the candidate model to help the controller navigate the search space” (P. 2, Sec. 2, First Paragraph). See Figure 1 on P. 2 illustrating the process of creating and training multiple candidate models.
Gupta discloses “We pay special attention to the design of the search space used for sampling the candidate neural network architectures. In particular, we augment the search space with building blocks known to achieve high overall utilization on the Edge TPU architecture. In addition, we prohibit the use of operations incompatible with the production software stack, thereby yielding models that are readily deployed on the target devices” (P. 1, Sec. 1).).
Before the effective filing date of the claimed invention, it would have been obvious to modify the combined neural architecture search method of Bhardwaj/Gupta/Seo with the candidate model generation technique disclosed by Gupta to train multiple candidate models. By training multiple candidate models, the likelihood of generating the best model to perform a particular task can be increased, thereby ensuring that a final selected model provides the best performance as possible.
With respect to claim 22, the combined neural architecture search method of Bhardwaj/Gupta/Seo teaches:
the system of claim 11, the first type of processing unit being a central processing unit and the second type of processing unit being a neural processing unit having the multiply and accumulate circuitry ((Col. 1, Lines 57-59) “single-instruction-multiple-data (SIMD) functional units with hardware loop to support a range of deep neural network (DNN) layer types”
(Col. 1, Line 64 to Col. 2, Line 3) “The programmable large-scale hardware accelerator includes a plurality of IMC processing elements (PEs), each comprising a set of IMC macros which are configured to run in parallel. The plurality of IMC PEs are configured to run at least one of serially or in parallel.”
(Col. 11, Lines 43-44) “different IMC PEs 14 can run serially (e.g., DNN layer-by-layer) or in parallel”
(Col. 10, Lines 42-44) “IMC PE 14 architecture also supports the fully-connected (FC) layers whose basic computation is also a MAC operation.”
A SIMD processor (‘central processing unit’) supports non-MAC operations, see (Col. 2, Lines 5-8). A large-scale hardware accelerator includes IMC PEs (‘second type of processing unit’) that are used to execute the MAC operations of a deep neural network (DNN), therefore IMC PEs are a neural processing unit having multiply and accumulate circuitry.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the combined neural architecture search method of Bhardwaj/Gupta/Seo with the technique disclosed by Seo to partition a neural network based on MAC operation compatibility. By partitioning a neural network based on MAC operation compatibility, a neural network’s MAC operations can be efficiently executed on a dedicated processor, thereby reducing latency and maximizing execution efficiency.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Jian et al. (“Radio Frequency Fingerprinting on the Edge”) teaches pruning weights of convolution layers to generate candidate models and performing emulation to evaluate the performance of the generated candidate models.
Brothers et al. (US 20160358070 A1) teaches iteratively modifying a portion of a neural network to generate a second neural network until a performance requirement is met.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PEDRO J MORALES whose telephone number is (571)272-6106. The examiner can normally be reached 8:30 AM - 6:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MIRANDA M HUANG can be reached at (571)270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PEDRO J MORALES/Examiner, Art Unit 2124
/MIRANDA M HUANG/Supervisory Patent Examiner, Art Unit 2124
Prosecution Insights