DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is made non-final.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant’s submission filed on July 30, 2026 has been entered. Claims 1, 5, 8-10, 14, 17-19, and 20 are pending. Claims 1 and 10 are independent claims.
Response to Amendment
The rejections of claims 1, 5, 8-10, 14, 17-19, and 20 under 35 U.S.C. 101 set forth in the previous office action are withdrawn in view of the amendments made to independent claims 1 and 10.
Response to Arguments
Applicant’s arguments, filed July 30, 2026, regarding the rejections from the previous office action made under 35 U.S.C. 103 have been fully considered but are moot as they do not apply to the reference Shen being used in the current rejections of claims 1 and 10 and their associated dependent claims 5, 8-9, 20, and 14, 17-19, respectively, to teach the amended claim limitation directed to deriving latency data by executing blocks of a neural network on a target device.
Specifically, Shen teaches the amended claim limitation describing receiving the plurality of latency data, for example in Shen’s in a “lookup table,” by executing the compressible target blocks or Shen’s “layers” on a target hardware and determining how to prune compressible layers based on the lookup table (Shen, Page 6, Section 3.4 and Page 14, Section A, see claim 1’s rejection below). While Shen fails to teach the providing of the latency data to a user, Welsh teaches providing information to a user through a screen (Welsh, Fig. 8, ¶48, see rejection below). Both Shen and Welsh are in the same field of endeavor of machine learning; specifically, both describe executing parts of a machine learning model on a target hardware in order to optimize the model for the hardware (Shen, Page 4, Section 3.1 and Welsh, ¶¶30-31 “the facility provides the device with code representing a portion of the subject model, and receives an indication of the device's performance when running the portion of the subject model, such as total execution time… the facility uses the tracker to distribute multiple optimizations of the subject model, or of portions of the subject model, to multiple devices in order to test each optimization”). One of ordinary skill in the art would be motivated to combine these features of Shen and Welsh, for example displaying Shen’s lookup table information on Welsh’s screen for the user, to “to quickly and efficiently optimize the subject model to increase its performance on a [user] specified hardware target” (Welsh, ¶33, see rejection below).
Additionally, while the previous office action relied on PGPub US 20220318563 A1, which failed to teach the “compression ratio values” of the amended claim 1, the current rejection now relies on a Machine Translation of Japanese Patent JP 7586079 B2, or Yashima. Although Yashima appears to describe the same invention as the US PGPub, Yashima also explicitly recites applying a “target compression ratio” which is set per layer by a user (Yashima, ¶182, see claim 1’s rejection below). Hence, Yashima now teaches the amended claim limitation directed to how “the user sets compression ratio values per compressible target block” as described on page 12 of the Remarks.
With the additions of the references Shen and the Machine Translation of Yashima teaching the subject matter introduced in the amendments, the rejections under 35 U.S.C. 103 stand.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5, 9, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Shen et al. (“HALP: Hardware-Aware Latency Pruning,” 2021, hereinafter Shen) in view of Welsh et al. (US 20220172119 A1) and further in view of Yashima (Machine Translation of JP 7586079 B2, hereinafter Yashima).
Regarding claim 1:
Regarding the limitation a method for providing a neural network model that is performed by a computing device, comprising: receiving, at a processor of the computing device, a trained model that has been trained based on a data set and a target device identified in a device farm using information about the target device that has been inputted by a user, Shen teaches a method for providing a neural network that is performed on a computing device (Page 2, ¶3 “we propose hardware-aware latency pruning (HALP)… we pre-analyze the operator-level latency values by creating a look-up table for every layer of the model on the target hardware,” Fig. 2, Caption: “Target hardware is NVIDIA Titan V GPU,” Page 7, Section 4.2, ¶1 “we also export the models”) comprising: receiving, at a processor of the computing device, a trained model that has been trained based on a data set (Page 6, Section 3.4, ¶1 “The pruning process takes a trained network as input and prunes it iteratively… The algorithm gradually trims down neurons,” wherein an “algorithm” implies instructions stored in memory that are executed using a processor of the computing device). However, Shen fails to teach a trained model that has been trained based on a data set and a target device identified in a device farm using information about the target device that has been inputted by a user.
Welsh, in the same field of endeavor, teaches optimizing a model based on optimization result data and a target device identified in a device farm (¶¶27-28 “the facility optimizes a subject model by using the optimization result data as a starting point for optimizing the subject model… the facility manages a “device farm” in which target devices of a variety of types are used to test the execution of candidate implementations of the subject model… the facility chooses one or more of these target devices to optimize the subject model based on comparing the device's hardware and the hardware target of the subject model”) using information about the target device that has been inputted by a user (Fig. 9 – 901, ¶50 “FIG. 9 is a model hardware target selection screen… The hardware target list 901 includes one or more hardware targets which a user can select to identify which hardware targets the facility should use when optimizing the model”).
Shen and Welsh are analogous art to the claimed invention as both are in the same field of endeavor of machine learning. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the target device in the device farm of Welsh with the methodology of Shen. The motivation to do so is to reduce “the dynamic display area, processing, storage, and/or data transmission resources needed to perform a certain task, thereby enabling the task to be permitted by less capable, capacious, and/or expensive hardware devices, and/or be performed with lesser latency, and/or preserving more of the conserved resources for use in performing other tasks” (Welsh, ¶33).
Shen further teaches identifying a plurality of compressible target blocks among a plurality of blocks included in the trained model, wherein the identifying comprises excluding predefined non-compressible blocks from the plurality of blocks, wherein the predefined non-compressible blocks are defined based on structural characteristics of the blocks in the trained model (Page 4, Section 3.1, ¶4 “we prune on batch normalization layers,” Pages 5-6, Section 3.3, ¶¶1-3 “We next explore grouping neurons so that a number of them can be jointly considered and removed enabling faster pruning… neurons are grouped by the order of importance… we not only group neurons within a layer, we also group the neurons sharing the same channel index from the connected layers… by grouping neurons together for a ResNet50, the total number of (grouped) neurons can be greatly reduced from 26,560to only 215,” Page 9, Section 4.5, ¶1 “we take the popular architecture Single Shot Detector (SSD)… We pretrain a SSD512 detector with ResNet50 as the backbone… We only prune the backbone network in the detector,” Page 17, ¶¶2-3 “we leave the first convolutional layer of ResNets unpruned… When we train a SSD-ResNet50, the main difference between our model and the model described in the original paper is in the backbone, where the VGG is replaced by the ResNet50… The backbone is followed by 6 additional coupled convolution layers… A BatchNorm layer is added after each convolution layer,” wherein identifying a plurality of compressible target blocks such as “batch normalization layers” to be pruned in a “ResNet50” comprises excluding predefined non-compressible blocks based on structural characteristics such as “the first convolutional layer of ResNets” or the “6 additional coupled convolution layers” and their respective “BatchNorm” layers in “a SSD-ResNet50” where only the “ResNet50” backbone is pruned) and a compression method (Abstract: “Hardware-Aware Latency Pruning (HALP) that formulates structural pruning”).
Regarding the limitation receiving a plurality of latency data from the target device, wherein each latency data of the plurality of latency data is associated with a respective one of the plurality of compressible target blocks, and wherein each latency data of the plurality of latency data is derived by executing an associated block of the plurality of compressible target blocks by the target device, Shen further teaches receiving a plurality of latency data from a target hardware, wherein each latency data of the plurality of latency data is associated with a respective one of the plurality of compressible target blocks, and wherein each latency data of the plurality of latency data is derived by executing an associated block of the plurality of compressible target blocks by the target hardware (Page 4, Section 3.1, ¶6 “Latency contribution. We empirically obtain the layer latency… by pre-building a layer-wise look-up table with pre-measured latencies. This layer latency corresponds to the aggregation of the neuron latency contribution of each neuron in the layer,” Page 5, ¶1 “we first rank globally neurons by importance and then consider their latency contribution… If we remove the least important neuron in layer l, then the number of neurons will change from pl to pl – 1, leading to a latency reduction… finding the right combination of neurons to keep imposes a combinatorial problem… we tackle it via reward maximization considering latency and accuracy traits,” wherein a “layer latency” of “layer l” in which neurons are to be removed encompasses a latency data associated with a respective one of the compressible target blocks, Page 6, Section 4, ¶1 “we target a NVIDIA TITAN V GPU for main experiments,” Page 7, ¶3 “Our approach is not limited to a single platform. We can compute the latency look-up table to apply HALP to a different platform. We repeat the ResNet50 experiments as described earlier on a JetsonTX2,” Page 14, Section A, ¶2 “For latency lookup table construction, we target a NVIDIA TITAN V GPU with batch size 256 for latency measurement… and target a Jetson TX2… We pre-generate a layer latency look-up table on the platform by iteratively reducing of the number of neurons in a layer to characterize the latency with NVIDIA cuDNN… We profile each latency measurement 100 times and take the average to avoid randomness,” wherein to “compute the latency lookup table” by characterizing the per-layer latency “on the platform” such as a target “NVIDIA TITAN V GPU” encompasses receiving a plurality of latency data from a target hardware, and wherein each latency data of the plurality of latency data or each layer latency is derived by executing an associated block or layer of compressible target blocks on a target hardware). However, Shen fails to teach the target device.
Welsh teaches the target device (28 “the facility manages a “device farm” in which target devices of a variety of types are used”).
Regarding the limitation providing, to the user, information on the plurality of compressible target blocks, wherein the information includes the latency data associated with each of the plurality of compressible target blocks, Shen teaches information on the plurality of compressible target blocks, wherein the information includes the latency data associated with each of the plurality of compressible target blocks (Page 4, Section 3.1, ¶6 “a layer-wise look-up table with pre-measured latencies. This layer latency corresponds to the aggregation of the neuron latency contribution of each neuron in the layer,” wherein “a layer-wise lookup table” containing “layer latency” encompasses information on the plurality of compressible target blocks, wherein the information includes latency data…). However, Shen fails to teach providing, to the user, information…
Welsh teaches providing, to the user, information on machine learning models and hardware (Fig. 8 – 801, ¶48 “The model list 801 displays information describing one or more models, as well as an optimize button 802 for each of the described models. When the user activates an optimize button 802, the facility begins the process of optimizing the model… the facility displays a model hardware target selection screen as part of beginning the process of optimizing the mode”).
Regarding the limitation receiving, from the user, a set of compression parameters including compression ratio values each indicating a degree of compression to be applied to a respective one of the plurality of compressible target blocks, Shen teaches a set of compression parameters including values each indicating a degree of compression to be applied to a respective one of the plurality of compressible target blocks (Page 5, ¶1 “finding the right combination of neurons to keep imposes a combinatorial problem,” Section 3.2, ¶2 “We augment the knapsack solver to consider the reordered neurons with descending importance score so that all the preceding neurons will removed before it,” Algorithm 1, Lines 16-22 depict “kept neurons” calculated for each layer “l” or compressible target block, implying a set of compression parameters including the number of neurons to be pruned per layer, each indicating a degree of compression to be applied to a respective one of the compressible target blocks). However, Shen fails to teach receiving, from the user, a set of compression parameters including compression ratio values…
Yashima, in the same field of endeavor, teaches receiving, from the user,… compression ratio values (¶¶180-182 “The calculation amount of each operation layer is indicated as a ratio of the usage amount of the memory of each operation layer… The user can grasp which operation layer may be a bottleneck… by checking the calculation amount of each operation layer… the user may be allowed to set an accuracy deterioration allowable value, which is an index indicating to what extent deterioration in accuracy is allowed, and a target compression ratio for compression by the compression method selected”).
Shen further teaches compressing the plurality of compressible target blocks based on the set of compression parameters to generate a compressed trained model (Section 3.4, ¶1 “The pruning process takes a trained network as input and prunes it iteratively to satisfy the requirement of a given latency budget… We also calculate each neuron’s importance score… we calculate the average importance score for each neuron and rank them accordingly. Then we count the number of neurons remaining in each layer and dynamically adjust the latency contributions… we group neurons… and calculate group’s importance and latency reduction. Then we get the nearest preceding group for each layer… We execute the Algo. 1 to select the neurons being remained with current latency milestone… Once pruning finishes we fine-tune the network to recover accuracy”).
Regarding the limitation and providing download data corresponding to the compressed trained model so that the compressed trained model is deployed on the target device, Shen teaches and providing download data corresponding to the compressed trained model so that the compressed trained model is deployed (Page 8, Section 4.2, ¶1 “To make it closer to the real application in production, we also export the models into onnx format and test the inference speed with TensorRT” and Table 2 depicts testing a model on target hardware “TITAN V GPU,” wherein the compressed trained model is deployed is implicit when testing the model). However, Shen fails to teach deployed on the target device.
Welsh teaches a model deployed on the target device (Fig. 10 – 1002, ¶57 “when a user activates the download button 1002, the facility presents a model download screen,” Fig. 12 – 1203, ¶59 “When a user activates the download button 1203, the facility packages the optimized machine learning model for each of the selected hardware targets and packaging options, and allows the user to download the packaged machine learning model”).
Shen, Welsh, and Yashima are analogous art to the claimed invention as all are from the same field of endeavor of machine learning. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the target device, device farm, providing of information to a user, and deploying of a model of Welsh and the user-selected compression ratios of Yashima with the compressible layers, latency lookup table, and overall pruning methodology of Shen. The motivation to do so is to reduce “the dynamic display area, processing, storage, and/or data transmission resources needed to perform a certain task, thereby enabling the task to be permitted by less capable, capacious, and/or expensive hardware devices, and/or be performed with lesser latency, and/or preserving more of the conserved resources for use in performing other tasks” (Welsh, ¶33) and “to easily design a neural network corresponding to a desired task, and it is possible to optimize the structure of the neural network corresponding to a wide range of tasks” (Yashima, ¶112).
Regarding claim 5, Shen in view of Welsh and further in view of Yashima teaches the method of claim 1 (and thus the rejection of claim 1 is incorporated).
Shen further teaches wherein the information on the plurality of compressible target blocks included in the trained model includes at least one of identification information of the block, a latency corresponding to the block, or a quantity of channels included in the block (Page 15, ¶1 “Fig. 8 shows the latency behavior of the example layers on the target platform when reducing the number of input and output channels,” Page 16, Fig. 8, Caption: “examples of pruned layers from HALP model,” Figure 8 depicts information or “latency (ms)” of various layers or the plurality of compressible target blocks in the trained model, which are indexed by decreasing “input” and “output channels” or a quantity of channels included in the block).
Regarding claim 9, Shen in view of Welsh and further in view of Yashima teaches the method of claim 1 (and thus the rejection of claim 1 is incorporated).
Regarding the limitation further comprising: performing, at the processor, at least one quantization or calibration operation on the compressed trained model based on the information about the target device, Shen teaches further comprising: performing, at the processor, at least one quantization or calibration operation on the compressed trained model based on information about a target hardware (Page 8, Section 4.2, ¶1 “we also export the models into onnx format and test the inference speed with TensorRT. We run the inference of the model with FP32, FP16 and also INT8. For INT8, we quantize the model using entropy calibration with 2560 randomly selected ImageNet training images. Since the INT8 TensorCore speedup is not supported in TITAN V GPU, we only report the quantized results on RTX3080 GPU”). However, Shen fails to teach the information about the target device.
Welsh teaches the information about the target device (¶28 “the facility manages a “device farm” in which target devices of a variety of types are used… the facility chooses one or more of these target devices to optimize the subject model based on comparing the device's hardware and the hardware target of the subject model”).
Shen and Welsh are analogous art to the claimed invention as both are in the same field of endeavor of machine learning. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the target device of Welsh with the methodology of Shen. The motivation to do so is to reduce “the dynamic display area, processing, storage, and/or data transmission resources needed to perform a certain task, thereby enabling the task to be permitted by less capable, capacious, and/or expensive hardware devices, and/or be performed with lesser latency, and/or preserving more of the conserved resources for use in performing other tasks” (Welsh, ¶33).
Regarding claim 20, Shen in view of Welsh and further in view of Yashima teaches the method of claim 1 (and thus the rejection of claim 1 is recorded.
Yashima teaches a non-transitory computer-readable recording medium storing a program that causes a computing device to execute… (Fig. 27 – 1001, 1008, ¶¶199-200 “a storage unit 1008 including a hard disk, a non-volatile memory… the CPU1001 loads, for example, a program stored in the storage unit 1008… thereby performing the series of processes described above”).
Claims 10, 14, 18, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Shen in view of Yashima and further in view of Welsh.
Regarding claim 10:
Regarding the limitation an electronic apparatus for providing a neural network model, comprising: a communication interface, configured to send and receive data via a data network, including at least one communication circuit, Shen teaches an electronic apparatus for providing a neural network model (Page 2, ¶3 “we propose hardware-aware latency pruning (HALP),” Page 5, Algorithm 1 implies code or software running on an electronic apparatus, Page 7, Section 4.2, ¶1 “we also export the models”). However, Shen fails to teach comprising: a communication interface, configured to send and receive data via a data network, including at least one communication circuit.
Yashima teaches comprising: a communication interface, configured to send and receive data via a data network (Fig. 1 – 10-30, ¶20 “The network 20 has a function of connecting the information processing terminal 10 and the information processing server 30”), including at least one communication circuit (Fig. 1 – 30, Fig. 2 – 100, Fig. 27 – 1009, ¶194 “each process executed by the information processing apparatus 100 described above may be performed by any one of the information processing terminals 10 and the information-processing server 30,” Fig. 27 – 1009, ¶199 “a communication unit 1009 including a network interface”).
Yashima further teaches a memory configured to store at least one operation instruction; and a processor, wherein execution of the at least one operation instruction causes the processor to… (Fig. 2 – 110, 150, ¶29 “Each unit of the control unit 110 is realized by a processor constituting the control unit 110 executing a predetermined program stored in the storage unit 150”).
Claims 10, 14, and 18 are apparatus claims that parallel the methods of claims 1, 5, and 9, respectively. Therefore, claims 10, 14, and 18 are rejected under substantially the same rationale as set forth above with respect to claims 1, 5, and 9, respectively.
Regarding claim 19, Shen in view of Yashima and further in view of Welsh teaches the electronic apparatus of claim 10 (and thus the rejection of claim 10 is incorporated).
Shen further teaches wherein the processor is further configured to determine a compression configuring value of the trained model based on the plurality of latency data (Page 5, ¶1 “we first rank globally neurons by importance and then consider their latency contribution… We assign the potential latency reduction to neurons in the layer by the importance order… finding the right combination of neurons to keep imposes a combinatorial problem… we tackle it via reward maximization considering latency and accuracy traits,” Section 3.2, ¶2 “We augment the knapsack solver to consider the reordered neurons with descending importance score so that all the preceding neurons will removed before it,” Algorithm 1, Lines 16-22 depict determining “kept neurons,” wherein determining the number of neurons to keep instead of prune encompasses to determine a compression configuring value of the trained model based on the plurality of latency data or the “latency contribution” of a neuron to a layer l’s “layer latency”).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Shen in view of Welsh and further in view of Yashima, and further in view of Choudhury et al. (WO 2020260991 A1, hereinafter Choudhury).
Regarding claim 8, Shen in view of Welsh and further in view of Yashima teaches the method of claim 1 (and thus the rejection of claim 1 is incorporated).
Regarding the limitation further comprising: receiving, at the processor, a user command for retraining the compressed trained model, Shen teaches the compressed trained model (Page 6, Section 3.4, ¶1 “The pruning process takes a trained network as input and prunes it iteratively”). However, Shen fails to teach further comprising: receiving, at the processor, a user command for retraining the compressed trained model.
Welsh teaches further comprising: receiving, at the processor, a user command (Fig. 9 – 902, ¶50 “When a user activates the run button 902, the facility continues the process to optimize a machine learning model”). However, Welsh fails to teach a user command for retraining…
Choudhury, in the same field of endeavor, teaches a module for retraining (Fig. 1 – 108-110, ¶27 “The retraining module 110 retrains the compressed model output by the weighted decomposition module 108 resulting in a trained compressed model”).
Choudhury further teaches generating a retrained model based on the compressed trained model (Fig. 1 – 110, 112, ¶27 “The retraining module 110 retrains the compressed model… resulting in a trained compressed model 112”).
Regarding the limitation and providing download data corresponding to the retrained model, Welsh teaches providing download data corresponding to an optimized model (Fig. 10 – 1002, ¶57 “when a user activates the download button 1002, the facility presents a model download screen,” Fig. 12 – 1203, ¶59 “When a user activates the download button 1203, the facility packages the optimized machine learning model… and allows the user to download the packaged machine learning model”). However, Welsh fails to teach the retrained model.
Choudhury teaches the retrained model (Fig. 5 – 510, ¶32 “Step 510 includes outputting the compressed trained model,” wherein it is implicit that “the compressed trained model” may be “a trained compressed model” produced by “the retraining module”).
Shen, Welsh, and Choudhury are analogous art to the claimed invention as all are from the same field of endeavor of machine learning. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the user command and download data of Welsh and the retraining mechanism of Choudhury with the methodology of Shen. The motivation to do so is to reduce “the dynamic display area, processing, storage, and/or data transmission resources needed to perform a certain task, thereby enabling the task to be permitted by less capable, capacious, and/or expensive hardware devices, and/or be performed with lesser latency, and/or preserving more of the conserved resources for use in performing other tasks” (Welsh, ¶33) and to reduce the “size and/or flops of neural networks compared to existing compression techniques” (Choudhury, ¶79).
Claim 17 is an apparatus that parallels the method of claim 8. Therefore, claim 17 is rejected under substantially the same rationale as set forth above with respect to claim 8.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WILLIAM M LEE whose telephone number is (571)272-4761. The examiner can normally be reached Mon-Fri. 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached at (571)272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WILLIAM M LEE/
Examiner, Art Unit 2145
/CESAR B PAULA/Supervisory Patent Examiner, Art Unit 2145
