Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 12/02/2025 has been entered.
Response to Remarks
Claim Rejections – 35 U.S.C. 103
Applicant’s amendments have been fully considered but they are not persuasive.
Applicant argues (pg. 10-12): “Awan and Yang fail to teach a plurality of candidate blocks which perform a same function while having different structures.”
Examiner respectfully disagrees. In Yang [0081] and [0082], the candidate modules/blocks may be residual blocks, recurrent blocks, attention block, or squeeze-and excitation block, among others. These blocks have a specified function depending on the type of the block. Therefore, two candidate blocks with the same type, such as a residual block, will both have the same function of using skip connections. Therefore, the blocks may be interchangeable, or mutually substitutable. The structure of the candidate blocks, however, may differ: Yang [¶ 0086]: “In at least one embodiment, components 202 comprises architectures 218 or architecture definitions to be used during component selection 206. In at least one embodiment, architectures 218 is one or more data values indicating neural network layer or block layout as well as relationships between one or more layers or blocks in one or more candidate neural networks 224”. Among the different candidate neural networks that the candidate blocks may form, the components, which comprise the architecture and the candidate block, gives how the structure of the block is to be configured, which is separate from the function, as described above. These may be different even amongst blocks of the same function. Applicant’s argument that Yang is in the context of architecture search, where there is evaluation of different neural network architectures composed of various modules does not refute this, as architecture is defined to specify the layout/relationship of layers in a candidate block/module in Yang.
Applicant argues (pg. 12-13): “Awan and Yang fail to teach selection of ‘a candidate block having a shortest inference time’ for the chosen accelerator.”
Examiner respectfully disagrees. Yang describes the criteria for choosing candidate blocks in the algorithm: Yang [¶ 0096]: “In at least one embodiment, time required to perform neural network operations is a consideration in determining or otherwise selecting an optimal neural network 320 by an evolutionary algorithm 306.” This may be the inference time of a neural network – the candidate block that comprises a neural network with the shortest inference time may be chosen for the overall neural network. Furthermore, the deep learning framework selects different candidate blocks: “each neural network or model can be trained independently by an automated deep learning framework 106, in an embodiment, neural network or model training is offloaded to individual computing units of one or more parallel processing units (PPUs)”. Thus, the algorithm may indeed select different candidate blocks for different accelerators if two instances of the algorithm choose different blocks on two separate PPUs.
Applicant argues (pg. 13-15): “Awan and Yang fail to teach inference using a neural network that excludes non-selected candidate blocks.”
Examiner respectfully disagrees. Applicant argues that since Yang’s candidate modules are used in an automated architecture search framework, it evaluates candidate architectures and eventually chooses one as the optimal neural network. Applicant argues that once a particular architecture is selected in Yang, it is used as a whole. Examiner respectfully disagrees as Figure 1 in Yang shows that the algorithm, while it does select for an optimal overall neural network, also has individual component selection. In the component selection, criteria, such as the inference time, is used to select block components of the neural network. Applicant argues that the “connections” discussed in paragraph 90 of Yang describe how levels, layers, or blocks may be connected within a candidate architecture. Applicant argues that this is part of defining candidate models, and not a disclosure of dynamically excluding non-selected blocks. Examiner respectfully disagrees. As described above, defining the candidate model is not mutually exclusive from selection of blocks/components in Yang. In fact, the former is comprised by the latter, amongst other elements. In Yang, by selecting candidate blocks amongst a plurality of potential blocks, it follows that the blocks not selected must be excluded from the connections (i.e. a lack of connection between blocks that are not chosen.)
Applicant argues (pg. 15-17): “the combination of Awan and Yang still does not render claim 1 obvious”
Examiner respectfully disagrees. From the arguments above, Yang is indeed in the same field of endeavor as the present invention as it addresses candidate blocks with same function and different structures, the criteria of shortest inference time, and excluding non-selected blocks. Awan is also in the same field of endeavor as the present invention as it analyzes the inference time of layers in a neural network and relates it to the accelerators used. See rejection below for an updated motivation to combine.
The foregoing applies to all independent claims and their dependent claims.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(B) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 11, 15 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claims 1, 11, 15 recite the limitation “… wherein the plurality of candidate blocks are configured to perform a same function while having different structures” in lines 18-19. There is insufficient antecedent basis for this limitation in the claim. While the Specification mentions structure of a neural network (in paragraph 0042), it does not mention the structure of individual/set of blocks, as in the claim. Furthermore, the structure of a neural network is only attempted to be defined using an example in Figure 2, where it is asserted that 210 and 220, while having the same function, have different structure. This implies that any difference in how a block within the neural network is composed (i.e. made up of a single layer or multiple layers) constitutes a difference in the overall structure of a neural network. For examination purposes the examiner will treat “having different structures” as any significant difference in how the candidate blocks are composed, such as having different architecture or different layers composing it. The examiner suggests amending the language to make the distinction between structure of a neural network and structure of a block clearer. Examiner also suggests defining the structure of a neural network in the claims.
Claim Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-8, 11, 13-15 are rejected under 35 U.S.C. 103 as being unpatentable over Awan et al. (“An In-depth Performance Characterization of CPU- and GPU-based DNN Training on Modern Architectures”) hereinafter known as Awan in view of Yang et al. (US 20220058466 A1) hereinafter known as Yang.
Regarding independent claim 1, Awan teaches:
An electronic device for performing inference by using a neural network, the electronic device comprising: (Awan [Page 3, Column 1, Paragraph 2]: “DNNs automatically infer features…” Awan teaches a device for performing inference by a deep neural network. Awan [Page 4, Figure 2]: Awan teaches that the network is made up of layers. These layers are modular, as can be seen in the figure: the layers used for various operations are repeated for the various accelerators used. This shows that the layers/blocks are common and form a selectable block set.)
…
and a processor comprising a plurality of accelerators, and configured to execute the one or more instructions to: obtain, by the processor, inference time information about the neural network for each of the plurality of accelerators, based on the information about the neural network; (Awan [Page 6, Table 2]: Awan teaches the plurality of accelerators that are used to process the neural network. Awan [Page 7, Table 3(a)]: Awan teaches the training time for the inference for each of the accelerators that were used on AlexNet.)
determine, by the processor, an accelerator for performing the inference according to the neural network from among the plurality of accelerators, based on the inference time information about the neural network; (Awan [Page 8, Column 1, Paragraph 2]: “Pascal P100 GPU is still the overall best performer across all architectures and provides up to 18% improvement over KNL for AlexNet training” Awan teaches that a particular GPU was determined based on the inference time information, namely the performance which includes the training speed. Awan [Page 5, Figure 5]: Awan teaches the inference time information for the various accelerators.)
…
…
Awan does not explicitly teach:
…
a memory configured to store one or more instructions and information about the neural network, wherein the neural network comprises a common block and a selectable block set including a plurality of candidate blocks;
…
…
select, by the processor, a candidate block having a shortest inference time corresponding to the accelerator from among a plurality of candidate blocks included in the selectable block set based on inference time information corresponding to each of the plurality of candidate blocks, wherein the plurality of candidate blocks are configured to perform a same function while having different structures;
and perform, by the processor, the inference using the neural network comprising the common block and the selected candidate block and excluding any candidate block that is not selected, by controlling a flow of the neural network, so that output data of a block prior to the selectable block set is provided as input to the selected candidate block.
However, Yang teaches:
…
a memory configured to store one or more instructions and information about the neural network, wherein the neural network comprises a common block and a selectable block set including a plurality of candidate blocks; (Yang [¶ 0125]: “In at least one embodiment, code and/or code and/or data storage 701 may be cache memory, dynamic randomly addressable memory (“DRAM”), static randomly addressable memory (“SRAM”), non-volatile memory (e.g., flash memory), or other storage.” Yang teaches memory configured to store information about the neural network. Yang [¶ 0064]: “In at least one embodiment, an automated deep learning framework 106 receives, as input, components 112 … components 112 comprise candidate modules and/or blocks … components 112 comprise architectures, … architectures is a data value indicating how one or more neural network layers in one or more neural networks to be generated by an automated deep learning framework” Yang teaches receiving the inference time information, and thus this information, in order to be processed, must be stored in some type of storage indicated above. [Yang ¶ 0064]: “components 112 comprise candidate modules and/or blocks”. Yang teaches that the neural network can have candidate blocks that can be selected into the neural network.)
…
…
select, by the processor, a candidate block having a shortest inference time corresponding to the accelerator from among a plurality of candidate blocks included in the selectable block set based on inference time information corresponding to each of the plurality of candidate blocks, wherein the plurality of candidate blocks are configured to perform a same function while having different structures; (Yang [¶ 0096]: “In at least one embodiment, time required to perform neural network operations is a consideration in determining or otherwise selecting an optimal neural network 320 by an evolutionary algorithm 306.” Yang teaches that the criteria for choosing the candidate block may be the inference time of a neural network that includes the candidate block. Yang [¶ 0081]: “candidate modules 208 and/or blocks comprise at least convolutional blocks or layers as well as residual blocks 210, recurrent blocks 212, attention blocks 214, squeeze-and-excitation blocks 216, or any other type of neural network block” Yang teaches that the candidate blocks may be of various types, for example residual blocks. This shows that the function of the different blocks may be the same (i.e. functioning as a residual block) while having different architectures/structures. Yang [¶ 0086]: “In at least one embodiment, components 202 comprises architectures 218 or architecture definitions to be used during component selection 206. In at least one embodiment, architectures 218 is one or more data values indicating neural network layer or block layout as well as relationships between one or more layers or blocks in one or more candidate neural networks 224”. Yang teaches that the different candidate neural networks that the candidate blocks may form, the components, which comprise the architecture and the candidate block, gives how the structure of the block is to be configured, which is separate from the function,)
and perform, by the processor, the inference using the neural network comprising the common block and the selected candidate block and excluding any candidate block that is not selected, by controlling a flow of the neural network, so that output data of a block prior to the selectable block set is provided as input to the selected candidate block. ([Yang ¶ 0090]: “a second half of levels, layers, or blocks in a candidate neural network 224 architecture are connected with any other type of neural network layer capable of connecting levels, layers, or blocks in said candidate neural network 224 architecture.” Yang teaches that the layers/blocks are connected to each other, which shows the flow of the neural network as the output of a prior block be the input of the next block. Note that each of the blocks can be selected or not selected, as they are candidate blocks, as shown in the following. [Yang ¶ 0064]: “components 112 comprise candidate modules and/or blocks”. Yang [Figure 1]: Yang teaches that the component selection, which comprises the algorithm for building the neural network, is selecting certain components, while disregarding/not selecting the other components.)
Awan and Yang are in the same field of endeavor as the present invention, as the
references are directed to accessing the performance of blocks of a neural network. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to combine determining the inference time of a particular layer/block by a particular accelerator as taught in Awan with selecting a block from a set of candidate blocks by the shortest inference time as taught in Yang. Yang provides this additional functionality. As such, it would have been obvious to one of ordinary skill in the art to modify the teachings of Awan to include teachings of Yang because the combination would allow for each block/layer’s inference time to be analyzed so that the fastest block can be selected for the neural network. This has the potential benefit of increasing the efficiency of inference tasks by neural networks, as the choice of accelerator and blocks can be determined based on the shortest time it takes to complete the inferences.
Regarding dependent claim 3, Awan and Yang teach:
The electronic device of claim 1,
Yang teaches:
wherein the neural network is trained so that a difference between operation results output using the plurality of candidate blocks included in the selectable block set is less than a preset value. (Yang [Figure 6]: “Step 618 shows that Yang teaches that the candidate model with the highest accuracy on these inference tasks become the optimal neural network. As the accuracy is being compared here, the difference in accuracy between some candidate block is indubitably less than the value 100%.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 4, Awan and Yang teach:
The electronic device of claim 1,
Awan teaches:
wherein the plurality of accelerators includes at least one from among a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), and a digital signal processor (DSP). (Awan [Page 6, Table 2]: Awan teaches the plurality of accelerators that are used to process the neural network. Among these accelerators include various CPUs and GPUs.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 5, Awan and Yang teach:
The electronic device of claim 1,
Awan teaches:
wherein the information about the neural network comprises, information indicating the candidate block corresponding to the accelerator from among the plurality of candidate blocks according to a type of the accelerator, wherein the processor is further configured to obtain an inference time associated with the neural network using the each of the plurality of accelerators, based on the information indicating the candidate block. (Awan [Page 5, Figure 6]: Awan teaches a layer-wise breakdown of performance for different accelerators. This information shows the plurality of layer/blocks corresponding to the plurality of accelerators and the inference time associated between each layer and accelerator. Given a choice of accelerator, the graph shows layers/blocks corresponding to that accelerator. Out of these, a criterion, such as shortest training time, may be used to select a candidate block. Awan teaches an inference time associated with the neural network, for each of the accelerators, given a candidate block.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 6, Awan and Yang teach:
The electronic device of claim 1,
Awan teaches:
wherein the inference time information about the neural network comprises inference time information about each of the plurality of candidate blocks, using the each of the plurality of accelerators. (Awan [Page 5, Figure 6]: Awan teaches a layer-wise breakdown of performance for different accelerators. This information shows the plurality of layer/blocks corresponding to the plurality of accelerators and the inference time associated between each layer and accelerator.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 7, Awan and Yang teach:
The electronic device of claim 1,
Awan teaches:
wherein the processor is further configured to execute the one or more instructions to determine an accelerator having a shortest inference time of the neural network from among the plurality of accelerators as the accelerator. (Awan [Page 5, Figure 5]: Awan teaches a bar chart that shows the various training times for different accelerators. As the training time is needed to create the model to make inference, the sizes of the bars determine the accelerator having the shortest inference time.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 8, Awan and Yang teach:
The electronic device of claim 1,
Yang teaches:
wherein the processor is further configured to execute the one or more instructions to store the inference time information about the neural network for the each of the plurality of accelerators in the memory. (Yang [¶ 0125]: “In at least one embodiment, code and/or code and/or data storage 701 may be cache memory, dynamic randomly addressable memory (“DRAM”), static randomly addressable memory (“SRAM”), non-volatile memory (e.g., flash memory), or other storage.” Yang teaches memory configured to store information about the neural network. Yang [¶ 0064]: “In at least one embodiment, an automated deep learning framework 106 receives, as input, components 112 … components 112 comprise candidate modules and/or blocks … components 112 comprise architectures, … architectures is a data value indicating how one or more neural network layers in one or more neural networks to be generated by an automated deep learning framework” Yang teaches receiving the inference time information, and thus this information, in order to be processed, must be stored in some type of storage indicated above.)
The reasons to combine are substantially similar to those of claim 1.
Claims 11, 13, 14 are rejected on the same grounds under 35 U.S.C. 103 as claims 1, 3, 4, as they are substantially similar, respectively. Mutatis mutandis.
Claim 15 is substantially similar to claim 1, but has the following additional elements:
Regarding independent claim 15, Awan and Yang teach:
A non-transitory computer-readable recording medium having stored thereon instructions which, when executed by at least one processor of a device including a plurality of accelerators and capable of performing inference by using a neural network, cause the at least one processor to: (Yang [¶ 0132]: In at least one embodiment, inference and/or training logic 715 illustrated in FIG. 7A may be used in conjunction with central processing unit (“CPU”) hardware, graphics processing unit (“GPU”) hardware or other hardware, such as field programmable gate arrays (“FPGAs”). Yang teaches a processor that includes accelerators. Yang [¶ 0125]: “In at least one embodiment, code and/or code and/or data storage 701 may be cache memory, dynamic randomly addressable memory (“DRAM”), static randomly addressable memory (“SRAM”), non-volatile memory (e.g., flash memory), or other storage.” Yang teaches non-transitory storage medium to store information about the neural network.)
The reasons to combine are substantially similar to those of claim 1.
Claims 2, 12 are rejected under 35 U.S.C. 103 as being unpatentable over Awan in view of Yang in view of Jang (US 20210058653 A1) hereinafter known as Jang.
Regarding dependent claim 2, Awan and Yang teach:
The electronic device of claim 1,
Yang teaches:
wherein the information about the neural network comprises a structure of the neural network and at least one weight of the neural network, … (Yang [¶ 0064]: “In at least one embodiment, an automated deep learning framework 106 receives, as input, components 112 … components 112 comprise candidate modules and/or blocks … components 112 comprise architectures, … architectures is a data value indicating how one or more neural network layers in one or more neural networks to be generated by an automated deep learning framework” Yang teaches inference time information about the neural network, which includes the structure and how the layers of the neural network are to be generated, which indubitably includes information regarding the weights of the layers.)
Awan and Yang do not explicitly teach:
… wherein the electronic device further comprises a communication interface configured to receive a neural network model file comprising the information about the neural network from an external device.
However, Jang teaches:
… wherein the electronic device further comprises a communication interface configured to receive a neural network model file comprising the information about the neural network from an external device. (Jang [¶ 0088]: “the neural network file calculation unit 132b may generate a neural network file, including information such as an activation function for each layer and parameters (a weight, a bias, a learning rate, etc.) set in a corresponding artificial neural network algorithm” Jang teaches a neural network file that contains information regarding the structure of the network and weights. Jang [¶ 0092]: According to the embodiment of the present invention, the communication unit 210 may perform a communication function for receiving the neural network file and the video data from the server 100. Jang teaches receiving the neural network model file from an external device, a server.)
Jang is in the same field as the present invention, since it is directed to performing operations using neural networks via receiving a file containing the details of the neural network. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to combine choosing the best suited accelerator for the neural network, and then, choosing the blocks that give the shortest inference time, as taught in Awan as modified by Yang with receiving the file containing the details of the neural network from an external device as taught in Jang. Jang provides this additional functionality. As such, it would have been obvious to one of ordinary skill in the art to modify the teachings of Awan as modified by Yang to include teachings of Jang because the combination would allow for the choice of accelerator to be made with information, such as the structure and the weights, regarding the neural network. This combination would even allow for such information to come from an external device. This has the potential benefit of speeding up the inference operations of the neural network, as the accelerator can be chosen based on the characteristics of the neural network that minimizes this inference time.
Claim 12 is rejected on the same grounds under 35 U.S.C. 103 as claim 2, as they are substantially similar. Mutatis mutandis.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KYU HYUNG HAN whose telephone number is (703) 756-5529. The examiner can normally be reached on MF 9-5.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached on (571) 270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Kyu Hyung Han/
Examiner
Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123