DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claims 1 and 6 have been considered but are moot because the arguments do not apply in view of newly found reference Yang being used in combination with Dasgupta and Tan in the current rejection. See the new rejection below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over US PG Pub 2022/0019880 to Dasgupta (“Dasgupta”) in view of US PG Pub 2020/0143227 to Tan (“Tan”) and US PG Pub 2021/0056378 to Yang (“Yang”).
Regarding claim 1, “A neural network construction device, comprising: a processor to execute a program; and a memory to store the program which, when executed by the processor, performs processes of” reads on the hardware and neural architecture that perform operations including obtaining a specification of a function and a plurality of hardware design parameters (abstract, ¶0005) disclosed by Dasgupta and represented in Fig. 1.
As to “specifying a plurality of operations allocatable to a plurality of layers constituting a neural network model on a basis of a construction condition for constructing the neural network model” Dasgupta discloses (¶0036) that the system provides a diagram of data flow through an accelerator performing inference of a neural architecture as represented in Fig. 4; (¶0042-¶0050) the system describes generating multiple layers for neural architectures candidates as represented in Figs. 3 and 5.
As to “determining whether or not a total necessary time which is inferred on a basis of the allocated operations and which is a total of necessary times of the respective operations satisfies a predetermined requested necessary time” Dasgupta discloses (¶0052-¶0057, ¶0048) that the system detects a latency bottleneck in a layer of the neural architecture according to the latency model of the neural architecture and estimates execution time where total latency for each layer is obtained; (¶0057-¶0061) the latency is compared with the target time constraint and determines if the architecture layer satisfies latency requirement as represented in Figs. 6 and 7.
As to “training the neural network model having the generated first operation combination” Dasgupta discloses (¶0062-¶0068) that all layers reuse latency of loading weights to train neural architecture.
As to “determining whether or not accuracy of inference by the neural network model after the training satisfies predetermined requested inference accuracy” Dasgupta discloses (¶0062-¶0066) that the performance metrics is evaluated; (¶0083, ¶0089) the test that resulted in the lowest latency, or the sample that resulted in the greatest accuracy with a latency of performance of inference that is lower than the threshold latency value; the system calculates the latency of performance of neural architecture inference by an accelerator.
As to “in response to determining that the total necessary time does not satisfy the requested necessary time, or determining that the accuracy of the inference does not satisfy the requested inference accuracy, generating a second operation combination different from the first operation combination and training the neural network model having the generated second operation combination” Dasgupta discloses (¶0066-¶0070, ¶0077, ¶0081) that the system updates architecture based on the results by iteratively using another architecture layer where the next architecture layer proceeds to another iteration of training the neural network as represented in Fig. 3; (¶0071-¶0073) the system iteratively searches until acceptable architecture layer is found as represented in Fig. 3, where (¶0058-¶0065, ¶0083, ¶0098) the system selects from among all the neural architectures, such as when the hyper-parameter value determination at is a simple importation of the sample used in the Monte Carlo test that resulted in the lowest latency, or the sample that resulted in the greatest accuracy with a latency of performance of inference that is lower than the threshold latency value.
Dasgupta meets all the limitations of the claim except “generating a first operation combination by allocating a corresponding one of the plurality of operations to each of the plurality of layers.” However, Tan discloses (¶0043-¶0048) a factorized hierarchical search space that includes of a sequence of factorized blocks, each block containing a list of layers defined by a hierarchical sub search space with different convolution operations and connections; (¶0050-¶0054) a factorized hierarchical search space partitions network layers into groups and searches for the operations and connections per group.
As to “wherein the construction condition…defines a search space and hardware for the neural network model to be executed, the search space defining the plurality of operations which are allocatable to the plurality of layers” Tan discloses (¶0013) that the CRM stores instructions that is executed by the processor cause the system to perform operations where the operations include defining an initial network structure for an artificial neural network where a plurality of search spaces are respectively associated with a plurality of blocks that defines searchable parameters and further modifies these parameters to generate new network structure (construction condition defining a search space); (¶0030-¶0031, ¶0054) the neural architecture search using a hierarchical search space that defines search spaces associated with blocks of neural network where an operation to be performed by each layer in the block. Therefore, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the invention to modify Dasgupta’s system by allocating a corresponding operation to each of the plurality of layers as taught by Tan in order to improve neural architecture search performance (Tan - ¶0010).
Combination of Dasgupta and Tan meets all the limitations of the claim except “wherein the construction condition is input by a user and defines a search space and hardware for the neural network model to be executed.” However, Yang discloses (¶0036, ¶0045, ¶0048) that the neural network architecture system receives training data, validation data, and the computation resource constraints for training a neural network to perform a particular machine learning task, where the search space for the resource constrained optimization task is mapped to a continuous search space; (¶0068-¶0069) the system receives user-defined computational resource constraints when implementing neural network generated using a process as represented in Fig. 2. Therefore, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the invention to modify Dasgupta and Tan’s systems by using the construction condition that is input by a user as taught by Yang in order to provide search constraints and resource requirements required by the neural architecture system.
Regarding claim 2, “The neural network construction device according to claim 1, wherein the plurality of operations includes a skip connection operation that outputs input data as it is and a no-connection operation that does not output any data” Tan discloses (¶0031, ¶0043-¶0048, ¶0050-¶0054, ¶0059-¶0063, ¶0124) that the searchable parameters included in the sub-search space associated with each block can include: a number of layers included in the block; an operation to be performed by each of the number of layers included in the block; a kernel size; a skip operation to be performed; an input size; and/or an output filter size.
Regarding claim 3, “The neural network construction device according to claim 1, wherein the total necessary time includes a necessary time in a layer to which none of the plurality of operations has been allocated, and which is among the plurality of layers” Dasgupta discloses (¶0052-¶0057, ¶0048) that the system detects a latency bottleneck in a layer of the neural architecture according to the latency model of the neural architecture and estimates execution time where total latency for each layer is obtained; (¶0057-¶0061) the latency is compared with the target time constraint and determines if the architecture layer satisfies latency requirement as represented in Figs. 6 and 7.
Regarding claim 4, “The neural network construction device according to claim 1, wherein the processes further include: executing the neural network model having the generated first operation combination under the construction condition; and determining whether or not an actual necessary time which the execution of the neural network model having the generated first operation combination takes satisfies the predetermined requested necessary time” Dasgupta discloses (¶0052-¶0057, ¶0048) that the system detects a latency bottleneck in a layer of the neural architecture according to the latency model of the neural architecture and estimates execution time where total latency for each layer is obtained; (¶0057-¶0061) the latency is compared with the target time constraint and determines if the architecture layer satisfies latency requirement as represented in Figs. 6 and 7.
Regarding claim 6, see rejection similar to claim 1.
Claims 5 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Dasgupta in view of Tan and Yang as applied to claims 1 and 6 above, and further in view of US PG Pub 2021/0012194 to Laskaridis (“Laskaridis”).
Regarding claim 5, combination of Dasgupta, Tan, and Yang meets all the limitations of the claim except “An image processing device, comprising: a memory to store a neural network model constructed by the neural network construction device according to claim 1; a camera to acquire image data for the neural network model to perform inference; and a monitor or a speaker to output a result of the inference by the neural network model.” However, Laskaridis discloses (¶0118, ¶0121) that the system implements a variable accuracy neural network on an apparatus where the apparatus is a smartphone or a computing device that comprises an image capture device such as a camera as represented in Fig. 8; (¶0126) the image is captured/obtained by the camera to be processed by a neural network as represented in Fig. 9, and (¶0124, ¶0131) outputting a processing result via an interface. Therefore, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the invention to modify Dasgupta, Tan, and Yang’s systems by using a neural network model on an image processing device with a camera and an output interface as taught by Laskaridis in order to develop an improved neural network architecture suitable for implementation on device and resource-constrained systems (Laskaridis - ¶00012).
Regarding claim 7, see rejection similar to claim 5.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PINKAL R CHOKSHI whose telephone number is (571)270-3317. The examiner can normally be reached Monday - Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, BRIAN T PENDLETON can be reached at (571)272-7527. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PINKAL R CHOKSHI/Primary Examiner, Art Unit 2425
Prosecution Insights