CTNF 18/470,798 CTNF 88083 DETAILED ACTION Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Claims 1-18 are pending in this office action. Claim Objections 07-05-06 Claims 10-18 objected to under 37 CFR 1.75 as being a substantial duplicate of claims 1-9 When two claims in an application are duplicates or else are so close in content that they both cover the same thing, despite a slight difference in wording, it is proper after allowing one claim to object to the other as being a substantial duplicate of the allowed claim. See MPEP § 608.01(m). Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. Claims 1-18 are rejected under 35 U.S.C. 103 as being unpatentable Bokhari et al US20190187963A1 in view of Breternitz et al US20180314945A1 and Kim et al US20230297487A1. As per claim 1 and 10, Bokhari discloses a method for generating a computer-executable code for implementing an artificial neural network, the method comprising: obtaining a neural network(INN), the neural network having a plurality of neural layers: [0002] “ CNNs are typically depicted in the form of interconnected layers of feature maps (eg 1304 and 1305 in FIG. 13) and can be implemented using interconnected Processing Units ie PUs (also referred to as Processing Elements ie PEs or “accelerators”) which can, for example, be fabricated on a System on Chip (SoC) such as 1314 (also referred to as a CNN accelerator SoC) in FIG. 13.”; each layer being capable of being executed according to different implementation solutions and impacting a required memory allocation for the execution of the neural network and/or an execution time of the neural network: [0012] “Therefore, it is often required to maximize the use of an on-chip memory module such as 1310 , 1315 , 1320 , while minimizing the accesses made to an external memory module such as 1309 . In a SoC such as 1314 , the use of shared on-chip memory modules such as 1310 is specified by the programmer. For example, while processing the CNN layer 1305 , the module 1310 can be used to store (a) some input data generated from the layer 1304 (which is input to the layer 1305 ), or (b) output data generated from layer 1305 , or both. The allocation of on-chip memory modules such as 1310 to input and output data is an important design decision as it impacts both execution time and energy consumption of the SoC executing the CNN application .”; determining an optimal required memory allocation size for the execution of the neural network from possible implementation solutions for each layer of the neural network; [0128] The selection process at step 113 takes information from the memory constraints 103, from the target accelerator hardware 110 and from layer information in the CNN process 101 and applies that information to the external memory access estimation models 306 received from the step 104, allowing an improved or optimal or close to optimal scheduling scheme and memory allocation to be chosen efficiently for each layer of the CNN . The step 113 consists of two sub-steps, 400 and 500, described hereinafter in more detail with reference to FIGS. 4 and 5 respectively”; determining an optimal execution time of the neural network from the possible implementation solutions for each layer of the neural network: [0057] The optimal or close to optimal scheduling scheme 1322 and memory allocation for the particular CNN process layer 1304 is defined as the scheduling scheme and memory allocation for execution on the SoC 1314 for the layer 1304 in question which satisfies given criteria with reduced or minimal design costs. For example, optimal or close to optimal memory allocation and scheduling schemes will efficiently use the on-chip memory module (such as shared on-chip memory module 1310 ) and reduce or minimize the number of accesses made to the external memory module 1309 , thus reducing the execution time of CNN process 1303 and the energy consumption of the SoC 1314 ”. In order to avoid performing time-expensive simulations but still be able to find the optimal or close to optimal selection of memory allocation and scheduling schemes for layers of the CNN process, an estimation method is required in order to accurately predict the design costs, such as the required on-chip memory size, the external memory accesses and the execution time of the CNN process 1303 .; estimating a performance loss or a performance gain in terms of execution time and required memory allocation for each implementation solution of each layer of the neural network: [0019] In another known method, optimal sizes for buffers such as 1309 are determined for each layer such as 1304 , 1305 , . . . 1306 of the CNN process using the same scheduling scheme for all the layers 1304, 1305, . . . 1306. Selecting the same schedule such as 1322 for all the layers in the CNN process is suboptimal with respect to reducing design costs such as external memory, execution time of the CNN process and the size of the on-chip memory such as 1320 ”; developing an executable code(ONN) for implementing the INN by choosing, for each layer of the INN, an implementation solution from possible implementation solutions of this layer according to the performance loss in terms of execution time and/or allocation of this implementation solution with respect to the execution time and/or the maximum memory allocation: [0128]“The selection process at step 113 takes information from the memory constraints 103 , from the target accelerator hardware 110 and from layer information in the CNN process 101 and applies that information to the external memory access estimation models 306 received from the step 104 , allowing an improved or optimal or close to optimal scheduling scheme and memory allocation to be chosen efficiently for each layer of the CNN. The step 113 consists of two sub steps , 400 and 500 , described hereinafter in more detail with reference to FIGS. 4 and 5 respectively”; But not explicitly: defining a maximum execution time threshold of the neural network and/or a maximum required memory allocation threshold for the execution of the neural network; Breternitz discloses: defining a maximum required memory allocation threshold for the execution of the neural network; [0036]“In various embodiments, the specification can include, or otherwise be indicative of , the amount of memory available to the various compute units of the target device . Next, the optimizer calculates a memory utilization threshold based on the specification of the target device (block 720 )”; [0038] If the optimizer determines that the calculated memory utilization is less than a programmable threshold (conditional block 740 , “yes” leg), then the optimizer replaces the given pattern in the representation with a combined kernel (block 745 ). In one embodiment, the memory utilization threshold calculated in block 720 is utilized as the programmable threshold in conditional block 740 . It would have obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to combine the teachings of cited references. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to incorporate the teachings of Berternitz into teachings of Bokhari for optimizing a source code representation of a neural network by replacing the two or more adjacent layers in the source code representation with a single combined layer. Additionally, the system generates an optimized representation of the neural network, wherein the optimized representation includes the single combined layer. When the executable version of the neural network is implemented on a target machine, the single combined layer can be invoked with a single kernel call.[Breternitz 0012]. But not explicitly: defining a maximum execution time threshold of the neural network Kim discloses: defining a maximum execution time threshold of the neural network : [0073] “Specifically, the partitioner 130 may partition the weighted node graph into the plurality of partitions such that the execution time of each of the plurality of partitions has a difference equal to or less than a threshold . For example, the partitioner 130 may partition the weighted node graph into the plurality of partitions such that the plurality of partitions have the same execution tim e.”; It would have obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to combine the teachings of cited references. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to incorporate the teachings of Kim into teachings of Bokhari and Berternitz for estimating execution time of a neural network in a multi-core accelerator, the method including generating trace information including operation timing information for each core of the multi-core accelerator, and calculating the execution time of the neural network reflecting communication overhead between cores of the multi-core accelerator and memory access time for each core of the cores, based on the trace information.[Kim 0008]. As per claim 2 and 11 the rejection of claim 1 is incorporated and furthermore Bokhari discloses: wherein the optimal required memory allocation size for the execution of the neural network is determined by evaluating possible placements of data generated by the layers of the neural network: [0012]” For example, while processing the CNN layer 1305, the module 1310 can be used to store (a) some input data generated from the layer 1304 (which is input to the layer 1305), or (b) output data generated from layer 1305, or both . The allocation of on-chip memory modules such as 1310 to input and output data is an important design decision as it impacts both execution time and energy consumption of the SoC executing the CNN application.”; As per claim 3 and 12 the rejection of claim 1 is incorporated and furthermore Bokhari discloses: wherein the optimal execution time of the neural network is determined by choosing, for each layer, the quickest implementation solution from the possible implementation solutions of this layer: [0078] “Each scheduling scheme will have an impact on execution time, memory accesses and the on-chip memory size requirement. The disclosed RAMABS arrangements aim to select one of the scheduling schemes from a predefined set of scheduling schemes 1308 for each layer based on the layer parameters set out in Table 1.” [0057] “For example, optimal or close to optimal memory allocation and scheduling schemes will efficiently use the on-chip memory module (such as shared on-chip memory module 1310) and reduce or minimize the number of accesses made to the external memory module 1309, thus reducing the execution time of CNN process 1303 and the energy consumption of the SoC 1314 ”; As per claim 4 and 13 the rejection of claim 1 is incorporated and furthermore Bokhari discloses: wherein estimating the performance gain or loss in terms of execution time of an implementation solution of a layer comprises evaluating the impact of the implementation solution on the execution time of the neural network: [0078] “ However, the programmer has the flexibility to schedule four outer loops 1702 in different combinations , resulting in different scheduling schemes . Each scheduling scheme will have an impact on execution time, memory accesses and the on-chip memory size requirement. The disclosed RAMABS arrangements aim to select one of the scheduling schemes from a predefined set of scheduling schemes 1308 for each layer based on the layer parameters set out in Table 1.”; As per claim 5 and 14 the rejection of claim 1 is incorporated and furthermore Bokhari discloses: wherein estimating the performance gain or loss in terms of required memory allocation of an implementation solution of a layer comprises evaluating the impact of the implementation solution on the required memory allocation for the execution of the neural network: [0012]” The cost (both in terms of energy and execution time) of accessing the external memory module 1309 is much higher than the cost of accessing the on-chip memory module 1310 . Therefore, it is often required to maximize the use of an on-chip memory module such as 1310, 1315, 1320, while minimizing the accesses made to an external memory module such as 1309 . In a SoC such as 1314 , the use of shared on-chip memory modules such as 1310 is specified by the programmer. For example, while processing the CNN layer 1305 , the module 1310 can be used to store (a) some input data generated from the layer 1304 (which is input to the layer 1305 ), or (b) output data generated from layer 1305 , or both. The allocation of on-chip memory modules such as 1310 to input and output data is an important design decision as it impacts both execution time and energy consumption of the SoC executing the CNN application.. ”; As per claim 6 and 15 the rejection of claim 1 is incorporated and furthermore Bokhari discloses: calculating a compromise cost for each implementation solution so as to construct a first list ordering the implementation solutions of the layers according to the compromise cost thereof with respect to the required memory allocation and/or a second list ordering the implementation solutions of the layers according to the compromise cost thereof with respect to the execution time: [0130] At the step 500 , performed by the processor 1205 executing the software program 1233 and described hereinafter in more detail with reference to FIG. 5, a scheduling scheme 508 is selected for each CNN layer, the selected scheduling scheme reducing or minimizing the external memory access cost for that layer. This set of scheduling scheme selections for each CNN layer reduces or minimizes the overall external memory access cost for the CNN implementation . The selection of a scheduling scheme is based on the optimal on-chip memory allocation selected at the step 400 and the relevant external memory access estimation model 306 . Hence a best scheduling scheme and memory allocation 107 is selected for each CNN layer. As per claim 7 and 16 the rejection of claim 6 is incorporated and furthermore Bokhari discloses: wherein the executable code for implementing the neural network is developed by taking the implementation solution from the first list and/or the second list in increasing order of the compromise cost [0156] “ A following step 507, performed by the processor 1205 executing the software program 1233, selects the scheduling scheme with the lowest determined external memory cost 506 (the scheduling scheme having lowest value for eqn [19], when eqn [20] constraint is applied) as the best scheduling scheme 508 for the CNN Layer 503 . The corresponding best memory allocation 413 for that best scheduling scheme 508 is also selected for the implementation of the CNN layer 503 ”; But not explicitly: comparing the performance loss in terms of execution time and/or allocation of each implementation solution from the list with respect to the maximum execution time threshold and/or the maximum memory allocation threshold Breternitz discloses: comparing the performance loss in terms of allocation of each implementation solution from the list with respect to the maximum memory allocation threshold [0036]“In various embodiments, the specification can include, or otherwise be indicative of , the amount of memory available to the various compute units of the target device . Next, the optimizer calculates a memory utilization threshold based on the specification of the target device (block 720 )”; [0038] If the optimizer determines that the calculated memory utilization is less than a programmable threshold (conditional block 740 , “yes” leg), then the optimizer replaces the given pattern in the representation with a combined kernel (block 745 ). In one embodiment, the memory utilization threshold calculated in block 720 is utilized as the programmable threshold in conditional block 740 . It would have obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to combine the teachings of cited references. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to incorporate the teachings of Berternitz into teachings of Bokhari for optimizing a source code representation of a neural network by replacing the two or more adjacent layers in the source code representation with a single combined layer. Additionally, the system generates an optimized representation of the neural network, wherein the optimized representation includes the single combined layer. When the executable version of the neural network is implemented on a target machine, the single combined layer can be invoked with a single kernel call.[Breternitz 0012]. But not explicitly: comparing the performance loss in terms of execution time of each implementation solution from the list with respect to the maximum execution time threshold ; Kim discloses: comparing the performance loss in terms of execution time of each implementation solution from the list with respect to the maximum execution time threshold : [0073] “Specifically, the partitioner 130 may partition the weighted node graph into the plurality of partitions such that the execution time of each of the plurality of partitions has a difference equal to or less than a threshold . For example, the partitioner 130 may partition the weighted node graph into the plurality of partitions such that the plurality of partitions have the same execution tim e.”; It would have obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to combine the teachings of cited references. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to incorporate the teachings of Kim into teachings of Bokhari and Berternitz for estimating execution time of a neural network in a multi-core accelerator, the method including generating trace information including operation timing information for each core of the multi-core accelerator, and calculating the execution time of the neural network reflecting communication overhead between cores of the multi-core accelerator and memory access time for each core of the cores, based on the trace information.[Kim 0008]. As per claim 8 and 17 the rejection of claim 1 is incorporated and furthermore Bokhari discloses: A non-transitory computer-readable media storing instructions, which, when executed by one or more processors, cause the one or more processors to implement the method according to claim 1: [0105] The software may be stored in a computer readable medium , including the storage devices described below, for example. The software is loaded into the computer system 1200 from the computer readable medium, and then executed by the computer system 1200 . As per claim 9 and 18 the rejection of claim 1 is incorporated and furthermore Bokhari discloses: a memory storing instructions according to claim 1; and one or more processors electrically connected to the memory, wherein the one or more processors are configured to execute the instructions. [0103] “For example, the processor 1205 is coupled to the system bus 1204 using a connection 1218. Likewise, the memory 1206 and optical disk drive 1212 are coupled to the system bus 1204 by connections 1219. Examples of computers on which the described arrangements can be practised include IBM-PC's and compatibles, Sun Sparcstations, Apple Mac or like computer systems. Pertinent arts: US 20200379740 A1 Peak can be used as follow: a method for generating a computer-executable code for implementing an artificial neural network, the method comprising: [0026] “The code generator 230 can take a NN model document file from the neural network model document files 244 and transform the NN model document file into code in a particular programming language to execute, once compiled, on a specialized processor of a target device. The neural network compiler 215 takes the generated code from the code generator 230 and compiles the code into a neural network binary executable , which can be stored in neural network executables 242 and then be deployed to one or more different target devices for execution (e.g., the wireless audio output device 104 )”; Obtaining a neural network(INN), the neural network having a plurality of neural layers: [0025]“In an example, each of the neural network model document files 244 can at least include information that represents a set of operations that are to be performed by corresponding nodes from different layers of a given neural network model .”; each layer being capable of being executed according to different implementation solutions and impacting a required memory allocation for the execution of the neural network and/or an execution time of the neural network: [0040]”The code generator 230 may further perform a graph coloring optimization on the NN model document file. Graph coloring, as referred to herein, refers to an optimization for memory allocation of layers of the neural network that involves, in an example, determining which memory allocations are reused by the layers. An example of a memory allocation technique is described in further detail in FIG. 5 below”; defining a maximum required memory allocation threshold for the execution of the neural network: [0069]“The following discussion describes how the code generator 230 determines memory allocations for the network . In one or more implementations, the total amount of memory available for allocation may be determined based at least in part on an amount of available memory of a given target device , e.g., a specialized processor provided by the wireless audio output device 104”; determining an optimal required memory allocation size for the execution of the neural network from possible implementation solutions for each layer of the neural network: [0037] “The data layers 350 may correspond to code for each layer of the NN, and the memory allocations 344 correspond to code for allocating memory portions based on a determined size of each layer of the NN and/or based on an amount of memory available at the target device . Determining a respective size of each layer of the NN is discussed in more detail further below in FIG. 5”; [0040]“Graph coloring, as referred to herein, refers to an optimization for memory allocation of layers of the neural network that involves , in an example, determining which memory allocations are reused by the layers”; estimating a performance loss or a performance gain in terms of execution time and required memory allocation for each implementation solution of each layer of the neural network: [0016] “ Further, the performance of the neural network may improve by avoiding using dynamic memory allocation and deallocation techniques , which are often performed during running of the neural network model . Additionally, some processors may not allow for or may not feasibly perform dynamic memory allocations, such as some specialized processors provided on a given electronic device. Thus, the subject technology described herein enables a neural network to be run on such specialized, e.g. resource-constrained, processors. These benefits therefore are understood as improving the computing functionality of a given electronic device , such as an end user device which may generally have less computational resources available than, e.g., one or more cloud-based servers.”; and developing an executable code(ONN) for implementing the INN by choosing, for each layer of the INN, an implementation solution from possible implementation solutions of this layer according to the performance loss in terms of execution time and/or allocation of this implementation solution with respect the maximum memory allocation threshold. [0037]“The data formats 346 (e.g., data blobs, arrays, array of arrays, matrices) may correspond to code corresponding to the NN data types 324 and/or include code for specifying a compatible binary format for NN data that is to utilized by a given specialized processor of a target platform (e.g., the wireless audio output device 104). The data layers 350 may correspond to code for each layer of the NN, and the memory allocations 344 correspond to code for allocating memory portions based on a determined size of each layer of the NN and/or based on an amount of memory available at the target device . Determining a respective size of each layer of the NN is discussed in more detail further below in FIG. 5.”; US20200387799A1: The compiler may add an overwrite instruction to the plurality of blocks that, when executed by one or more execution engines, triggers an overwrite action. The overwrite action causes the instructions of subsequent blocks to be overwritten with NOP instructions. US11003429 B1: Using the graph, the compiler can determine estimated execution times for the operations represented by each node in the graph. The compiler can schedule operations by determining an estimated execution time for set of dependent operations that depend from an operation. Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRAHIM BOURZIK whose telephone number is (571)270-7155. The examiner can normally be reached Monday-Friday (8-4:30). Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Wei Y Mui can be reached at 571-270-2738. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /BRAHIM BOURZIK/ Examiner, Art Unit 2191 /WEI Y MUI/ Supervisory Patent Examiner, Art Unit 2191 Application/Control Number: 18/470,798 Page 2 Art Unit: 2191 Application/Control Number: 18/470,798 Page 4 Art Unit: 2191 Application/Control Number: 18/470,798 Page 5 Art Unit: 2191 Application/Control Number: 18/470,798 Page 6 Art Unit: 2191 Application/Control Number: 18/470,798 Page 7 Art Unit: 2191 Application/Control Number: 18/470,798 Page 8 Art Unit: 2191 Application/Control Number: 18/470,798 Page 9 Art Unit: 2191 Application/Control Number: 18/470,798 Page 10 Art Unit: 2191 Application/Control Number: 18/470,798 Page 11 Art Unit: 2191 Application/Control Number: 18/470,798 Page 12 Art Unit: 2191 Application/Control Number: 18/470,798 Page 13 Art Unit: 2191 Application/Control Number: 18/470,798 Page 14 Art Unit: 2191 Application/Control Number: 18/470,798 Page 15 Art Unit: 2191 Application/Control Number: 18/470,798 Page 16 Art Unit: 2191 Application/Control Number: 18/470,798 Page 17 Art Unit: 2191
Prosecution Insights