DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 have been examined.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: a task-scheduling unit to schedule computing tasks in claim 19.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3, 10, 12 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication 20190340010 by Lee et al. (“Lee”) in view of U.S. Patent Application Publication 20190213139 by Sharpe et al. (“Sharpe”).
Regarding claim 1. Lee discloses:
1. A computer-implemented method for reducing inference latency of an artificial intelligence (AI) system, the method comprising: See Lee, at least Figs. 8 and 12, broadly depicting a method.
obtaining an AI model; compiling the AI model to generate at least one Directed Acyclic Graph (DAG), Lee, See Fig. 12, depicting compiler operations. Also see ¶ 0102, “The compiler 232 loads 1201 the neural network. The neural network may be a DNN, ANN, CNN, RNN, DBN, or any combination thereof and may be represented by a DAG.” Also ¶ 0104, “Specifically, the compiler 232 determines the memory space in the data buffer 318 for storing a tile of input data, intermediate data generated in a previous cycle of neural engines 314 for further processing in a subsequent cycle of neural engines, and slice of output data resulting from the processing at neural engines 314.” Also see ¶ 0081, “The network optimizations performed by the network optimization module 901 include, among others, converting a generic DAG corresponding to the neural network to a DAG of tasks specific to or configured for processing by the neural processor circuit 218.”
which comprises determining an … address associated with a piece of intermediate data to be transferred from a primary memory shared among multiple AI accelerators to a secondary memory, Lee, Fig. 3, elements 314A-314N and 318, depicting multiple AI accelerators sharing data buffer 318. Also see Fig. 4 along with ¶ 0051, “Input buffer circuit 402 is a circuit that stores a portion of the input data 322 as it is received from the data buffer 318 …” Also ¶ 0088, “If shared input data is non-resident and will be stored in the data buffer 318 …” Also see Fig. 6, broadly depicting slices, tiles, and work units.
Note that Lee discloses determining data sizes for use in transferring data from buffer 318 to buffer 402. Lee does not expressly disclose use of an offset. This is taught by Sharpe. See Sharp, ¶ 0015, “An additional instruction is generated for each global variable within the same memory page that includes an offset that is relative to the base address. In this manner, the compiler generates a reduced number of instructions to load an address thereby resulting in a smaller-sized image file.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Sharpe’s offset addressing with Lee’s data in order to utilize efficient addressing for use in computing devices with limited memory resources as suggested by Sharpe.
Lee also discloses:
wherein the AI accelerators, the primary memory, and the secondary memory are located on a same system on a chip (SoC); Lee, Figs. 2-4 depicting accelerators 314A-3147N, and memories 318 and 402 located on SoC 204.
scheduling computing tasks for the inference, Lee, ¶ 0088, “The scheduler module 905 is a software module or a hardware module that determines the schedule of tasks related to the converted DAG.”
which comprises determining a base address associated with the DAG in the secondary memory; and Lee, Fig. 6, broadly depicting slices, tiles, and work units, each requiring some form of base address along with an offset. Also see ¶ 0072, “Rasterizer 718 in data buffer 318 then instructs 808 data buffer 318 to send a work unit to one or more neural engines 314. The work unit is then stored in input buffer circuits 402 of the one or more neural engines 314.” Also see Sharpe, ¶ 0015 as cited above.
performing the inference based on the DAG, which comprises transferring the piece of intermediate data from the primary memory to the secondary memory based on the offset address and the base address. Lee, ¶ 0092, “In one embodiment, the allocation module 907 allocates a portion of the data buffer 318 to store a tile of the input data. … A portion of the tile 1103 (e.g., a work unit) is transmitted to a neural engine 314 and processed by the neural engine 314 during a single cycle of the neural engine 314.”
Regarding claim 3, Lee also discloses:
3. The method of claim 1, wherein receiving the AI model comprises storing weight files associated with the AI model in an off-chip memory. Lee, ¶ 0045, “Kernel DMA 324 is a read circuit that fetches kernel information from a source (e.g., system memory 230) and sends kernel information 326A through 326N to each of the neural engines 314.” Note that kernel data in this context applies to a broad but reasonable interpretation of “weight files.”
Regarding claim 10, Lee discloses:
10. A computing system, comprising: a processor; and a memory coupled to the processor and storing instructions that when executed by the processor cause the processor to perform a method for reducing inference latency of an artificial intelligence (AI) system, the method comprising: See at least ¶ 0022, “, a memory (which may include one or more computer readable storage mediums), a memory controller, one or more central processing units (CPUs) …”
All further limitations of claim 10 have been addressed in the rejection of claim 1 above.
Regarding claim 12, parent claim 10 is addressed above.
All further limitations of claim 12 have been addressed in the rejection of claim 3 above.
Regarding claim 19, Lee discloses:
19. An artificial intelligence (AI) system, comprising: See Lee, Figs. 2-3, broadly depicting a system.
… data-loading firmware … See Lee, ¶ 0023, “firmware.”
All further limitations of claim 19 have been addressed in the rejection of claim 1 above.
Claim(s) 2, 4, 11, 13 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee in view of Sharpe as applied above, and further in view of U.S. Patent Application Publication 20190187963 by Bokhari et al. (“Bokhari”).
Regarding claim 2, Lee does not expressly disclose:
2. The method of claim 1, wherein the primary memory or the secondary memory includes a static random-access memory (SRAM). This is taught by Bokhari. See Bokhari ¶ 0011, “Each PU such as 1321 may have an associated local (ie on-chip) memory module 1320 (also commonly referred to as on-chip memory or SRAM).” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Bokhari’s SRAM with Lee’s memory in order to minimize external memory accesses as suggested by Bokhari (see ¶ 0071).
Regarding claim 4, Lee also discloses:
4. The method of claim 3, further comprising, [loading] … at least a portion of the weight files from the off-chip memory into the secondary memory, thereby reducing the inference latency resulting from loading the weight files into the primary memory to allow the multiple AI accelerators to perform computations based on the weight files. Lee, ¶ 0090, “In one embodiment, the allocation module 907 prioritizes storing data in and retrieving data from the data buffer 318 instead of system memory 230 to reduce the time, power, and data transfer bandwidth associated with providing the input data to the neural engines 314. By storing the input data in the data buffer 318, the input data is local to the neural engines 314 compared to if the input data were stored in the system memory 230 resulting in less power consumption and improved processing time of the input data.”
Lee does not expressly disclose: preloading, prior to the inference, … This is taught by Bokhari. See Bokhari, ¶ 0075, “In case of limited SM 607, a scheduling scheme should be applied to bring input FMs or kernel weights from the external memory 613 to the shared memory 607 before processing.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Bokhari’s scheduling with Lee’s loading in order to reduce latency and power as suggested by Bokhari (see ¶ 0071).
Regarding claims 11 and 13, parent claims 10 and 12 are addressed above.
All further limitations of claims 11 and 13 have been addressed in the above rejections of claims 2 and 4, respectively.
Regarding claim 20, parent claim 19 is addressed above.
All further limitations of claim 20 have been addressed in the rejection of claim 4 above.
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee in view of Sharpe and Bokhari as applied above, and further in view of U.S. Patent Application Publication 20160188207 by Choi et al. (“Choi”).
Regarding claim 5, Lee does not expressly disclose:
5. The method of claim 4, further comprising preloading one or more weight files associated with a second model into the secondary memory. This is taught by Choi. See ¶ 0230, “The scheduler 1380 can intelligently offload, select, partition, divide, or combination thereof, data, processes, analyses, functions, learning, or combination thereof, for providing first models 1372 and second models 1374 with the first compute device 1312, the second compute device 1314, the third compute device 1316, or combination thereof.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Choi’s model loading with the preloading of Lee and Bokhari in order to intelligently load a variety of models as essentially suggested by Choi.
Claim(s) 6-7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee in view of Sharpe as applied above, and further in view of U.S. Patent 9047334 to Cheriton (“Cheriton”).
Regarding claim 6, Lee also discloses:
6. The method of claim 1, wherein performing the inference comprises loading weight files associated with the DAG into the primary memory, and Lee, ¶ 0045, “Kernel DMA 324 is a read circuit that fetches kernel information from a source (e.g., system memory 230) and sends kernel information 326A through 326N to each of the neural engines 314.” Note that kernel data in this context applies to a broad but reasonable interpretation of “weight files.”
Lee does not expressly disclose: wherein loading the weight files comprises skipping a weight file that pre-exists in the primary memory. This is taught by Cheriton. See Cheriton col. 8, lines 36-41, “While new PLIDs and new relationships are formed to construct a new DAG, any portion of the old DAG that is unchanged, such as PLID 306, is preserved and reused, thus allowing the new DAG to be created efficiently and reducing computation overhead significantly.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Cheriton’s reuse with Lee’s data in order to reduce computation overhead as suggested by Cheriton.
Regarding claim 7, Lee does not expressly disclose:
7. The method of claim 6, wherein skipping the weight file comprises: determining that the DAG remains unchanged from a previous inference; determining that a base address in the primary memory corresponding to the DAG remains unchanged from the previous inference; and determining that the weight file is persistent during the previous inference. This is taught by Cheriton. See Cheriton Col. 4 lines 13-18, “Each line is addressed by a physical line identification (PLID) and has a unique content that is immutable for the life of the line. The memory is accessed as a number of segments, where each segment is structured as a directed acyclic graph ("DAG") of memory lines. A segment table maps each segment to the PLID that represents the root of the DAG.” Also see col. 8, lines 36-41, “While new PLIDs and new relationships are formed to construct a new DAG, any portion of the old DAG that is unchanged, such as PLID 306, is preserved and reused, thus allowing the new DAG to be created efficiently and reducing computation overhead significantly.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Cheriton’s reuse with Lee’s data in order to reduce computation overhead as suggested by Cheriton.
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee in view of Sharpe and Cheriton as applied above, and further in view of U.S. Patent Application Publication 20180129613 by Faldu et al. (“Faldu”).
Regarding claim 8, Lee also discloses:
8. The method of claim 6, wherein generating the DAG comprises generating a memory operator associated with the weight file and See Lee, ¶ 0028, “data fetch and read operations.”
Lee does not expressly disclose: setting a persistent bit in the memory operator. This is taught by Faldu. See Faldu, Fig. 4 and ¶ 0045, “In various embodiments, Reuse Bits 402 may include any suitable number of data bits configured to store information indicative of potential reuse in a cache of a particular vertex of a graph referenced in Data 401.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Faldu’s reuse bit with Lee’s memory operations in order to identify data that is subject to reuse as suggested by Faldu (see ¶ 0026).
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee in view of Sharpe and Cheriton as applied above, and further in view of U.S. Patent Application Publication 20210271631 by Snelgrove et al. (“Snelgrove”).
Regarding claim 9, Lee also discloses:
9. The method of claim 6, wherein scheduling the computing tasks comprises generating a data-loading command associated with the DAG and See Lee, ¶ 0028, “data fetch and read operations.”
Lee does not expressly disclose: setting a skip_weight bit in the data-loading command. This is taught by Snelgrove. See ¶ 0118, “The rightmost PEs can also skip the bad column of PEs and share with the leftmost PEs. This can be achieved by each PE having a DUD register that can be written from SRAM. An application can then initially detect the bad or failed PEs, and set the DUD bits for the entire column of PEs of the broken PE. When SRAM is then to be shared in a group that contains a DUD column, the controller can read the DUD bit and skip over the PE in that column.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Snelgrove’s skip bits with Lee’s data in order to identify data that should not be loaded as essentially suggested by Snelgrove.
Claim(s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee in view of Sharpe, Bokhari, and Choi.
Regarding claim 14, parent claim 13 is addressed above.
All further limitations of claim 14 have been addressed in the rejection of claim 5 above.
Claim(s) 15-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee in view of Sharpe, Bokhari and Cheriton.
Regarding claims 15-16, parent claim 11 is addressed above.
All further limitations of claims 15-16 have been addressed in the above rejections of claims 6-7, respectively.
Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee in view of Sharpe, Bokhari, Cheriton and Faldu.
Regarding claim 17, parent claim 15 is addressed above.
All further limitations of claim 17 have been addressed in the rejection of claim 8 above.
Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee in view of Sharpe, Bokhari, Cheriton and Snelgrove.
Regarding claim 18, parent claim 15 is addressed above.
All further limitations of claim 18 have been addressed in the rejection of claim 9 above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to James D Rutten whose telephone number is (571)272-3703. The examiner can normally be reached M-F 9:00-5:30 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at (571)272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/James D. Rutten/Primary Examiner, Art Unit 2121