DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The following action is in response to the original filing of 10/06/2023 and preliminary amendment of 11/20/2023.
By the amendment, claims 1 and 10 have been amended.
Claims 1-18 are pending and have been considered below.
Specification
The specification amendment of 11/20/2023 has been reviewed and entered.
Drawings
The replacement drawings of 11/20/2023 have been reviewed and entered.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-18 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract ideas without significantly more.
Regarding claims 1 and 10:
Step 1, MPEP 2106.03:
These limitations have been determined, under Step 1, to be statutory categories of invention:
A method of performing automatic tuning on a deep learning model [..] (claim 1)
A system for automatically tuning on a deep learning model, comprising: at least a processor [..] (claim 10)
Step 2A Prong One MPEP 2106.04, 2106.04(a):
These limitations represent, under Step 2A Prong One, mental processes such as concepts that can be practically performed in the human mind, or by a human using pen and paper as a physical aid, including observations, evaluations, judgments and opinions, MPEP 2106.04(a)(2)(III), for example when given configuration parameters of a tuned model, a human can estimate first performance metrics according to costs and second performance metrics according to gathered statistical data and find the best configuration based on the analysis to apply an optimal configuration:
[..] utilizing .. instruction-based learned cost .. to estimate a first type of operational performance metrics based on a tuned configuration ..; [..]
[..] utilizing statistical data .. to determine a second type of operational performance metrics based on the tuned configuration ..; [..]
[..] performing an .. process to obtain a plurality of optimal configurations based on the first type of operational performance metrics and the second type of operational performance metrics; [..]
[..] configure .. according to one of the plurality of optimal configurations [..]
Step 2A Prong Two, MPEP 2106.04(d):
These limitations represent, under Step 2A Prong Two, mere instructions to implement the abstract idea using generic computing tools, MPEP 2106.05(f):
[..] A system .., comprising: at least one processor; and one or more computer readable storage media storing computer-readable instructions that when used by the one or more computer processors, cause the one or more computer processors to perform operations [..] (claim 10)
[..] a deep learning model [..]
[..] an instruction-based learned cost model [..]
[..] an auto-tuning process [..]
[..] configure the deep learning model [..]
These limitations represent, under Step 2A Prong Two, mere data gathering, MPEP 2106.05:
[..] a tuned configuration of layer fusion and tensor tiling [..]
[..] statistical data gathered during a compilation process of the deep learning model [..]
Step 2B, MPEP 2106.05:
These limitations are considered, under Step 2B, insignificant extra-solution activity as being recited at a high level of generality, MPEP 2106.05(d):
[..] A system .., comprising: at least one processor; and one or more computer readable storage media storing computer-readable instructions that when used by the one or more computer processors, cause the one or more computer processors to perform operations [..] (claim 10)
[..] a deep learning model [..]
[..] an instruction-based learned cost model [..]
[..] an auto-tuning process [..]
[..] configure the deep learning model [..]
These limitations are considered, under Step 2B, insignificant extra-solution activity of data gathering/selecting a particular type of data, MPEP 2106.05(g):
[..] a tuned configuration of layer fusion and tensor tiling [..]
[..] statistical data gathered during a compilation process of the deep learning model [..]
Regarding dependent claims 2-3, 8, 11-12 and 17, these dependent claims additionally recite elements that apply/perform a configuration or process to generate an outcome (claims 2-3, 11-12) or utilizing a tool to generate an outcome (claims 8, 17). The analysis incorporates the Step analysis of its respective parent. The additional limitations represent, under Step 2A Prong Two, mere instructions to implement the abstract idea using generic computing tools, MPEP 2106.05(f); under Step 2B, insignificant extra-solution activity as being recited at a high level of generality, MPEP 2106.05(d), and mere instructions to apply to obtain a solution/outcome, MPEP 2106.05(f).
Regarding dependent claims 4-5 and 13-14, these dependent claims additionally recite elements that apply/perform a configuration or process to generate an outcome and using the outcome as input (claims 4, 13), and obtaining information and performing a process with the obtained information (claims 5, 14). The analysis incorporates the Step analysis of its respective parent. The additional limitations represent, under Step 2A Prong Two, mere instructions to implement the abstract idea using generic computing tools, MPEP 2106.05(f), and mere data gathering, MPEP 2106.05; under Step 2B, insignificant extra-solution activity as mere instructions to apply to obtain a solution/outcome, MPEP 2106.05(f), and data gathering/selecting, MPEP 2106.05(g).
Regarding dependent claims 6-7 and 15-16, these dependent claims additionally recite elements of what type of data the first and second metrics comprise (claims 6, 7, 15, 16). The analysis incorporates the Step analysis of its respective parent. The additional limitations represent, under Step 2A Prong Two, mere data gathering, MPEP 2106.05; under Step 2B, insignificant extra-solution activity as mere data gathering/selecting, MPEP 2106.05(g).
Regarding dependent claims 9 and 18, these dependent claim further recites limitations that include representing the tuned configuration in a particular sequence and single numbered form, additionally reciting elements of what the sequence and single number represent (claims 9, 18). The analysis incorporates the Step analysis of its respective parent. These limitations represent, these limitations represent, under Step 2A Prong One, mental processes such as concepts that can be practically performed in the human mind, or by a human using pen and paper as a physical aid, including observations, evaluations, judgments and opinions, MPEP 2106.04(a)(2)(III). The additional limitations represent, under Step 2A Prong Two, mere instructions to implement the abstract idea using generic computing tools, MPEP 2106.05(f); under Step 2B, insignificant extra-solution activity as being recited at a high level of generality, MPEP 2106.05(d).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-8 and 10-17 are rejected under 35 U.S.C. 103 as being unpatentable over Sarah et al., US 2022/0035878 A1 published 02/03/2022 [SARAH] in view of Zhou et al., US 2023/0176840 A1 published 06/08/2023 [ZHOU].
Regarding claim 1, SARAH discloses a method of performing automatic tuning on a deep learning model (pp. 3: tuning of ML, pp. 13: of DNN), comprising:
utilizing an instruction-based learned cost model to estimate a first type of operational performance metrics based on a tuned configuration of layer and tensor architecture (pp. 45-46: MOCG model optimizer generates candidate ML architectures and finds/estimates an optimal set of ML parameters that minimizes a cost function);
utilizing statistical data gathered during a compilation process of the deep learning model to determine a second type of operational performance metrics based on the tuned configuration of layer and tensor architecture (pp. 51: performance metric evaluator computes actual performance metrics by training and operating/compiling the model and collecting measurements);
performing an auto-tuning process to obtain a plurality of optimal configurations based on the first type of operational performance metrics and the second type of operational performance metrics (pp. 53: evaluating the performance to find optimal ML architecture parameters and associated performance metrics to store for later retrieval, pp. 34: performance indicators, Fig. 9, pp. 103: auto-tuning process via looping from 903 to 902); and
configure the deep learning model according to one of the plurality of optimal configurations (Fig. 9, pp. 103: once optimal configuration is determined it is used to update the ML config and repeat the process).
While SARAH discloses that the tuned configuration is for layer and tensor architecture (pp. 21: hyperparameters for architectural aspects of ML include at least layer number and type, output channels, kernel size, etc., pp. 38: tensor architecture, pp. 185, pp. 209), SARAH fails to explicitly disclose wherein the tuned configuration is of layer fusion and tensor tiling.
ZHOU discloses methods for optimizing deep learning models (pp. 6), an analogous art. In particular, ZHOU discloses that it is old and well known in the AI arts that optimizing ML compilers is based on tuned configurations of layer and tensor architectures, including specifically fusion, tiling, layout and scheduling (pp. 3). Therefore it would have been obvious to one having ordinary skill in the art and the teachings of SARAH and ZHOU before them before the effective filing of the claimed invention to combine the known art teaching of ML compiler configurations based on layer and tensor fusion, layout and tiling, and/or scheduling, as taught by ZHOU, with the tuned configuration of layer and tensor architecture of SARAH, yielding the predictable result of the determined first and second type of operational performance metrics of SARAH and ZHOU being based on tuned configuration of layer fusion and tensor tiling. One would have been motivated to make this combination in order to solve the most common optimization problems for ML compilers, as suggested by ZHOU (pp. 3).
Regarding claim 2, SARAH and ZHOU disclose the method of claim 1, and SARA further discloses:
applying the plurality of optimal configurations to a hardware simulation device to find a best configuration (pp. 103: get hardware results to find optimal configuration, pp. 65-66: simulation results); and
configure the deep learning model according to the best configuration (Fig. 9, pp. 103).
Regarding claim 3, SARAH and ZHOU disclose the method of claim 1, and SARAH further discloses:
performing the auto-tuning process to generate the tuned configuration of layer fusion and tensor tiling (Fig. 9, pp. 103: update the ML config); and
performing the compilation process according to the tuned configuration of layer fusion and tensor tiling (Fig. 9, pp. 102: evaluate ML using actual performance).
Regarding claim 4, SARAH and ZHOU disclose the method of claim 1, and ZHOU further discloses:
performing the compilation process to convert the deep learning model into a set of instructions (pp. 68-70: generate encoded graph embedding from graph of operations); and
inputting the set of instructions to the instruction-based learned cost model to estimate the first type of operational performance metrics (pp. 71: pass graph embedding to policy network to estimate optimization actions).
It would have been obvious to one having ordinary skill in the art and the teachings of SARAH and ZHOU before them before the effective filing of the claimed invention to combine the converting a deep learning model into instructions to input into a learned cost model to estimate operational metrics, as taught by ZHOU, with the cost model and compilation process for performing auto-tuning of SARAH and ZHOU. One would have been motivated to make this combination in order to provide better performing optimizations, as suggested by ZHOU (pp. 25-26).
Regarding claim 5, SARAH and ZHOU disclose the method of claim 1, and SARAH further discloses:
obtaining information regarding at least one of a search space, heuristics-found configurations, and a tuning algorithm configuration (pp. 30: search space, pp. 32: heuristic warm start, pp. 45: tuning algorithms); and
performing the auto-tuning process based on the information regarding the at least one of the search space, the heuristics-found configurations, and the tuning algorithm configuration (Fig. 2).
Regarding claim 6, SARAH and ZHOU disclose the method of claim 1, and SARAH further discloses wherein the first type of operational performance metrics comprises at least one of latency and power consumption regarding execution of the deep learning model (pp. 14-15: metrics include accuracy, latency, power consumption).
Regarding claim 7, SARAH and ZHOU disclose the method of claim 1, and SARAH further discloses wherein the second type of operational performance metrics comprises at least one of dynamic random-access memory (DRAM) access and memory footprint regarding execution of the deep learning model and a compile time of the compilation process of the deep learning model (pp. 15: metrics include hardware characteristics such as cache memory or clock speed, pp. 145).
Regarding claim 8, SARAH and ZHOU disclose the method of claim 1, and SARAH further discloses wherein the step of performing the auto-tuning process comprises:
utilizing an auto-tuner including a plurality of sub-tuners with shared tuning parameters to perform tasks of the auto-tuning process in parallel (pp. 213: parallel implementation to increase bandwidth/throughput on edge systems).
Regarding claims 10-17, claims 10-17 recite limitations similar to claims 1-8, respectively, and are similarly rejected.
Claims 9 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over SARAH in view of ZHOU and in further view of Vasilache, Nicolas, et al. "Tensor comprehensions: Framework-agnostic high-performance machine learning abstractions." arXiv preprint arXiv:1802.04730 (2018) [VASILACHE].
Regarding claim 9, SARAH and ZHOU disclose the method of claim 1, and SARAH further discloses:
representing the tuned configuration of layer fusion and tensor tiling in a form of a combination of a number sequence (pp. 62: number format selection).
However, neither SARAH nor ZHOU specifically disclose wherein the number sequence represents a tiling configuration corresponding to a layer and a single number represents a fusion configuration corresponding to preceding layers.
VASILACHE discloses methods for selecting optimization algorithms to deal with hardware features (page 2 pp. 9: “We present a novel domain-specific flow capable of generating highly-optimized kernels for tensor expressions, leveraging optimizations across operators and optimizations that take into account the size and shape of data.”), an analogous art. In particular, VASLIACHE discloses that a syntax for representing tensor and layer configurations (page 3 pp. 2: “Tensor Comprehensions (TC): a high-level language to express tensor computations arising in ML with a syntax generalizing the Einstein notation. It supports shape and size inference, flexible element-wise syntax with both named and positional parameters.”), wherein the syntax includes a number sequence representing tiling configuration to a layer and a single number representing fusion configuration of preceding layers (page 11 Fig. 3(c) fused and tiled
PNG
media_image1.png
36
182
media_image1.png
Greyscale
, page 11 pp. 2: “Inside each filtered branch, Band nodes define an identity schedule with as many one-dimensional schedule functions as loop iterators for the statement.”, page 13 pp. 1: “Loop tiling is implemented as a separate step after the scheduling took place and performed as a schedule tree transformation. Essentially, it converts a permutable schedule band into a chain of two bands with the outer band containing tile loops and the inner band containing point loops with fixed trip count.”, pp 2: “Dependence analysis shows that loops i and j are parallel. Therefore, we can tile them and sink the point loops below the band of the reduction k loop, resulting in the schedule tree in Figure 3.d. Innermost nested bands with point loops can be joined together into a single band after checking for permutability.”, page 11 Fig. 3(d) fused, tiled and sunk
PNG
media_image2.png
69
338
media_image2.png
Greyscale
).
Therefore it would have been obvious to one having ordinary skill in the art and the teachings of SARAH, ZHOU and VASILACHE before them before the effective filing of the claimed invention to combine the syntax for representing tensor and layer configurations includes a number sequence representing tiling configuration to a layer and a single number representing fusion configuration of preceding layers, as taught by VASILACHE, with the numbered representation of the layer fusion and tensor tiling configuration. One would have been motivated to implement this combination to better represent critical GUP targets of complex scheduling and mapping transformation (page 2 pp. 5: “The transformation language of these frameworks does not seem to be able to represent complex scheduling and mapping transformations which are often crucial to GPU targets with partitioned memory architectures.”), allowing for efficient memory management, as suggested by VASILACHE (page 3 pp. 1: “allows for efficient memory management and mapping to complex parallel platforms”)
Regarding claim 18, claim 18 recites limitations similar to claim 9 and is similarly rejected.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Ou, Dao-li et al.
CN 112711422 A
OPTIMIZATION METHOD AND SYSTEM OF NEURAL NETWORK COMPILATION
Wang; Leyuan et al.
US 11797876 B1
UNIFIED OPTIMIZATION FOR CONVOLUTIONAL NEURAL NETWORK MODEL INFERENCE ON INTEGRATED GRAPHICS PROCESSING UNITS
Yu; Hao et al.
US 12450475 B1
TUNED EXECUTABLE CODE SERVICE FOR MACHINE LEARNING MODELS
Baskaran; Muthu M. et al.
US 20150309779 A1
SYSTEMS AND METHODS FOR POWER OPTIMIZATION OF PROCESSORS
Datta; Pallab et al.
US 20200042856 A1
SCHEDULER FOR MAPPING NEURAL NETWORKS ONTO AN ARRAY OF NEURAL CORES IN AN INFERENCE PROCESSING UNIT
Horesh; Lior et al.
US 20200151580 A1
GENERATING AND MANAGING DEEP TENSOR NEURAL NETWORKS
Venkat; Anand et al.
US 20210103434 A1
METHODS, SYSTEMS, ARTICLES OF MANUFACTURE AND APPARATUS TO AUTOMATICALLY OPTIMIZE SOFTWARE PROGRAMS
Chang; Andre Xian Ming et al.
US 20220066760 A1
DEEP NEURAL NETWORKS COMPILER FOR A TRACE-BASED ACCELERATOR
Cho; Junguk et al.
US 20220129315 A1
DEEP LEARNING AUTOTUNING TASK OPTIMIZATION
LI; Jiajun et al.
US 20220188613 A1
SGCNAX: A SCALABLE GRAPH CONVOLUTIONAL NEURAL NETWORK ACCELERATOR WITH WORKLOAD BALANCING
Cho; Junguk et al.
US 20220198317 A1
CONTEXT-AWARE AND STATELESS DEEP LEARNING AUTOTUNING FRAMEWORK
Ju; Dz-ching
US 20220342666 A1
ACCELERATION OF OPERATIONS
Jain; Nilesh et al.
US 20240007414 A1
METHODS, SYSTEMS, ARTICLES OF MANUFACTURE AND APPARATUS TO OPTIMIZE RESOURCES IN EDGE NETWORKS
Ling; Andrew Chaang et al.
US 20240020537 A1
METHODOLOGY TO GENERATE EFFICIENT MODELS AND ARCHITECTURES FOR DEEP LEARNING
Zhang; Dan et al.
US 20240220768 A1
OPTIMIZING OFF-CHIP MEMORY ACCESSES ON A NEURAL NETWORK HARDWARE ACCELERATOR
Zhang; Dan et al.
US 20240370693 A1
FULL-STACK HARDWARE ACCELERATOR SEARCH
Banerjee, Tania, Mohamed Gadou, and Sanjay Ranka. "A genetic algorithm based approach for multi-objective hardware/software co-optimization." Sustainable Computing: Informatics and Systems 10 (2016): 36-47.
Chen, Changbo, and Haoyu Chi. "Auto-tuning matrix multiplication and convolution for deep learning on CPUs." (2021).
Phothilimthana, Phitchaya Mangpo, et al. "A flexible approach to autotuning multi-pass machine learning compilers." 2021 30th International Conference on Parallel Architectures and Compilation Techniques (PACT). IEEE, 2021.
Zhao, Jie, et al. "AKG: automatic kernel generation for neural processing units using polyhedral transformations." Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation. 2021.
Zhang, Dan, et al. "A full-stack search technique for domain optimized deep learning accelerators." Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. 2022.
Hagedorn, Bastian, et al. "Graphene: An ir for optimized tensor computations on gpus." Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. 2023.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW L TANK whose telephone number is (571)270-1692. The examiner can normally be reached Monday-Thursday 9a-6p.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Ell can be reached at 571-270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW L TANK/Primary Examiner, Art Unit 2141
Prosecution Insights