DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 10, 15-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al. (“TVM: An Automated End-to-End Optimizing Compiler for Deep Learning”, October 8–10, 2018 • Carlsbad, CA, USA, pages 579-593) in view of Cox et al. (“MASA: Responsive Multi-DNN Inference on the Edge”, 22-26 March 2021).
Regarding claim 1.
Chen teaches a computer-implemented method for multi-model implementation comprising: transforming, by a neural computing optimizer (NCO), each of multiple neural network models into a hardware-specific format that fits in a heterogeneous hardware platform (see page 586, “Given the rich set of schedule primitives, our remaining problem is to find optimal operator implementations for each layer of a DL model. Here, TVM creates a specialized operator for the specific input shape and layout associated with each layer.”, also see page 588, “we evaluated TVM on four types of platforms: (1) a server-class GPU, (2) an embedded GPU, (3) an embedded CPU, and (4) a DL accelerator implemented on a low-power FPGA SoC.”, i.e. TVM transforms DL models into optimized hardware and the four types support heterogenous hardware platforms);
establishing, a model tree for the transformed multiple neural network models to represent a collaborative relationship among the transformed multiple neural network models for implementation in the heterogeneous hardware platform (see page 591, “High-level computation graph DSLs are a typical way to represent and perform high-level optimizations.”);
mapping, by a neural computing accelerator (NCA), the model tree into the heterogeneous hardware platform for deployment (see page 590, “We built a driver library for VDLA with a C runtime API that constructs instructions and pushes them to the target accelerator for execution. Our code generation algorithm then translates the accelerator program to a series of calls into the runtime API.”);
and scheduling, by the NCA, one or more transformed neural network models for action using corresponding mapped resources in the heterogeneous hardware platform (see page 586-587 sections 5.1-5.3, “At a high level, we would like to consider as many configurations as possible and let the optimizer manage the selection burden. Consequently, the optimizer must search over billions of possible configurations for the real world DL workloads used in our experiments.”, also see page 587, section 5.4, “A distributed device pool scales up the running of on hardware trials and enables fine-grained resource sharing among multiple optimization jobs. TVM implements a customized, RPC-based distributed device pool that enables clients to run programs on a specific type of device.”).
Chen do not specifically teach establishing, a model tree for the transformed multiple neural network models to represent a collaborative relationship among the transformed multiple neural network models for implementation in the heterogeneous hardware platform; and scheduling across multiple neural networks.
Cox teaches establishing, a model tree for the transformed multiple neural network models to represent a collaborative relationship among the transformed multiple neural network models for implementation in the heterogeneous hardware platform; and scheduling across multiple neural networks (see page 4, under figure 3, “The scheduler constructs the dependency graph for tasks of each DNN and then distributes those tasks to free workers based on their dependency graph, as shown in Fig. 4.”, also same page section c. “The scheduler sends the loading and execution tasks of all layers of DNNs that are requested for specific images to workers.”).
Both Chen and Cox pertain to the problem of neural network optimization, thus being analogous. It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine Chen and Cox to teach the above limitations. The motivation for doing so would be “we design and implement MASA, a responsive memory aware multi-DNN execution framework, an on-device middleware featuring on modeling inter- and intra-network dependency and leveraging complimentary memory usage of each layer. MASA can consistently ensure the average response time when deterministically and stochastically executing multiple DNN-based image analyses. We extensively evaluate MASA on three configurations of Raspberry Pi and a large set of popular DNN models triggered by different generation patterns of images. Our evaluation results show that MASA can achieve lower average response times by up to 90% on devices with small memory, i.e., 512 MB to 1 GB, compared to the state of the art multi-DNN scheduling solutions.” (see Chen Abstract).
Regarding claim 2.
Chen and Cox teaches the computer-implemented method of claim 1,
Chen further teaches wherein the multiple neural network models are deep neural network (DNN) models (see page 588, “we evaluated TVM on four types of platforms: (1) a server-class GPU, (2) an embedded GPU, (3) an embedded CPU, and (4) a DL accelerator implemented on a low-power FPGA SoC. The benchmarks are based on real world DL inference workloads, including ResNet [16], MobileNet [19], the LSTM Language Model [48], the Deep Q Network (DQN) [28] and Deep Convolutional Generative Adversarial Networks (DCGAN)”).
Regarding claim 3.
Chen and Cox teaches the computer-implemented method of claim 1,
Chen further teaches wherein the DNN models are vision-based DNN models (see page 588, “we evaluated TVM on four types of platforms: (1) a server-class GPU, (2) an embedded GPU, (3) an embedded CPU, and (4) a DL accelerator implemented on a low-power FPGA SoC. The benchmarks are based on real world DL inference workloads, including ResNet [16], MobileNet [19], the LSTM Language Model [48], the Deep Q Network (DQN) [28] and Deep Convolutional Generative Adversarial Networks (DCGAN)”, i.e. including ResNet [16], MobileNet [19 are visual based DNN models).
Regarding claim 4.
Chen and Cox teaches the computer-implemented method of claim 2,
Cox further teaches wherein the multiple neural network models are collaborative DNN models having a dependency, the collaborative DNN models comprise a first DNN model in a first hierarchical level and multiple DNN models in a second hierarchical level, any multiple DNN model in the second hierarchical level is executed after the first DNN model in the first hierarchical level (see page 4, “The scheduler constructs the dependency graph for tasks of each DNN and then distributes those tasks to free workers based on their dependency graph, as shown in Fig. 4.”, also see section c, “The scheduler follows two principles: (i) the task dependency, i.e., loading task of layer j should be completed before starting its execution task, and (ii) a hybrid order of layer type and memory constraint within and across multiple DNNs. As such, MASA incorporates the memory dependency at the inter- and intra-network levels. The loading and execution tasks of all layers across all specified DNNs are kept sorted into two groups: the waiting and ready group. Tasks start in the waiting group. Once all their dependencies are satisfied they are moved to the ready group. Dependencies are resolved based on the dependencies graphs defined by the multi-DNN job and each DNN model.”).
The motivation utilized in the combination of claim 1, super, applies equally as well to claim 4.
Claim 10 recites a system to perform the method recited in claim 1. Therefore the rejection of claim 1 above applies equally here.
Regarding claim 15.
Chen and Cox teaches the computer-implemented method of claim 10,
Cox further teaches wherein the NCO schedules the one or more transformed neural network models for operation in a route comprising multiple actions to implement a task pipeline (see page 2, “The network preparator separates layer execution into two tasks, namely memory loading, and execution. The scheduler handles both loading and execution tasks for each layer of all DNNs by dynamically checking the inter-network dependency, the available memory space, and the estimated memory demand of all available layers1. We implement MASA on top of Caffe, and then extensively evaluate MASA on a large set of multi-DNN inference scenarios and a representative set of hardware, namely Rasberry Pi. Our evaluation shows that the memory-aware features of MASA significantly reduce not only the response times, and resource footprints but also energy consumption, compared to the state of the art multi-DNN engines.”, also see page 4, “The scheduler constructs the dependency graph for tasks of each DNN and then distributes those tasks to free workers based on their dependency graph, as shown in Fig. 4. Loading tasks have to be strictly executed prior to the execution tasks of the same layers. More importantly, the scheduler dynamically monitors the memory usages and estimated memory requirement for tasks of all DNNs that are ready. The estimated memory requirement is obtained from the profiler. The detailed algorithms of the scheduler is described in Section III-C. When all tasks from a DNN are completed, the inference results are sent as a response to the application” ).
The motivation utilized in the combination of claim 1, super, applies equally as well to claim 15.
Claim 16 recites a non-transitory computer-readable medium or media comprising one or more sequences of instructions to perform the method recited in claim 1. Therefore the rejection of claim 1 above applies equally here.
Claim(s) 5-6, 11-13, and 17-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al. (“TVM: An Automated End-to-End Optimizing Compiler for Deep Learning”, October 8–10, 2018 • Carlsbad, CA, USA, pages 579-593) in view of Cox et al. (“MASA: Responsive Multi-DNN Inference on the Edge”, 22-26 March 2021) in further in view of Rivas-Gomez et al. (“Exploring the Vision Processing Unit as Co-processor for Inference”, 2018 IEEE).
Regarding claim 5.
Chen and Cox teaches the computer-implemented method of claim 4,
Chen further teaches wherein the heterogeneous hardware platform is an edge device comprising one or more central processing units (CPUs), one or more graphic processing units (GPUs) (see page 588, “we evaluated TVM on four types of platforms: (1) a server-class GPU, (2) an embedded GPU, (3) an embedded CPU, and (4) a DL accelerator implemented on a low-power FPGA SoC”), and multiple vision processing units (VPUs),each VPU comprises multiple cores.
Chen and Cox do not teach multiple vision processing units (VPUs),each VPU comprises multiple cores.
Rivas-Gomez teaches multiple vision processing units (VPUs),each VPU comprises multiple cores (see page 593, “we analyze three implementations inside the NCSw framework that target a CPU, a GPU, and a multi-VPU configuration, respectively. We evaluate these implementations in terms of inference performance and confidence error. For this purpose, we use the Intel-optimized Caffe-MKL fork (v1.0.7) for Intel processors, the NVIDIA optimized Caffe-cuDNN fork (v0.16.4) for NVIDIA graphic cards, and the Neural Compute SDK (v1.12.00.01) for the Myriad 2 VPU on the NCS.”, also see page 593, “we demonstrate equivalent performance results by using a parallel, multi VPU configuration with eight NCS devices.”, also see page 590, “the Myriad 2 VPU is designed featuring 12 highly-parallelizable vector processors, named Streaming Hybrid Architecture Vector Engines (SHAVE). Each SHAVE processor contains wide register files and several functional units. These are controlled by Variable-Length Long Instruction Word (VLLIW) packets.”).
Chen, Cox and Rivas-Gomez pertain to the problem of neural network optimization, thus being analogous. It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine Chen, Cox and Rivas-Gomez to teach the above limitations. The motivation for doing so would be “we explore the so-called Vision Processing Unit (VPU), a highly-parallel vector processor with a power envelope of less than 1W. We evaluate this chip during inference using a pre-trained GoogLeNet convolutional network model and a large image dataset from the ImageNet ILSVRC challenge. Preliminary results indicate that a multi VPU configuration provides similar performance compared to reference CPU and GPU implementations, while reducing the thermal-design power (TDP) up to 8× in comparison.” (see Rivas-Gomez Abstract).
Regarding claim 6.
Chen, Cox and Rivas-Gomez teaches the computer-implemented method of claim 5,
Rivas-Gomez further teaches wherein the first DNN model is deployed into using the one or more CPUs and the one or more GPUs, the multiple DNN models in the second hierarchical level are deployed among the a plurality of VPUs (see page 593, “we analyze three implementations inside the NCSw framework that target a CPU, a GPU, and a multi-VPU configuration, respectively. We evaluate these implementations in terms of inference performance and confidence error. For this purpose, we use the Intel-optimized Caffe-MKL fork (v1.0.7) for Intel processors, the NVIDIA optimized Caffe-cuDNN fork (v0.16.4) for NVIDIA graphic cards, and the Neural Compute SDK (v1.12.00.01) for the Myriad 2 VPU on the NCS.”, also see page 593, “we demonstrate equivalent performance results by using a parallel, multi VPU configuration with eight NCS devices.”, also see page 590, “the Myriad 2 VPU is designed featuring 12 highly-parallelizable vector processors, named Streaming Hybrid Architecture Vector Engines (SHAVE). Each SHAVE processor contains wide register files and several functional units. These are controlled by Variable-Length Long Instruction Word (VLLIW) packets.”).
Chen, Cox and Rivas-Gomez pertain to the problem of neural network optimization, thus being analogous. It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine Chen, Cox and Rivas-Gomez to teach the above limitations. The motivation for doing so would be “we explore the so-called Vision Processing Unit (VPU), a highly-parallel vector processor with a power envelope of less than 1W. We evaluate this chip during inference using a pre-trained GoogLeNet convolutional network model and a large image dataset from the ImageNet ILSVRC challenge. Preliminary results indicate that a multi VPU configuration provides similar performance compared to reference CPU and GPU implementations, while reducing the thermal-design power (TDP) up to 8× in comparison.” (see Rivas-Gomez Abstract).
Claim 11 recites a system to perform the method recited in claim 5. Therefore the rejection of claim 5 above applies equally here.
Claim 12 recites a system to perform the method recited in claim 4. Therefore the rejection of claim 4 above applies equally here.
Claim 13 recites a system to perform the method recited in claim 6. Therefore the rejection of claim 6 above applies equally here.
Claim 17 recites a non-transitory computer-readable medium or media comprising one or more sequences of instructions to perform the method recited in claim 5. Therefore the rejection of claim 5 above applies equally here.
Claim 18 recites a non-transitory computer-readable medium or media comprising one or more sequences of instructions to perform the method recited in claim 4. Therefore the rejection of claim 4 above applies equally here.
Claim 19 recites a non-transitory computer-readable medium or media comprising one or more sequences of instructions to perform the method recited in claim 6. Therefore the rejection of claim 6 above applies equally here.
Allowable Subject Matter
Claims 7-9, 14 and 20 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Related prior arts:
Rotem et al. (“Glow: Graph Lowering Compiler Techniques for Neural Networks”, 3 Apr 2019) teaches design of Glow, a machine learning compiler for heterogeneous hardware. It is a pragmatic approach to compilation that enables the generation of highly optimized code for multiple targets. Glow lowers the traditional neural network dataflow graph into a two-phase strongly-typed intermediate representation.
Baek et al. (“A Multi-Neural Network Acceleration Architecture”, 2020) teaches AI-MultiTasking (AI-MT), a novel accelerator architecture which enables a cost-effective, high performance multi-neural network execution. The key idea of AI-MT is to fully utilize the accelerator’s computation resources and memory bandwidth by matching compute- and memory intensive tasks from different networks and executing them in parallel.
DALLI et al. (US 20210350211 A1) teaches distributed architecture may include a parallel execution step which may combine parallel XNNs into an aggregate model by calculating the average (or weighted average) from the parallel models. A distributed hybrid XNN/XAI architecture may include multiple independent models which can work independently without relying on the full distributed architecture.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IMAD M KASSIM whose telephone number is (571)272-2958. The examiner can normally be reached 10:30AM-5:30PM, M-F (E.S.T.).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael J. Huntley can be reached at (303) 297 - 4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/IMAD KASSIM/Primary Examiner, Art Unit 2129
Prosecution Insights