DETAILED ACTION
This non-final office action is responsive to application 18/389,848 as submitted 20 Dec. 2023.
Claim status is currently pending and under examination for claims 1-15 of which independent claims are 1, 8 and 15.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55. The application has an effective filing date of 12/21/22.
Information Disclosure Statement
As required by M.P.E.P. 609(c), the applicant’s submissions of the Information Disclosure Statement dated 12/20/23 is acknowledged by the examiner and the cited references have been considered in the examination of the claims now pending. As required by M.P.E.P. 609 C(2), a copy of the PTOL-1449 initialed and dated by the examiner is attached to the instant office action.
Claim Objections
Claims 1, 5, 8, 12 and 15 objected to because of the following informalities:
Claims 1, 8 and 15 limitation generating recites “the the” redundancy of terminology.
Claims 5 and 12 limitation determine recites “the computing resource information and the computing resource information” redundancy of terminology.
Appropriate correction is requested.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-4, 8-11 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over:
Zawish et al., “Complexity-Driven CNN Compression for Resource-constrained Edge AI” hereinafter Zawish (arXiv: 2208.12816v1).
Hou et al., “Network Pruning via Resource Reallocation” hereinafter Hou (arXiv: 2103.01847v1), in view of
Passov et al., “Gator: Customizable Channel Pruning of Neural Networks with Gating” hereinafter Passov (arXiv: 2205.15404v1).
With respect to claim 1, Zawish teaches:
A method for inferring a result data corresponding to an input data using an adaptive deep learning model in a mobile terminal including a memory and a processor {Zawish [P.4] Alg.1 is method to [P.5 ¶1] “accelerate the inference of a CNN on a computationally constrained device” with [P.10 ¶2] “CNN inference, measuring the CPU and RAM usage involves determining the scale of these resources” CNN is the model and mobile terminal is e.g. [P.1 ¶1] “IoT devices such as smartphones” Fig 2 and/or [P.10 Last¶] “Nvidia Jetson Nano, which is often utilized for edge AI applications. This edge device comprises a GPU with…RAM”}, the method comprising:
determining computing resource information of the mobile terminal {Zawish [P.4] Alg.1 at Lines 7,18 “FLOPs” and “Memory” are the resources, [P.3] FLOPs-aware and Memory-aware pruning determined as defined process with calculations to consider complexity which is described in terms of computational resources including processors and RAM, storage budget that is restricted for an edge/IoT device shown Fig 2 introduces smartphone [P.1 ¶1]. [P.10 ¶2]};
determining a basic deep learning model stored in the memory of the mobile terminal {Zawish [P.5 ¶2] “CNN… deep models to run at the edge, they must fit within the target device’s RAM” Fig 3 CNN e.g. MobileNetV2, VGG-16, ResNet-50, AlexNet [P.2 Sect. I.A], Alg.1 Line1 Input CNN};
inputting the input data into the adaptive deep learning model in the mobile terminal to determine the inferred result data to be outputted from the adaptive deep learning model {Zawish [P.4 Sect.B] “for a dataset D = {xi, yi}Zi=1 with Z inputs and their corresponding labels, the pruning mechanism aims to find a CNN model M’ with fewer parameters as compared to the baseline CNN model M” Alg.1 inputs, output is e.g. [P.8 ¶2] “y = F(x,{Wi})+x …y represents the output vector” so as for [P.10 ¶2] “performing a CNN inference” Fig 2, test results on CIFAR dataset at Tables V-VI, [P.6,7 Last¶]}.
Zawish further generates a compressed CNN model transformed by pruning said resources. However, Zawish does not explicitly disclose “allocable resources” which is met by Hou:
generating the adaptive deep learning model by transforming the basic deep learning model based on allocable resources determined with reference to the computing resource information of the the mobile terminal {Hou [P.4 Sect.3 Step4] “Resource reallocation… assign the remaining resources in the resource pool to those groups of layers… takes into account the ever-changing layer significance” Eq.5 details implementation, and “we eventually obtain the desired compact model” e.g. MobileNet [Abst, P.1 ¶1] “for resource-limited edge devices and mobile applications” subject to a [P.3 Last2¶] “target resource constraint M as λ (0 ≤ λ ≤ 1), which is a tunable hyperparameter” said resources are [P.2 ¶2] “store the saved resources (e.g., FLOPs, parameters or latency) in a resource pool” Further see Fig 2 pruning},
Hou is directed to resource constraints for deep learning model optimization and estimation thus being analogous. A person having ordinary skill in the art would have considered it obvious prior to the effective filing date to perform resource allocation for model transformations of Hou in combination for a motivation [P.1-2 Pg.Brk] “By shifting resources from less crucial layers to more significant layers, one can easily acquire a desired slim model from off-the-shelf network architectures… By estimating layer importance via some standard criteria, resource reallocation is then performed to assign resources to different layers based on the estimated layer significance” and is “robust to the criterion to determine the layer importance.”
However, the combination of Zawish and Hou does not appear to expressly disclose fewer layers which is disclosed by Passov:
wherein the adaptive deep learning model has a number of layers less than the basic deep learning model {Passov [P.12 ¶1] “reducing the number of layers” by “prune entire ResNet blocks” where ResNet is a known CNN/deep learning model shown Fig 2};
Passov is directed to deep learning model optimization thus being analogous. A person having ordinary skill in the art would have considered it obvious prior to the effective filing date to reduce the number of layers per Passov in combination to arrive at the invention as claimed as applying known techniques to known methods ready for improvement to yield predictable results and/or for a motivation [P.2 ¶4] “practical compression criteria, such as memory reduction, FLOPs reduction, and hardware-specific speedup” e.g. [P.12 ¶2] “Gator can accommodate a variety of auxiliary cost functions to optimize FLOPs, memory and latency reduction due to pruning. Latency reduction, in particular, is important for allowing to produce very light networks designed for small devices.”
With respect to claim 2, the combination of Zawish, Hou and Passov teaches the method of claim 1, wherein
the computing resource information of the mobile terminal includes at least one of storage capacity information, memory usage information, and processor usage information of the mobile terminal {Zawish [P.4-5] memory-aware information and processor usage information as FLOPS-aware where the information is characterized as complexity-based, and further stats are provided Tbls. I-VI. Also, example resource-constrained hardware specifications (i.e. GB RAM, and CPU or nano GPU) are disclosed [P.6 ¶3], [P.10 Last¶]}.
With respect to claim 3, the combination of Zawish, Hou and Passov teaches the method of claim 2, wherein
the mobile terminal further comprises a graphic processing unit (GPU) or a neural network processing unit (NPU) {Zawish [P.10 Last¶] “resource-constrained edge device, i.e., Nvidia Jetson Nano, which is often utilized for edge AI applications. This edge device comprises a GPU”}, and
wherein the computing resource information includes GPU usage information or NPU usage information {Zawish [P.10] Tbl.VI gives information evaluated on Nvidia Jetson Nano, e.g. FLOPs. Additionally see Hou [P.6-7 Sect4.1] “GPU memory usage”}.
With respect to claim 4, the combination of Zawish, Hou and Passov teaches the method of claim 1, wherein
the computing resource information changes over time {Zawish [P.9 ¶3] “speedups, i.e. reduction in FLOPs” floating operations (per second) as well as “speedups, i.e., inference time which is also known as latency” plotted Figs 4-5 in ms milliseconds and cpu/memory utilization}.
With respect to claim 8, the rejection of claim 1 is incorporated. The difference in scope being an apparatus comprising memory and processor to execute stored instructions with model and performing limitations of method claim 1. Zawish discloses [P.6 ¶3] “We implemented the networks using Keras deep learning API with TensorFlow as a backend on the Nvidia Tesla K20 GPU with 8GB memory” similarly at [P.10 ¶3-5]. The remainder of this claim is rejected for the same rationale as claim 1.
With respect to claim 9, the combination of Zawish, Hou and Passov teaches the apparatus of claim 8, and further teaches the limitation of claim 2. Therefore, the rejection of claim 2 is applied to claim 9.
With respect to claim 10, the combination of Zawish, Hou and Passov teaches the apparatus of claim 9, and further teaches the limitation of claim 3. Therefore, the rejection of claim 3 is applied to claim 10.
With respect to claim 11, the combination of Zawish, Hou and Passov teaches the apparatus of claim 8, and further teaches the limitation of claim 4. Therefore, the rejection of claim 4 is applied to claim 11.
With respect to claim 15, the rejection of claim 1 is incorporated. The difference in scope being a non-transitory computer readable storage medium storing instructions executed by processor to perform limitations of method claim 1. Zawish discloses [P.4] Alg.1 code and [P.6 ¶3] “We implemented the networks using Keras deep learning API with TensorFlow as a backend on the Nvidia Tesla K20 GPU with 8GB memory” similarly at [P.10 ¶3-5]. The remainder of this claim is rejected for the same rationale as claim 1.
Claims 5 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Zawish, Hou and Passov in view of Yeom et al., US PG Pub No 2023/0214657A1 hereinafter Yeom.
With respect to claim 5, the combination of Zawish, Hou and Passov teaches the method of claim 1, wherein the generating the adaptive deep learning model includes:
estimating resource information required to process the basic deep learning model {Yeom [0097] “estimate FLOPs of the pruned second model” similar at [0057] “minimize a distance between current FLOPs and targeted FLOPs” see [0036] Alg.1 and Tables 2-3};
determining the allocable resources based on the computing resource information and the computing resource information required to process the basic deep learning model {Yeom [0048] “in the case of running a neural network on a device with limited computational resources, pruning to efficiently reduce the FLOPs is a common solution” thus resources determined limited e.g. Fig 4, Tbl.4 [0085] Jetson-Nano or Raspberry Pi, then FLOPs resources are adjusted. Fig 1 “Targeted FLOPs” [0036]}; and
generating the adaptive deep learning model by transforming the basic deep learning model with reference to the allocable resources {Yeom [0048] “generate any size of pruned models by constraining the FLOPs” implemented [0036] Alg.1 which outputs the pruned model}.
Yeom is directed to inference using adapted deep learning models with resource constraints thus being analogous. A person having ordinary skill in the art would have considered it obvious prior to the effective filing date to estimate FLOPs for generating pruned models per Yeom in combination to arrive at the invention as claimed for a motivation [0086] “lightweight method according to example embodiments shows that inference time for pruned models is improved in every target edge device.”
With respect to claim 12, the combination of Zawish, Hou and Passov teaches the apparatus of claim 8, and further combination with Yeom teaches the limitation of claim 5. Therefore, the rejection of claim 5 is applied to claim 12.
Claims 6 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Zawish, Hou and Passov in view of Jeon et al., “Target Capacity Filter Pruning Method for Optimized Inference Time Based on YOLOv5 in Embedded Systems” hereinafter Jeon.
With respect to claim 6, the combination of Zawish, Hou and Passov teaches the method of claim 1, wherein the allocable resources are determined {Hou [P.4 Sect.4 Step 4] “resource reallocation” Eq.5}
with reference to the computing resource information at the time of starting inference of the adaptive deep learning model {Jeon [P.70845 Sect. III.D] Eq.11 “it is possible to determine in advance the FLOPs that the network must have to satisfy a specific inference time” as “correlation between inference time and the amount of computation… F and F* represent the FLOPs of the baseline network and the FLOPs of the pruned network, respectively. Moreover, T and T* represent the inference time of the baseline network and the inference time of the pruned network, respectively”}.
Jeon is directed to inference time optimization for deep learning models in resource limited systems thus being analogous. A person having ordinary skill in the art would have considered it obvious prior to the effective filing date to correlate inference time with FLOP resources for baseline and pruned networks per Jeon in combination with Hou’s resource reallocation to arrive at the invention as claimed for a motivation [P.70845 Sect. III.D] “If the relationship between the amount of computation and the inference time as described above is used, it is possible to easily predict and generate a network with the target inference speed.”
Claims 7 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Zawish, Hou, Passov and Yeom in view of Li et al., US PG Pub No 2023/0084203A1 hereinafter Li.
With respect to claim 7, the combination of Zawish, Hou, Passov and Yeom teaches the method of claim 5. Li teaches wherein the determining of the allocable resources includes
determining a target downsize ratio associated with a minimum ratio of ratios of predetermined first values included in the resource information required to process the basic deep learning model and predetermined second values included in the computing resource information {Li [0147] “target pruning ratio… LCE may continue to decrease” i.e. decrease to minimum e.g. [0050] “minimizing the loss of each pruned network”, and [0077-78] equation gives ratio similar to instant spec and uses loss function includes lambda λ to control tradeoff between accuracy and compression}, and
Li is directed to inference with adapted deep learning models in resource limited environments thus being analogous. A person having ordinary skill in the art would have considered it obvious prior to the effective filing date to determine target pruning ratio per Li in combination for a motivation [0003,5] “compression may be needed for large neural networks, e.g., a convolutional neural network (CNN), to reduce computation overheads that these networks may be deployed in resource limited devices without a significant drop in accuracy… improved efficiency and performance.”
However, Li does not disclose the following limitation which is met by Passov:
wherein the generating the adaptive deep learning model includes adjusting the number of layers included in the basic deep learning model based on the target downsize ratio {Passov [P.12 ¶1] “reducing the number of layers” as “prune entire ResNet blocks” shown Fig 3, further disclosing [P.4 ¶1] “layer pruning ratios minimizing either FLOPs or memory metrics for a given target” Fig 1 gating zeros the weights such that zero-value of binary (0,1) is 100% pruning ratio}.
A person having ordinary skill in the art would have considered it obvious prior to the effective filing date to reduce the number of layers per Passov in combination to arrive at the invention as claimed as applying known techniques to known methods ready for improvement to yield predictable results and/or for a motivation [P.2 ¶4] “practical compression criteria, such as memory reduction, FLOPs reduction, and hardware-specific speedup” e.g. [P.12 ¶2] “Gator can accommodate a variety of auxiliary cost functions to optimize FLOPs, memory and latency reduction due to pruning. Latency reduction, in particular, is important for allowing to produce very light networks designed for small devices.”
With respect to claim 14, the combination of Zawish, Hou, Passov and Yeom teaches the apparatus of claim 12, and further combination with Li teaches the limitation of claim 7. Therefore, the rejection of claim 7 with equal motivation is applied to claim 14.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Choe et al., US PG Pub No 2023/0259776A1 UIF Yonsei (KR) discloses argmin pruning ratios
Shim et al., “Layer-wise Pruning of Transformer Attention Heads for Efficient Language Modeling” arXiv: 2110.03252v1 Seoul Nat. Univ. discloses gating mechanism
Oh et al., “Layerweaver: Maximizing Resource Utilization of Neural Processing Units via Layer-Wise Scheduling” Sungkyunkwan/SNU, Figs 1, 6
Song et al., “CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Prediction” arXiv: 2203.04570v1 see Alg.2 layer-aware pruning ratio
Dutta et al., US PG Pub No 2024/0046099A1 discloses optimal pruning ratio acceleration
Kamma et al., “Pruning Ratio Optimization with Layer-Wise Pruning Method for Accelerating Convolutional Neural Networks” see Alg.1
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Chase P Hinckley whose telephone number is (571)272-7935. The examiner can normally be reached M-F 9:00 - 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda M. Huang can be reached at 571-270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHASE P. HINCKLEY/Examiner, Art Unit 2124
Prosecution Insights