DETAILED ACTION
Claims 1-17 are pending in the application.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Examiner’s Notes
The Examiner cites particular sections in the references as applied to the claims below for the convenience of the applicant(s). Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the applicant(s) fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the Examiner.
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Should applicant desire to obtain the benefit of foreign priority under 35 U.S.C. 119(a)-(d) prior to declaration of an interference, a certified English translation of the foreign application must be submitted in reply to this action. 37 CFR 41.154(b) and 41.202(e).
Failure to provide a certified translation may result in no benefit being accorded for the non-English application.
Information Disclosure Statement
The listing of references in the specification is not a proper information disclosure statement (see the non-patent literature documents listed on pages 6-7). 37 CFR 1.98(b) requires a list of all patents, publications, or other information submitted for consideration by the Office, and MPEP § 609.04(a) states, "the list may not be incorporated into the specification but must be submitted in a separate paper." Therefore, unless the references have been cited by the examiner on form PTO-892, they have not been considered.
Specification
The use of the term BLUETOOTH, which is a trade name or a mark used in commerce, has been noted in this application. The term should be accompanied by the generic terminology; furthermore the term should be capitalized wherever it appears or, where appropriate, include a proper symbol indicating use in commerce such as ™, SM , or ® following the term.
Although the use of trade names and marks used in commerce (i.e., trademarks, service marks, certification marks, and collective marks) are permissible in patent applications, the proprietary nature of the marks should be respected and every effort made to prevent their use in any manner which might adversely affect their validity as commercial marks.
Claim Objections
Claims 1-16 objected to because of the following informalities:
Claim 1: The abbreviation “NPL” in line 2 should have been in its full form at least for the first time it is used in the claim; i.e. it should have been --Neural Processing Unit (NPU)— and “Neural Processing Unit (NPU)” in line 6 should have been –NPL--.
Claim 1: “a multi-precision” (line 6) and “multiple” (line 8) should have been –the multi-precision— and –the multiple—, respectively.
Claims 2-8 inherit the features of claim 1 and are objected to accordingly.
Claim 8: “adjust” (line 1) and “the execution” (line 2) should have been –adjusts— and –execution—, respectively.
Claim 9: The abbreviation “NPL” in line 2 should have been in its full form at least for the first time it is used in the claim; i.e. it should have been --Neural Processing Unit (NPU)— and “Neural Processing Unit (NPU)” in line 5 should have been –NPL--.
Claim 9: “a multi-precision” (line 5) and “multiple” (line 7) should have been –the multi-precision— and –the multiple—, respectively.
Claims 10-16 inherit the features of claim 9 and are objected to accordingly.
Appropriate corrections are required. Applicant is advised to review the entire claims for further needed corrections.
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-8 and 17 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 is directed to an apparatus comprising a memory and a processor, and recites “the processor is configured to” (line 5) which identifies an intended use for the claimed apparatus. More specifically, it is not clear if the apparatus actually implements the functions recited therewith, or if the apparatus is merely required to have a configuration (e.g. a hardware processor) that can implement these functions.
For the following analysis, the Examiner will consider the apparatus implementing the recited functions (e.g. the memory comprising instructions and the processor being configured to execute these instructions to implement the recited functions).
Claims 2-8 inherit the features of claim 1 and are rejected accordingly.
Claim 17 recites “A computer-readable recording medium for storing a computer program” (line 1) which identifies an intended use for the claimed medium. More specifically, it is not clear if the medium actually stores the computer program, or if the medium is merely intended for storing the computer program.
For the following analysis, the Examiner will consider the medium actually storing the computer program; i.e. –A computer-readable recording medium storing a computer program—.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 17 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Claim 17 is directed to a “computer-readable recording medium”. However, the claim does not limit the recording medium to non-transitory embodiments. Furthermore, even though the specification provides non-limiting examples for the “computer-readable recording medium” (see page 8, lines 21-27), it does not provide an definition for the recording medium that explicitly limits the medium to non-transitory embodiments.
As such, the “computer-readable recording medium” recited in claim 17 encompasses transitory embodiments which are non-statutory. See MPEP §2106.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-17 are rejected under 35 U.S.C. 103 as being unpatentable over Venieris et al. (“NAWQ-SR: A Hybrid-Precision NPU Engine for Efficient On-Device Super-Resolution”; March 8, 2023; from IDS filed on February 02, 2026; hereinafter “Venieris”) in view of Li et al. (US 2024/0311196 A1; hereinafter “Li”).
With respect to claim 1, Venieris teaches: An apparatus for providing execution plans of multiple mixed-precision deep learning models based on a multi-precision NPU (see e.g. page 2, column 1, paragraph 3: “hybrid-precision execution scheme together with a methodology for optimizing the deployment of SR DNNs to the latency and quality requirements of the target application. By considering the multiple precisions supported by a given NPU, our framework adapts each layer’s wordlength through a single-shot optimization algorithm that co-optimizes the per-layer quantization of the DNN and the scheduling of its layers on the NPU”), the apparatus comprising:
a memory (see e.g. Venieris, page 4, column 2, Fig. 3: “Main Memory”); and
a processor electrically connected to the memory (see e.g. Venieris, page 10, column 1, paragraph 1: “SR systems using heterogeneous processors, CPU, and NPU”),
wherein the processor is configured to:
form a multi-precision Neural Processing Unit (NPU) including a processing element (PE) (see e.g. Venieris, page 1, abstract: “NAWQ-SR exploits the multi-precision capabilities of modern mobile NPUs”; page 6, column 2, paragraph 3: “To guide the on-device execution, the Neural Image Codec introduces a dispatcher that, given the per-layer quantization configuration ql, schedules execution to the appropriate hardware processor of the NPU, using the specified bitwidth, scale factor and zero point… the dispatcher partitions the DNN into groups of consecutive layers based on their target bitwidth (e.g. INT8 or A16W8) and range estimation technique (dl), scheduling execution on a per-partition basis”; and page 7, column 1, paragraph 1: “dispatcher coordinates with the NPU executor to perform inference on a target processor (e.g. either HVX or HTA in SDM865’s NPU) that supports the requested partition’s bitwidth”)
generate multiple mixed-precision deep learning models that perform multiple precision operations requiring different degrees of precision in a model execution process (see e.g. Venieris, page 2, column 1, paragraph 3: “By considering the multiple precisions supported by a given NPU, our framework adapts each layer’s wordlength through a single-shot optimization algorithm that co-optimizes the per-layer quantization of the DNN and the scheduling of its layers on the NPU”; page 4, column 2, paragraph 4: “Hybrid-Precision Quantization Strategy. To implement multi-wordlength DNNs, a hybrid-precision quantization strategy needs to be defined. The proposed strategy utilizes different wordlength bl, scale factor sl and zero point zl for each layer l, such that a value x is quantized to a b-bit integer xquant as xquant = [x . sl – zl]. To introduce different wordlengths among layers, quantization is performed such that all values within each activations tensor at the input of each layer have a single wordlength, scale factor and zero point”; and page 5, column 2, Algorithm 1), and
generate the execution plans for executing the multiple mixed-precision deep learning models on the multi-precision NPU (see e.g. Venieris, page 4, column 2, paragraph 1: “optimization of the DNN execution so that different operations are performed using different precision”; and page 6, column 2, paragraph 3: “Dispatcher. To guide the on-device execution, the Neural Image Codec introduces a dispatcher that, given the per-layer quantization configuration ql, schedules execution to the appropriate hardware processor of the NPU, using the specified bitwidth, scale factor and zero point… dispatcher partitions the DNN into groups of consecutive layers based on their target bitwidth (e.g. INT8 or A16W8) and range estimation technique (dl), scheduling execution on a per-partition basis”).
Venieris does not but Li teaches:
composed of multiple Micro-PEs (see e.g. Li, paragraph 27: “splits data and a task into micro tasks and maps the tasks to a plurality of processing elements (PEs), and orchestrates data movement between the PEs”),
Venieris and Li are analogous art because they are in the same field of endeavor: managing processing element operations utilizing neural network models. Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention to modify Venieris with the teachings of Li. The motivation/suggestion would be to improve processing efficiency by implementing finer-grained execution management for the PEs.
With respect to claim 2, Venieris as modified teaches: The apparatus of claim 1, wherein the processor generates the multiple mixed-precision deep learning models through Hardware-Aware Mixed-precision Quantization (HAWQ) (see e.g. Venieris, page 11, column 2, paragraph 6: “Metric-based methods assign bitwidths by estimating the layerwise resilience to low precision with metrics that are relatively cheap to calculate, such as the Hessian-based metric adopted by HAWQ”).
With respect to claim 3, Venieris as modified teaches: The apparatus of claim 1, wherein the processor generates the execution plan to ensure efficient distribution of precision operations for each model, taking into account a structure and characteristics of the multiple mixed-precision deep learning models (see e.g. page 3, column 2, paragraph 5: “Towards addressing the shortcomings of existing mobile SR systems, we propose NAWQ-SR, an NPU-centric framework that maximizes the efficiency of on-device SR. NAWQSR leverages the fact that different parts of SR neural architectures have nonuniform precision needs, in order to partition the execution across the NPU’s heterogeneous units”; and page 6, column 2, paragraph 3: “Dispatcher. To guide the on-device execution, the Neural Image Codec introduces a dispatcher that, given the per-layer quantization configuration ql, schedules execution to the appropriate hardware processor of the NPU, using the specified bitwidth, scale factor and zero point. To ensure efficient execution, this process is performed in a number of steps. First, the dispatcher adopts a partitioning strategy to reduce the communication between the codec components and the target processors. Specifically, the dispatcher partitions the DNN into groups of consecutive layers based on their target bitwidth (e.g. INT8 or A16W8) and range estimation technique (dl), scheduling execution on a per-partition basis. As such, the scheduling of consecutive layers that need to interact with the same components is coalesced, amortizing the cost of communication between components”).
With respect to claim 4, Venieris as modified teaches: The apparatus of claim 1, wherein the processor generates the execution plan to ensure efficient resource utilization while minimizing execution time of each of the multiple mixed-precision deep learning models using a dynamic programming method (see e.g. page 1, abstract: “NAWQ-SR exploits the multi-precision capabilities of modern mobile NPUs in order to minimize latency, while meeting user-specified quality constraints”; page 2, column 1, paragraph 2: “NAWQ-SR introduces an NPU-centric approach, comprising a novel hybrid-precision execution paradigm and a runtime neural image codec that exploit the multi-precision processing capabilities of modern mobile NPUs to minimize latency while meeting the user-specified quality targets”; and page 5, column 2, paragraph 3: “4.2 Dynamic Range Adaptation”).
With respect to claim 5, Venieris as modified teaches: The apparatus of claim 4, wherein the processor generates the execution plan as a result of applying the dynamic programming method based on a result of measuring execution times of all precision operations for each mixed-precision deep learning model through a pre-simulation method or an actual execution method (see e.g. Venieris, page 2, column 1, paragraph 4: “selectively applies adaptive arithmetic precision on quantization-sensitive layers, enhancing them with wider representational power at run time. We dynamically adapt the quantization parameters of the selected layers in a per-sample input-dependent manner”; page 5, column 2, paragraph 4: “dynamic range estimation (DRE). DRE adapts the scale factor and zero point of an activations tensor at run time, based on the actual range of its values for the particular input sample”; and page 5, column 2, Table 5).
With respect to claim 6, Venieris as modified teaches: The apparatus of claim 1, wherein the processor dynamically allocates at least one [Micro-]PE that executes each of the multiple precision operations in a process of executing the multiple mixed-precision deep learning models according to the execution plan (see e.g. Venieris, page 6, column 2, paragraph 3: “a dispatcher that, given the per-layer quantization configuration ql, schedules execution to the appropriate hardware processor of the NPU, using the specified bitwidth, scale factor and zero point”; and page 7, column 1, paragraph 1: “dispatcher coordinates with the NPU executor to perform inference on a target processor (e.g. either HVX or HTA in SDM865’s NPU) that supports the requested partition’s bitwidth”).
Venieris does not but Li teaches:
Micro (see e.g. Li, paragraph 27: “splits data and a task into micro tasks and maps the tasks to a plurality of processing elements (PEs), and orchestrates data movement between the PEs”),
Venieris and Li are analogous art because they are in the same field of endeavor: managing processing element operations utilizing neural network models. Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention to modify Venieris with the teachings of Li. The motivation/suggestion would be to improve processing efficiency by implementing finer-grained execution management for the PEs.
With respect to claim 7, Venieris as modified teaches: The apparatus of claim 6, wherein the processor controls precision operations of a model with high execution priority among the multiple mixed-precision deep learning models to be preferentially executed according to the execution plan (see e.g. Venieris, page 6, column 1, paragraph 3: “to effectively utilize DRE, we have to devise a method of: i) quantifying the resilience of each layer to low precision, and ii) an algorithm that leverages this information to selectively apply DRE to a subset of the DNN’s layers” and paragraphs 4-5: “the layers are sorted in a decreasing order of quality drop (line 8). DRE Layer Selection. After selecting the highest performing bitwidths via QuantSR-WLopt and estimating the layerwise resilience to quantization through LRA, NAWQSR picks a subset of layers, to have their scale factors and zero points computed at run time”).
With respect to claim 8, Venieris as modified teaches: The apparatus of claim 6, wherein the processor dynamically adjust the execution plan by tracking and monitoring the execution time and resource usage for precision operations of each mixed-precision deep learning model (see e.g. Venieris, page 5, column 2, paragraph 4: “NAWQ-SR introduces a new design dimension to the quantization strategy, which we name dynamic range estimation (DRE). DRE adapts the scale factor and zero point of an activations tensor at run time, based on the actual range of its values for the particular input sample”; page 6, column 1, paragraph 3: “to effectively utilize DRE, we have to devise a method of: i) quantifying the resilience of each layer to low precision, and ii) an algorithm that leverages this information to selectively apply DRE to a subset of the DNN’s layers” and paragraphs 4-5: “the layers are sorted in a decreasing order of quality drop (line 8). DRE Layer Selection. After selecting the highest performing bitwidths via QuantSR-WLopt and estimating the layerwise resilience to quantization through LRA, NAWQSR picks a subset of layers, to have their scale factors and zero points computed at run time”).
With respect to claims 9-13: Claims 9-13 are directed to a method corresponding to the active functions implemented by the apparatus disclosed in claims 1-5, respectively; please see the rejections directed to claims 1-5 above which also cover the limitations recited in claims 9-13.
With respect to claim 14, Venieris as modified teaches: The method of claim 9, wherein the generating the execution plans comprises analyzing data dependency that occurs during the execution of each mixed-precision deep learning model and then optimizing the execution plan, taking into account the data dependency (see e.g. Venieris, page 5, column 2, paragraph 4: “NAWQ-SR introduces a new design dimension to the quantization strategy, which we name dynamic range estimation (DRE). DRE adapts the scale factor and zero point of an activations tensor at run time, based on the actual range of its values for the particular input sample”; and page 6, column 1, paragraph 1: “DRE adapts the scale factor and zero point in an input dependent manner, occupying the full range of values for the activations of the current input”).
With respect to claim 15, Venieris as modified teaches: The method of claim 9, wherein the generating the execution plans comprises detecting errors and exceptions that occur during the execution of the mixed-precision deep learning model (see e.g. Venieris, page 6, column 1, paragraph 1: “The primary limitation that leads to degraded output quality is manifested in cases where the estimated dynamic range does not capture the actual encountered range of an input. In these cases, the statically determined precision underutilizes the representation range of the selected wordlength, leading to excessive numerical error and, in turn, quality drop”) and adding processing plans for the detected errors and exceptions to the execution plan (see e.g. Venieris, page 6, column 1, paragraph 1: “Instead, DRE adapts the scale factor and zero point in an input dependent manner, occupying the full range of values for the activations of the current input” and paragraph 4: “Layerwise Resilience Analysis”).
With respect to claim 16: Claim 16 is directed to a method corresponding to the active functions implemented by the apparatus disclosed in claim 6; please see the rejection directed to claim 6 above which also covers the limitations recited in claim 16.
With respect to claim 17: Claim 17 is directed to a computer-readable recording medium for storing instructions to implement the active functions corresponding to the functions implemented by the apparatus comprising a memory (i.e. a computer-readable recording medium) as disclosed in claim 1; please see the rejection directed to claim 1 above which also covers the limitations recited in claim 17.
CONCLUSION
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Wang et al. (“HAQ: Hardware-Aware Automated Quantization with Mixed Precision”; 2019) discloses a Hardware-Aware Automated Quantization (HAQ) framework as a model quantization technique to compress and accelerate deep neural network (DNN) inference by automatically determining a quantization policy via a hardware simulator that generates direct feedback signals (latency and energy) from a hardware accelerator (see abstract).
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Umut Onat whose telephone number is (571)270-1735. The examiner can normally be reached M-Th 9:00-7:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kevin L Young can be reached at (571) 270-3180. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/UMUT ONAT/Primary Examiner, Art Unit 2194
Prosecution Insights