DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This action is in response to the submission filed 13th April 2026 for application 17/210,644.
Currently no claims are amended. Claims 3, 8, 9, 12, 17, 18, 21, 26, 27, and 30 are cancelled. Claims 1, 2, 4-7, 10, 11, 13-16, 19, 20, 22-25, 28, and 29 are pending and have been examined.
Information Disclosure Statement
An information disclosure statement (IDS) was submitted on 13th April 2026. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Arguments
Applicant’s arguments, see pages 9-12, filed 13 April 2026, with respect to the 35 USC 103 rejection of independent claim 1 (and similarly in independent claims 10, 19, and 28), Applicant specifically argues on Page 11 (Paragraphs 4 & 5) that however, Applicant submits that the combination of Wang, Gross, and Moazzemi fails to describe or make obvious, at least, "determining an AI quality of service (QoS) value based on the temperature data, wherein the QoS value represents a target for accuracy of a result generated by the AI processor and a throughput of the AI processor," as recited in claim 1. The Office submitted that Moazzemi discloses these features. Office Action, pp. 7-8. Applicant respectfully disagrees.
For example, as discussed above, Moazzemi describes "heuristic and optimization methods used to manage resources such as power, energy, temperature." Moazzemi, Abstract. Moazzemi discusses "dynamic management methods" that consider "thermal issues such as overheating or faults." Id., p. 63 (section B). Moazzemi describes that "[t]emperature aware scheduling for multi- threaded processors can reduce hot spots" and that "[t]hermal management became a prominent challenge in dark silicon era" Id. Moazzemi then discusses quality-of-service (QoS) as a separate metric that is "a primary metric to qualitatively evaluate the system's efficiency in satisfying applications' requirements" and that includes "metrics such as frame rate (multi-media) [25], latency-per-query (web search and financial) [26], throughput (data analytics and streaming) [27], responsiveness (user centric) [28], end-to-end latency and privacy (social media) [29], etc."
Applicant continues to argue on Page 12 (Paragraphs 2-4) that however, Moazzemi's discussions of "temperature" and "QoS" are not related and are instead described as different types of metrics/resources that can be managed. See, e.g., Id. at Abstract ("We first cover heuristic and optimization methods used to manage resources such as power, energy, temperature, Quality-of-Service (QoS) and reliability of the system"). Moazzemi does not describe that any "quality of service (QoS) value" is "based on the temperature data," as claimed. For example, Moazzemi's "temperature aware scheduling" is not related at all to a "QoS." Similarly, "limit[ing] temperature increase" using "modeling" also bears no relation to "QoS."
Rather, as noted above, Moazzemi describes "temperature" and "QoS" as separate resources. For example, Moazzemi explains that a "target resource metrics while studying these methods are power, energy, temperature, Quality-of-Service and reliability." Id. at p., 68. Moazzemi's Figure 1, reproduced below for reference, further illustrates this distinction:
As can be seen, Figure 1 of Moazzemi lists "QoS" and "temperature" as distinct metrics. Therefore, Moazzemi (individually or in combination with Wang and/or Gross) does not describe "determining an AI quality of service (QoS) value based on the temperature data," as claimed.
Examiner’s Response: Examiner respectfully disagrees because Applicant’s arguments are fully considered but are not persuasive. Although Applicant argues that reference Moazzemi does not describe that any "quality of service (QoS) value" is "based on the temperature data," as claimed Examiner disagrees because reference Moazzemi ties quality of service and temperature together in several places throughout the reference. For example, on Page 63, Column 2, Section B clearly states that often circuits that perform in their highest computation power suffer from thermal issues such as overheating or faults due to thermal emergencies. Here, “performing in their highest computation power” corresponds to quality of service because “Quality of service” is a broad term and “performing in their highest computation power” falls within the scope of interpretation of “Quality of service” as QoS in AI refers to managing and prioritizing resources to ensure AI systems deliver consistent, reliable, and timely performance. Furthermore, the reference continues to cite in the same sentence, “from thermal issues such as overheating or faults due to thermal emergencies”, which clearly show that it is based on temperature data. Hence, reference Moazzemi teaches “determining an AI quality of service (QoS) value based on the temperature data” as claimed in the independent claim 1 (and similarly in independent claims 10, 19, and 28).
Applicant’s arguments, see pages 9-12, filed 13 April 2026, with respect to the 35 USC 103 rejection of independent claim 1 (and similarly in independent claims 10, 19, and 28), Applicant further submits on Page 13 (Paragraphs 1-4) that Moazzemi (individually or in combination with Wang and/or Gross) fails to describe that "the QoS value represents a target for accuracy of a result generated by the AI processor and a throughput of the AI processor," as claimed. Rather, as noted above, Moazzemi describes that QoS is "a primary metric to qualitatively evaluate the system's efficiency in satisfying applications' requirements," including "metrics such as frame rate (multi-media) [25], latency-per-query (web search and financial) [26], throughput (data analytics and streaming) [27], responsiveness (user centric) [28], end-to-end latency and privacy (social media) [29], etc." Moazzemi, p. 63. To the extent that Moazzemi mentions "throughput," there is no discussion in Moazzemi (individually or when combined with Wang and/or Gross) regarding a QoS metric that "represents a target for accuracy of a result generated by the AI processor and a throughput of the AI processor," as claimed.
For at least the reasons discussed above, Applicant respectfully submits that the combination of Wang, Gross, and Moazzemi fails to disclose or make obvious all features of claim 1. The Turakhia, Du, and Jiang references, alone or in combination with Wang, Gross, and/or Moazzemi, also fail to describe or make obvious the features of claim 1, and were not used in such a manner. Therefore, it is respectfully submitted that claim 1 is in condition for allowance.
While differing in scope, independent claim 10, 19, and 28 has recite features that are similar to the distinguishing features of claim 1 discussed above. Therefore, it is respectfully submitted that claim 10, 19, and 28 are also in condition for allowance for at least the same reasons.
Accordingly, reconsideration of independent claims 1, 10, 19, and 28 and withdrawal of the rejections under 35 U.S.C. § 103 are requested.
Examiner’s Response: : Examiner respectfully disagrees because Applicant’s arguments are fully considered but are not persuasive. Although Applicant argues that there is no discussion in Moazzemi (individually or when combined with Wang and/or Gross) regarding a QoS metric that "represents a target for accuracy of a result generated by the AI processor and a throughput of the AI processor," as claimed, Examiner disagrees because reference Moazzemi cites Qos with accuracy and throughput in several places. For example, Page 64, Column 1, Paragraph 2 states, we abstractly classify QoS management techniques as performance-bound and/or accuracy-bound, as shown in Figure 2. We present major underlying approaches and strategies for performance bound QoS guarantees through provisioning compute, memory and network bandwidth resources and accuracy-bound QoS through quality monitoring and control. Also, Page 65, Column 1, Paragraph 2 clearly states that some techniques use the target accuracy requirement as a feedback to explore accuracy-performance. Furthermore, Page 64, Column 1, Last but one Paragraph cites, combining a set of cores, memory and network bandwidth into a package to provision isolated resources for applications as per their QoS requirement to provide infrastructure as a service, where cores correspond to AI processors. Furthermore, Page 64, Column 2, Section 2 cites Accuracy Bound QoS. Lastly, Figure 2 also shows Quality of service representing accuracy bound classification of QoS management. Next, Page 64, Column 2, Paragraph 3, states that under workload diversity, smart co-location - scheduling an optimized combination of latency and throughput sensitive applications together, exploits under-utilized resources to satisfy QoS of both types of applications. All these techniques feature user/application defined QoS metrics such as latency and throughput bounds. This clearly shows that QoS represents both a target accuracy and throughput of the AI processor. Hence, reference Moazzemi teaches “wherein the QoS value represents a target for accuracy of a result generated by the AI processor and a throughput of the AI processor”, as claimed in the independent claim 1 (and similarly in independent claims 10, 19, and 28).
Applicant’s arguments, see page 13, with respect to the dependent claims, Applicant respectfully submits that pursuant to 35 U.S.C. § 112(d), the dependent claims incorporate by reference all the limitations of the claim to which they refer and include their own patentable features, and are therefore in condition for allowance. Furthermore, although Applicant has not discussed the specific rejections to all of the dependent claims, Applicant does not necessarily agree with the characterizations of the prior art made by the Office. Moreover, because each dependent claim includes its own patentable features, individual consideration of each on its own merits is respectfully requested.
Examiner’s Response: Applicant’s arguments have been fully considered but they are not persuasive because the dependent claims depend from one of the independent claims 1, 10, 19, or 28 and as explained above and as shown in the detailed rejection below, the combination of cited references teach every element of the claims. Also, the applicant fails to provide any information regarding which particular limitation of which dependent claim applicant does not agree with and any reasons or explanations as to why the applicant does not agree. Hence, the claims are rejected as well.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 2, 4-6, 28, and 29 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al (HAQ: Hardware-Aware Automated Quantization with Mixed Precision, 2019) in view of Gross et al (US 8164434 B2) and further in view of Moazzemi et al (Trends in On-Chip Dynamic Resource Management, 2018).
Regarding claim 1
Wang teaches: dynamically adjusting an Al quantization level for a segment of a neural network ([Page 8613, Column 2, Paragraph 2] To this end, we propose the Hardware-Aware Automated Quantization (HAQ) framework that leverages reinforcement learning to automatically predict the quantization policy given the hardware’s feedback. The RL agent decides the bitwidth of a given neural network in a layer-wise manner. For each layer, the agent receives the layer configuration and statistics as observation, and it then outputs the action which is the bitwidth of weights and activations. We then leverage the hardware accelerator as the environment to obtain the direct feedback from hardware to guide the RL agent to satisfy the resource constraints. [Page 8615] Resource Constraints. In real-world applications, we have limited computation budgets (i.e., latency, energy, and model size). We would like to find the quantization policy with the best performance given the constraint. [Page 8619, Column 2, Paragraph 1] Our framework succeeds in learning to adjust its bitwidth policy under different constraints. Note: Automatically predict given the hardware's feedback corresponds to dynamically adjusting. Layer-wise manner corresponds to a segment of the neural network.);
and processing the segment of the neural network using the adjusted Al quantization level ([Page 8613, Column 2, Paragraph 2] After all layers are quantized, we finetune the quantized model for one more epoch, and feed the validation accuracy after short-term retraining as the reward signal to our RL agent).
However, Wang is not relied upon to teach: A method comprising: receiving temperature data associated with a temperature of an artificial intelligence (Al) processor; determining an AI quality of service (QoS) value based on the temperature data, wherein the QoS value represents a target for accuracy of a result generated by the AI processor and a throughput of the AI processor; and adjust in response to the Qos value.
Gross teaches, in an analogous system: A method comprising: receiving temperature data associated with a temperature of an artificial intelligence (Al) processor ([Column 2, Lines 20-22] During operation, the electronic device receives temperature measurements from thermal sensors in the computer system. [Column 5, Lines 47-55] Referring back to FIG. 1A, analysis device 124 then validates the telemetry data (such as the temperature and/or fan-speed measurements) using a pattern-recognition model with the telemetry data (e.g., temperature and/or fan-speed measurements) as inputs. In particular, the pattern-recognition model may be a nonlinear, nonparametric regression model, such as a multivariate state estimation technique (MSET) and/or a kernel regression model. Note that MSET refers to a class of pattern-recognition techniques. [Column 5, Lines 63-67] Hence, the term MSET as used in this specification can refer to (among other things) any technique outlined in Gribok et al., including: ordinary least squares (OLS), support vector machines (SVM), artificial neural networks (ANNs), MSET, or regularized MSET (RMSET));
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Wang to incorporate the teachings of Gross to use temperature data associated with a temperature of an artificial intelligence (Al) processor. One would have been motivated to do this modification because doing so would give the benefit of validating at least some of telemetry data 440 using one or more pattern-recognition models as taught by Gross [Column 11, Lines 7 and 8].
Moazzemi teaches, in an analogous system: determining an AI quality of service (QoS) value based on the temperature data, wherein the QoS value represents a target for accuracy of a result generated by the AI processor and a throughput of the AI processor ([Abstract, Paragraph 1] We first cover heuristic and optimization methods used to manage resources such as power, energy, temperature, Qualityof-Service (QoS) and reliability of the system. [Page 63, Column 2, Section B] Delivering high performance does not only come with the cost of power consumption. Often circuits that perform in their highest computation power suffer from thermal issues such as overheating or faults due to thermal emergencies. Temperature aware scheduling for multi-threaded processors can reduce hot spots. [Page 64, Column 1, Paragraph 2] We abstractly classify QoS management techniques as performance-bound and/or accuracy-bound, as shown in Figure 2. We present major underlying approaches and strategies for performance bound QoS guarantees through provisioning compute, memory and network bandwidth resources and accuracy-bound QoS through quality monitoring and control. [Page 64, Column 2, Paragraph 3] Under workload diversity, smart co-location - scheduling an optimized combination of latency and throughput sensitive applications together, exploits under-utilized resources to satisfy QoS of both types of applications. All these techniques feature user/application defined QoS metrics such as latency and throughput bounds. [Page 64, Column 2, Section 2) Accuracy Bound QoS:] However, reasoning for accuracy loss, guaranteed error bound and control on quality of result (QoR) is crucial for viability of approximation techniques. [Page 64, Column 1, Last but one Paragraph] Combining a set of cores, memory and network bandwidth into a package to provision isolated resources for applications as per their QoS requirement is proposed in [35], to provide infrastructure as a service. [Page 65, Column 1, Paragraph 2] Some techniques use the target accuracy requirement as a feedback to explore accuracy-performance. Note: Also see Figure 1 showing QoS and temperature. See Fig. 2 showing QoS and accuracy. Performing in their highest computation power also corresponds to the quality of service that is produced and "from thermal issues" corresponds to “based on temperature data”);
adjust in response to the QoS value ([Page 63, Column 2, Last Paragraph] Runtime QoS management becomes necessary and challenging [Page 64, Column 1, Paragraph 1] with i) variable workload characteristics ii) variable QoS requirements of applications, iii) identification and translation of QoS metrics into system level parameters for provisioning and iv) resource contention and arbitration among concurrent applications. Meeting QoS requirements of applications are largely based on: • the nature of computation - compute, memory and I/O intensity, streaming inputs and batch processing • the nature of end result - numerical, perceptive, soft and hard real-time, and user-interaction. We abstractly classify QoS management techniques as performance-bound and/or accuracy-bound, as shown in Figure 2. We present major underlying approaches and strategies for performance bound QoS guarantees through provisioning compute, memory and network bandwidth resources and accuracy-bound QoS through quality monitoring and control. Note: Run-time QoS management and Fig. 2 showing calibration corresponds to adjusting).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combined teachings of Wang and Gross to incorporate the teachings of Moazzemi to determine an AI quality of service (QoS) value based on the temperature data, wherein the QoS value represents a target for accuracy of a result generated by the AI processor and a throughput of the AI processor and adjust in response to the QoS value. One would have been motivated to do this modification because doing so would give the benefit of classify QoS management techniques as performance-bound and/or accuracy-bound as taught by Moazzemi [Page 64, Column 1, Paragraph 2].
Regarding claim 2
The system of Wang, Gross, and Moazzemi teaches: The method of claim 1 (as shown above).
Wang further teaches: wherein dynamically adjusting the Al quantization level for the segment of the neural network comprises: increasing the Al quantization level in response to an operating condition information indicating a level that increased constraint of a processing ability of the Al processor, and decreasing the Al quantization level in response to operating condition information indicating a level that decreased constraint of the processing ability of the Al processor ([Page 8619, Column 1, Last Paragraph] Discussions. In Figure 5, we visualize the bitwidth allocation strategy for MobileNet-V2. From this figure, we can observe that our framework assigns more bitwidths to the weights in depthwise convolution layers than pointwise convolution layers. Intuitively, this is because the number of parameters in the former is much smaller than the latter. Comparing Figure 4 and Figure 5, the policies are drastically [Page 8619, Column 2, Paragraph 1] different under different optimization objectives (fewer bitwiths for depthwise convolutions under latency optimization, more bitwidths for depthwise convolutions under model size optimization). Our framework succeeds in learning to adjust its bitwidth policy under different constraints).
However, Wang is not relied upon to teach: the temperature data.
Gross teaches, in an analogous system: the temperature data ([Column 2, Lines 20-22] During operation, the electronic device receives temperature measurements from thermal sensors in the computer system).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Wang to incorporate the teachings of Gross to use temperature data. One would have been motivated to do this modification because doing so would give the benefit of validating the temperature measurements using a pattern-recognition model based at least on the temperature measurements as taught by Gross [Column 2, Lines 24-26].
Regarding claim 4
The system of Wang, Gross, and Moazzemi teaches: The method of claim 1 (as shown above).
Wang further teaches: wherein dynamically adjusting the Al quantization level for the segment of the neural network comprises adjusting the Al quantization level for quantizing weight values to be processed by the segment of the neural network ([Page 8614, Column 1, Last but one Paragraph] Our framework further explores the automated quantization for network weights and activations, and it takes the hardware architectures into consideration. [Page 8615, Column 2, Section 3.4, Paragraph 2] Specifically, for each weight value w in the kth layer, we first truncate it into the range of [−c, c], and we then quantize it linearly into ak bits).
Regarding claim 5
The system of Wang, Gross, and Moazzemi teaches: The method of claim 1 (as shown above).
Wang further teaches: wherein dynamically adjusting the Al quantization level for the segment of the neural network comprises adjusting the Al quantization level for quantizing activation values to be processed by the segment of the neural network ([Page 8614, Column 1, Last but one Paragraph] Our framework further explores the automated quantization for network weights and activations, and it takes the hardware architectures into consideration).
Regarding claim 6
The system of Wang, Gross, and Moazzemi teaches: The method of claim 1 (as shown above).
Wang further teaches: wherein dynamically adjusting the Al quantization level for the segment of the neural network comprises adjusting the Al quantization level for quantizing weight values and activation values to be processed by the segment of the neural network ([Page 8614, Column 1, Last but one Paragraph] Our framework further explores the automated quantization for network weights and activations, and it takes the hardware architectures into consideration. [Page 8615, Column 2, Section 3.4, Paragraph 1] We linearly quantize the weights and activations of each layer).
Regarding claim 28
Wang teaches: means for dynamically adjusting an Al quantization level for a segment of a neural network ([Page 8613, Column 2, Paragraph 2] To this end, we propose the Hardware-Aware Automated Quantization (HAQ) framework that leverages reinforcement learning to automatically predict the quantization policy given the hardware’s feedback. The RL agent decides the bitwidth of a given neural network in a layer-wise manner. For each layer, the agent receives the layer configuration and statistics as observation, and it then outputs the action which is the bitwidth of weights and activations. We then leverage the hardware accelerator as the environment to obtain the direct feedback from hardware to guide the RL agent to satisfy the resource constraints. [Page 8615] Resource Constraints. In real-world applications, we have limited computation budgets (i.e., latency, energy, and model size). We would like to find the quantization policy with the best performance given the constraint. [Page 8619, Column 2, Paragraph 1] Our framework succeeds in learning to adjust its bitwidth policy under different constraints. Note: Automatically predict given the hardware's feedback corresponds to dynamically adjusting. Layer-wise manner corresponds to a segment of the neural network);
and means for processing the segment of the neural network using the adjusted Al quantization level ([Page 8613, Column 2, Paragraph 2] After all layers are quantized, we finetune the quantized model for one more epoch, and feed the validation accuracy after short-term retraining as the reward signal to our RL agent).
However, Wang is not relied upon to teach: An artificial intelligence (AI) processor, comprising: means for receiving AI processor temperature data associated with operation of the AI processor; means for determining an AI quality of service (QoS) value based on the temperature data, wherein the QoS value represents a target for accuracy of a result generated by the AI processor and a throughput of the AI processor; and adjusting in response to the QoS value.
Gross teaches, in an analogous system: An artificial intelligence (AI) processor, comprising: means for receiving AI processor temperature data associated with operation of the AI processor ([Column 2, Lines 20-22] During operation, the electronic device receives temperature measurements from thermal sensors in the computer system. [Column 5, Lines 47-55] Referring back to FIG. 1A, analysis device 124 then validates the telemetry data (such as the temperature and/or fan-speed measurements) using a pattern-recognition model with the telemetry data (e.g., temperature and/or fan-speed measurements) as inputs. In particular, the pattern-recognition model may be a nonlinear, nonparametric regression model, such as a multivariate state estimation technique (MSET) and/or a kernel regression model. Note that MSET refers to a class of pattern-recognition techniques. [Column 5, Lines 63-67] Hence, the term MSET as used in this specification can refer to (among other things) any technique outlined in Gribok et al., including: ordinary least squares (OLS), support vector machines (SVM), artificial neural networks (ANNs), MSET, or regularized MSET (RMSET));
adjusting in response to the temperature data ([Column 11, Lines 4-6] Then, optional resampling module 432 may resample and/or de-quantize at least some of telemetry data 440, such as temperature measurements).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Wang to incorporate the teachings of Gross to use an artificial intelligence (AI) processor, comprising: means for receiving AI processor temperature data associated with operation of the AI processor and adjusting in response to the temperature data. One would have been motivated to do this modification because doing so would give the benefit of validating at least some of telemetry data 440 using one or more pattern-recognition models as taught by Gross [Column 11, Lines 7 and 8].
Moazzemi teaches, in an analogous system: means for determining an AI quality of service (QoS) value based on the temperature data, wherein the QoS value represents a target for accuracy of a result generated by the AI processor and a throughput of the AI processor ([Abstract, Paragraph 1] We first cover heuristic and optimization methods used to manage resources such as power, energy, temperature, Qualityof-Service (QoS) and reliability of the system. [Page 63, Column 2, Section B] Delivering high performance does not only come with the cost of power consumption. Often circuits that perform in their highest computation power suffer from thermal issues such as overheating or faults due to thermal emergencies. Temperature aware scheduling for multi-threaded processors can reduce hot spots. [Page 64, Column 1, Paragraph 2] We abstractly classify QoS management techniques as performance-bound and/or accuracy-bound, as shown in Figure 2. We present major underlying approaches and strategies for performance bound QoS guarantees through provisioning compute, memory and network bandwidth resources and accuracy-bound QoS through quality monitoring and control. [Page 64, Column 2, Paragraph 3] Under workload diversity, smart co-location - scheduling an optimized combination of latency and throughput sensitive applications together, exploits under-utilized resources to satisfy QoS of both types of applications. All these techniques feature user/application defined QoS metrics such as latency and throughput bounds. [Page 64, Column 2, Section 2) Accuracy Bound QoS:] However, reasoning for accuracy loss, guaranteed error bound and control on quality of result (QoR) is crucial for viability of approximation techniques. [Page 64, Column 1, Last but one Paragraph] Combining a set of cores, memory and network bandwidth into a package to provision isolated resources for applications as per their QoS requirement is proposed in [35], to provide infrastructure as a service. [Page 65, Column 1, Paragraph 2] Some techniques use the target accuracy requirement as a feedback to explore accuracy-performance. Note: Also see Figure 1 showing QoS and temperature. See Fig. 2 showing QoS and accuracy. Performing in their highest computation power also corresponds to the quality of service that is produced and "from thermal issues" corresponds to “based on temperature data”);
adjusting in response to the QoS value ([Page 63, Column 2, Last Paragraph] Runtime QoS management becomes necessary and challenging [Page 64, Column 1, Paragraph 1] with i) variable workload characteristics ii) variable QoS requirements of applications, iii) identification and translation of QoS metrics into system level parameters for provisioning and iv) resource contention and arbitration among concurrent applications. Meeting QoS requirements of applications are largely based on: • the nature of computation - compute, memory and I/O intensity, streaming inputs and batch processing • the nature of end result - numerical, perceptive, soft and hard real-time, and user-interaction. We abstractly classify QoS management techniques as performance-bound and/or accuracy-bound, as shown in Figure 2. We present major underlying approaches and strategies for performance bound QoS guarantees through provisioning compute, memory and network bandwidth resources and accuracy-bound QoS through quality monitoring and control. Note: Run-time QoS management and Fig. 2 showing calibration corresponds to adjusting).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combined teachings of Wang and Gross to incorporate the teachings of Moazzemi to determine an AI quality of service (QoS) value based on the temperature data, wherein the QoS value represents a target for accuracy of a result generated by the AI processor and a throughput of the AI processor and adjusting in response to the QoS value. One would have been motivated to do this modification because doing so would give the benefit of classify QoS management techniques as performance-bound and/or accuracy-bound as taught by Moazzemi [Page 64, Column 1, Paragraph 2].
Regarding claim 29
The system of Wang, Gross, and Moazzemi teaches: The Al processor of claim 28 (as shown above).
Wang further teaches: wherein means for dynamically adjusting the Al quantization level for the segment of the neural network comprises: means for increasing the Al quantization level in response to indicating a level of that increased constraint of a processing ability of the Al processor, and means for decreasing the Al quantization level in response to indicating a level of that decreased constraint of the processing ability of the Al processor ([Page 8619, Column 1, Last Paragraph] Discussions. In Figure 5, we visualize the bitwidth allocation strategy for MobileNet-V2. From this figure, we can observe that our framework assigns more bitwidths to the weights in depthwise convolution layers than pointwise convolution layers. Intuitively, this is because the number of parameters in the former is much smaller than the latter. Comparing Figure 4 and Figure 5, the policies are drastically [Page 8619, Column 2, Paragraph 1] different under different optimization objectives (fewer bitwiths for depthwise convolutions under latency optimization, more bitwidths for depthwise convolutions under model size optimization). Our framework succeeds in learning to adjust its bitwidth policy under different constraints).
However, Wang is not relied upon to teach: the temperature data.
Gross teaches, in an analogous system: the temperature data ([Column 2, Lines 20-22] During operation, the electronic device receives temperature measurements from thermal sensors in the computer system).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the artificial intelligence processor of Wang to incorporate the teachings of Gross to use temperature data. One would have been motivated to do this modification because doing so would give the benefit of validating the temperature measurements using a pattern-recognition model based at least on the temperature measurements as taught by Gross [Column 2, Lines 24-26].
Claims 7, 10, 11, 13-16, 19, 20, and 22-25 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al (HAQ: Hardware-Aware Automated Quantization with Mixed Precision, 2019) in view of Gross et al (US 8164434 B2) and Moazzemi et al (Trends in On-Chip Dynamic Resource Management, 2018) and further in view of Turakhia et al (US 20180164866 A1).
Regarding claim 7
The system of Wang, Gross, and Moazzemi teaches: The method of claim 1 (as shown above).
Wang further teaches: wherein: the Al quantization level is configured to indicate dynamic bits of a value to be processed by the neural network to quantize ([Page 8614, Column 2, Section 3] We model the quantization task as a reinforcement learning problem (Figure 2). We use the actor-critic model with DDPG agent to give the action: bits for each layer);
and processing the segment of the neural network using the adjusted Al quantization level ([Page 8614, Column 2, Section 3.1] Our agent processes the neural network in a layer-wise manner. [Page 3, Column 1, Paragraph 1] Table 1. Comparison of ImageNet validation accuracy among different rounding schemes for 4-bit quantization of the first layer of Resnet18. Note: 4-bit quantization corresponds to the Al quantization level. Each layer corresponds to a segment of the neural network).
However, the system of Wang, Gross, and Moazzemi does not explicitly disclose: bypassing portions of a multiplier accumulator (MAC) associated with the dynamic bits of the value.
Turakhia teaches, in an analogous system: bypassing portions of a multiplier accumulator (MAC) associated with the dynamic bits of the value ([0015] FIG. 5 is a diagram illustrating an example of a modified multiplier-accumulator unit that bypasses the multiplier and adder when at least one of the operands for the multiplier is zero).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Wang, Gross, and Moazzemi to incorporate the teachings of Turakhia to bypass portions of a multiplier accumulator (MAC) associated with the dynamic bits of the value. One would have been motivated to do this modification because doing so would give the benefit of implementing a combinational logic as taught by Turakhia [0039].
Regarding claim 10
Wang teaches: and dynamically adjust an Al quantization level for a segment of a neural network neural network ([Page 8613, Column 2, Paragraph 2] To this end, we propose the Hardware-Aware Automated Quantization (HAQ) framework that leverages reinforcement learning to automatically predict the quantization policy given the hardware’s feedback. The RL agent decides the bitwidth of a given neural network in a layer-wise manner. For each layer, the agent receives the layer configuration and statistics as observation, and it then outputs the action which is the bitwidth of weights and activations. We then leverage the hardware accelerator as the environment to obtain the direct feedback from hardware to guide the RL agent to satisfy the resource constraints. [Page 8615] Resource Constraints. In real-world applications, we have limited computation budgets (i.e., latency, energy, and model size). We would like to find the quantization policy with the best performance given the constraint. [Page 8619, Column 2, Paragraph 1] Our framework succeeds in learning to adjust its bitwidth policy under different constraints. Note: Automatically predict given the hardware's feedback corresponds to dynamically adjusting. Layer-wise manner corresponds to a segment of the neural network);
to process the segment of the neural network using the adjusted Al quantization level ([Page 8613, Column 2, Paragraph 2] After all layers are quantized, we finetune the quantized model for one more epoch, and feed the validation accuracy after short-term retraining as the reward signal to our RL agent).
However, Wang does not explicitly disclose: An artificial intelligence (AI) processor, comprising: a dynamic quantization controller configured to: receive temperature data associated with operation of an AI processor; determine an AI quality of service (QoS) value based on the temperature data, wherein the QoS value represents a target for accuracy of a result generated by the AI processor and a throughput of the AI processor; adjust in response to the QoS value; and a multiplier accumulator (MAC) array configured.
Gross teaches, in an analogous system: An artificial intelligence (AI) processor, comprising: a dynamic quantization controller configured to: receive temperature data associated with operation of an AI processor ([Column 2, Lines 20-22] During operation, the electronic device receives temperature measurements from thermal sensors in the computer system. [Column 5, Lines 47-55] Referring back to FIG. 1A, analysis device 124 then validates the telemetry data (such as the temperature and/or fan-speed measurements) using a pattern-recognition model with the telemetry data (e.g., temperature and/or fan-speed measurements) as inputs. In particular, the pattern-recognition model may be a nonlinear, nonparametric regression model, such as a multivariate state estimation technique (MSET) and/or a kernel regression model. Note that MSET refers to a class of pattern-recognition techniques. [Column 5, Lines 63-67] Hence, the term MSET as used in this specification can refer to (among other things) any technique outlined in Gribok et al., including: ordinary least squares (OLS), support vector machines (SVM), artificial neural networks (ANNs), MSET, or regularized MSET (RMSET));
adjust in response to the temperature data ([Column 11, Lines 4-6] Then, optional resampling module 432 may resample and/or de-quantize at least some of telemetry data 440, such as temperature measurements).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Wang to incorporate the teachings of Gross to use an artificial intelligence (AI) processor, comprising: a dynamic quantization controller configured to: receive temperature data associated with operation of an AI processor and adjust in response to the temperature data. One would have been motivated to do this modification because doing so would give the benefit of validating at least some of telemetry data 440 using one or more pattern-recognition models as taught by Gross [Column 11, Lines 7 and 8].
Moazzemi teaches, in an analogous system: determine an AI quality of service (QoS) value based on the temperature data, wherein the QoS value represents a target for accuracy of a result generated by the AI processor and a throughput of the AI processor ([Abstract, Paragraph 1] We first cover heuristic and optimization methods used to manage resources such as power, energy, temperature, Qualityof-Service (QoS) and reliability of the system. [Page 63, Column 2, Section B] Delivering high performance does not only come with the cost of power consumption. Often circuits that perform in their highest computation power suffer from thermal issues such as overheating or faults due to thermal emergencies. Temperature aware scheduling for multi-threaded processors can reduce hot spots. [Page 64, Column 1, Paragraph 2] We abstractly classify QoS management techniques as performance-bound and/or accuracy-bound, as shown in Figure 2. We present major underlying approaches and strategies for performance bound QoS guarantees through provisioning compute, memory and network bandwidth resources and accuracy-bound QoS through quality monitoring and control. [Page 64, Column 1, Last but one Paragraph] Combining a set of cores, memory and network bandwidth into a package to provision isolated resources for applications as per their QoS requirement is proposed in [35], to provide infrastructure as a service. [Page 64, Column 2, Paragraph 3] Under workload diversity, smart co-location - scheduling an optimized combination of latency and throughput sensitive applications together, exploits under-utilized resources to satisfy QoS of both types of applications. All these techniques feature user/application defined QoS metrics such as latency and throughput bounds. [Page 64, Column 2, Section 2) Accuracy Bound QoS:] However, reasoning for accuracy loss, guaranteed error bound and control on quality of result (QoR) is crucial for viability of approximation techniques. [Page 65, Column 1, Paragraph 2] Some techniques use the target accuracy requirement as a feedback to explore accuracy-performance. Note: Also see Figure 1 showing QoS and temperature. See Fig. 2 showing QoS and accuracy. Performing in their highest computation power also corresponds to the quality of service that is produced and "from thermal issues" corresponds to “based on temperature data”);
adjust in response to the QoS value ([Page 63, Column 2, Last Paragraph] Runtime QoS management becomes necessary and challenging [Page 64, Column 1, Paragraph 1] with i) variable workload characteristics ii) variable QoS requirements of applications, iii) identification and translation of QoS metrics into system level parameters for provisioning and iv) resource contention and arbitration among concurrent applications. Meeting QoS requirements of applications are largely based on: • the nature of computation - compute, memory and I/O intensity, streaming inputs and batch processing • the nature of end result - numerical, perceptive, soft and hard real-time, and user-interaction. We abstractly classify QoS management techniques as performance-bound and/or accuracy-bound, as shown in Figure 2. We present major underlying approaches and strategies for performance bound QoS guarantees through provisioning compute, memory and network bandwidth resources and accuracy-bound QoS through quality monitoring and control. Note: Run-time QoS management and Fig. 2 showing calibration corresponds to adjusting).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combined teachings of Wang and Gross to incorporate the teachings of Moazzemi to determine an AI quality of service (QoS) value based on the temperature data, wherein the QoS value represents a target for accuracy of a result generated by the AI processor and a throughput of the AI processor and adjusting in response to the QoS value. One would have been motivated to do this modification because doing so would give the benefit of classify QoS management techniques as performance-bound and/or accuracy-bound as taught by Moazzemi [Page 64, Column 1, Paragraph 2].
Turakhia teaches, in an analogous system: and a multiplier accumulator (MAC) array configured ([0015] FIG. 5 is a diagram illustrating an example of a modified multiplier-accumulator unit).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combined teachings of Wang, Gross, and Moazzemi to incorporate the teachings of Turakhia to use a multiplier accumulator (MAC). One would have been motivated to do this modification because doing so would give the benefit of implementing a combinational logic as taught by Turakhia [0039].
Regarding claim 11
The system of Wang, Gross, Moazzemi, and Turakhia teaches: The Al processor of claim 10 (as shown above).
Wang further teaches: wherein the dynamic quantization controller is configured such that dynamically adjusting the Al quantization level for the segment of the neural network comprises: increasing the Al quantization level in response to indicating a level that increased constraint of a processing ability of the Al processor, and decreasing the Al quantization level in response to indicating a level that decreased constraint of the processing ability of the Al processor ([Page 8619, Column 1, Last Paragraph] Discussions. In Figure 5, we visualize the bitwidth allocation strategy for MobileNet-V2. From this figure, we can observe that our framework assigns more bitwidths to the weights in depthwise convolution layers than pointwise convolution layers. Intuitively, this is because the number of parameters in the former is much smaller than the latter. Comparing Figure 4 and Figure 5, the policies are drastically [Page 8619, Column 2, Paragraph 1] different under different optimization objectives (fewer bitwiths for depthwise convolutions under latency optimization, more bitwidths for depthwise convolutions under model size optimization). Our framework succeeds in learning to adjust its bitwidth policy under different constraints).
However, Wang is not relied upon to teach: the temperature data.
Gross teaches, in an analogous system: the temperature data ([Column 2, Lines 20-22] During operation, the electronic device receives temperature measurements from thermal sensors in the computer system).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the AI processor of Wang to incorporate the teachings of Gross to use temperature data. One would have been motivated to do this modification because doing so would give the benefit of validating the temperature measurements using a pattern-recognition model based at least on the temperature measurements as taught by Gross [Column 2, Lines 24-26].
Regarding claim 13
The system Wang, Gross, Moazzemi, and Turakhia teaches: The Al processor of claim 10 (as shown above).
Wang further teaches: wherein the dynamic quantization controller is configured such that dynamically adjusting the Al quantization level for the segment of the neural network comprises adjusting the Al quantization level for quantizing weight values to be processed by the segment of the neural network ([Page 8614, Column 1, Last but one Paragraph] Our framework further explores the automated quantization for network weights and activations, and it takes the hardware architectures into consideration. [Page 8615, Column 2, Section 3.4, Paragraph 2] Specifically, for each weight value w in the kth layer, we first truncate it into the range of [−c, c], and we then quantize it linearly into ak bits).
Regarding claim 14
The system Wang, Gross, Moazzemi, and Turakhia teaches: The Al processor of claim 10 (as shown above).
Wang further teaches: wherein the dynamic quantization controller is configured such that dynamically adjusting the Al quantization level for the segment of the neural network comprises adjusting the Al quantization level for quantizing activation values to be processed by the segment of the neural network ([Page 8614, Column 1, Last but one Paragraph] Our framework further explores the automated quantization for network weights and activations, and it takes the hardware architectures into consideration).
Regarding claim 15
The system Wang, Gross, Moazzemi, and Turakhia teaches: The Al processor of claim 10 (as shown above).
Wang further teaches: wherein the dynamic quantization controller is configured such that dynamically adjusting the Al quantization level for the segment of the neural network comprises adjusting the Al quantization level for quantizing weight values and activation values to be processed by the segment of the neural network ([Page 8614, Column 1, Last but one Paragraph] Our framework further explores the automated quantization for network weights and activations, and it takes the hardware architectures into consideration. [Page 8615, Column 2, Section 3.4, Paragraph 1] We linearly quantize the weights and activations of each layer).
Regarding claim 16
The system Wang, Gross, Moazzemi, and Turakhia teaches: The Al processor of claim 10 (as shown above).
Wang further teaches wherein: the Al quantization level is configured to indicate dynamic bits of a value to be processed by the neural network to quantize ([Page 8614, Column 2, Section 3] We model the quantization task as a reinforcement learning problem (Figure 2). We use the actor-critic model with DDPG agent to give the action: bits for each layer);
processing the segment of the neural network using the adjusted Al quantization level ([Page 8614, Column 2, Section 3.1] Our agent processes the neural network in a layer-wise manner. [Page 3, Column 1, Paragraph 1] Table 1. Comparison of ImageNet validation accuracy among different rounding schemes for 4-bit quantization of the first layer of Resnet18. Note: 4-bit quantization corresponds to the Al quantization level. Each layer corresponds to a segment of the neural network).
However, the system of Wang, Gross, and Moazzemi does not explicitly disclose: the MAC array is configured such that processing comprises bypassing portions of a MAC associated with the dynamic bits of the value.
Turakhia teaches, in an analogous system: the MAC array is configured such that processing comprises bypassing portions of a MAC associated with the dynamic bits of the value ([0015] FIG. 5 is a diagram illustrating an example of a modified multiplier-accumulator unit that bypasses the multiplier and adder when at least one of the operands for the multiplier is zero).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combined teachings of Wang, Gross, and Moazzemi to incorporate the teachings of Turakhia wherein the MAC array is configured such that processing comprises bypassing portions of a MAC associated with the dynamic bits of the value. One would have been motivated to do this modification because doing so would give the benefit of implementing a combinational logic as taught by Turakhia [0039].
Regarding claim 19
Wang teaches: and dynamically adjust an Al quantization level for a segment of a neural network ([Page 8613, Column 2, Paragraph 2] To this end, we propose the Hardware-Aware Automated Quantization (HAQ) framework that leverages reinforcement learning to automatically predict the quantization policy given the hardware’s feedback. The RL agent decides the bitwidth of a given neural network in a layer-wise manner. For each layer, the agent receives the layer configuration and statistics as observation, and it then outputs the action which is the bitwidth of weights and activations. We then leverage the hardware accelerator as the environment to obtain the direct feedback from hardware to guide the RL agent to satisfy the resource constraints. [Page 8615] Resource Constraints. In real-world applications, we have limited computation budgets (i.e., latency, energy, and model size). We would like to find the quantization policy with the best performance given the constraint. [Page 8619, Column 2, Paragraph 1] Our framework succeeds in learning to adjust its bitwidth policy under different constraints. Note: Automatically predict given the hardware's feedback corresponds to dynamically adjusting. Layer-wise manner corresponds to a segment of the neural network);
to process the segment of the neural network using the adjusted Al quantization level ([Page 8613, Column 2, Paragraph 2] After all layers are quantized, we finetune the quantized model for one more epoch, and feed the validation accuracy after short-term retraining as the reward signal to our RL agent).
However, Wang does not explicitly disclose: A computing device, comprising: an artificial intelligence (Al) processor comprising a dynamic quantization controller configured to: receive temperature data associated with operation of an Al processor; and in response to the temperature data; and the Al processor further comprising a multiplier accumulator (MAC) array configured.
Gross teaches, in an analogous system: A computing device, comprising: an artificial intelligence (Al) processor comprising a dynamic quantization controller configured to: receive temperature data associated with operation of an Al processor ([Column 2, Lines 20-22] During operation, the electronic device receives temperature measurements from thermal sensors in the computer system. [Column 5, Lines 47-55] Referring back to FIG. 1A, analysis device 124 then validates the telemetry data (such as the temperature and/or fan-speed measurements) using a pattern-recognition model with the telemetry data (e.g., temperature and/or fan-speed measurements) as inputs. In particular, the pattern-recognition model may be a nonlinear, nonparametric regression model, such as a multivariate state estimation technique (MSET) and/or a kernel regression model. Note that MSET refers to a class of pattern-recognition techniques. [Column 5, Lines 63-67] Hence, the term MSET as used in this specification can refer to (among other things) any technique outlined in Gribok et al., including: ordinary least squares (OLS), support vector machines (SVM), artificial neural networks (ANNs), MSET, or regularized MSET (RMSET));
adjust in response to the temperature data ([Column 11, Lines 4-6] Then, optional resampling module 432 may resample and/or de-quantize at least some of telemetry data 440, such as temperature measurements).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Wang to incorporate the teachings of Gross to use a computing device, comprising: an artificial intelligence (Al) processor comprising a dynamic quantization controller configured to: receive temperature data associated with operation of an Al processor and adjust in response to the temperature data. One would have been motivated to do this modification because doing so would give the benefit of validating at least some of telemetry data 440 using one or more pattern-recognition models as taught by Gross [Column 11, Lines 7 and 8].
Moazzemi teaches, in an analogous system: determine an AI quality of service (QoS) value based on the temperature data, wherein the QoS value represents a target for accuracy of a result generated by the AI processor and a throughput of the AI processor ([Abstract, Paragraph 1] We first cover heuristic and optimization methods used to manage resources such as power, energy, temperature, Qualityof-Service (QoS) and reliability of the system. [Page 63, Column 2, Section B] Delivering high performance does not only come with the cost of power consumption. Often circuits that perform in their highest computation power suffer from thermal issues such as overheating or faults due to thermal emergencies. Temperature aware scheduling for multi-threaded processors can reduce hot spots. [Page 64, Column 1, Paragraph 2] We abstractly classify QoS management techniques as performance-bound and/or accuracy-bound, as shown in Figure 2. We present major underlying approaches and strategies for performance bound QoS guarantees through provisioning compute, memory and network bandwidth resources and accuracy-bound QoS through quality monitoring and control. [Page 64, Column 2, Paragraph 3] Under workload diversity, smart co-location - scheduling an optimized combination of latency and throughput sensitive applications together, exploits under-utilized resources to satisfy QoS of both types of applications. All these techniques feature user/application defined QoS metrics such as latency and throughput bounds. [Page 64, Column 2, Section 2) Accuracy Bound QoS:] However, reasoning for accuracy loss, guaranteed error bound and control on quality of result (QoR) is crucial for viability of approximation techniques. [Page 64, Column 1, Last but one Paragraph] Combining a set of cores, memory and network bandwidth into a package to provision isolated resources for applications as per their QoS requirement is proposed in [35], to provide infrastructure as a service. [Page 65, Column 1, Paragraph 2] Some techniques use the target accuracy requirement as a feedback to explore accuracy-performance. Note: Also see Figure 1 showing QoS and temperature. See Fig. 2 showing QoS and accuracy. Performing in their highest computation power also corresponds to the quality of service that is produced and "from thermal issues" corresponds to “based on temperature data”);
adjust in response to the QoS value ([Page 63, Column 2, Last Paragraph] Runtime QoS management becomes necessary and challenging [Page 64, Column 1, Paragraph 1] with i) variable workload characteristics ii) variable QoS requirements of applications, iii) identification and translation of QoS metrics into system level parameters for provisioning and iv) resource contention and arbitration among concurrent applications. Meeting QoS requirements of applications are largely based on: • the nature of computation - compute, memory and I/O intensity, streaming inputs and batch processing • the nature of end result - numerical, perceptive, soft and hard real-time, and user-interaction. We abstractly classify QoS management techniques as performance-bound and/or accuracy-bound, as shown in Figure 2. We present major underlying approaches and strategies for performance bound QoS guarantees through provisioning compute, memory and network bandwidth resources and accuracy-bound QoS through quality monitoring and control. Note: Run-time QoS management and Fig. 2 showing calibration corresponds to adjusting).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combined teachings of Wang and Gross to incorporate the teachings of Moazzemi to determine an AI quality of service (QoS) value based on the temperature data, wherein the QoS value represents a target for accuracy of a result generated by the AI processor and a throughput of the AI processor and adjusting in response to the QoS value. One would have been motivated to do this modification because doing so would give the benefit of classify QoS management techniques as performance-bound and/or accuracy-bound as taught by Moazzemi [Page 64, Column 1, Paragraph 2].
Turakhia teaches, in an analogous system: and the Al processor further comprising a multiplier accumulator (MAC) array configured ([0047] In some examples, the compiler 132 can divide the neural network model into portions (e.g., neural network 200)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combined teachings of Wang, Gross, and Moazzemi to incorporate the teachings of Turakhia to use the Al processor further comprising a multiplier accumulator (MAC) array configured. One would have been motivated to do this modification because doing so would give the benefit of implementing a combinational logic as taught by Turakhia [0039].
Regarding claim 20
The system of Wang, Gross, Moazzemi, and Turakhia teaches: The computing device of claim 19 (as shown above).
Wang further teaches: wherein the dynamic quantization controller is configured to dynamically adjust the Al quantization level for the segment of the neural network by: increasing the Al quantization level in response to indicating a level that increased constraint of a processing ability of the Al processor, and decreasing the Al quantization level in response to indicating a level that decreased constraint of the processing ability of the Al processor ([Page 8619, Column 1, Last Paragraph] Discussions. In Figure 5, we visualize the bitwidth allocation strategy for MobileNet-V2. From this figure, we can observe that our framework assigns more bitwidths to the weights in depthwise convolution layers than pointwise convolution layers. Intuitively, this is because the number of parameters in the former is much smaller than the latter. Comparing Figure 4 and Figure 5, the policies are drastically [Page 8619, Column 2, Paragraph 1] different under different optimization objectives (fewer bitwiths for depthwise convolutions under latency optimization, more bitwidths for depthwise convolutions under model size optimization). Our framework succeeds in learning to adjust its bitwidth policy under different constraints).
However, Wang is not relied upon to teach: the temperature data.
Gross teaches, in an analogous system: the temperature data ([Column 2, Lines 20-22] During operation, the electronic device receives temperature measurements from thermal sensors in the computer system).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the computing device of Wang to incorporate the teachings of Gross to use temperature data. One would have been motivated to do this modification because doing so would give the benefit of validating the temperature measurements using a pattern-recognition model based at least on the temperature measurements as taught by Gross [Column 2, Lines 24-26].
Regarding claim 22
The system of Wang, Gross, Moazzemi, and Turakhia teaches: The computing device of claim 19 (as shown above).
Wang further teaches: wherein the dynamic quantization controller is configured to dynamically adjust the Al quantization level for the segment of the neural network by adjusting the Al quantization level for quantizing weight values to be processed by the segment of the neural network ([Page 8614, Column 1, Last but one Paragraph] Our framework further explores the automated quantization for network weights and activations, and it takes the hardware architectures into consideration. [Page 8615, Column 2, Section 3.4, Paragraph 2] Specifically, for each weight value w in the kth layer, we first truncate it into the range of [−c, c], and we then quantize it linearly into ak bits).
Regarding claim 23
The system of Wang, Gross, Moazzemi, and Turakhia teaches: The computing device of claim 19 (as shown above).
Wang further teaches: wherein the dynamic quantization controller is configured to dynamically adjust the Al quantization level for the segment of the neural network by adjusting the Al quantization level for quantizing activation values to be processed by the segment of the neural network ([Page 8614, Column 1, Last but one Paragraph] Our framework further explores the automated quantization for network weights and activations, and it takes the hardware architectures into consideration).
Regarding claim 24
The system of Wang, Gross, Moazzemi, and Turakhia teaches: The computing device of claim 19 (as shown above).
Wang further teaches: wherein the dynamic quantization controller is configured to dynamically adjust the Al quantization level for the segment of the neural network by adjusting the Al quantization level for quantizing weight values and activation values to be processed by the segment of the neural network ([Page 8614, Column 1, Last but one Paragraph] Our framework further explores the automated quantization for network weights and activations, and it takes the hardware architectures into consideration. [Page 8615, Column 2, Section 3.4, Paragraph 1] We linearly quantize the weights and activations of each layer).
Regarding claim 25
The system Wang, Gross, Moazzemi, and Turakhia teaches: The computing device of claim 19 (as shown above).
Wang further teaches wherein: the Al quantization level is configured to indicate dynamic bits of a value to be processed by the neural network to quantize ([Page 8614, Column 2, Section 3] We model the quantization task as a reinforcement learning problem (Figure 2). We use the actor-critic model with DDPG agent to give the action: bits for each layer);
process the segment of the neural network using the adjusted Al quantization level ([Page 8614, Column 2, Section 3.1] Our agent processes the neural network in a layer-wise manner. [Page 3, Column 1, Paragraph 1] Table 1. Comparison of ImageNet validation accuracy among different rounding schemes for 4-bit quantization of the first layer of Resnet18. Note: 4-bit quantization corresponds to the Al quantization level. Each layer corresponds to a segment of the neural network).
However, the system of Wang, Gross, Moazzemi does not explicitly disclose: the MAC array is configured to bypassing portions of a MAC associated with the dynamic bits of the value.
Turakhia teaches, in an analogous system: the MAC array is configured to bypassing portions of a MAC associated with the dynamic bits of the value ([0015] FIG. 5 is a diagram illustrating an example of a modified multiplier-accumulator unit that bypasses the multiplier and adder when at least one of the operands for the multiplier is zero).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combined teachings of Wang, Gross, and Moazzemi to incorporate the teachings of Turakhia wherein the MAC array is configured to bypassing portions of a MAC associated with the dynamic bits of the value. One would have been motivated to do this modification because doing so would give the benefit of implementing a combinational logic as taught by Turakhia [0039].
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Kim et al (AutoScale: Energy Efficiency Optimization for Stochastic Edge Inference Using Reinforcement Learning, 2020) discloses offloading at model granularity, since model partitioning at layer granularity introduces additional context switching overhead (from transferring intermediate inputs to another execution target).
Liu et al (Stealthy and Robust Glitch Injection Attack on Deep Learning Accelerator for Target with Variational Viewpoint, 2020) discloses a new glitch injection attack on DNN accelerator that is capable of misclassifying a target under variational viewpoints. The glitches injected into the computation clock signal induce transitory but disruptive errors in the intermediate results of the multiply-and-accumulate (MAC) operations.
Bartolini et al (Thermal and Energy Management of High-Performance Multicores: Distributed and Self-Calibrating Model-Predictive Controller, 2013) discloses a scalable, fully distributed, energy-aware thermal management solution for single-chip multicore platforms. The model-predictive controller complexity is drastically reduced by splitting it in a set of simpler interacting controllers, each one allocated to a core in the system. Locally, each node selects the optimal frequency to meet temperature constraints while minimizing the performance penalty and system energy. Comparable performance with state-of-the-art MPC controllers is achieved by letting controllers exchange a limited amount of information at runtime on a neighborhood basis. In addition, we address model uncertainty by supporting learning of the thermal model with a novel distributed self-calibration approach that matches well the controller architecture.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHAITANYA RAMESH JAYAKUMAR whose telephone number is (571)272-3369. The examiner can normally be reached Mon-Fri 9am-1pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas can be reached at (571)272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.R.J./Examiner, Art Unit 2128
/OMAR F FERNANDEZ RIVAS/Supervisory Patent Examiner, Art Unit 2128
Prosecution Insights