Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
Acknowledgment is made of the information disclosure statement filed on January 12, 2026. U.S. patent applications, foreign patents, and non-patent literature documents have been considered.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7-12, 14-18, and 20 are rejected under 35 U.S.C. § 103 as being unpatentable over Gong et. al. (U.S. Pat. Pub. 2023/0118802), herein referred to as “Gong”, in view of Gadelrab et. al. (U.S. Pat. Pub. 2021/0279635), herein referred to as “Gadelrab”.
Regarding Claim 1,
Gong discloses: A method comprising: converting floating-point variables of a trained machine-learning model (ML model) to integers having a first bit-width supported by hardware on which the ML model is to execute an inference stage, to provide first converted variables
[0038] In some embodiments, additional optimization of the inference model may be accomplished through mixed-precision auto-tuning. FIG. 5 provides a flowchart illustrating a process 510 for tuning an inference neural network model according to one or more embodiments, with reference to components and features described herein including but not limited to the figures and associated description. Mixed-precision auto-tuning may be applied to an inference model as implemented in inference engine 116 (FIG. 1) or inference engine 416 (FIG. 4). The tuning process may include completing a series of test runs of the model to determine if model accuracy may be improved by making changes to precisions used in quantizing. The process begins (block 512) with a current state of the inference model. At block 514, the inference model is run and a determination of the accuracy of results is obtained. At block 516, the results are compared to accuracy criteria. Accuracy criteria may include, e.g., mean average precision (mAP). Other accuracy criteria may be used for the accuracy test of block 516, such as, for example, the optimal mean squared error (OMSE) process described below. If the accuracy of the results pass the accuracy criteria assessment, the process continues at block 530 (described below). If the accuracy of the results do not pass the accuracy criteria assessment of block 516, then the process proceeds to block 518.
[0036] Returning to FIG. 4, model weights quantization 424 may be implemented by quantizing floating point weights on a per-channel basis using symmetric quantization, such that each high precision weight value (e.g., fp32) is quantized into a signed integer value (e.g., int8).
[0039] At block 518, it is determined if another precision for inputs and/or weights is available for selection. For example, int16 quantization could be selected as an alternative for int8 quantization.
Note: Paragraphs [0036] and [0039] are presented that a FP32 value can be quantized to either an 8-bit or 16-bit integer value, thus providing for a first bit-width/first variable.
Gong does not explicitly disclose the remaining limitations of Claim 1.
However, Gadelrab discloses: computing a baseline memory bandwidth of the ML model based on a sum of memory bandwidths of the first converted variables.
[0086] FIG. 3 illustrates an example sequence of operations 400 that may be performed to activate and deactivate a high efficiency quantized mode for performing inferences on data.
[0088] At block 304, the system generates quantized weights from the floating point weights. The quantized weights may be generated by reducing the floating point weights into one or more integer approximations of the weights (e.g., 16-bit, 8-bit, 4-bit, etc. integers). As discussed, by reducing floating point weights into integer approximations of the weights, embodiments of the present disclosure may allow for machine learning models to be executed on more power efficient processors that may not be capable of performing floating point operations or may not be capable of performing such operations with acceptable performance.
[0091] In some embodiments, the quantized parameters for each layer of the machine learning model may be quantized to a common bit size (e.g., the quantized parameters may be quantized to an n bit representation for every layer in the machine learning model). In some embodiments, the quantized parameters may be quantized to different bit sizes for each layer of the machine learning model such that layers of the machine learning model with smaller dynamic range (e.g., a smaller difference between maximum and minimum values) are quantized to an integer representation using a smaller number of bits, while layers of the machine learning model with larger dynamic range (e.g., a larger difference between maximum and minimum values) are quantized to an integer representation using a larger number of bits.
Note: The concept of “bandwidth” is being interpreted as bit size, to where the different bit numbers are different bandwidths as stated in applicant’s specification. Here, the “baseline” is the integer representation of the smaller number of bits which can be based on one or more integer bit value as shown in paragraph [0088]. The “first bit-width” can be the smallest bit value or part of the smallest dynamic range as part of being quantized into different bit sizes/summation.
Gadelrab further discloses: computing a maximum memory bandwidth of the ML model based on a sum of memory bandwidths of the floating-point variables when converted to integers having a second bit-width supported by the hardware, wherein the second bit-width is greater than the first bit-width.
[0086] FIG. 3 illustrates an example sequence of operations 400 that may be performed to activate and deactivate a high efficiency quantized mode for performing inferences on data.
[0088] At block 304, the system generates quantized weights from the floating point weights. The quantized weights may be generated by reducing the floating point weights into one or more integer approximations of the weights (e.g., 16-bit, 8-bit, 4-bit, etc. integers). As discussed, by reducing floating point weights into integer approximations of the weights, embodiments of the present disclosure may allow for machine learning models to be executed on more power efficient processors that may not be capable of performing floating point operations or may not be capable of performing such operations with acceptable performance.
[0091] In some embodiments, the quantized parameters for each layer of the machine learning model may be quantized to a common bit size (e.g., the quantized parameters may be quantized to an n bit representation for every layer in the machine learning model). In some embodiments, the quantized parameters may be quantized to different bit sizes for each layer of the machine learning model such that layers of the machine learning model with smaller dynamic range (e.g., a smaller difference between maximum and minimum values) are quantized to an integer representation using a smaller number of bits, while layers of the machine learning model with larger dynamic range (e.g., a larger difference between maximum and minimum values) are quantized to an integer representation using a larger number of bits.
Note: Similar to the above, the “second-bit width” can be quantized/summed to different bit size, which can be in the smaller or larger dynamic range, thus providing that a second value can be greater than the first value.
Gadelrab further discloses: selecting a target memory bandwidth that is between the baseline memory bandwidth and the maximum memory bandwidth.
[0091] In some embodiments, the quantized parameters for each layer of the machine learning model may be quantized to a common bit size (e.g., the quantized parameters may be quantized to an n bit representation for every layer in the machine learning model). In some embodiments, the quantized parameters may be quantized to different bit sizes for each layer of the machine learning model such that layers of the machine learning model with smaller dynamic range (e.g., a smaller difference between maximum and minimum values) are quantized to an integer representation using a smaller number of bits, while layers of the machine learning model with larger dynamic range (e.g., a larger difference between maximum and minimum values) are quantized to an integer representation using a larger number of bits.
Note: Here, the “target bandwidth” can be located in either small or large dynamic ranges. The maximum value, then, can occur at the larger (or smaller) dynamic range.
Gadelrab also discloses: converting one or more of the floating-point variables to integers having the second bit-width, without exceeding the target memory bandwidth
[0088] At block 304, the system generates quantized weights from the floating point weights. The quantized weights may be generated by reducing the floating point weights into one or more integer approximations of the weights (e.g., 16-bit, 8-bit, 4-bit, etc. integers). As discussed, by reducing floating point weights into integer approximations of the weights, embodiments of the present disclosure may allow for machine learning models to be executed on more power efficient processors that may not be capable of performing floating point operations or may not be capable of performing such operations with acceptable performance.
Note: Here, there are integer values that can be less than the target value found in the respective dynamic ranges.
Gong discloses: such that the ML model operates as a mixed-precision ML model.
[0038] In some embodiments, additional optimization of the inference model may be accomplished through mixed-precision auto-tuning. FIG. 5 provides a flowchart illustrating a process 510 for tuning an inference neural network model according to one or more embodiments, with reference to components and features described herein including but not limited to the figures and associated description. Mixed-precision auto-tuning may be applied to an inference model as implemented in inference engine 116 (FIG. 1) or inference engine 416 (FIG. 4). The tuning process may include completing a series of test runs of the model to determine if model accuracy may be improved by making changes to precisions used in quantizing. The process begins (block 512) with a current state of the inference model. At block 514, the inference model is run and a determination of the accuracy of results is obtained. At block 516, the results are compared to accuracy criteria. Accuracy criteria may include, e.g., mean average precision (mAP). Other accuracy criteria may be used for the accuracy test of block 516, such as, for example, the optimal mean squared error (OMSE) process described below. If the accuracy of the results pass the accuracy criteria assessment, the process continues at block 530 (described below). If the accuracy of the results do not pass the accuracy criteria assessment of block 516, then the process proceeds to block 518.
Gong and Gadelrab are considered to be analogous because they pertain to utilizing data processing by machine learning/neural networks. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gong to include the concepts of first- and second-bit widths, where the first bit-width serves as a baseline, the second bit-width is greater than the first bit-width, and a target bit-width between the baseline and maximum bitwidth as taught by Gadelrab so as to improve fidelity, performance, and interpretability.
Regarding Claim 2,
Gong discloses: The method of claim 1, wherein the ML model comprises a neural network.
[0038] In some embodiments, additional optimization of the inference model may be accomplished through mixed-precision auto-tuning. FIG. 5 provides a flowchart illustrating a process 510 for tuning an inference neural network model according to one or more embodiments, with reference to components and features described herein including but not limited to the figures and associated description. Mixed-precision auto-tuning may be applied to an inference model as implemented in inference engine 116 (FIG. 1) or inference engine 416 (FIG. 4). The tuning process may include completing a series of test runs of the model to determine if model accuracy may be improved by making changes to precisions used in quantizing. The process begins (block 512) with a current state of the inference model. At block 514, the inference model is run and a determination of the accuracy of results is obtained. At block 516, the results are compared to accuracy criteria. Accuracy criteria may include, e.g., mean average precision (mAP). Other accuracy criteria may be used for the accuracy test of block 516, such as, for example, the optimal mean squared error (OMSE) process described below. If the accuracy of the results pass the accuracy criteria assessment, the process continues at block 530 (described below). If the accuracy of the results do not pass the accuracy criteria assessment of block 516, then the process proceeds to block 518.
Regarding Claim 3,
Gong does not explicitly disclose the limitations of Claim 3.
However, Gadelrab discloses: The method of claim 1, further comprising using the integers having the first bit-width as proxies for the baseline memory bandwidth; and using the integers having the second bit-width as proxies for the maximum memory bandwidth.
[0086] FIG. 3 illustrates an example sequence of operations 400 that may be performed to activate and deactivate a high efficiency quantized mode for performing inferences on data.
[0088] At block 304, the system generates quantized weights from the floating point weights. The quantized weights may be generated by reducing the floating point weights into one or more integer approximations of the weights (e.g., 16-bit, 8-bit, 4-bit, etc. integers). As discussed, by reducing floating point weights into integer approximations of the weights, embodiments of the present disclosure may allow for machine learning models to be executed on more power efficient processors that may not be capable of performing floating point operations or may not be capable of performing such operations with acceptable performance.
[0091] In some embodiments, the quantized parameters for each layer of the machine learning model may be quantized to a common bit size (e.g., the quantized parameters may be quantized to an n bit representation for every layer in the machine learning model). In some embodiments, the quantized parameters may be quantized to different bit sizes for each layer of the machine learning model such that layers of the machine learning model with smaller dynamic range (e.g., a smaller difference between maximum and minimum values) are quantized to an integer representation using a smaller number of bits, while layers of the machine learning model with larger dynamic range (e.g., a larger difference between maximum and minimum values) are quantized to an integer representation using a larger number of bits.
Note: As with Claim 1, the “baseline” is the integer representation of the smaller number of bits which can be based on one or more integer bit value as shown in paragraph [0088] and can serve as a “proxy”. The “second-bit width” can be quantized/summed to different bit size, which can be in the smaller or larger dynamic range, thus providing that a second value can be greater than the first value, and can potentially be the maximum value.
Gong and Gadelrab are considered to be analogous because they pertain to utilizing data processing by machine learning/neural networks. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gong to include the concepts of using the integers having the first bit-width as proxies for the baseline memory bandwidth and using the integers having the second bit-width as proxies for the maximum memory bandwidth as taught by Gadelrab so as to improve fidelity, performance, and interpretability.
Regarding Claim 4,
Gong does not explicitly disclose the limitations of Claim 3.
However, Gadelrab discloses: The method of claim 1, wherein the converting the one or more floating-point variables to integers having the second-bit width comprises: sorting the floating-point variable based on memory bandwidth of the floating-point variables
[0087] As illustrated, the operations 300 may begin at block 302, where a system receives floating point weights for a machine learning model to be executed on the system. The floating point weights may be weights previously determined by a model training system and may be used for performing inferences using the machine learning model by executing the machine learning model on high performance processors, such as processors capable of performing operations on large bit-size floating point numbers (e.g., 16-bit half precision floating point, 32-bit single precision floating point, 64-bit double precision floating point, etc.).
Note: The “sorting” and “iteratively processing” occurs with the weights previously determined with other floating-point values.
and iteratively processing the floating-point variables based on the sorting, comprising, for each of one or more of the floating-point variables
[0087] As illustrated, the operations 300 may begin at block 302, where a system receives floating point weights for a machine learning model to be executed on the system. The floating point weights may be weights previously determined by a model training system and may be used for performing inferences using the machine learning model by executing the machine learning model on high performance processors, such as processors capable of performing operations on large bit-size floating point numbers (e.g., 16-bit half precision floating point, 32-bit single precision floating point, 64-bit double precision floating point, etc.).
[0091] In some embodiments, the quantized parameters for each layer of the machine learning model may be quantized to a common bit size (e.g., the quantized parameters may be quantized to an n bit representation for every layer in the machine learning model). In some embodiments, the quantized parameters may be quantized to different bit sizes for each layer of the machine learning model such that layers of the machine learning model with smaller dynamic range (e.g., a smaller difference between maximum and minimum values) are quantized to an integer representation using a smaller number of bits, while layers of the machine learning model with larger dynamic range (e.g., a larger difference between maximum and minimum values) are quantized to an integer representation using a larger number of bits.
converting the floating-point variable to an integer having the second bit- width to provide a second converted variable; computing a revised memory bandwidth of the ML model based on a memory bandwidth of the second converted variable; and retaining the second converted variable if the revised memory bandwidth does not exceed the target memory bandwidth
[0086] FIG. 3 illustrates an example sequence of operations 400 that may be performed to activate and deactivate a high efficiency quantized mode for performing inferences on data.
[0088] At block 304, the system generates quantized weights from the floating point weights. The quantized weights may be generated by reducing the floating point weights into one or more integer approximations of the weights (e.g., 16-bit, 8-bit, 4-bit, etc. integers). As discussed, by reducing floating point weights into integer approximations of the weights, embodiments of the present disclosure may allow for machine learning models to be executed on more power efficient processors that may not be capable of performing floating point operations or may not be capable of performing such operations with acceptable performance.
[0091] In some embodiments, the quantized parameters for each layer of the machine learning model may be quantized to a common bit size (e.g., the quantized parameters may be quantized to an n bit representation for every layer in the machine learning model). In some embodiments, the quantized parameters may be quantized to different bit sizes for each layer of the machine learning model such that layers of the machine learning model with smaller dynamic range (e.g., a smaller difference between maximum and minimum values) are quantized to an integer representation using a smaller number of bits, while layers of the machine learning model with larger dynamic range (e.g., a larger difference between maximum and minimum values) are quantized to an integer representation using a larger number of bits.
Gong and Gadelrab are considered to be analogous because they pertain to utilizing data processing by machine learning/neural networks. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gong to include the concepts of sorting and iteratively processing floating-point variables as well can converting floating point variables to integers as taught by Gadelrab so as to improve fidelity, performance, and interpretability.
Regarding Claim 7,
Gong does not disclose all the limitations of Claim 7.
However, Gadelrab discloses: The method of claim 1, wherein the converting the one or more floating-point variables to integers having the second bit-width comprises: iteratively converting the one or more floating-point variables based on a performance metric.
[0087] As illustrated, the operations 300 may begin at block 302, where a system receives floating point weights for a machine learning model to be executed on the system. The floating point weights may be weights previously determined by a model training system and may be used for performing inferences using the machine learning model by executing the machine learning model on high performance processors, such as processors capable of performing operations on large bit-size floating point numbers (e.g., 16-bit half precision floating point, 32-bit single precision floating point, 64-bit double precision floating point, etc.).
[0088] At block 304, the system generates quantized weights from the floating point weights. The quantized weights may be generated by reducing the floating point weights into one or more integer approximations of the weights (e.g., 16-bit, 8-bit, 4-bit, etc. integers). As discussed, by reducing floating point weights into integer approximations of the weights, embodiments of the present disclosure may allow for machine learning models to be executed on more power efficient processors that may not be capable of performing floating point operations or may not be capable of performing such operations with acceptable performance.
Gong and Gadelrab are considered to be analogous because they pertain to utilizing data processing by machine learning/neural networks. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gong to include the concept of iteratively converting floating-point variables based on a performance metric as taught by Gadelrab so as to improve fidelity, performance, and interpretability.
Regarding Claim 8,
Gong does not disclose all the limitations of Claim 8.
However, Gadelrab discloses: The method of claim 7, wherein the performance metric comprises the target memory bandwidth.
[0087] As illustrated, the operations 300 may begin at block 302, where a system receives floating point weights for a machine learning model to be executed on the system. The floating point weights may be weights previously determined by a model training system and may be used for performing inferences using the machine learning model by executing the machine learning model on high performance processors, such as processors capable of performing operations on large bit-size floating point numbers (e.g., 16-bit half precision floating point, 32-bit single precision floating point, 64-bit double precision floating point, etc.).
[0088] At block 304, the system generates quantized weights from the floating point weights. The quantized weights may be generated by reducing the floating point weights into one or more integer approximations of the weights (e.g., 16-bit, 8-bit, 4-bit, etc. integers). As discussed, by reducing floating point weights into integer approximations of the weights, embodiments of the present disclosure may allow for machine learning models to be executed on more power efficient processors that may not be capable of performing floating point operations or may not be capable of performing such operations with acceptable performance.
[0091] In some embodiments, the quantized parameters for each layer of the machine learning model may be quantized to a common bit size (e.g., the quantized parameters may be quantized to an n bit representation for every layer in the machine learning model). In some embodiments, the quantized parameters may be quantized to different bit sizes for each layer of the machine learning model such that layers of the machine learning model with smaller dynamic range (e.g., a smaller difference between maximum and minimum values) are quantized to an integer representation using a smaller number of bits, while layers of the machine learning model with larger dynamic range (e.g., a larger difference between maximum and minimum values) are quantized to an integer representation using a larger number of bits.
Gong and Gadelrab are considered to be analogous because they pertain to utilizing data processing by machine learning/neural networks. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gong to include the concept of the performance metric comprising the target memory bandwidth as taught by Gadelrab so as to improve fidelity, performance, and interpretability.
Regarding Claim 9,
Claim 9 is rejected on the same grounds of rejection set forth in claim 1.
Gong discloses: A system comprising: a processor; and a memory storing instructions, which when executed by the processor, cause the processor to: convert floating-point variables of a trained machine-learning model (ML model) to integers having a first bit-width supported by hardware on which the ML model is to execute an inference stage, to provide first converted variables
[0038] In some embodiments, additional optimization of the inference model may be accomplished through mixed-precision auto-tuning. FIG. 5 provides a flowchart illustrating a process 510 for tuning an inference neural network model according to one or more embodiments, with reference to components and features described herein including but not limited to the figures and associated description. Mixed-precision auto-tuning may be applied to an inference model as implemented in inference engine 116 (FIG. 1) or inference engine 416 (FIG. 4). The tuning process may include completing a series of test runs of the model to determine if model accuracy may be improved by making changes to precisions used in quantizing. The process begins (block 512) with a current state of the inference model. At block 514, the inference model is run and a determination of the accuracy of results is obtained. At block 516, the results are compared to accuracy criteria. Accuracy criteria may include, e.g., mean average precision (mAP). Other accuracy criteria may be used for the accuracy test of block 516, such as, for example, the optimal mean squared error (OMSE) process described below. If the accuracy of the results pass the accuracy criteria assessment, the process continues at block 530 (described below). If the accuracy of the results do not pass the accuracy criteria assessment of block 516, then the process proceeds to block 518.
[0036] Returning to FIG. 4, model weights quantization 424 may be implemented by quantizing floating point weights on a per-channel basis using symmetric quantization, such that each high precision weight value (e.g., fp32) is quantized into a signed integer value (e.g., int8).
[0039] At block 518, it is determined if another precision for inputs and/or weights is available for selection. For example, int16 quantization could be selected as an alternative for int8 quantization.
Note: Paragraphs [0036] and [0039] are presented that a FP32 value can be quantized to either an 8-bit or 16-bit integer value, thus providing for a first bit-width/first variable.
Gong does not explicitly disclose the remaining limitations of Claim 1.
However, Gadelrab discloses: compute a baseline memory bandwidth of the ML model based on a sum of memory bandwidths of the first converted variables.
[0086] FIG. 3 illustrates an example sequence of operations 400 that may be performed to activate and deactivate a high efficiency quantized mode for performing inferences on data.
[0088] At block 304, the system generates quantized weights from the floating point weights. The quantized weights may be generated by reducing the floating point weights into one or more integer approximations of the weights (e.g., 16-bit, 8-bit, 4-bit, etc. integers). As discussed, by reducing floating point weights into integer approximations of the weights, embodiments of the present disclosure may allow for machine learning models to be executed on more power efficient processors that may not be capable of performing floating point operations or may not be capable of performing such operations with acceptable performance.
[0091] In some embodiments, the quantized parameters for each layer of the machine learning model may be quantized to a common bit size (e.g., the quantized parameters may be quantized to an n bit representation for every layer in the machine learning model). In some embodiments, the quantized parameters may be quantized to different bit sizes for each layer of the machine learning model such that layers of the machine learning model with smaller dynamic range (e.g., a smaller difference between maximum and minimum values) are quantized to an integer representation using a smaller number of bits, while layers of the machine learning model with larger dynamic range (e.g., a larger difference between maximum and minimum values) are quantized to an integer representation using a larger number of bits.
Note: The concept of “bandwidth” is being interpreted as bit size, to where the different bit numbers are different bandwidths as stated in applicant’s specification. Here, the “baseline” is the integer representation of the smaller number of bits which can be based on one or more integer bit value as shown in paragraph [0088]. The “first bit-width” can be the smallest bit value or part of the smallest dynamic range as part of being quantized into different bit sizes/summation.
Gadelrab further discloses: compute a maximum memory bandwidth of the ML model based on a sum of memory bandwidths of the floating-point variables when converted to integers having a second bit-width supported by the hardware, wherein the second bit-width is greater than the first bit-width.
[0086] FIG. 3 illustrates an example sequence of operations 400 that may be performed to activate and deactivate a high efficiency quantized mode for performing inferences on data.
[0088] At block 304, the system generates quantized weights from the floating point weights. The quantized weights may be generated by reducing the floating point weights into one or more integer approximations of the weights (e.g., 16-bit, 8-bit, 4-bit, etc. integers). As discussed, by reducing floating point weights into integer approximations of the weights, embodiments of the present disclosure may allow for machine learning models to be executed on more power efficient processors that may not be capable of performing floating point operations or may not be capable of performing such operations with acceptable performance.
[0091] In some embodiments, the quantized parameters for each layer of the machine learning model may be quantized to a common bit size (e.g., the quantized parameters may be quantized to an n bit representation for every layer in the machine learning model). In some embodiments, the quantized parameters may be quantized to different bit sizes for each layer of the machine learning model such that layers of the machine learning model with smaller dynamic range (e.g., a smaller difference between maximum and minimum values) are quantized to an integer representation using a smaller number of bits, while layers of the machine learning model with larger dynamic range (e.g., a larger difference between maximum and minimum values) are quantized to an integer representation using a larger number of bits.
Note: Similar to the above, the “second-bit width” can be quantized/summed to different bit size, which can be in the smaller or larger dynamic range, thus providing that a second value can be greater than the first value.
Gadelrab further discloses: select a target memory bandwidth that is between the baseline memory bandwidth and the maximum memory bandwidth.
[0091] In some embodiments, the quantized parameters for each layer of the machine learning model may be quantized to a common bit size (e.g., the quantized parameters may be quantized to an n bit representation for every layer in the machine learning model). In some embodiments, the quantized parameters may be quantized to different bit sizes for each layer of the machine learning model such that layers of the machine learning model with smaller dynamic range (e.g., a smaller difference between maximum and minimum values) are quantized to an integer representation using a smaller number of bits, while layers of the machine learning model with larger dynamic range (e.g., a larger difference between maximum and minimum values) are quantized to an integer representation using a larger number of bits.
Note: Here, the “target bandwidth” can be located in either small or large dynamic ranges. The maximum value, then, can occur at the larger (or smaller) dynamic range.
Gadelrab also discloses: convert one or more of the floating-point variables to integers having the second bit-width, without exceeding the target memory bandwidth
[0088] At block 304, the system generates quantized weights from the floating point weights. The quantized weights may be generated by reducing the floating point weights into one or more integer approximations of the weights (e.g., 16-bit, 8-bit, 4-bit, etc. integers). As discussed, by reducing floating point weights into integer approximations of the weights, embodiments of the present disclosure may allow for machine learning models to be executed on more power efficient processors that may not be capable of performing floating point operations or may not be capable of performing such operations with acceptable performance.
Note: Here, there are integer values that can be less than the target value found in the respective dynamic ranges.
Gong discloses: such that the ML model operates as a mixed-precision ML model.
[0038] In some embodiments, additional optimization of the inference model may be accomplished through mixed-precision auto-tuning. FIG. 5 provides a flowchart illustrating a process 510 for tuning an inference neural network model according to one or more embodiments, with reference to components and features described herein including but not limited to the figures and associated description. Mixed-precision auto-tuning may be applied to an inference model as implemented in inference engine 116 (FIG. 1) or inference engine 416 (FIG. 4). The tuning process may include completing a series of test runs of the model to determine if model accuracy may be improved by making changes to precisions used in quantizing. The process begins (block 512) with a current state of the inference model. At block 514, the inference model is run and a determination of the accuracy of results is obtained. At block 516, the results are compared to accuracy criteria. Accuracy criteria may include, e.g., mean average precision (mAP). Other accuracy criteria may be used for the accuracy test of block 516, such as, for example, the optimal mean squared error (OMSE) process described below. If the accuracy of the results pass the accuracy criteria assessment, the process continues at block 530 (described below). If the accuracy of the results do not pass the accuracy criteria assessment of block 516, then the process proceeds to block 518.
Gong and Gadelrab are considered to be analogous because they pertain to utilizing data processing by machine learning/neural networks. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gong to include the concepts of first- and second-bit widths, where the first bit-width serves as a baseline, the second bit-width is greater than the first bit-width, and a target bit-width between the baseline and maximum bitwidth as taught by Gadelrab so as to improve fidelity, performance, and interpretability.
Regarding Claim 10,
Claim 10 is rejected on the same grounds of rejection set forth in claim 2.
Regarding Claim 11,
Claim 11 is rejected on the same grounds of rejection set forth in claim 3.
Regarding Claim 12,
Claim 12 is rejected on the same grounds of rejection set forth in claim 4.
Regarding Claim 14,
Claim 14 is rejected on the same grounds of rejection set forth in claim 7.
Regarding Claim 15,
Claim 15 is rejected on the same grounds of rejection set forth in claim 1.
Gong discloses: A non-transitory computer readable medium comprising stored instructions, which when executed by the processor, cause the processor to: convert floating-point variables of a trained machine-learning model (ML model) to integers having a first bit-width supported by hardware on which the ML model is to execute an inference stage, to provide first converted variables
[0038] In some embodiments, additional optimization of the inference model may be accomplished through mixed-precision auto-tuning. FIG. 5 provides a flowchart illustrating a process 510 for tuning an inference neural network model according to one or more embodiments, with reference to components and features described herein including but not limited to the figures and associated description. Mixed-precision auto-tuning may be applied to an inference model as implemented in inference engine 116 (FIG. 1) or inference engine 416 (FIG. 4). The tuning process may include completing a series of test runs of the model to determine if model accuracy may be improved by making changes to precisions used in quantizing. The process begins (block 512) with a current state of the inference model. At block 514, the inference model is run and a determination of the accuracy of results is obtained. At block 516, the results are compared to accuracy criteria. Accuracy criteria may include, e.g., mean average precision (mAP). Other accuracy criteria may be used for the accuracy test of block 516, such as, for example, the optimal mean squared error (OMSE) process described below. If the accuracy of the results pass the accuracy criteria assessment, the process continues at block 530 (described below). If the accuracy of the results do not pass the accuracy criteria assessment of block 516, then the process proceeds to block 518.
[0036] Returning to FIG. 4, model weights quantization 424 may be implemented by quantizing floating point weights on a per-channel basis using symmetric quantization, such that each high precision weight value (e.g., fp32) is quantized into a signed integer value (e.g., int8).
[0039] At block 518, it is determined if another precision for inputs and/or weights is available for selection. For example, int16 quantization could be selected as an alternative for int8 quantization.
Note: Paragraphs [0036] and [0039] are presented that a FP32 value can be quantized to either an 8-bit or 16-bit integer value, thus providing for a first bit-width/first variable.
Gong does not explicitly disclose the remaining limitations of Claim 1.
However, Gadelrab discloses: compute a baseline memory bandwidth of the ML model based on a sum of memory bandwidths of the first converted variables.
[0086] FIG. 3 illustrates an example sequence of operations 400 that may be performed to activate and deactivate a high efficiency quantized mode for performing inferences on data.
[0088] At block 304, the system generates quantized weights from the floating point weights. The quantized weights may be generated by reducing the floating point weights into one or more integer approximations of the weights (e.g., 16-bit, 8-bit, 4-bit, etc. integers). As discussed, by reducing floating point weights into integer approximations of the weights, embodiments of the present disclosure may allow for machine learning models to be executed on more power efficient processors that may not be capable of performing floating point operations or may not be capable of performing such operations with acceptable performance.
[0091] In some embodiments, the quantized parameters for each layer of the machine learning model may be quantized to a common bit size (e.g., the quantized parameters may be quantized to an n bit representation for every layer in the machine learning model). In some embodiments, the quantized parameters may be quantized to different bit sizes for each layer of the machine learning model such that layers of the machine learning model with smaller dynamic range (e.g., a smaller difference between maximum and minimum values) are quantized to an integer representation using a smaller number of bits, while layers of the machine learning model with larger dynamic range (e.g., a larger difference between maximum and minimum values) are quantized to an integer representation using a larger number of bits.
Note: The concept of “bandwidth” is being interpreted as bit size, to where the different bit numbers are different bandwidths as stated in applicant’s specification. Here, the “baseline” is the integer representation of the smaller number of bits which can be based on one or more integer bit value as shown in paragraph [0088]. The “first bit-width” can be the smallest bit value or part of the smallest dynamic range as part of being quantized into different bit sizes/summation.
Gadelrab further discloses: compute a maximum memory bandwidth of the ML model based on a sum of memory bandwidths of the floating-point variables when converted to integers having a second bit-width supported by the hardware, wherein the second bit-width is greater than the first bit-width.
[0086] FIG. 3 illustrates an example sequence of operations 400 that may be performed to activate and deactivate a high efficiency quantized mode for performing inferences on data.
[0088] At block 304, the system generates quantized weights from the floating point weights. The quantized weights may be generated by reducing the floating point weights into one or more integer approximations of the weights (e.g., 16-bit, 8-bit, 4-bit, etc. integers). As discussed, by reducing floating point weights into integer approximations of the weights, embodiments of the present disclosure may allow for machine learning models to be executed on more power efficient processors that may not be capable of performing floating point operations or may not be capable of performing such operations with acceptable performance.
[0091] In some embodiments, the quantized parameters for each layer of the machine learning model may be quantized to a common bit size (e.g., the quantized parameters may be quantized to an n bit representation for every layer in the machine learning model). In some embodiments, the quantized parameters may be quantized to different bit sizes for each layer of the machine learning model such that layers of the machine learning model with smaller dynamic range (e.g., a smaller difference between maximum and minimum values) are quantized to an integer representation using a smaller number of bits, while layers of the machine learning model with larger dynamic range (e.g., a larger difference between maximum and minimum values) are quantized to an integer representation using a larger number of bits.
Note: Similar to the above, the “second-bit width” can be quantized/summed to different bit size, which can be in the smaller or larger dynamic range, thus providing that a second value can be greater than the first value.
Gadelrab further discloses: select a target memory bandwidth that is between the baseline memory bandwidth and the maximum memory bandwidth.
[0091] In some embodiments, the quantized parameters for each layer of the machine learning model may be quantized to a common bit size (e.g., the quantized parameters may be quantized to an n bit representation for every layer in the machine learning model). In some embodiments, the quantized parameters may be quantized to different bit sizes for each layer of the machine learning model such that layers of the machine learning model with smaller dynamic range (e.g., a smaller difference between maximum and minimum values) are quantized to an integer representation using a smaller number of bits, while layers of the machine learning model with larger dynamic range (e.g., a larger difference between maximum and minimum values) are quantized to an integer representation using a larger number of bits.
Note: Here, the “target bandwidth” can be located in either small or large dynamic ranges. The maximum value, then, can occur at the larger (or smaller) dynamic range.
Gadelrab also discloses: convert one or more of the floating-point variables to integers having the second bit-width, without exceeding the target memory bandwidth
[0088] At block 304, the system generates quantized weights from the floating point weights. The quantized weights may be generated by reducing the floating point weights into one or more integer approximations of the weights (e.g., 16-bit, 8-bit, 4-bit, etc. integers). As discussed, by reducing floating point weights into integer approximations of the weights, embodiments of the present disclosure may allow for machine learning models to be executed on more power efficient processors that may not be capable of performing floating point operations or may not be capable of performing such operations with acceptable performance.
Note: Here, there are integer values that can be less than the target value found in the respective dynamic ranges.
Gong discloses: such that the ML model operates as a mixed-precision ML model.
[0038] In some embodiments, additional optimization of the inference model may be accomplished through mixed-precision auto-tuning. FIG. 5 provides a flowchart illustrating a process 510 for tuning an inference neural network model according to one or more embodiments, with reference to components and features described herein including but not limited to the figures and associated description. Mixed-precision auto-tuning may be applied to an inference model as implemented in inference engine 116 (FIG. 1) or inference engine 416 (FIG. 4). The tuning process may include completing a series of test runs of the model to determine if model accuracy may be improved by making changes to precisions used in quantizing. The process begins (block 512) with a current state of the inference model. At block 514, the inference model is run and a determination of the accuracy of results is obtained. At block 516, the results are compared to accuracy criteria. Accuracy criteria may include, e.g., mean average precision (mAP). Other accuracy criteria may be used for the accuracy test of block 516, such as, for example, the optimal mean squared error (OMSE) process described below. If the accuracy of the results pass the accuracy criteria assessment, the process continues at block 530 (described below). If the accuracy of the results do not pass the accuracy criteria assessment of block 516, then the process proceeds to block 518.
Gong and Gadelrab are considered to be analogous because they pertain to utilizing data processing by machine learning/neural networks. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gong to include the concepts of first- and second-bit widths, where the first bit-width serves as a baseline, the second bit-width is greater than the first bit-width, and a target bit-width between the baseline and maximum bitwidth as taught by Gadelrab so as to improve fidelity, performance, and interpretability.
Regarding Claim 16,
Claim 16 is rejected on the same grounds of rejection set forth in claim 2.
Regarding Claim 17,
Claim 17 is rejected on the same grounds of rejection set forth in claim 3.
Regarding Claim 18,
Claim 18 is rejected on the same grounds of rejection set forth in claim 4.
Regarding Claim 20,
Claim 20 is rejected on the same grounds of rejection set forth in claim 7.
Claims 5-6, 13, and 19 are rejected under 35 U.S.C. § 103 as being unpatentable over Gong in view of Gadelrab, held further in view of Hetherington et. al. (U.S. Pat. Pub. 2020/0302318), herein referred to as “Hetherington”.
Regarding Claim 5,
Gong does not disclose all the limitations of Claim 5.
Gong discloses: The method of claim 4, wherein the ML model comprises a neural network
[0038] In some embodiments, additional optimization of the inference model may be accomplished through mixed-precision auto-tuning. FIG. 5 provides a flowchart illustrating a process 510 for tuning an inference neural network model according to one or more embodiments, with reference to components and features described herein including but not limited to the figures and associated description. Mixed-precision auto-tuning may be applied to an inference model as implemented in inference engine 116 (FIG. 1) or inference engine 416 (FIG. 4). The tuning process may include completing a series of test runs of the model to determine if model accuracy may be improved by making changes to precisions used in quantizing. The process begins (block 512) with a current state of the inference model. At block 514, the inference model is run and a determination of the accuracy of results is obtained. At block 516, the results are compared to accuracy criteria. Accuracy criteria may include, e.g., mean average precision (mAP). Other accuracy criteria may be used for the accuracy test of block 516, such as, for example, the optimal mean squared error (OMSE) process described below. If the accuracy of the results pass the accuracy criteria assessment, the process continues at block 530 (described below). If the accuracy of the results do not pass the accuracy criteria assessment of block 516, then the process proceeds to block 518.
Gadelrab discloses: wherein the sorting comprises sorting the floating-point variables based further on proximities of the floating-point variables.
[0087] As illustrated, the operations 300 may begin at block 302, where a system receives floating point weights for a machine learning model to be executed on the system. The floating point weights may be weights previously determined by a model training system and may be used for performing inferences using the machine learning model by executing the machine learning model on high performance processors, such as processors capable of performing operations on large bit-size floating point numbers (e.g., 16-bit half precision floating point, 32-bit single precision floating point, 64-bit double precision floating point, etc.).
Gong and Gadelrab are considered to be analogous because they pertain to utilizing data processing by machine learning/neural networks. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gong to include the concept of sorting floating-point as taught by Gadelrab so as to improve fidelity, performance, and interpretability.
Hetherington further teaches a root of a network graph representing the neural network.
[0098] Step 401 assigns tree nodes to tree levels based on distance from root node 311. Step 402 maps tree levels to rule levels, such as by fractions as explained above. An embodiment may limit tree traversal according to a configurable maximum tree depth threshold, such that deepest tree level(s) are ignored during rule generation.
[0002] Machine learning (ML) and deep learning are becoming ubiquitous for two main reasons, which include their ability to solve complex problems in a variety of domains and the growth in the performance and efficiency of computers to support ML. However, as the complexity of problems continue to increase, so too does the complexity of the ML models applied to these problems. Deep learning is a prime example of this trend. Traditional machine learning algorithms, such as neural networks, may only contain a few layers of densely connected neurons, whereas deep learning algorithms, such as convolutional neural networks, may contain tens to hundreds of layers of neurons performing vastly different operations.
[0004] Importance of a feature for a tree node can be determined by measuring the ability of a selected value of the feature to optimize the entropy or impurity, as explained later herein, of a class split at that node. Feature importance can be determined by evaluating the distance of a node to the root of the tree. .
Gong in view of Gadelrab and Hetherington are considered to be analogous because they pertain to utilizing data processing by machine learning/neural networks. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gong in view of Gadelrab to include the root of a network graph concept as taught Hetherington in order to improve fidelity, performance, and interpretability.
Regarding Claim 5,
Gong does not disclose all the limitations of Claim 5.
Gadelrab discloses: The method of claim 1, wherein the converting the one or more
floating-point variables to integers having the second bit-width comprises: converting the one or more floating-point variables to integers having the second bit-width based on one or more of size and proximity of the floating-point variables
[0086] FIG. 3 illustrates an example sequence of operations 400 that may be performed to activate and deactivate a high efficiency quantized mode for performing inferences on data.
[0088] At block 304, the system generates quantized weights from the floating point weights. The quantized weights may be generated by reducing the floating point weights into one or more integer approximations of the weights (e.g., 16-bit, 8-bit, 4-bit, etc. integers). As discussed, by reducing floating point weights into integer approximations of the weights, embodiments of the present disclosure may allow for machine learning models to be executed on more power efficient processors that may not be capable of performing floating point operations or may not be capable of performing such operations with acceptable performance.
[0091] In some embodiments, the quantized parameters for each layer of the machine learning model may be quantized to a common bit size (e.g., the quantized parameters may be quantized to an n bit representation for every layer in the machine learning model). In some embodiments, the quantized parameters may be quantized to different bit sizes for each layer of the machine learning model such that layers of the machine learning model with smaller dynamic range (e.g., a smaller difference between maximum and minimum values) are quantized to an integer representation using a smaller number of bits, while layers of the machine learning model with larger dynamic range (e.g., a larger difference between maximum and minimum values) are quantized to an integer representation using a larger number of bits.
Gong and Gadelrab are considered to be analogous because they pertain to utilizing data processing by machine learning/neural networks. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gong to include the concept of converting floating-point variables to integers having the second bit-width based on one or more of size and proximity of the floating-point variables as taught by Gadelrab so as to improve fidelity, performance, and interpretability.
Hetherington further teaches a root of a network graph representing the neural network.
[0098] Step 401 assigns tree nodes to tree levels based on distance from root node 311. Step 402 maps tree levels to rule levels, such as by fractions as explained above. An embodiment may limit tree traversal according to a configurable maximum tree depth threshold, such that deepest tree level(s) are ignored during rule generation.
[0002] Machine learning (ML) and deep learning are becoming ubiquitous for two main reasons, which include their ability to solve complex problems in a variety of domains and the growth in the performance and efficiency of computers to support ML. However, as the complexity of problems continue to increase, so too does the complexity of the ML models applied to these problems. Deep learning is a prime example of this trend. Traditional machine learning algorithms, such as neural networks, may only contain a few layers of densely connected neurons, whereas deep learning algorithms, such as convolutional neural networks, may contain tens to hundreds of layers of neurons performing vastly different operations.
[0004] Importance of a feature for a tree node can be determined by measuring the ability of a selected value of the feature to optimize the entropy or impurity, as explained later herein, of a class split at that node. Feature importance can be determined by evaluating the distance of a node to the root of the tree. .
Gong in view of Gadelrab and Hetherington are considered to be analogous because they pertain to utilizing data processing by machine learning/neural networks. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gong in view of Gadelrab to include the root of a network graph concept as taught Hetherington in order to improve fidelity, performance, and interpretability.
Regarding Claim 13,
Claim 13 is rejected on the same grounds of rejection set forth in claim 5.
Regarding Claim 19,
Claim 19 is rejected on the same grounds of rejection set forth in claim 5.
Response to Arguments
Applicant’s response filed on January 26, 2026 is acknowledged.
The following claims were amended as part of applicant’s response: 1-20.
There are no new claims and no canceled claims.
Claims 1-20 are pending.
Applicant's arguments filed with respect to independent claims 1, 9, and 15 have been fully considered but they are not persuasive. Applicant generally states that primary reference Gong and secondary reference Gadelrab do not apply as they are a trial-and-error approach versus a computational approach. Id. at 8-9. Applicant merely restates cited portions as part of the office action and nothing more. Examiner respectfully disagrees; the art reads on the limitations as claimed. While the Applicant argues the art is a trial-and-error approach to which the Examiner disagrees, the claim language does not explicitly forbid a trial-and-error approach (as the Applicant states), and the Examiner is using the broadest reasonable interpretation to apply the art to the claims. Therefore, the rejection is maintained.
Conclusion
THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JESSE P. SAMLUK whose telephone number is (571)270-5607. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Derrick Ferris can be reached on 571-272-3123. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JESSE P. SAMLUK/Examiner, Art Unit 2411
/DERRICK W FERRIS/Supervisory Patent Examiner, Art Unit 2411
Prosecution Insights