Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 04/24/2025 has been entered.
Response to Arguments
Regarding the rejection of claims under 35 U.S.C. 103, Applicant’s arguments are directed towards amended portions of the claims that have not been previously examined. New grounds of rejection are given below.
Claim Objections
Claims 14-20 objected to because of the following informalities. These claims, either directly or through dependency, recite an amended limitation, “optimizing the modified digital file for execution of the neural network by the integrated circuit.” However, as this limitation is part of a list introduced by either “causes a processor to” or “configured to,” the word “optimizing” should be “optimize.” Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1-20, either directly or through dependency, recite in relevant part:
wherein the at least one detected representation format is a format not supported by optimization functionality and the predefined representation format is supported by the optimization functionality;
optimizing the modified digital file for execution of the neural network by the integrated circuit;
The term “optimizing” in this claim is ambiguous. Neither the claim nor the specification define the characteristics required for the modified digital file to be optimized for execution. The specification provides examples, such as in paragraph 0063: “The integration software is programmed to optimize the neural network. In particular, the integration software allows for example to optimize a network topology, an order of execution of the elements of the neural networks or else to optimize a memory allocation which can be performed during the execution of the neural network.” These examples do not explain what makes a network topology or memory allocation optimized, or how such optimization can be measured or determined. As a further example, the specification in paragraph 0123 states, “More particularly, the integration software can be programmed to support only predefined data representation formats, in particular for optimizing the neural network,” but this statement does not explain what makes the neural network optimized.
The ambiguity of “optimizing” in this context further renders the term “supported by optimization functionality” ambiguous. A person having ordinary skill in the art would be unable to ascertain what is intended by “optimizing” and whether a number format is, or is not, “supported by optimization functionality” and therefore the person would be unable to ascertain the metes and bounds of the claim.
In further examination below, the term “optimizing” will be read as meaning “performing a process intended to improve,” i.e., altering the digital file with the intention that the execution of the represented neural network by the integrated circuit is improved. Likewise, “wherein the at least one detected representation format is a format not supported by optimization functionality and the predefined representation format is supported by the optimization functionality” will be read as meaning that the use of the “predefined representation format” is intended to result in an improvement over the “the at least one detected representation format” in the execution of the represented neural network by the integrated circuit.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5 and 14-20 rejected under 35 U.S.C. 103 over Martin et al., US Pre-Grant Publication No. 20190087718 (hereafter Martin) in view of Imber, US Pre-Grant Publication No. 2019/0236449 (hereafter Imber) and Glow Documentation, revision posted July 2019, https://github.com/pytorch/glow/blob/4554c5d7cbd79f078a9046eb98cb3ddb79f42223/docs/Quantization.md (hereafter Glow).
Regarding claim 1:
Martin teaches:
“A method for implementing an artificial neural network in an integrated circuit, the method comprising”: Martin, paragraph 0006, “Described herein are hardware implementations of DNNs [an artificial neural network] and related methods with a variable output data format” ; Martin, paragraph 0009, “The hardware implementation of a DNN may be embodied in hardware on an integrated circuit [in an integrated circuit].”
“obtaining an initial digital file representative of a neural network configured according to one or more data representation formats”: Martin, paragraph 0009, “There may be provided a non-transitory computer readable storage medium having stored thereon a computer readable description of a hardware implementation of a DNN [an initial digital file representative of a neural network] that, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying a hardware implementation of a DNN.”
“wherein the one or more data representation formats include formats for representing weights assigned to layers of the neural network, input data for each layer, and output data for each layer”: Martin, paragraph 0035, “For example, in some cases, in addition to receiving, for a set of input data for a hardware pass, information indicating the format of the corresponding output data, the hardware implementation may also receive information indicating the desired internal format of input data and/or output data for one or more of the layers associated with the hardware pass [formats for … input data for each layer, and output data for each layer]”; Martin, paragraph 0061, “Information indicating the weight format(s) used by a particular layer is then provided to the hardware implementation to allow the hardware implementation to process the weights correctly [formats for representing weights assigned to layers of the neural network].”
“and wherein the formats include at least one of signed or unsigned formats, symmetric or asymmetric formats, or different integer representations”: Martin, paragraph 0055, “In some cases, the signed integer may be represented in two's complement format, and in other cases other signed integer formats may be used [signed … formats].”
“detecting at least one representation format representing at least part of data of the neural network, at least one representation format of the input data for each layer of the neural network, and at least one representation format of the output data for each layer of the neural network”: Martin, paragraph 0035, “For example, in some cases, in addition to receiving, for a set of input data for a hardware pass, information indicating the format of the corresponding output data, the hardware implementation may also receive information indicating the desired internal format of input data and/or output data for one or more of the layers associated with the hardware pass [detecting at least one representation format representing at least part of data of the neural network, at least one representation format of the input data for each layer of the neural network, and at least one representation format of the output data for each layer of the neural network].”
(bold only) “converting the at least one detected representation format of the weights, the at least one detected representation format of the input data for each layer, and the at least one detected representation format of the output data for each layer into a predefined representation format by shifting data values by a power-of-two offset based on a number of bits used to represent the data to obtain a modified digital file representative of the neural network”: Martin, paragraph 0033, “In particular, when a hardware implementation receives the set of input data to be processed, the hardware implementation also receives information indicating the desired format for the output data of the hardware pass and the hardware implementation is configured to, prior to outputting the processed data, convert the processed data to the desired format [converting … to obtain a modified digital file representative of the neural network]”; Martin, paragraph 0045, “In some cases, one or more software tools may be used (e.g. during a training phase of the DNN, or, during a mapping phase when the DNN layers are being mapped to hardware passes) to determine an appropriate or optimum output data format for each hardware pass. This may comprise identifying an appropriate or optimum data format for one or more of the layers associated with the hardware pass. For example, in some cases determining an appropriate or optimum output data format for a hardware pass may comprise identifying an appropriate or optimum output data format for the last layer associated with the hardware pass (i.e. the layer that will be processed last in the hardware pass). For example, if a hardware pass is associated with layer a, layer b and layer c which are to be processed in the listed order then determining the optimum data format for the hardware pass may comprise determining an appropriate or optimum data format for layer c. Since the optimum output data format may be different between layers the optimum data format may be different between hardware passes [converting … the at least one detected representation format of the output data for each layer into a predefined representation format]”; Martin, paragraph 0061, “Where the hardware implementation supports a variable weight format software (which may be the same software, or different software, to that used to identify the appropriate or optimum output data format for each hardware pass) may be configured to determine an appropriate or optimum format (or formats) for the weights of each relevant layer (i.e. each layer with associated weights, such as a convolution layer) [converting the at least one detected representation format of the weights]. Specifically, the software may determine an appropriate or optimal single weight format for all weights of a layer, or an appropriate or optimal weight format for each filter of the layer. Information indicating the weight format(s) used by a particular layer is then provided to the hardware implementation to allow the hardware implementation to process the weights correctly. In some cases, the information indicating the weight formats for each such layer (e.g. each convolution layer) may be stored in memory which is read by the hardware implementation. In other cases, the information indicating the weight formats may be stored locally with respect to the hardware implementation (e.g. in a register) or dynamically provided to the hardware implementation in another manner.”
“wherein the at least one detected representation format is a format not supported by optimization functionality and the predefined representation format is supported by the optimization functionality”: Martin, paragraph 0045, “In some cases, one or more software tools may be used (e.g. during a training phase of the DNN, or, during a mapping phase when the DNN layers are being mapped to hardware passes) to determine an appropriate or optimum output data format for each hardware pass. This may comprise identifying an appropriate or optimum data format for one or more of the layers associated with the hardware pass. For example, in some cases determining an appropriate or optimum output data format for a hardware pass may comprise identifying an appropriate or optimum output data format for the last layer associated with the hardware pass (i.e. the layer that will be processed last in the hardware pass). For example, if a hardware pass is associated with layer a, layer b and layer c which are to be processed in the listed order then determining the optimum data format for the hardware pass may comprise determining an appropriate or optimum data format for layer c. Since the optimum output data format may be different between layers the optimum data format may be different between hardware passes [wherein the at least one detected representation format is a format not supported by optimization functionality and the predefined representation format is supported by the optimization functionality, interpreted as preferring one number format over another to improve the hardware execution of the neural network].”
“optimizing the modified digital file for execution of the neural network by the integrated circuit”: Martin, paragraph 0045, “In some cases, one or more software tools may be used (e.g. during a training phase of the DNN, or, during a mapping phase when the DNN layers are being mapped to hardware passes) to determine an appropriate or optimum output data format for each hardware pass. This may comprise identifying an appropriate or optimum data format for one or more of the layers associated with the hardware pass. For example, in some cases determining an appropriate or optimum output data format for a hardware pass may comprise identifying an appropriate or optimum output data format for the last layer associated with the hardware pass (i.e. the layer that will be processed last in the hardware pass) [optimizing the modified digital file for execution of the neural network by the integrated circuit, interpreted as performing a process to improve the execution of the neural network on hardware]. For example, if a hardware pass is associated with layer a, layer b and layer c which are to be processed in the listed order then determining the optimum data format for the hardware pass may comprise determining an appropriate or optimum data format for layer c. Since the optimum output data format may be different between layers the optimum data format may be different between hardware passes.”
“integrating the modified digital file into a memory of the integrated circuit”: Martin, paragraph 0130, “The IC manufacturing system 902 is configured to receive an IC definition dataset (e.g. defining a hardware implementation of a DNN as described in any of the examples herein), process the IC definition dataset, and generate an IC according to the IC definition dataset (e.g. which embodies a hardware implementation of a DNN as described in any of the examples herein). The processing of the IC definition dataset configures the IC manufacturing system 902 to manufacture an integrated circuit embodying a hardware implementation of a DNN as described in any of the examples herein [integrating the modified digital file into a memory of the integrated circuit].”
“and executing the artificial neural network on the integrated circuit using the modified digital file integrated into the memory to process input data and generate an output”: Martin, paragraph 0007, “A first aspect provides a method in a hardware implementation of a Deep Neural Network ‘DNN’ configured to implement the DNN by processing data using one or more hardware passes [executing the artificial neural network on the integrated circuit], the method comprising: receiving a set of input data for a hardware pass of the hardware implementation, the set of input data representing at least a portion of input data for a particular layer of the DNN; receiving information indicating a desired output data format for the hardware pass; processing the set of input data according to one or more layers of the DNN associated with the hardware pass to produce processed data [process input data and generate an output], the one or more layers comprising the particular layer of the DNN; and converting the processed data into the desired output data format [using the modified digital file] for the hardware pass to produce output data for the hardware pass.”
Martin does not explicitly teach (bold only) “converting the at least one detected representation format of the weights, the at least one detected representation format of the input data for each layer, and the at least one detected representation format of the output data for each layer into a predefined representation format by shifting data values by a power-of-two offset based on a number of bits used to represent the data to obtain a modified digital file representative of the neural network.”
Imber teaches (bold only) “converting the at least one detected representation format of the weights, the at least one detected representation format of the input data for each layer, and the at least one detected representation format of the output data for each layer into a predefined representation format by shifting data values by a power-of-two offset based on a number of bits used to represent the data to obtain a modified digital file representative of the neural network”: Imber, paragraph 0052, “At block 308, a fixed point number format is selected [converting … into a predefined representation format] for each of one or more sets of values input to, or output from, the current layer that minimises the output error of the instantiation of the DNN. A set of values input to, or output from, a layer may be all or a portion of the input data values for a layer, all or a portion of the weights of a layer, all or a portion of the biases of a layer, or all or a portion of the output data values of a layer [the weights, the at least one detected representation format of the input data for each layer, and the at least one detected representation format of the output data for each layer]. Each layer may have one or more sets of values for which a fixed point number format is to be selected. For example, some layers may have a first set of values that represent the input data values to the layer which can be represented in a fixed point number format and a second set of values that represent the output data values of the layer which can be represented in a fixed point number format. In other examples, some layers (such as convolution layers and fully-connected layers) may have a third set of values that represent the weights of the layer which can be represented using a fixed point number format and a fourth set of values that represents the biases of the layer can be represented in a fixed point number format. Since different sets of values of a layer may have different ranges etc. different fixed point number formats may be selected for each set of values.”
Imber and Martin are analogous arts as they are both related to the determination of number formats for hardware implementations of neural networks. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the selection of number formats for layer weights, inputs, and outputs from Imber to the teachings of Martin to arrive at the present invention, in order to make efficient use of hardware resources when implementing neural networks, as stated in Imber, paragraphs 0002-0003, “DNNs have been implemented in applications where power resources are not a significant factor. Despite this, DNNs have application in a number of different technical fields in which the resources of the hardware used to implement the DNNs is such that power consumption, processing capabilities, or silicon area are limited. There is therefore a need to implement hardware that is configured to implement a DNN in an efficient manner, i.e. in a manner that requires less silicon area or less processing power when operating. Moreover, DNNs can be configured in a number of different ways for a variety of different applications. There is therefore also a need for hardware for implementing a DNN to be flexible to be able to support a variety of DNN configurations.”
Glow teaches (bold only) “converting the at least one detected representation format of the weights, the at least one detected representation format of the input data for each layer, and the at least one detected representation format of the output data for each layer into a predefined representation format by shifting data values by a power-of-two offset based on a number of bits used to represent the data to obtain a modified digital file representative of the neural network”: Glow, “Caffe2 Quantized Model Support” section, “Supported the conversion from uint8 quantized activations to int8 quantized activations. For the quantized Caffe2 ops, the activations are quantized to uint8. In Glow, the activations are quantized to int_8. Therefore, for the offset read from quantized Caffe2 model, we need to subtract 128(i.e. INT8_MIN) [shifting data values by a power-of-two offset based on a number of bits used to represent the data] to make the activations become int8.”
Glow and Martin are analogous arts as they are both related to the quantization of neural networks. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the signed to unsigned type conversion of Glow with the teachings of Martin to arrive at the present invention, in order to allow greater interoperability with existing models, unsigned-to-signed conversions among the adaptations for hardware acceleration, as stated in Glow, “Caffe2 Quantized Model Support” section, “Supported the conversion from uint8 quantized activations to int8 quantized activations. For the quantized Caffe2 ops, the activations are quantized to uint8. In Glow, the activations are quantized to int_8. Therefore, for the offset read from quantized Caffe2 model, we need to subtract 128(i.e. INT8_MIN) [shifting data values by a power-of-two offset based on a number of bits used to represent the data] to make the activations become int8.”
Regarding claim 2:
Martin as modified by Imber and Glow teaches the method according to claim 1.
Martin further teaches “wherein the converting the at least one detected representation format of at least part of the data is carried out for at least one layer of the neural network“: Martin, paragraph 0086, “As described above, different layers typically have different input and output ranges. Accordingly, having variable internal data formats allows different layers [for at least one layer of the neural network] that are processed during the same hardware pass to use different data formats which can improve the efficiency of the processing of those layers.”
Regarding claim 3:
Martin as modified by Imber and Glow teaches the method according to claim 1.
Martin further teaches “wherein the converting the at least one detected representation format of at least part of the data is carried out for each layer of the neural network“: Martin, paragraph 0086, “As described above, different layers typically have different input and output ranges. Accordingly, having variable internal data formats allows different layers [for at least one layer of the neural network] that are processed during the same hardware pass to use different data formats which can improve the efficiency of the processing of those layers.”
Regarding claim 4:
Martin as modified by Imber and Glow teaches the method according to claim 1.
Martin further teaches “adding a first conversion layer at an input of the neural network configured to modify a value of input data that is inputted to the neural network according to the predefined representation format; and adding a second layer for conversion at an output of the neural network configured to modify a value of output data of a last neural network layer according to a format for representing the output data of the initial digital file”: Martin, paragraph 0025, “Each layer receives input data, processes the input data in accordance with the layer to produce output data, which is provided to the next layer as the input data or is output as the final output of the DNN. Accordingly, the first layer receives the original input data to the DNN ( e.g. an image) as the input data. All other layers receive the output data from the previous layer (which may also be referred to as intermediate data) as the input data”; Martin, paragraph 0035, “The hardware implementation may then be configured to perform a conversion of the input data for that layer to the desired format of input data prior to processing the data [a first conversion layer at an input of the neural network configured to modify a value of input data that is inputted to the neural network according to the predefined representation format] and/or perform a conversion of the output data to the desired output data format [a second layer for conversion at an output of the neural network configured to modify a value of output data of a last neural network layer according to a format for representing the output data of the initial digital file] before passing the output data for that layer to the next layer or prior to converting the output data to the desired output data format.”
Regarding claim 5:
Martin as modified by Imber and Glow teaches the method according to claim 1.
Martin further teaches “wherein the neural network comprises a succession of neural network layers, and the data of the neural network include weights assigned to the neural network layers as well as input data and output data that is generated and used by the neural network layers”: Martin, paragraph 0025, “Each layer receives input data, processes the input data in accordance with the layer to produce output data, which is provided to the next layer as the input data or is output as the final output of the DNN. Accordingly, the first layer receives the original input data to the DNN ( e.g. an image) as the input data. All other layers receive the output data [output data] from the previous layer (which may also be referred to as intermediate data) as the input data [input data]”; Martin, paragraph 0027, “As can be seen in FIG. 1, the format of data used in a DNN may be formed of a plurality of planes. The input data may be arranged as P planes of data, where each plane has a dimension x × y. A DNN may comprise one or more convolution layers each of which has associated therewith a plurality of filters formed by a plurality of weights w0 ... wn [weights assigned to the neural network layers].”
Regarding claim 14:
Martin teaches:
“A non-transitory computer-readable media storing computer instructions which, when executed by a processor, causes the processor to”: Martin, paragraph 0011, “There may be provided non-transitory computer readable storage medium having stored thereon computer readable instructions that, when executed at a computer system, cause the computer system to perform the methods as described herein.”
“detect at least one representation format of weights assigned to layers of a neural network, at least one representation format of input data for each layer of the neural network, and at least one representation format of output data for each layer of the neural network, wherein the neural network is represented by an initial digital file”: Martin, paragraph 0009, “There may be provided a non-transitory computer readable storage medium having stored thereon a computer readable description of a hardware implementation of a DNN [neural network is represented by an initial digital file] that, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying a hardware implementation of a DNN”; Martin, paragraph 0035, “For example, in some cases, in addition to receiving, for a set of input data for a hardware pass, information indicating the format of the corresponding output data, the hardware implementation may also receive information indicating the desired internal format of input data and/or output data for one or more of the layers associated with the hardware pass [format of … at least one representation format of input data for each layer of the neural network, and at least one representation format of output data for each layer of the neural network]”; Martin, paragraph 0061, “Information indicating the weight format(s) used by a particular layer is then provided to the hardware implementation to allow the hardware implementation to process the weights correctly [formats for representing weights assigned to layers of the neural network].”
“wherein the at least one representation format of the weights, the at least one representation format of the input data for each layer, and the at least one representation format of the output data for each layer, include at least one of signed or unsigned formats, symmetric or asymmetric formats, or different integer representations”: Martin, paragraph 0055, “In some cases, the signed integer may be represented in two's complement format, and in other cases other signed integer formats may be used [signed … formats].”
(bold only) “convert the at least one detected representation format of the weights, the at least one detected representation format of the input data for each layer, and the at least one detected representation format of the output data for each layer into a predefined representation format by shifting data values by a power-of-two offset based on a number of bits used to represent the data to obtain a modified digital file representative of the neural network”: Martin, paragraph 0033, “In particular, when a hardware implementation receives the set of input data to be processed, the hardware implementation also receives information indicating the desired format for the output data of the hardware pass and the hardware implementation is configured to, prior to outputting the processed data, convert the processed data to the desired format [convert … to obtain a modified digital file representative of the neural network]”; Martin, paragraph 0045, “In some cases, one or more software tools may be used (e.g. during a training phase of the DNN, or, during a mapping phase when the DNN layers are being mapped to hardware passes) to determine an appropriate or optimum output data format for each hardware pass. This may comprise identifying an appropriate or optimum data format for one or more of the layers associated with the hardware pass. For example, in some cases determining an appropriate or optimum output data format for a hardware pass may comprise identifying an appropriate or optimum output data format for the last layer associated with the hardware pass (i.e. the layer that will be processed last in the hardware pass). For example, if a hardware pass is associated with layer a, layer b and layer c which are to be processed in the listed order then determining the optimum data format for the hardware pass may comprise determining an appropriate or optimum data format for layer c. Since the optimum output data format may be different between layers the optimum data format may be different between hardware passes [convert … the at least one detected representation format of the output data for each layer into a predefined representation format]”; Martin, paragraph 0061, “Where the hardware implementation supports a variable weight format software (which may be the same software, or different software, to that used to identify the appropriate or optimum output data format for each hardware pass) may be configured to determine an appropriate or optimum format (or formats) for the weights of each relevant layer (i.e. each layer with associated weights, such as a convolution layer) [convert the at least one detected representation format of the weights]. Specifically, the software may determine an appropriate or optimal single weight format for all weights of a layer, or an appropriate or optimal weight format for each filter of the layer. Information indicating the weight format(s) used by a particular layer is then provided to the hardware implementation to allow the hardware implementation to process the weights correctly. In some cases, the information indicating the weight formats for each such layer (e.g. each convolution layer) may be stored in memory which is read by the hardware implementation. In other cases, the information indicating the weight formats may be stored locally with respect to the hardware implementation (e.g. in a register) or dynamically provided to the hardware implementation in another manner.”
“wherein the at least one detected representation format is a format not supported by optimization functionality and the predefined representation format is supported by the optimization functionality”: Martin, paragraph 0045, “In some cases, one or more software tools may be used (e.g. during a training phase of the DNN, or, during a mapping phase when the DNN layers are being mapped to hardware passes) to determine an appropriate or optimum output data format for each hardware pass. This may comprise identifying an appropriate or optimum data format for one or more of the layers associated with the hardware pass. For example, in some cases determining an appropriate or optimum output data format for a hardware pass may comprise identifying an appropriate or optimum output data format for the last layer associated with the hardware pass (i.e. the layer that will be processed last in the hardware pass). For example, if a hardware pass is associated with layer a, layer b and layer c which are to be processed in the listed order then determining the optimum data format for the hardware pass may comprise determining an appropriate or optimum data format for layer c. Since the optimum output data format may be different between layers the optimum data format may be different between hardware passes [wherein the at least one detected representation format is a format not supported by optimization functionality and the predefined representation format is supported by the optimization functionality, interpreted as preferring one number format over another to improve the hardware execution of the neural network].”
“optimizing the modified digital file for execution of the neural network by the integrated circuit”: Martin, paragraph 0045, “In some cases, one or more software tools may be used (e.g. during a training phase of the DNN, or, during a mapping phase when the DNN layers are being mapped to hardware passes) to determine an appropriate or optimum output data format for each hardware pass. This may comprise identifying an appropriate or optimum data format for one or more of the layers associated with the hardware pass. For example, in some cases determining an appropriate or optimum output data format for a hardware pass may comprise identifying an appropriate or optimum output data format for the last layer associated with the hardware pass (i.e. the layer that will be processed last in the hardware pass) [optimizing the modified digital file for execution of the neural network by the integrated circuit, interpreted as performing a process to improve the execution of the neural network on hardware]. For example, if a hardware pass is associated with layer a, layer b and layer c which are to be processed in the listed order then determining the optimum data format for the hardware pass may comprise determining an appropriate or optimum data format for layer c. Since the optimum output data format may be different between layers the optimum data format may be different between hardware passes.”
“integrate the modified digital file into a memory of an integrated circuit”: Martin, paragraph 0130, “The IC manufacturing system 902 is configured to receive an IC definition dataset (e.g. defining a hardware implementation of a DNN as described in any of the examples herein), process the IC definition dataset, and generate an IC according to the IC definition dataset (e.g. which embodies a hardware implementation of a DNN as described in any of the examples herein). The processing of the IC definition dataset configures the IC manufacturing system 902 to manufacture an integrated circuit embodying a hardware implementation of a DNN as described in any of the examples herein [integrate the modified digital file into a memory of an integrated circuit].”
“execute an artificial neural network using the modified digital file to process input data and generate an output”: Martin, paragraph 0007, “A first aspect provides a method in a hardware implementation of a Deep Neural Network ‘DNN’ configured to implement the DNN by processing data using one or more hardware passes [execute an artificial neural network], the method comprising: receiving a set of input data for a hardware pass of the hardware implementation, the set of input data representing at least a portion of input data for a particular layer of the DNN; receiving information indicating a desired output data format for the hardware pass; processing the set of input data according to one or more layers of the DNN associated with the hardware pass to produce processed data [process input data and generate an output], the one or more layers comprising the particular layer of the DNN; and converting the processed data into the desired output data format [using the modified digital file] for the hardware pass to produce output data for the hardware pass.”
Martin does not explicitly teach (bold only) “convert the at least one detected representation format of the weights, the at least one detected representation format of the input data for each layer, and the at least one detected representation format of the output data for each layer into a predefined representation format by shifting data values by a power-of-two offset based on a number of bits used to represent the data to obtain a modified digital file representative of the neural network.”
Imber teaches (bold only) “convert the at least one detected representation format of the weights, the at least one detected representation format of the input data for each layer, and the at least one detected representation format of the output data for each layer into a predefined representation format by converting between signed and unsigned representations while preserving quantization parameters to obtain a modified digital file representative of the neural network”: Imber, paragraph 0052, “At block 308, a fixed point number format is selected [convert … into a predefined representation format] for each of one or more sets of values input to, or output from, the current layer that minimises the output error of the instantiation of the DNN. A set of values input to, or output from, a layer may be all or a portion of the input data values for a layer, all or a portion of the weights of a layer, all or a portion of the biases of a layer, or all or a portion of the output data values of a layer [the weights, the at least one detected representation format of the input data for each layer, and the at least one detected representation format of the output data for each layer]. Each layer may have one or more sets of values for which a fixed point number format is to be selected. For example, some layers may have a first set of values that represent the input data values to the layer which can be represented in a fixed point number format and a second set of values that represent the output data values of the layer which can be represented in a fixed point number format. In other examples, some layers (such as convolution layers and fully-connected layers) may have a third set of values that represent the weights of the layer which can be represented using a fixed point number format and a fourth set of values that represents the biases of the layer can be represented in a fixed point number format. Since different sets of values of a layer may have different ranges etc. different fixed point number formats may be selected for each set of values.”
Imber and Martin are analogous arts as they are both related to the determination of number formats for hardware implementations of neural networks. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the selection of number formats for layer weights, inputs, and outputs from Imber to the teachings of Martin to arrive at the present invention, in order to make efficient use of hardware resources when implementing neural networks, as stated in Imber, paragraphs 0002-0003, “DNNs have been implemented in applications where power resources are not a significant factor. Despite this, DNNs have application in a number of different technical fields in which the resources of the hardware used to implement the DNNs is such that power consumption, processing capabilities, or silicon area are limited. There is therefore a need to implement hardware that is configured to implement a DNN in an efficient manner, i.e. in a manner that requires less silicon area or less processing power when operating. Moreover, DNNs can be configured in a number of different ways for a variety of different applications. There is therefore also a need for hardware for implementing a DNN to be flexible to be able to support a variety of DNN configurations.”
Glow teaches (bold only) “convert the at least one detected representation format of the weights, the at least one detected representation format of the input data for each layer, and the at least one detected representation format of the output data for each layer into a predefined representation format by converting between signed and unsigned representations while preserving quantization parameters to obtain a modified digital file representative of the neural network”: Glow, “Caffe2 Quantized Model Support” section, “Supported the conversion from uint8 quantized activations to int8 quantized activations. For the quantized Caffe2 ops, the activations are quantized to uint8. In Glow, the activations are quantized to int_8. Therefore, for the offset read from quantized Caffe2 model, we need to subtract 128(i.e. INT8_MIN) to make the activations become int8 [converting between signed and unsigned representations while preserving quantization parameters].”
Glow and Martin are analogous arts as they are both related to the quantization of neural networks. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the signed to unsigned type conversion of Glow with the teachings of Martin to arrive at the present invention, in order to allow greater interoperability with existing models, unsigned-to-signed conversions among the adaptations for hardware acceleration, as stated in Glow, “Caffe2 Quantized Model Support” section, “Supported the conversion from uint8 quantized activations to int8 quantized activations. For the quantized Caffe2 ops, the activations are quantized to uint8. In Glow, the activations are quantized to int_8. Therefore, for the offset read from quantized Caffe2 model, we need to subtract 128(i.e. INT8_MIN) [shifting data values by a power-of-two offset based on a number of bits used to represent the data] to make the activations become int8.”
Regarding claim 15:
Martin as modified by Imber and Glow teaches the non-transitory computer-readable media of claim 14.
Martin further teaches “wherein the instructions, when executed by the processor, integrates the modified digital file into a memory of an integrated circuit“: Martin, paragraph 0130, “The IC manufacturing system 902 is configured to receive an IC definition dataset (e.g. defining a hardware implementation of a DNN as described in any of the examples herein), process the IC definition dataset, and generate an IC according to the IC definition dataset (e.g. which embodies a hardware implementation of a DNN as described in any of the examples herein). The processing of the IC definition dataset configures the IC manufacturing system 902 to manufacture an integrated circuit embodying a hardware implementation of a DNN as described in any of the examples herein [integrate the modified digital file into a memory of an integrated circuit].”
Regarding claim 16:
Martin teaches:
“A computer-based tool, comprising”: Martin, paragraph 0010, “There may be provided an integrated circuit manufacturing system [A computer-based tool comprising] comprising: a non-transitory computer readable storage medium having stored thereon a computer readable description of the hardware implementation of a DNN; a layout processing system configured to process the computer readable description so as to generate a circuit layout description of an integrated circuit embodying the hardware implementation of a DNN; and an integrated circuit generation system configured to manufacture the hardware implementation of a DNN according to the circuit layout description.”
“an input configured to receive an initial digital file representative of a neural network configured according to one or more data representation formats”: Martin, paragraph 0009, “There may be provided a non-transitory computer readable storage medium having stored thereon a computer readable description of a hardware implementation of a DNN [an initial digital file representative of a neural network] that, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying a hardware implementation of a DNN.”
“wherein the one or more data representation formats include formats for representing weights assigned to layers of the neural network, input data for each layer, and output data for each layer”: Martin, paragraph 0035, “For example, in some cases, in addition to receiving, for a set of input data for a hardware pass, information indicating the format of the corresponding output data, the hardware implementation may also receive information indicating the desired internal format of input data and/or output data for one or more of the layers associated with the hardware pass [formats for … input data for each layer, and output data for each layer]”; Martin, paragraph 0061, “Information indicating the weight format(s) used by a particular layer is then provided to the hardware implementation to allow the hardware implementation to process the weights correctly [formats for representing weights assigned to layers of the neural network].”
“and wherein the formats include at least one of signed or unsigned formats, symmetric or asymmetric formats, or different integer representations”: Martin, paragraph 0055, “In some cases, the signed integer may be represented in two's complement format, and in other cases other signed integer formats may be used [signed … formats].”
“a processing unit communicatively coupled to the input and configured to” : Martin, paragraph 0123, “The algorithms and methods described herein could be performed by one or more processors executing code that causes the processor(s) [a processing unit] to perform the algorithms/methods.”
“detect at least one representation format representing at least part of data of the neural network, at least one representation format of the input data for each layer of the neural network, and at least one representation format of the output data for each layer of the neural network”: Martin, paragraph 0035, “For example, in some cases, in addition to receiving, for a set of input data for a hardware pass, information indicating the format of the corresponding output data, the hardware implementation may also receive information indicating the desired internal format of input data and/or output data for one or more of the layers associated with the hardware pass [detect at least one representation format representing at least part of data of the neural network, at least one representation format of the input data for each layer of the neural network, and at least one representation format of the output data for each layer of the neural network]”
(bold only) “convert the at least one detected representation format of the weights, the at least one detected representation format of the input data for each layer, and the at least one detec