DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 lines 17-21 recite “a hardware controller configured to determine the at least one scaling factor based on the first multiply-and-accumulate result, or a software controller configured to determine the at least one scaling factor based on the first multiply-and-accumulate result”. It is unclear whether the claim requires only one of a hardware controller or a software controller, or that the apparatus is configured to selectively choose from a hardware or software controller. For purposes of examination, Examiner interprets as only one of the hardware controller or the software controller is required by the claim. Claims 2-7 inherit the same deficiency as claim 1 based on dependence. Claim 8 recites substantially the same limitation and is rejected for the same reason. Claims 9-14 inherit the same deficiency as claim 8 based on dependence.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7 are rejected under 35 U.S.C. 103 as being unpatentable over US 20220066740 A1 Radhadevi et al., (hereinafter “Radhadevi”) in view of US 11899518 B2 Kirshenboim et al, (hereinafter “Kirshenboim”)
Regarding claim 1, Radhadevi teaches the following:
a first plurality of inputs representing an activation input vector (fig 1A/1B, [0100-0102], Input_1.. Input_16, [0002], [0003], [0010] vector dot product inputs for a convolution neural network (CNN) being an activation input vector);
a second plurality of inputs representing a weight input vector (fig 1A/1B, [0100-0103], input weights, 8 bit coefficient);
an analog multiplier-and-accumulator to generate a first analog voltage representing a first multiply-and-accumulate result for the said first inputs and the second inputs (Fig 1A/1B, Fig 1C showing the multiplier, [0101-0105], with accumulate performed in a column basis);
a voltage multiplier that takes the said first analog voltage and produces a second analog voltage representing a second multiply-and-accumulate result by multiplying at least one scaling factor to the first analog voltage (fig 9-912, fig 9B-2, [0165-0166]);
an analog to digital converter configured to convert the said second analog voltage multiply-and-accumulate result into a digital signal (fig 1A/1B [0105] ADC); and
a hardware controller configured to determine the at least one scaling factor based on the first multiply-and-accumulate result, or a software controller configured to determine the at least one scaling factor based on the first multiply-and- accumulate result (fig 9, [0163], controller controlling scaling caps 912 based on the accumulation in from the multiply accumulate).
Radhadevi does not explicitly disclose a limited-precision operation during a neural network inference operation. However, in the same field of endeavor, Kirshenboim discloses an apparatus similar to Radhadevi for performing an analog multiply and accumulate operation for a neural network processor (abstract, fig 2, fig 3). Kirshenboim further discloses limiting the precision at the analog to digital converter (fig 3 310, 308). It would have been obvious to one of ordinary skill in the art before the effective filing data do add the mechanism for limited precision disclosed by Kirshenboim at the ADC of Radhadevi. It would have been obvious to achieve the benefit of reducing power consumption during inference such as on an edge device (col 3 line 1-33, col 6 lines 22-32).
Regarding claim 2, in addition to the teachings addressed in the claim 1 analysis, Radhadevi teaches the following:
wherein the at least one scaling factor comprises a plurality of independent scaling factors determined during different aspects of a neural network, one independent scaling factor per switched capacitor operation(fig 9 912 scaling caps having different scaling factors based on switch positions, [0163], determined by controller controlling scaling caps 912 based on the accumulation in from the multiply accumulate).
Radhadevi discloses different aspects of a neural network, but does not explicitly disclose the scaling factors determined during training, or application to a layer of a neural network. However Kirshenboim discloses initialization of weights and precision during training of the neural network (fig 4, col 12 line 58- col 14 line 42) . It would have been obvious to one of ordinary skill in the art before the effective filing date to include the determining of the at least one scaling factor used to amplify the first plurality of inputs or to amplify the analog voltage multiply-and-accumulate result during the training as disclosed by Kirshenboim. Furthermore, Kirshenboim discloses training different setting parameter values for each layer of a number of layers in a neural network (col 5 line 50-57). It would further have been obvious to determine the scaling factors of a layer of a neural network comprising a plurality of layers. It would have been obvious to achieve optimum control of various parameters including the dynamic control of the ADC (col 13 line 66 – col 14 line 7).
Regarding claim 3, Radhadevi in view of Kirshenboim teach the claim 1 analysis. Radhadevi discloses different aspects of a neural network, but does not explicitly disclose determining the at least one scaling factor determined during training of a neural network. However, Kirshenboim discloses initialization of weights and precision during training of the neural network (fig 4, col 12 line 58- col 14 line 42) . It would have been obvious to one of ordinary skill in the art before the effective filing date to include the determining of the at least one scaling factor during the training as disclosed by Kirshenboim. It would have been obvious to achieve optimum control of various parameters including the dynamic control of the ADC (col 13 line 66 – col 14 line 7).
Regarding claim 4, Radhadevi in view of Kirshenboim teach the claim 3 limitations. Radhadevi further discloses that the at least one scaling factor determined during training and used at inference is further an integer value ([0165] scaled by 2).
Regarding claim 5, in addition to the teachings addressed in the claim 1 analysis, Radhadevi teaches the following:
an accumulation store charge configured to accumulate a charge corresponding to the second analog voltage multiply-and- accumulate result for a number of iterations (abstract, [0005] operation over 4 clock cycles, fig 1E-1H, fig 2E).
Regarding claim 6, in addition to the teachings addressed in the claim 1 analysis, Radhadevi teaches the following:
a programmable controller configured to control the voltage multiplier, based on the at least one scaling factor (fig 9 controlling the voltage multiplier 912 based on scaling factors of the capacitors, and programmable as shown in the various charge caps of fig 9B-3).
Regarding claim 7, in addition to the teachings addressed in the claim 1 analysis, Radhadevi teaches the following:
wherein the voltage multiplier comprises a plurality of switched capacitors configured in series or parallel (fig 9 912).
Claims 8-20 are rejected under 35 U.S.C. 103 as being unpatentable over Radhadevi in view of Kirshenboim in view of A. Sebastian et al., Memory devices and applications for in-memory computing, Focus|Review article, nature nanotechnology, 2020 (hereinafter “Sebastian”)
Regarding claim 8, Radhadevi teaches the following:
a first plurality of inputs representing an original activation input vector (fig 1A/1B, [0100-0102], Input_1.. Input_16, [0002], [0003], [0010] vector dot product inputs for a convolution neural network (CNN) being an activation input vector);
a third plurality of inputs representing a weight input vector (fig 1A/1B, [0100-0103], input weights, 8 bit coefficient);
an analog multiplier-and-accumulator to generate an analog voltage representing a first multiply-and-accumulate result for the said first inputs and the third inputs (Fig 1A/1B, Fig 1C showing the multiplier, [0101-0105], with accumulate performed in a column basis);
an analog to digital converter configured to convert the said analog voltage multiply-and-accumulate result into a digital signal (fig 1A/1B [0105] ADC); and
a hardware controller configured to determine the at least one scaling factor based on the first multiply-and-accumulate result, or a software controller configured to determine the at least one scaling factor based on the first multiply-and- accumulate result (fig 9, [0163], controller controlling scaling caps 912 based on the accumulation in from the multiply accumulate).
Radhadevi does not explicitly disclose a limited-precision operation during a neural network inference operation, or a plurality of voltage multipliers that take the said first plurality of inputs and produce a second plurality of inputs by multiplying at least one scaling factor to voltages of the original activation input vector. However, in the same field of endeavor, Kirshenboim discloses an apparatus similar to Radhadevi for performing an analog multiply and accumulate operation for a neural network processor (abstract, fig 2, fig 3). Kirshenboim further discloses limiting the precision at the analog to digital converter (fig 3 310, 308). It would have been obvious to one of ordinary skill in the art before the effective filing data do add the mechanism for limited precision disclosed by Kirshenboim at the ADC of Radhadevi. It would have been obvious to achieve the benefit of reducing power consumption during inference such as on an edge device (col 3 line 1-33, col 6 lines 22-32).
Furthermore, as to the plurality of voltage multipliers, Sebastian teaches computing devices similar to Radhadevi and Kirshenboim for performing crossbar computing including multiplication and accumulation for deep learning (p. 536 second column – p. 538, fig 5, fig 6). Sebastian further discloses scaling the input ranges of the device (p. 539 last paragraph). It would have been obvious to one of ordinary skill in the art before the effective filing date include a plurality of voltage multipliers as in the input Scaling of Sebastian at the input of Radhadevi in view of Kirshenboim by substituting the voltage multiplier of Radhadevi at the inputs. It would have been obvious to achieve the benefit of properly scaling the input range such that the crossbar falls within the limited dynamic range of the ADC (Sebastian p. 539 last sentence).
Regarding claim 9, in addition to the teachings addressed in the claim 8 analysis, Radhadevi teaches the following:
wherein the at least one scaling factor comprises a plurality of independent scaling factors determined during different aspects of a neural network, one independent scaling factor per switched capacitor operation(fig 9 912 scaling caps having different scaling factors based on switch positions, [0163], determined by controller controlling scaling caps 912 based on the accumulation in from the multiply accumulate).
Radhadevi discloses different aspects of a neural network, but does not explicitly disclose the scaling factors determined during training, or application to a layer of a neural network. However Kirshenboim discloses initialization of weights and precision during training of the neural network (fig 4, col 12 line 58- col 14 line 42) . It would have been obvious to one of ordinary skill in the art before the effective filing date to include the determining of the at least one scaling factor used to amplify the first plurality of inputs or to amplify the analog voltage multiply-and-accumulate result during the training as disclosed by Kirshenboim. Furthermore, Kirshenboim discloses training different setting parameter values for each layer of a number of layers in a neural network (col 5 line 50-57). It would further have been obvious to determine the scaling factors of a layer of a neural network comprising a plurality of layers. It would have been obvious to achieve optimum control of various parameters including the dynamic control of the ADC (col 13 line 66 – col 14 line 7).
Regarding claim 10, Radhadevi in view of Kirshenboim in view of Sebastian teach the claim 9 analysis. Radhadevi discloses different aspects of a neural network, but does not explicitly disclose determining the plurality of independent scaling factor determined during training of a neural network. However, Kirshenboim discloses initialization of weights and precision during training of the neural network (fig 4, col 12 line 58- col 14 line 42) . It would have been obvious to one of ordinary skill in the art before the effective filing date to include the determining of the at least one scaling factor during the training as disclosed by Kirshenboim. It would have been obvious to achieve optimum control of various parameters including the dynamic control of the ADC (col 13 line 66 – col 14 line 7).
Regarding claim 11, Radhadevi in view of Kirshenboim in view of Sebastian teach the claim 8 analysis. Radhadevi discloses different aspects of a neural network, but does not explicitly disclose determining the at least one scaling factor determined during training of a neural network. However, Kirshenboim discloses initialization of weights and precision during training of the neural network (fig 4, col 12 line 58- col 14 line 42) . It would have been obvious to one of ordinary skill in the art before the effective filing date to include the determining of the at least one scaling factor during the training as disclosed by Kirshenboim. It would have been obvious to achieve optimum control of various parameters including the dynamic control of the ADC (col 13 line 66 – col 14 line 7).
Regarding claim 12, Radhadevi in view of Kirshenboim in view of Sebastian teach the claim 11 limitations. Radhadevi further discloses that the at least one scaling factor determined during training and used at inference is further an integer value ([0165] scaled by 2).
Regarding claim 13, in addition to the teachings addressed in the claim 8 analysis, Radhadevi teaches the following:
an accumulation store charge configured to accumulate a charge corresponding to the second analog voltage multiply-and- accumulate result for a number of iterations (abstract, [0005] operation over 4 clock cycles, fig 1E-1H, fig 2E).
Regarding claim 14 in addition to the teachings addressed in the claim 8 analysis, Radhadevi teaches the following:
at least one programmable controller configured to control the plurality of voltage multipliers, based on the at least one scaling factor (fig 9 controlling the voltage multipliers 912 based on scaling factors of the capacitors, and programmable as shown in the various charge caps of fig 9B-3, wherein the voltage multiplier of Radhadevi is substitute at the input location disclosed by Sebastian).
Regarding claim 15, Radhadevi teaches the following:
receiving a first plurality of inputs representing an activation input vector (fig 1A/1B, [0100-0102], Input_1.. Input_16, [0002], [0003], [0010] vector dot product inputs for a convolution neural network (CNN) being an activation input vector);
receiving a second plurality of inputs representing a weight input vector (fig 1A/1B, [0100-0103], input weights, 8 bit coefficient);
generating, with an analog multiplier-and-accumulator, an analog voltage representing a multiply-and-accumulate result for the first plurality of inputs and the second plurality of inputs (Fig 1A/1B, Fig 1C showing the multiplier, [0101-0105], with accumulate performed in a column basis);
converting, with an analog to digital converter, the analog voltage multiply-and-accumulate result into a digital signal (fig 1A/1B [0105] ADC); and
determining, during different applications of the neural network, at least one scaling factor used to amplify the first plurality of inputs or to amplify the analog voltage multiply-and- accumulate result (fig 9, [0163], controller controlling scaling caps 912 based on the accumulation in from the multiply accumulate, [0003] the apparatus is applied in different applications of a convolutional neural network, which results in the determining being during either training or calibration or both of the neural network, [0165] scale by 2 for amplification, fig 10).
Radhadevi does not explicitly disclose a limited-precision operation during a neural network inference operation. However, in the same field of endeavor, Kirshenboim discloses an apparatus similar to Radhadevi for performing an analog multiply and accumulate operation for a neural network processor (abstract, fig 2, fig 3). Kirshenboim further discloses limiting the precision at the analog to digital converter (fig 3 310, 308). It would have been obvious to one of ordinary skill in the art before the effective filing data do add the mechanism for limited precision disclosed by Kirshenboim at the ADC of Radhadevi. It would have been obvious to achieve the benefit of reducing power consumption during inference such as on an edge device (col 3 line 1-33, col 6 lines 22-32).
Furthermore, Radhadevi, although disclosing determining during different applications of neural network, does not explicitly disclose the determining the at least one scaling factor during training or calibration of the neural network. However Kirshenbaum discloses initialization of weights and precision during training of the neural network (fig 4, col 12 line 58- col 14 line 42) . It would have been obvious to one of ordinary skill in the art before the effective filing date to include the determining of the at least one scaling factor used to amplify the first plurality of inputs or to amplify the analog voltage multiply-and-accumulate result during the training as disclosed by Kirshenbaum. It would have been obvious to achieve optimum control of various parameters including the dynamic control of the ADC (col 13 line 66 – col 14 line 7).
Furthermore, Radhadevi teaches amplifying the analog voltage multiply-and-accumulate result but does not explicitly disclose amplifying the first plurality of inputs. However, Sebastian teaches computing devices similar to Radhadevi and Kirshenboim for performing crossbar computing including multiplication and accumulation for deep learning (p. 536 second column – p. 538, fig 5, fig 6). Sebastian further discloses scaling the input ranges of the device (p. 539 last paragraph). It would have been obvious to one of ordinary skill in the art before the effective filing date include amplifying the first plurality of inputs as in the input Scaling of Sebastian at the input of Radhadevi in view of Kirshenboim by substituting the voltage multiplier of Radhadevi at the inputs, and determining a scaling factor for the input amplification as in Radhadevi. It would have been obvious to achieve the benefit of properly scaling the input range such that the crossbar falls within the limited dynamic range of the ADC (Sebastian p. 539 last sentence).
Regarding claim 16, in addition to the teachings addressed in the claim 15 analysis, Radhadevi teaches the following:
determining a plurality of independent scaling factors, comprising determining one independent scaling factor per switch capacitor operation determined during different aspects of a neural network, wherein the at least one scaling factor comprises the plurality of independent scaling factors (fig 9 912 scaling caps having different scaling factors based on switch positions, [0163], determined by controller controlling scaling caps 912 based on the accumulation in from the multiply accumulate in a neural network layer).
Radhadevi discloses different aspects of a neural network, but does not explicitly disclose application to a layer of a neural network. However, Kirshenboim discloses training different setting parameter values for each layer of a number of layers in a neural network (col 5 line 50-57). It would further have been obvious to determine the scaling factors of a layer of a neural network comprising a plurality of layers. It would have been obvious to achieve optimum control of various parameters including the dynamic control of the ADC (col 13 line 66 – col 14 line 7).
Regarding claim 17, in addition to the teachings addressed in the claim 15 analysis, Radhadevi teaches the following:
wherein amplifying the first plurality of inputs comprises producing, with a plurality of voltage multipliers, an amplified first plurality of inputs by multiplying the at least one scaling factor to voltages of the activation input vector (the combination as in the claim 15 mapping results in voltage multiplier of Radhadevi at the inputs generating the at least one scaling voltage to the activation input vector), the method further comprising generating, with the analog multiplier-and-accumulator, the analog voltage multiply-and- accumulate result for the amplified first plurality of inputs (as in the claim 15 mapping the generated analog voltage multiply and accumulate results for the first plurality of inputs).
Regarding claim 18, in addition to the teachings addressed in the claim 15 analysis, Radhadevi teaches the following:
wherein amplifying the analog voltage comprises producing, with a voltage multiplier, an amplified analog voltage multiply-and-accumulate result by applying the at least one scaling factor to the analog voltage multiply-and- accumulate result, the method further comprising converting, with the analog to digital converter, the amplified analog voltage multiply-and-accumulate result into the digital signal using the limited-precision operation during the inference operation of the neural network (fig 9, [0163], controller controlling scaling caps 912 based on the accumulation in from the multiply accumulate, [0003] the apparatus is applied in different applications of a convolutional neural network, which results in the determining being during either training or calibration or both of the neural network, [0165] scale by 2 for amplification, fig 1A/1B [0105] ADC, during the inference operation as in the claim 15 mapping).
Regarding claim 19, in addition to the teachings addressed in the claim 18 analysis, Radhadevi teaches the following:
configuring a plurality of switched capacitors of the voltage multiplier in series; or
configuring the plurality of switched capacitors of the voltage multiplier in parallel (fig 9 912).
Regarding claim 20, in addition to the teachings addressed in the claim 15 analysis, Radhadevi teaches the following:
accumulating a charge corresponding to the analog voltage multiply-and- accumulate result for a number of iterations (abstract, [0005] operation over 4 clock cycles, fig 1E-1H, fig 2E).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EMILY E LAROCQUE whose telephone number is (469)295-9289. The examiner can normally be reached on 10:00am - 1200pm, 2:00pm - 8pm ET M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Andrew Caldwell can be reached on 571-272-3701. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EMILY E LAROCQUE/Examiner, Art Unit 2182
Prosecution Insights