DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This non-final office action is in response to the application and preliminary amendment filed 8 March 2024.
Claims 11-20 are pending. Claims 11 and 19 are independent claims. Claims 1-10 are cancelled. Claims 11-20 are newly added.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d).
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 8 March 2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Drawings
The examiner accepts the drawings filed 8 March 2024.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 20 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
With respect to independent claim 20, the claim recites a “computer-readable recording media (line 1).” The applicant’s specification defines a “computer-readable recording media” as “all kinds of recording devices that stores data readable by a computer system” and “computer-readable recording media can be distributed to computer systems connected via a network (paragraph 0165).”
Therefore, the broadest reasonable interpretation of the term “computer-readable recording media” includes implementations where the computer-readable recording media is transitory, such as carrier waves and data signals. Such transitory implementations fail to define a process, machine, manufacture, or composition of matter. Therefore, the claim does not fall within at least one of the four categories of patent eligible subject matter, and is non-statutory.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 11-16 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Katayama (US 11816552, filed 26 October 2017) and further in view of Mathew et al. (US 2021/0224658, published 22 July 2021, hereafter Mathew; provided by application on IDS filed 8 March 2024).
As per independent claim 11, Katayama discloses a computer-implemented method for expanding an architecture of a neural network, the method comprising:
selecting a target producer neuron from neurons included in the neural network (column 3, lines 33-57: Here, a neuron of the neural network is an input neuron providing an input signal to a row of weights. Each weight in the row of weights has a settable value that can be multiplied by the input signal so that output flows from the weight to a respective hidden neuron to represent the weighted output)
replacing the target producer neuron with a plurality of producer neurons corresponding to the segments (column 3, lines 33-57: Here, the hidden neurons act as replacement neurons for providing output to an output neuron. The hidden neuron receives the output of the input neuron and applies a weight. This hidden neuron acts as a producer neuron because it generates output and provides the generated output to the output neurons)
setting parameters of each producer neuron such that each producer neuron processes an input of the target producer neuron (column 3, lines 33-57: Here, a neuron of the neural network is an input neuron providing an input signal to a row of weights. Each weight in the row of weights has a settable value (parameters) that can be multiplied by the input signal so that output flows from the weight to a respective hidden neuron to represent the weighted output)
setting parameters of a consumer neuron connected to the target producer neurons such that the consumer neurons process outputs of the plurality of producer neurons (column 3, lines 58-65: Here, an output (consumer) neuron is connected to the target producer (hidden) neurons. The output neuron processes the producer neurons based upon parameter (weights) of the output neurons)
Katayama fails to specifically disclose:
an activation clipped according to a given clipping range
dividing the given clipping range into a plurality of segments
However, Mathew, which is analogous to the claimed invention because it is directed toward clipping activations, discloses:
an activation clipped according to a given clipping range (paragraphs 0005 and 0023-0024: Here, a clipping value (range) is determined by expanding the remaining set of data to a nearest power of two value. The data set is then quantized using the clipping value)
dividing the given clipping range into a plurality of segments (paragraphs 0026-0028: Here, the clipped range is divided into a plurality of segments. This includes a low clipping value (l) and a high clipping value (h))
It would have been obvious to one of ordinary skill in the art at the time of the applicant’s effective filing date to have combined Mathew with Katayama, with a reasonable expectation of success, as it would have allowed for improved accuracy when deriving quantized information (Mathew: paragraph 0005).
As per dependent claim 12, Katayama and Mathew disclose the limitations similar to those in claim 11, and the same rejection is incorporated herein. Katayama discloses producer neurons outputting data (column 3, lines 33-57: Here, the hidden neurons act as replacement neurons for providing output to an output neuron. The hidden neuron receives the output of the input neuron and applies a weight. This hidden neuron acts as a producer neuron because it generates output and provides the generated output to the output neurons).
Katayama fails to specifically disclose an activation clipped according to a range of corresponding segment among the plurality of segments. However, Mathew discloses wherein neurons and an activation clipped according to a range of corresponding segment among the plurality of segments (paragraphs 0016 and 0023-0028: Here, the activation function is implemented in a neural network including a plurality of neurons. The clipping value (range) is determined by expanding the remaining set of data to a nearest power of two value. The data set is then quantized using the clipping value).
It would have been obvious to one of ordinary skill in the art at the time of the applicant’s effective filing date to have combined Mathew with Katayama, with a reasonable expectation of success, as it would have allowed for improved accuracy when deriving quantized information (Mathew: paragraph 0005).
As per dependent claim 13, Katayama and Mathew disclose the limitations similar to those in claim 11, and the same rejection is incorporated herein. Katayama discloses producer neurons and a target producer neuron (column 3, lines 33-57: Here, the hidden neurons act as replacement neurons for providing output to an output neuron. The hidden neuron receives the output of the input neuron and applies a weight. This hidden neuron acts as a producer neuron because it generates output and provides the generated output to the output neurons).
Katayama fails to specifically disclose wherein a plurality of activation outputs has the same precisions as precision of the clipped activation output. However, Mathew discloses wherein a plurality of activation outputs has the same precisions as precision of the clipped activation output (Figure 1; paragraphs 0026-0032: Here, the clipping function is performed to prevent degradation and maintain accuracy of the activation).
It would have been obvious to one of ordinary skill in the art at the time of the applicant’s effective filing date to have combined Mathew with Katayama, with a reasonable expectation of success, as it would have allowed for improved accuracy when deriving quantized information (Mathew: paragraph 0005).
As per dependent claim 14, Katayama and Mathew disclose the limitations similar to those in claim 11, and the same rejection is incorporated herein. Mathew further discloses converting the plurality of segments into segments that have the same size as the given clipping range and do not overlap (paragraph 0022: Here, the power-of-2, per-layer quantization specifies an 8-bit range having two segments, a negative range and a non-negative range between -128 to +127. In this instance the negative range and the non-negative range are the same size and do not overlap).
It would have been obvious to one of ordinary skill in the art at the time of the applicant’s effective filing date to have combined Mathew with Katayama, with a reasonable expectation of success, as it would have allowed for improved accuracy when deriving quantized information (Mathew: paragraph 0005).
As per dependent claim 15, Katayama and Mathew disclose the limitations similar to those in claim 11, and the same rejection is incorporated herein. Mathew further discloses wherein at least two of the plurality of segments have different sizes (paragraphs 0022 and 0027-0028: Here, if the quantized data is unsigned, it will be quantized between 0 and 255. In this instance, the negative/positive segments are different sizes. Alternatively, the clipping value may be used to clip the high and low values to create uneven high/low distribution).
It would have been obvious to one of ordinary skill in the art at the time of the applicant’s effective filing date to have combined Mathew with Katayama, with a reasonable expectation of success, as it would have allowed for improved accuracy when deriving quantized information (Mathew: paragraph 0005).
As per dependent claim 16, Katayama and Mathew disclose the limitations similar to those in claim 11, and the same rejection is incorporated herein. Katayama discloses producer neurons outputting data (column 3, lines 33-57: Here, the hidden neurons act as replacement neurons for providing output to an output neuron. The hidden neuron receives the output of the input neuron and applies a weight. This hidden neuron acts as a producer neuron because it generates output and provides the generated output to the output neurons).
Katayama fails to specifically disclose adjusting the range of each segment in consideration of a computation range. However, Mathew discloses adjusting the range of each segment in consideration of a computation range (paragraphs 0026-0028: Here, the clipped range is divided into a plurality of segments. This includes a low clipping value (l) and a high clipping value (h). This causes the range of each segment to be adjusted)
It would have been obvious to one of ordinary skill in the art at the time of the applicant’s effective filing date to have combined Mathew with Katayama, with a reasonable expectation of success, as it would have allowed for improved accuracy when deriving quantized information (Mathew: paragraph 0005).
With respect to independent claim 19, the claim recites the limitations substantially similar to those in claim 1. The rejection of claim 1 is incorporated herein by reference.
Katayama further discloses:
a memory in which instructions are stored (Figure 8; items 808 and 810; column 11, lines 7-17)
at least one processor (Figure 8, item 804; column 11, lines 7-17)
wherein the at least one processor is configure to execute instructions (claim 11)
As per dependent claim 20, Katayama and Mathew disclose the limitations similar to those in claim 11, and the same rejection is incorporated herein. Katayama further discloses a computer-readable recording medium recording a computer program (claim 11).
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Katayama and Mathew and further in view of Ross (US 2019/0171168, published 6 June 2019).
As per dependent claim 17, Katayama and Mathew disclose the limitations similar to those in claim 16, and the same rejection is incorporated herein. Katayama discloses producer neurons (column 3, lines 33-57). Katayama fails to specifically disclose:
wherein the setting of parameters of each producer neuron comprises setting parameters of each producer neuron based on the range of the segment corresponding to each producer neuron and adjusted range of the segment
the setting of parameters of the consumer neuron comprises setting parameters applied to the output of each producer neuron based on the range of the segment corresponding to each producer neuron and the adjusted range of the segment
However, Ross, which is analogous to the claimed invention because it is directed toward adjusting parameters of neurons, discloses:
wherein the setting of parameters of each neuron comprises setting parameters of each neuron based on the range of the segment corresponding to each neuron and adjusted range of the segment (paragraphs 0041-0043: Here, a plurality of process parameters are monitored. This includes monitoring sensors/actuators to identify a range of measurements. This range is used for identifying values, such as minimum, maximum, and ranges. These values are used to set parameters for the neuron)
the setting of parameters of the consumer neuron comprises setting parameters applied to the output of each neuron based on the range of the segment corresponding to each neuron and the adjusted range of the segment (paragraphs 0050-0051: Here, based upon the range a threshold is set. This threshold allows for identifying of deviation and errors for use in setting parameters)
It would have been obvious to one of ordinary skill in the art at the time of the applicant’s effective filing date to have combined Ross with Katayama-Mathews, with a reasonable expectation of success, as it would have allowed for monitoring and adjusting parameters based upon deviation from a tolerance range (Ross: paragraph 0050).
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Katayama and Mathew and further in view of Sather et al. (US 12165066, filed 14 March 2018, hereafter Sather).
As per dependent claim 18, Katayama and Mathew disclose the limitations similar to those in claim 1, and the same rejection is incorporated herein. Katayama discloses wherein each producer neuron processes the input using the same parameters as parameters of the target producer neuron (column 3, lines 33-57: Here, a neuron of the neural network is an input neuron providing an input signal to a row of weights. Each weight in the row of weights has a settable value (parameters) that can be multiplied by the input signal so that output flows from the weight to a respective hidden neuron to represent the weighted output), and the consumer neuron processes the outputs of the plurality of producer neurons using the same parameters applied to the output of the target producer (column 3, lines 58-65: Here, an output (consumer) neuron is connected to the target producer (hidden) neurons. The output neuron processes the producer neurons based upon parameter (weights) of the output neurons).
However, Katayama fails to specifically disclose an offset according to the plurality of segments. However, Sather, which is analogous to the claimed invention because it is directed toward processing outputs of hidden nodes, discloses an offset according to the plurality of segments (Figure 3; column 4, lines 30-45: Here, for each hidden or output neuron, a linear component is generated of a vector of weight coefficients, a vector of output values of prior nodes, and an offset. These outputs create a set of output points for use in calculating a true positive rate for the network).
It would have been obvious to one of ordinary skill in the art at the time of the applicant’s effective filing date to have combined Sather with Katayama-Mathew, with a reasonable expectation of success, as it would have allowed for improving identification of positives by a neural network by maximizing the true positive rate (Sather: column 2, lines 24-32).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Nagel et al. (US 2020/0302299): Discloses improving performance, accuracy, and precision of quantized models (paragraph 0036) including activation re-quantization and clipping (paragraph 0026)
Gao et al. (US 2020/0082269): Discloses quantization of full-precision weights in a neural network performed after back propagation is performed using a differentiable loss function to improve accuracy (paragraph 0092)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KYLE R STORK whose telephone number is (571)272-4130. The examiner can normally be reached 8am - 2pm; 4pm - 6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas can be reached at 571/272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KYLE R STORK/Primary Examiner, Art Unit 2128
