DETAILED ACTION
This action is responsive to the amendment filed 02/02/2025. Claims 1-2, 4-10, 12-14, and 16-19 are currently pending in the case. Claims 1-2, 9-10, 12-14, and 16 are currently amended. Claims 1, 9, 13, 18, and 19 are independent claims.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant's claim for domestic priority based on PCT application No. PCT/JP2021/024836 filed on 06/30/2021, which claims priority to a foreign application filed in Japan on 06/30/2020. Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 12/31/2025 is being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-2, 4-8, and are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim limitations “an execution model generation unit…” of claim 1, “a learning unit…” of claim 1, “a hardware generation unit…” of claim 2, “an execution model generation unit…” of claim 18, “a learning unit…” of claim 18, “an execution model generation unit…” of claim 19, and “a learning unit…” of claim 19 been evaluated under the three-prong test set forth in MPEP § 2181, subsection I, but the result is inconclusive. Thus, it is unclear whether this limitation should be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because it is unclear what constitutes the limitations, if each limitation is considered to be their respective “instructions”, then the limitations do not invoke 35 U.S.C. 112(f), however, if each limitation is considered to be their respective “instructions” as well as the “processor device”, then the units are considered to be generic placeholders and the limitations do invoke 35 U.S.C. 112(f). The boundaries of this claim limitation are ambiguous; therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
In response to this rejection, applicant must clarify whether this limitation should be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. Mere assertion regarding applicant’s intent to invoke or not invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph is insufficient. Applicant may:
(a) Amend the claim to clearly invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, by reciting “means” or a generic placeholder for means, or by reciting “step.” The “means,” generic placeholder, or “step” must be modified by functional language, and must not be modified by sufficient structure, material, or acts for performing the claimed function;
(b) Present a sufficient showing that 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, should apply because the claim limitation recites a function to be performed and does not recite sufficient structure, material, or acts to perform that function;
(c) Amend the claim to clearly avoid invoking 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, by deleting the function or by reciting sufficient structure, material or acts to perform the recited function; or
(d) Present a sufficient showing that 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, does not apply because the limitation does not recite a function or does recite a function along with sufficient structure, material or acts to perform that function.
A broad range or limitation together with a narrow range or limitation that falls within the broad range or limitation (in the same claim) may be considered indefinite if the resulting claim does not clearly set forth the metes and bounds of the patent protection desired. See MPEP § 2173.05(c). In the present instance, claim 5 recites the broad recitations "a convolution operation circuit that implements convolution operations" and "a quantization operation circuit that implements quantization operations", and the claim also recites "a convolution operation circuit for performing a convolution operation on the input data stored in the first memory" and "a quantization operation circuit for performing a quantization operation on the convolution operation output data stored in the second memory”, in claim 1, which is the narrower statement of the range/limitation. The claim(s) are considered indefinite because there is a question or doubt as to whether the feature introduced by such narrower language is (a) merely exemplary of the remainder of the claim, and therefore not required, or (b) a required feature of the claims.
Claims 4, 6-8, and 17 are rejected as being dependent upon a rejected base claim without curing any of the deficiencies.
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claims 5-8 are rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends. Claim 5 recites "a convolution operation circuit that implements convolution operations" and "a quantization operation circuit that implements quantization operations". The parent claim recites "a convolution operation circuit for performing a convolution operation on the input data stored in the first memory" and "a quantization operation circuit for performing a quantization operation on the convolution operation output data stored in the second memory”. Therefore, dependent claim 5 fails to further limit the subject matter of the claim upon which is depends, because claim 5 is broader than claim 1. Applicant may cancel the claim(s), amend the claim(s) to place the claim(s) in proper dependent form, rewrite the claim(s) in independent form, or present a sufficient showing that the dependent claim(s) complies with the statutory requirements.
Claims 6-8 are rejected as being dependent upon a rejected base claim without curing any of the deficiencies.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-2, 4-10, 12-14, and 16-19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1:
Step 1 Statutory Category: Claim 1 is directed to a device, which falls under one of the four statutory categories.
Step 2A Prong 1 Judicial exception: Claim 1 recites, in part, “… generates the neural network execution model based on hardware information regarding hardware in which the neural network execution model is operating and network information regarding the neural network”. This limitation, under the broadest reasonable interpretation, and in light of paragraph 0080 of applicant’s specification, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I). Further, the claim recites: “… generates learned parameters of the generated neural network execution model”. This limitation, under the claim interpretation under 35 U.S.C. 112(f) and applicant’s specification paragraph 0181, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I). Further, the claim recites: “partitions convolution operations implemented in the neural network execution model based on the generated neural network execution model in accordance with capacities of the first memory and the second memory”. This limitation, under the broadest reasonable interpretation, and according to applicant’s specification paragraph 0082, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I). Additionally, this limitation, under the broadest reasonable interpretation, covers the recitation of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), in this case an evaluation. See MPEP § 2106.04(a)(2)(III).
Step 2A Prong 2 Integration into a practical application: This judicial exception is not integrated into a practical application. In particular the claim recites: “a neural network generating device”, “an execution model generation unit comprising first instructions executed by a processor device of the neural network generating device”, “a learning unit comprising second instructions executed by the processor device of the neural network generating device”, “the neural network execution model includes a first memory for storing input data, a convolution operation circuit for performing a convolution operation on the input data stored in the first memory, a second memory for storing convolution operation output data of the convolution operation circuit, and a quantization operation circuit for performing a quantization operation on the convolution operation output data stored in the second memory”, and “the execution model generation unit…”. These limitations are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f). Further, the claim recites: “a neural network execution model for operating a neural network”. This limitation is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use, see MPEP §2106.05(h).
Step 2B Significantly more: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements: “a neural network generating device”, “an execution model generation unit comprising first instructions executed by a processor device of the neural network generating device”, “a learning unit comprising second instructions executed by the processor device of the neural network generating device”, “the neural network execution model includes a first memory for storing input data, a convolution operation circuit for performing a convolution operation on the input data stored in the first memory, a second memory for storing convolution operation output data of the convolution operation circuit, and a quantization operation circuit for performing a quantization operation on the convolution operation output data stored in the second memory”, and “the execution model generation unit…” amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept. Further, the additional element: “a neural network execution model for operating a neural network” generally links the use of the judicial exception to a particular technological environment or field of use. Elements that merely generally link the use of the judicial exception to a particular technological environment or field of use cannot provide an inventive concept. The claim is not patent eligible.
Regarding claim 2, the rejection of claim 1 is incorporated, and further the claim recites: “…generates a neural network hardware model based on the hardware information and the neural network execution model”. This limitation, under the broadest reasonable interpretation, recites mathematical concepts in addition to those identified in the rejection of the parent claim, thus the claim recites a judicial exception.
Further, the claim recites: “a hardware generation unit comprising third instructions executed by the processor device of the neural network generating device”. This limitation is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f). Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept. The claim is not patent eligible.
Regarding claim 4, the rejection of claim 1 is incorporated, and further, the claim recites: “…performs associated operations implemented when generating the learned parameters with higher precision than operations implemented by the neural network execution model”. This limitation is a continuation of the “… generates learned parameters of the generated neural network execution model” limitation identified as an abstract idea in the rejection of the parent claim, thus the claim recites a judicial exception.
Further, the claim recites: “the learning unit”. This limitation is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f). Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept. The claim is not patent eligible.
Regarding claim 5, the rejection of claim 1 is incorporated, and further, the claim recites: “the neural network execution model comprises a convolution operation circuit that implements convolution operations and a quantization operation circuit that implements quantization operations”. This limitation is an additional element that amounts to generally linking the use of the judicial exception to a particular technological environment or field of use. See MPEP §2106.05(h). Elements that merely generally link the use of the judicial exception to a particular technological environment or field of use cannot provide an inventive concept. The claim is not patent eligible.
Regarding claim 6, the rejection of claim 5 is incorporated, and further, the claim recites: “…performs convolution operations implemented when generating the learned parameter with higher precision than operations implemented by the convolution operation circuit”. This limitation is a continuation of the “… generates learned parameters of the generated neural network execution model” limitation identified as an abstract idea in the rejection of the parent claim, thus the claim recites a judicial exception.
Further, the claim recites: “the learning unit”. This limitation is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f). Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept. The claim is not patent eligible.
Regarding claim 7, the rejection of claim 5 is incorporated, and further, the claim recites: “…learns quantization parameters that the quantization operation circuit uses for the quantization operations”. This limitation, according to applicant’s specification paragraph 0181, recites mathematical concepts in addition to those identified in the rejection of the parent claim, thus the claim recites a judicial exception.
Further, the claim recites: “the learning unit”. This limitation is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f). Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept. The claim is not patent eligible.
Regarding claim 8, the rejection of claim 7 is incorporated, and further, the claim recites: “…learns a scaling factor for quantization operation output data quantized by the quantization parameters when learning the quantization parameters”. This limitation, under the broadest reasonable interpretation, covers the recitation of mathematical concepts in addition to those identified in the rejection of the parent claim, thus the claim recites a judicial exception.
Further, the claim recites: “the learning unit”. This limitation is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f). Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept. The claim is not patent eligible.
Regarding claim 9:
Step 1 Statutory Category: Claim 9 is directed to a method, which falls under one of the four statutory categories.
Step 2A Prong 1 Judicial exception: Claim 9 recites, in part, “generating the neural network execution model based on the hardware information and the network information”. This limitation, under the broadest reasonable interpretation, and in light of paragraph 0080 of applicant’s specification, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I). Further, the claim recites: “learning learned parameters of the generated neural network execution model”. This limitation, under the claim interpretation under 35 U.S.C. 112(f) and applicant’s specification paragraph 0181, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I). Further, the claim recites: “generating the neural network execution model comprises partitioning convolution operations implemented in the neural network execution model based on the generated neural network execution model in accordance with capacities of the first memory and the second memory”. This limitation, under the broadest reasonable interpretation, and according to applicant’s specification paragraph 0082, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I). Additionally, this limitation, under the broadest reasonable interpretation, covers the recitation of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), in this case an evaluation. See MPEP § 2106.04(a)(2)(III).
Step 2A Prong 2 Integration into a practical application: This judicial exception is not integrated into a practical application. In particular the claim recites: “acquiring hardware information regarding hardware in which the neural network execution model is operating” and “setting network information regarding the neural network”. These limitations, according to applicant’s specification paragraphs 0060 and 0063 respectively, are additional elements that amount to mere data gathering. It is necessary to acquire the data in order to use the recited judicial exception. Therefore, these limitations are insignificant extra-solution activity to the judicial exception, see MPEP §2106.05(g). Further, the claim recites: “a neural network execution model for operating a neural network”. This limitation is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §2106.05(h). Further, the claim recites: “the neural network execution model includes a first memory for storing input data, a convolution operation circuit for performing a convolution operation on the input data stored in the first memory, a second memory for storing convolution operation output data of the convolution operation circuit, and a quantization operation circuit for performing a quantization operation on the convolution operation output data stored in the second memory”. This limitation is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f).
Step 2B Significantly more: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements: “acquiring hardware information regarding hardware in which the neural network execution model is operating” and “setting network information regarding the neural network” amount to mere data gathering and are thus insignificant extra-solution activity to the judicial exception. Further, these elements are directed to receiving or transmitting data over a network which courts have recognized as well-understood, routine, and conventional when they are claimed in a generic manner, see MPEP §2106.05(d)(II). Further, the claim recites the additional element: “a neural network execution model for operating a neural network” generally links the use of the judicial exception to a particular technological environment or field of use. Elements that merely generally link the use of the judicial exception to a particular technological environment or field of use cannot provide an inventive concept. Further, the additional element “the neural network execution model includes a first memory for storing input data, a convolution operation circuit for performing a convolution operation on the input data stored in the first memory, a second memory for storing convolution operation output data of the convolution operation circuit, and a quantization operation circuit for performing a quantization operation on the convolution operation output data stored in the second memory” amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept. The claim is not patent eligible.
Regarding claim 10, the rejection of claim 9 is incorporated, and further, claim 10 is substantially similar to claim 2 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 12, the rejection of claim 9 is incorporated, and further, claim 12 is substantially similar to claim 4 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 13:
Step 1 Statutory Category: Claim 13 is directed to a machine, which falls under one of the four statutory categories.
Step 2A Prong 1 Judicial exception: Claim 13 recites, in part, “… generate the neural network execution model based on the hardware information and the network information”. This limitation, under the broadest reasonable interpretation, and in light of paragraph 0080 of applicant’s specification, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I). Further, the claim recites: “… learn learned parameters of the generated neural network execution model”. This limitation, under the claim interpretation under 35 U.S.C. 112(f) and applicant’s specification paragraph 0181, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I). Further, the claim recites: “partitioning convolution operations implemented in the neural network execution model based on the generated neural network execution model in accordance with capacities of the first memory and the second memory”. This limitation, under the broadest reasonable interpretation, and according to applicant’s specification paragraph 0082, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I). Additionally, this limitation, under the broadest reasonable interpretation, covers the recitation of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), in this case an evaluation. See MPEP § 2106.04(a)(2)(III).
Step 2A Prong 2 Integration into a practical application: This judicial exception is not integrated into a practical application. In particular the claim recites: “a non-transitory computer-readable recording medium”, “a computer”, “making the computer”, and “the neural network execution model includes a first memory for storing input data, a convolution operation circuit for performing a convolution operation on the input data stored in the first memory, a second memory for storing convolution operation output data of the convolution operation circuit, and a quantization operation circuit for performing a quantization operation on the convolution operation output data stored in the second memory”. These limitations are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f). Further, the claim recites: “… acquire hardware information regarding hardware in which the neural network execution model is operating” and “… set network information regarding the neural network”. These limitations, according to applicant’s specification paragraphs 0060 and 0063 respectively, are additional elements that amount to mere data gathering. It is necessary to acquire the data in order to use the recited judicial exception. Therefore, these limitations are insignificant extra-solution activity to the judicial exception, see MPEP §2106.05(g). Further, the claim recites: “a neural network execution model for operating a neural network”. This limitation is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §2106.05(h).
Step 2B Significantly more: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements: “a non-transitory computer-readable recording medium”, “a computer”, “making the computer”, and “the neural network execution model includes a first memory for storing input data, a convolution operation circuit for performing a convolution operation on the input data stored in the first memory, a second memory for storing convolution operation output data of the convolution operation circuit, and a quantization operation circuit for performing a quantization operation on the convolution operation output data stored in the second memory” amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept. Further, the additional elements: “… acquire hardware information regarding hardware in which the neural network execution model is operating” and “… set network information regarding the neural network” amount to mere data gathering and are thus insignificant extra-solution activity to the judicial exception. Further, these elements are directed to receiving or transmitting data over a network which courts have recognized as well-understood, routine, and conventional when they are claimed in a generic manner, see MPEP §2106.05(d)(II). Further, the claim recites the additional element: “a neural network execution model for operating a neural network” generally links the use of the judicial exception to a particular technological environment or field of use. Elements that merely generally link the use of the judicial exception to a particular technological environment or field of use cannot provide an inventive concept. The claim is not patent eligible.
Regarding claim 14, the rejection of claim 13 is incorporated, and further, claim 14 is substantially similar to claim 2 and claim 10 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 16, the rejection of claim 13 is incorporated, and further, claim 16 is substantially similar to claim 4 and claim 12 respectively, and is rejected in the same manner and reasoning applying.
Regarding claim 17, the rejection of claim 1 is incorporated, and further, the claim recites: “wherein: the first memory and the second memory are a first memory area and a second memory area in a same memory”. This limitation is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f). Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept. The claim is not patent eligible.
Regarding claim 18:
Step 1 Statutory Category: Claim 18 is directed to a device, which falls under one of the four statutory categories.
Step 2A Prong 1 Judicial exception: Claim 18 recites, in part, “…generates the neural network execution model based on hardware information regarding hardware in which the neural network execution model is operating and network information regarding the neural network”. This limitation, under the broadest reasonable interpretation, and in light of paragraph 0080 of applicant’s specification, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I). Further, the claim recites: “…generates learned parameters of the generated neural network execution model, wherein: … performs convolution operations implemented when generating the learned parameter through backpropagation with higher precision than operations implemented by the convolution operation circuit; and … performs the operations performed when calculating output data through forward propagation based on the convolution operation circuit”. This limitation, under the claim interpretation under 35 U.S.C. 112(f) and applicant’s specification paragraph 0181, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I).
Step 2A Prong 2 Integration into a practical application: This judicial exception is not integrated into a practical application. In particular the claim recites: “a neural network generating device”, “an execution model generation unit comprising first instructions executed by a processor device of the neural network generating device”, “a learning unit comprising second instructions executed by the processor device of the neural network generating device”, “the learning unit…” and “the neural network execution model comprises a convolution operation circuit that implements convolution operations and a quantization operation circuit that implements quantization operations”. These limitations are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f). Further, the claim recites: “a neural network execution model for operating a neural network”. This limitation is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §2106.05(h).
Step 2B Significantly more: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements: “a neural network generating device”, “an execution model generation unit comprising first instructions executed by a processor device of the neural network generating device”, “a learning unit comprising second instructions executed by the processor device of the neural network generating device”, “the learning unit…” and “the neural network execution model comprises a convolution operation circuit that implements convolution operations and a quantization operation circuit that implements quantization operations” amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept. Further, the additional element: “a neural network execution model for operating a neural network” generally links the use of the judicial exception to a particular technological environment or field of use. Elements that merely generally link the use of the judicial exception to a particular technological environment or field of use cannot provide an inventive concept. The claim is not patent eligible.
Regarding claim 19:
Step 1 Statutory Category: Claim 19 is directed to a device, which falls under one of the four statutory categories.
Step 2A Prong 1 Judicial exception: Claim 19 recites, in part, “…generates the neural network execution model based on hardware information regarding hardware in which the neural network execution model is operating and network information regarding the neural network”. This limitation, under the broadest reasonable interpretation, and in light of paragraph 0080 of applicant’s specification, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I). Further, the claim recites: “…generates learned parameters of the generated neural network execution model, wherein: … learns quantization parameters that the quantization operation circuit uses for the quantization operations; … learns a scaling factor for quantization operation output data quantized by the quantization parameters when learning the quantization parameters”. This limitation, under the claim interpretation under 35 U.S.C. 112(f) and applicant’s specification paragraph 0181, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I). Further, the claim recites: “performs the quantization operation using the quantization parameters incorporating the scaling factor”. This limitation, under the broadest reasonable interpretation, and in light of paragraph 0080 of applicant’s specification, covers the recitation of mathematical concepts, see MPEP §2106.04(a)(2)(I).
Step 2A Prong 2 Integration into a practical application: This judicial exception is not integrated into a practical application. In particular the claim recites: “a neural network generating device”, “an execution model generation unit comprising first instructions executed by a processor device of the neural network generating device”, “a learning unit comprising second instructions executed by the processor device of the neural network generating device”, “the learning unit…”, “the neural network execution model comprises a convolution operation circuit that implements convolution operations and a quantization operation circuit that implements quantization operations”, and “the quantization operation circuit…”. These limitations are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f). Further, the claim recites: “a neural network execution model for operating a neural network”. This limitation is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. See MPEP §2106.05(h).
Step 2B Significantly more: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements: “a neural network generating device”, “an execution model generation unit comprising first instructions executed by a processor device of the neural network generating device”, “a learning unit comprising second instructions executed by the processor device of the neural network generating device”, “the learning unit…”, “the neural network execution model comprises a convolution operation circuit that implements convolution operations and a quantization operation circuit that implements quantization operations”, and “the quantization operation circuit…” amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept. Further, the additional element: “a neural network execution model for operating a neural network” generally links the use of the judicial exception to a particular technological environment or field of use. Elements that merely generally link the use of the judicial exception to a particular technological environment or field of use cannot provide an inventive concept. The claim is not patent eligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 5, 9-10, 13-14, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Konishi et al., U.S. Patent Application Publication No. 20210056420, hereinafter referred to as “Konishi” in view of Shibata et al., U.S. Patent Application Publication No. 20210110236, hereinafter referred to as “Shibata”.
Regarding claim 1, Konishi teaches A neural network generating device that generates a neural network execution model for operating a neural network (Konishi, Paragraph 0005, Lines 1-4, “the present disclosure provides a neural network construction device that contributes to an improvement in the efficiency of obtaining an optimal neural network by narrowing candidate neural networks”), the neural network generating device comprising:
an execution model generation unit comprising first instructions executed by a processor device of the neural network generating device that (Konishi, Paragraph 0190, “Some or all of the structural elements included in each device in the above embodiments may be one system LSI (Large Scale Integration: large scale integrated circuit). The system LSI is a super multifunctional LSI manufactured by integrating a plurality of components onto a single chip. Specifically, the system LSI is a computer system configured of a microprocessor, read-only memory (ROM), random-access memory (RAM), and so on. A computer program is stored in the RAM. The system LSI achieves its function as a result of the microprocessor operating according to the computer program”), when executed, generates the neural network execution model based on hardware information regarding hardware in which the neural network execution model is operating and network information regarding the neural network (Konishi, Paragraph 0049, Lines 1-3, “the first condition may include a resource condition related to a computational resource of an embedded device”; Konishi, Paragraph 0053, Lines 1-4, “ the first condition may include information of at least one of a size of input data input to the neural network or a size of output data output from the neural network”; Konishi, Paragraph 0053, Lines 10-17, “More specifically, the size of the input data may be dimensionality of the input data, the size of the output data may be dimensionality of the output data, and the one or more candidate hyperparameters may include both a total number of layers in the neural network and a total number of nodes in the neural network. Furthermore, the first condition may further include information indicating that the neural network is a convolutional neural network”; The “resource condition” is considered to be “hardware information” and the “embedded device” is considered to be the “hardware in which the neural network execution model is operating”. Further, the “size of the input data”, “size of output data” and the indication of the type of the neural network are considered to be the “network information”; Konishi, Paragraph 0045, Lines 1-10, “A neural network construction device according to the above technique includes: an obtainer which obtains a first condition and a second condition, the first condition being used to determine a candidate hyperparameter that is a candidate of a hyperparameter of a neural network to be constructed … a generator which generates the model of the neural network using the candidate hyperparameter”; The “generator” is considered to be the “execution model generation unit”, the “model of the neural network” is considered to be the “neural network execution model”, and because the generator uses the “candidate hyperparameter” which is determined using the “first condition” which includes both the hardware and network information, thus the neural network execution model is generated “based on” the hardware and network information); and
a learning unit comprising second instructions executed by the processor device of the neural network generating device that (Konishi, Paragraph 0190, “Some or all of the structural elements included in each device in the above embodiments may be one system LSI (Large Scale Integration: large scale integrated circuit). The system LSI is a super multifunctional LSI manufactured by integrating a plurality of components onto a single chip. Specifically, the system LSI is a computer system configured of a microprocessor, read-only memory (ROM), random-access memory (RAM), and so on. A computer program is stored in the RAM. The system LSI achieves its function as a result of the microprocessor operating according to the computer program”), when executed, generates learned parameters of the generated neural network execution model (Konishi, Paragraph 0083, Lines 1-2, “Learning unit 19 performs, using the learning data, learning of the model generated by generator 13”; Konishi, Paragraph 0057, Lines 1-2, “As a result of parameters such as the weight being determined by such learning”).
Konishi does not explicitly teach wherein: the neural network execution model includes a first memory for storing input data, a convolution operation circuit for performing a convolution operation on the input data stored in the first memory, a second memory for storing convolution operation output data of the convolution operation circuit, and a quantization operation circuit for performing a quantization operation on the convolution operation output data stored in the second memory; and the execution model generation unit partitions convolution operations implemented in the neural network execution model based on the generated neural network execution model in accordance with capacities of the first memory and the second memory.
Shibata teaches wherein:
the neural network execution model includes a first memory for storing input data (Shibata, Paragraph 0090, Lines 7-10, “The primary memory includes the primary weight storage part 211, the primary input data memory part 112, and the primary operation result memory part 213”; The “primary input data memory part” is considered to be the “first memory”), a convolution operation circuit for performing a convolution operation on the input data stored in the first memory (Shibata, Paragraph 0092, “The convolutional operation part 43 executes the convolutional operation using the data stored in the primary input data storage part 212 and the weights stored in the primary weight storage part 211”), a second memory for storing convolution operation output data of the convolution operation circuit (Shibata, Paragraph 0091, Lines 4-7, “the primary operation result memory part 213 stores the result of the operation by the multiplication/addition part 204 (the result of a single convolution operation)”; The “primary operation result memory part” is considered to be the “second memory”), and a quantization operation circuit for performing a quantization operation on the convolution operation output data stored in the second memory (Shibata, Paragraph 0048, Lines 4-6, “The quantization part 101 quantizes a result of a convolutional operation in a convolutional neural network using input data and weights”; see also Shibata, Figure 1); and
the execution model generation unit partitions convolution operations implemented in the neural network execution model based on the generated neural network execution model in accordance with capacities of the first memory and the second memory (Shibata, Paragraph 0125, Lines 1-4, “the input data conversion part 402 extends one channel included in the input data of the first layer to a number of subchannels equal to the number of bits of the element that forms the input data of the first layer”; Shibata, Paragraph 0133, Lines 1-4, “The input data conversion part 402 supplies the subchannels generated as described above to the convolutional operation part 43 as logarithmic quantized input data (activation)”; Shibata, Paragraph 0134, “Since each element of the subchannel is an exponential power or truly zero, the multiplication required for the convolutional operation in the convolutional operation part 43 (or more precisely, the multiplication/addition part 204) can be realized by a shift operation. Also, in the convolutional operation, the result of the multiplication is equal to the result of the convolutional operation using the input data before the conversion and the result of the convolutional operation using the input data after the conversion, because the results of the multiplications are added-up after the multiplication of the corresponding elements of channels and weights” Performing the convolution operations using “shift operation[s]” and then adding the results, is considered to be partitioned convolution operations, further, since the method creates the subchannels according to “the number of bits of the element that forms the input data of the first layer”, it is considered to be performed based on the “generated neural network execution model”, since the “generated neural network execution model” is generated based on “computational resources” of the device, it is considered to be “in accordance with capacities of the first memory and second memory”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the execution model generation unit of Konishi to include partitioning convolution operations and including two memories for use with the convolution circuit and quantization circuit as taught by Shibata. The motivation for doing so would have been that performing the multiplication process in a single shift operation reduces the number of multiplications required for convolution and allows for a reduced hardware scale, and using multiple memories reduces the memory load of each respectively (Shibata, Paragraph 0022, “an input data to be multiplied in a convolutional operation is rounded to a power of two representation, and the multiplication process of the rounded input data and its weights is performed in a single shift operation. As a result, the number of multiplications required for convolutional operation can be reduced and the hardware scale for performing convolutional operation can be reduced”; Shibata, Paragraph 0049, Lines 1-3, “The above inferential device 100 is configured so that the multiplication process in the convolutional operation can be realized by the shift operation”).
Regarding claim 2, the rejection of claim 1 is incorporated, and further, the proposed combination teaches a hardware generation unit comprising third instructions executed by the processor device of the neural network generating device that (Konishi, Paragraph 0190, “Some or all of the structural elements included in each device in the above embodiments may be one system LSI (Large Scale Integration: large scale integrated circuit). The system LSI is a super multifunctional LSI manufactured by integrating a plurality of components onto a single chip. Specifically, the system LSI is a computer system configured of a microprocessor, read-only memory (ROM), random-access memory (RAM), and so on. A computer program is stored in the RAM. The system LSI achieves its function as a result of the microprocessor operating according to the computer program”), when executed, generates a neural network hardware model based on the hardware information and the neural network execution model (Konishi, Paragraph 0056, Lines 22-31, “the neural network construction device may further include: a learning unit; and an outputter. The obtainer may further obtain learning data on the neural network, the determination unit may output data indicating a model generated by the generator and determined as meeting the second condition, the learning unit may perform, using the learning data, learning of the model indicated in the data output by the determination unit, and the outputter may output at least a part of the model that has already been learned”; The “outputter” is considered to be the “hardware generation unit”; Konishi, Paragraph 0068, “obtain an optimal neural network by selecting the optimal neural network from among candidates narrowed by excluding neural networks that cannot meet the condition”; Konishi, Paragraph 0062, “ the outputter may output the model in a format of a source code in a language dependent on an arithmetic processing device. Furthermore, for example, the outputter may output the model in a format of a hardware description language”; Outputting the model in a “format of a source code in a language dependent on an arithmetic processing device” or “in a format of a hardware description language” is considered to be generating “based on the hardware information”; since the “optimal neural network” is selected for outputting, the generating is performed “based on the neural network execution model”).
Regarding claim 5, the rejection of claim 1 is incorporated, and further, the proposed combination teaches the neural network execution model comprises a convolution operation circuit that implements convolution operations and a quantization operation circuit that implements quantization operations (Shibata, Paragraph 0048, Lines 1-8, “An inferential device according to an exemplary embodiment includes a quantization part 101, a convolutional operation part 102, and an input data conversion part 103 (see FIG. 1). The quantization part 101 quantizes a result of a convolutional operation in a convolutional neural network using input data and weights. The convolutional operation part 102 performs convolutional operation using the quantized operation result as input data”; The “quantization part” is considered to be the “quantization operation circuit” and the “convolutional operation part” is considered to be the “quantization operation circuit”).
Regarding claim 9, Konishi teaches A neural network generating method for generating a neural network execution model for operating a neural network (Konishi, Paragraph 0005, Lines 1-4, “the present disclosure provides a neural network construction device that contributes to an improvement in the efficiency of obtaining an optimal neural network by narrowing candidate neural networks”), the neural network generating method comprising:
acquiring hardware information regarding hardware in which the neural network execution model is operating; setting network information regarding the neural network (Konishi, Paragraph 0045, Lines 1-6, “A neural network construction device according to the above technique includes: an obtainer which obtains a first condition and a second condition, the first condition being used to determine a candidate hyperparameter that is a candidate of a hyperparameter of a neural network to be constructed”; Konishi, Paragraph 0049, Lines 1-3, “the first condition may include a resource condition related to a computational resource of an embedded device”; Konishi, Paragraph 0053, Lines 1-4, “ the first condition may include information of at least one of a size of input data input to the neural network or a size of output data output from the neural network”; Konishi, Paragraph 0053, Lines 10-17, “More specifically, the size of the input data may be dimensionality of the input data, the size of the output data may be dimensionality of the output data, and the one or more candidate hyperparameters may include both a total number of layers in the neural network and a total number of nodes in the neural network. Furthermore, the first condition may further include information indicating that the neural network is a convolutional neural network”; The “resource condition” is considered to be “hardware information” and the “embedded device” is considered to be the “hardware in which the neural network execution model is operating”. Further, the “size of the input data”, “size of output data” and the indication of the type of the neural network are considered to be the “network information”);
generating the neural network execution model based on the hardware information and the network information (Konishi, Paragraph 0049, Lines 1-3, “the first condition may include a resource condition related to a computational resource of an embedded device”; Konishi, Paragraph 0053, Lines 1-4, “ the first condition may include information of at least one of a size of input data input to the neural network or a size of output data output from the neural network”; Konishi, Paragraph 0053, Lines 10-17, “More specifically, the size of the input data may be dimensionality of the input data, the size of the output data may be dimensionality of the output data, and the one or more candidate hyperparameters may include both a total number of layers in the neural network and a total number of nodes in the neural network. Furthermore, the first condition may further include information indicating that the neural network is a convolutional neural network”; The “resource condition” is considered to be “hardware information”; Further, the “size of the input data”, “size of output data” and the indication of the type of the neural network are considered to be the “network information”; Konishi, Paragraph 0045, Lines 1-10, “A neural network construction device according to the above technique includes: an obtainer which obtains a first condition and a second condition, the first condition being used to determine a candidate hyperparameter that is a candidate of a hyperparameter of a neural network to be constructed … a generator which generates the model of the neural network using the candidate hyperparameter”; The “generator” is considered to be the “execution model generation step”, the “model of the neural network” is considered to be the “neural network execution model”, and because the generator uses the “candidate hyperparameter” which is determined using the “first condition” which includes both the hardware and network information, thus the neural network execution model is generated “based on” the hardware and network information); and
learning learned parameters of the generated neural network execution model (Konishi, Paragraph 0083, Lines 1-2, “Learning unit 19 performs, using the learning data, learning of the model generated by generator 13”; Konishi, Paragraph 0057, Lines 1-2, “As a result of parameters such as the weight being determined by such learning”).
Konishi does not explicitly teach wherein: the neural network execution model includes a first memory for storing input data, a convolution operation circuit for performing a convolution operation on the input data stored in the first memory, a second memory for storing convolution operation output data of the convolution operation circuit, and a quantization operation circuit for performing a quantization operation on the convolution operation output data stored in the second memory; and generating the neural network execution model comprises partitioning convolution operations implemented in the neural network execution model based on the generated neural network execution model in accordance with capacities of the first memory and the second memory.
Shibata teaches wherein:
the neural network execution model includes a first memory for storing input data (Shibata, Paragraph 0090, Lines 7-10, “The primary memory includes the primary weight storage part 211, the primary input data memory part 112, and the primary operation result memory part 213”; The “primary input data memory part” is considered to be the “first memory”), a convolution operation circuit for performing a convolution operation on the input data stored in the first memory (Shibata, Paragraph 0092, “The convolutional operation part 43 executes the convolutional operation using the data stored in the primary input data storage part 212 and the weights stored in the primary weight storage part 211”), a second memory for storing convolution operation output data of the convolution operation circuit (Shibata, Paragraph 0091, Lines 4-7, “the primary operation result memory part 213 stores the result of the operation by the multiplication/addition part 204 (the result of a single convolution operation)”; The “primary operation result memory part” is considered to be the “second memory”), and a quantization operation circuit for performing a quantization operation on the convolution operation output data stored in the second memory (Shibata, Paragraph 0048, Lines 4-6, “The quantization part 101 quantizes a result of a convolutional operation in a convolutional neural network using input data and weights”; see also Shibata, Figure 1); and
generating the neural network execution model comprises partitioning convolution operations implemented in the neural network execution model based on the generated neural network execution model in accordance with capacities of the first memory and the second memory (Shibata, Paragraph 0125, Lines 1-4, “the input data conversion part 402 extends one channel included in the input data of the first layer to a number of subchannels equal to the number of bits of the element that forms the input data of the first layer”; Shibata, Paragraph 0133, Lines 1-4, “The input data conversion part 402 supplies the subchannels generated as described above to the convolutional operation part 43 as logarithmic quantized input data (activation)”; Shibata, Paragraph 0134, “Since each element of the subchannel is an exponential power or truly zero, the multiplication required for the convolutional operation in the convolutional operation part 43 (or more precisely, the multiplication/addition part 204) can be realized by a shift operation. Also, in the convolutional operation, the result of the multiplication is equal to the result of the convolutional operation using the input data before the conversion and the result of the convolutional operation using the input data after the conversion, because the results of the multiplications are added-up after the multiplication of the corresponding elements of channels and weights” Performing the convolution operations using “shift operation[s]” and then adding the results, is considered to be partitioned convolution operations, further, since the method creates the subchannels according to “the number of bits of the element that forms the input data of the first layer”, it is considered to be performed based on the “generated neural network execution model”, since the “generated neural network execution model” is generated based on “computational resources” of the device, it is considered to be “in accordance with capacities of the first memory and second memory”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the execution model generation unit of Konishi to include partitioning convolution operations and including two memories for use with the convolution circuit and quantization circuit as taught by Shibata. The motivation for doing so would have been that performing the multiplication process in a single shift operation reduces the number of multiplications required for convolution and allows for a reduced hardware scale, and using multiple memories reduces the memory load of each respectively (Shibata, Paragraph 0022, “an input data to be multiplied in a convolutional operation is rounded to a power of two representation, and the multiplication process of the rounded input data and its weights is performed in a single shift operation. As a result, the number of multiplications required for convolutional operation can be reduced and the hardware scale for performing convolutional operation can be reduced”; Shibata, Paragraph 0049, Lines 1-3, “The above inferential device 100 is configured so that the multiplication process in the convolutional operation can be realized by the shift operation”).
Regarding claim 10, the rejection of claim 9 is incorporated, and further, the proposed combination teaches generating a neural network hardware model based on the hardware information and the neural network execution model (Konishi, Paragraph 0056, Lines 22-31, “the neural network construction device may further include: a learning unit; and an outputter. The obtainer may further obtain learning data on the neural network, the determination unit may output data indicating a model generated by the generator and determined as meeting the second condition, the learning unit may perform, using the learning data, learning of the model indicated in the data output by the determination unit, and the outputter may output at least a part of the model that has already been learned”; The “outputter” is considered to be the “output step”; Konishi, Paragraph 0068, “obtain an optimal neural network by selecting the optimal neural network from among candidates narrowed by excluding neural networks that cannot meet the condition”; Konishi, Paragraph 0062, “ the outputter may output the model in a format of a source code in a language dependent on an arithmetic processing device. Furthermore, for example, the outputter may output the model in a format of a hardware description language”; Outputting the model in a “format of a source code in a language dependent on an arithmetic processing device” or “in a format of a hardware description language” is considered to be generating “based on the hardware information”; since the “optimal neural network” is selected for outputting, the generating is performed “based on the neural network execution model”).
Regarding claim 13, Konishi teaches A non-transitory computer-readable recording medium storing a neural network generating program for making a computer generate a neural network execution model for operating a neural network (Konishi, Paragraph 0008, Lines 1-8, “a recording medium according to one aspect of the present disclosure is a non-transitory computer-readable recording medium having recorded thereon a program to be executed by an arithmetic processing device included in a neural network construction device including the arithmetic processing device and a storage device. The program is executed by the arithmetic processing device to cause the neural network construction device to execute”; Konishi, Paragraph 0005, Lines 1-4, “the present disclosure provides a neural network construction device that contributes to an improvement in the efficiency of obtaining an optimal neural network by narrowing candidate neural networks”), the neural network generating program comprising:
making the computer acquire hardware information regarding hardware in which the neural network execution model is operating; making the computer set network information regarding the neural network (Konishi, Paragraph 0045, Lines 1-6, “A neural network construction device according to the above technique includes: an obtainer which obtains a first condition and a second condition, the first condition being used to determine a candidate hyperparameter that is a candidate of a hyperparameter of a neural network to be constructed”; Konishi, Paragraph 0049, Lines 1-3, “the first condition may include a resource condition related to a computational resource of an embedded device”; Konishi, Paragraph 0053, Lines 1-4, “ the first condition may include information of at least one of a size of input data input to the neural network or a size of output data output from the neural network”; Konishi, Paragraph 0053, Lines 10-17, “More specifically, the size of the input data may be dimensionality of the input data, the size of the output data may be dimensionality of the output data, and the one or more candidate hyperparameters may include both a total number of layers in the neural network and a total number of nodes in the neural network. Furthermore, the first condition may further include information indicating that the neural network is a convolutional neural network”; The “resource condition” is considered to be “hardware information” and the “embedded device” is considered to be the “hardware in which the neural network execution model is operating”. Further, the “size of the input data”, “size of output data” and the indication of the type of the neural network are considered to be the “network information”);
making the computer generate the neural network execution model based on the hardware information and the network information (Konishi, Paragraph 0049, Lines 1-3, “the first condition may include a resource condition related to a computational resource of an embedded device”; Konishi, Paragraph 0053, Lines 1-4, “ the first condition may include information of at least one of a size of input data input to the neural network or a size of output data output from the neural network”; Konishi, Paragraph 0053, Lines 10-17, “More specifically, the size of the input data may be dimensionality of the input data, the size of the output data may be dimensionality of the output data, and the one or more candidate hyperparameters may include both a total number of layers in the neural network and a total number of nodes in the neural network. Furthermore, the first condition may further include information indicating that the neural network is a convolutional neural network”; The “resource condition” is considered to be “hardware information”; Further, the “size of the input data”, “size of output data” and the indication of the type of the neural network are considered to be the “network information”; Konishi, Paragraph 0045, Lines 1-10, “A neural network construction device according to the above technique includes: an obtainer which obtains a first condition and a second condition, the first condition being used to determine a candidate hyperparameter that is a candidate of a hyperparameter of a neural network to be constructed … a generator which generates the model of the neural network using the candidate hyperparameter”; The “generator” is considered to be the “execution model generation step”, the “model of the neural network” is considered to be the “neural network execution model”, and because the generator uses the “candidate hyperparameter” which is determined using the “first condition” which includes both the hardware and network information, thus the neural network execution model is generated “based on” the hardware and network information); and
making the computer learn learned parameters of the generated neural network execution model (Konishi, Paragraph 0083, Lines 1-2, “Learning unit 19 performs, using the learning data, learning of the model generated by generator 13”; Konishi, Paragraph 0057, Lines 1-2, “As a result of parameters such as the weight being determined by such learning”).
Konishi does not explicitly teach wherein: the neural network execution model includes a first memory for storing input data, a convolution operation circuit for performing a convolution operation on the input data stored in the first memory, a second memory for storing convolution operation output data of the convolution operation circuit, and a quantization operation circuit for performing a quantization operation on the convolution operation output data stored in the second memory; and making the computer generate the neural network execution model comprises partitioning convolution operations implemented in the neural network execution model based on the generated neural network execution model in accordance with capacities of the first memory and the second memory.
Shibata teaches wherein:
the neural network execution model includes a first memory for storing input data (Shibata, Paragraph 0090, Lines 7-10, “The primary memory includes the primary weight storage part 211, the primary input data memory part 112, and the primary operation result memory part 213”; The “primary input data memory part” is considered to be the “first memory”), a convolution operation circuit for performing a convolution operation on the input data stored in the first memory (Shibata, Paragraph 0092, “The convolutional operation part 43 executes the convolutional operation using the data stored in the primary input data storage part 212 and the weights stored in the primary weight storage part 211”), a second memory for storing convolution operation output data of the convolution operation circuit (Shibata, Paragraph 0091, Lines 4-7, “the primary operation result memory part 213 stores the result of the operation by the multiplication/addition part 204 (the result of a single convolution operation)”; The “primary operation result memory part” is considered to be the “second memory”), and a quantization operation circuit for performing a quantization operation on the convolution operation output data stored in the second memory (Shibata, Paragraph 0048, Lines 4-6, “The quantization part 101 quantizes a result of a convolutional operation in a convolutional neural network using input data and weights”; see also Shibata, Figure 1); and
making the computer generate the neural network execution model comprises partitioning convolution operations implemented in the neural network execution model based on the generated neural network execution model in accordance with capacities of the first memory and the second memory (Shibata, Paragraph 0125, Lines 1-4, “the input data conversion part 402 extends one channel included in the input data of the first layer to a number of subchannels equal to the number of bits of the element that forms the input data of the first layer”; Shibata, Paragraph 0133, Lines 1-4, “The input data conversion part 402 supplies the subchannels generated as described above to the convolutional operation part 43 as logarithmic quantized input data (activation)”; Shibata, Paragraph 0134, “Since each element of the subchannel is an exponential power or truly zero, the multiplication required for the convolutional operation in the convolutional operation part 43 (or more precisely, the multiplication/addition part 204) can be realized by a shift operation. Also, in the convolutional operation, the result of the multiplication is equal to the result of the convolutional operation using the input data before the conversion and the result of the convolutional operation using the input data after the conversion, because the results of the multiplications are added-up after the multiplication of the corresponding elements of channels and weights” Performing the convolution operations using “shift operation[s]” and then adding the results, is considered to be partitioned convolution operations, further, since the method creates the subchannels according to “the number of bits of the element that forms the input data of the first layer”, it is considered to be performed based on the “generated neural network execution model”, since the “generated neural network execution model” is generated based on “computational resources” of the device, it is considered to be “in accordance with capacities of the first memory and second memory”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the execution model generation unit of Konishi to include partitioning convolution operations and including two memories for use with the convolution circuit and quantization circuit as taught by Shibata. The motivation for doing so would have been that performing the multiplication process in a single shift operation reduces the number of multiplications required for convolution and allows for a reduced hardware scale, and using multiple memories reduces the memory load of each respectively (Shibata, Paragraph 0022, “an input data to be multiplied in a convolutional operation is rounded to a power of two representation, and the multiplication process of the rounded input data and its weights is performed in a single shift operation. As a result, the number of multiplications required for convolutional operation can be reduced and the hardware scale for performing convolutional operation can be reduced”; Shibata, Paragraph 0049, Lines 1-3, “The above inferential device 100 is configured so that the multiplication process in the convolutional operation can be realized by the shift operation”).
Regarding claim 14, the rejection of claim 13 is incorporated, and further, the proposed combination teaches making the computer generate a neural network hardware model based on the hardware information and the neural network execution model (Konishi, Paragraph 0056, Lines 22-31, “the neural network construction device may further include: a learning unit; and an outputter. The obtainer may further obtain learning data on the neural network, the determination unit may output data indicating a model generated by the generator and determined as meeting the second condition, the learning unit may perform, using the learning data, learning of the model indicated in the data output by the determination unit, and the outputter may output at least a part of the model that has already been learned”; The “outputter” is considered to be the “output step”; Konishi, Paragraph 0068, “obtain an optimal neural network by selecting the optimal neural network from among candidates narrowed by excluding neural networks that cannot meet the condition”; Konishi, Paragraph 0062, “ the outputter may output the model in a format of a source code in a language dependent on an arithmetic processing device. Furthermore, for example, the outputter may output the model in a format of a hardware description language”; Outputting the model in a “format of a source code in a language dependent on an arithmetic processing device” or “in a format of a hardware description language” is considered to be generating “based on the hardware information”; since the “optimal neural network” is selected for outputting, the generating is performed “based on the neural network execution model”).
Regarding claim 17, the rejection of claim 1 is incorporated, and further, the proposed combination teaches wherein: the first memory and the second memory are a first memory area and a second memory area m a same memory (Shibata, Paragraph 0090, Lines 7-10, “The primary memory includes the primary weight storage part 211, the primary input data memory part 112, and the primary operation result memory part 213”).
Claims 4, 6, 12, 16, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Konishi in view of Shibata in further view of Nagel et al., Up or Down? Adaptive Rounding for Post-Training Quantization, 04/22/2020, https://arxiv.org/pdf/2004.10568v1, hereinafter referred to as Nagel.
Regarding claim 4, the rejection of claim 1 is incorporated.
The proposed combination does not explicitly teach the learning unit performs associated operations implemented when generating the learned parameters with higher precision than operations implemented by the neural network execution model.
Nagel teaches the learning unit performs associated operations implemented when generating the learned parameters with higher precision than operations implemented by the neural network execution model (Nagel, Page 1, Section 1, Paragraph 4, “post-training quantization methods (Nagel et al., 2019; Cai et al., 2020; Choukroun et al., 2019; Banner et al., 2019), which can be more easily applied in practice. These types of methods allow for network quantization to happen on-the-fly when deploying models, without the user of the model spending time and energy on quantization” Because the quantization is not performed until after training, the operations performed by the learning unit would be at full precision, and afterward the quantization would occur, resulting in the operations performed by the neural network execution model having a lower precision).
It would have been obvious, to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the neural network generating device taught by the proposed combination to include post-training quantization as taught by Nagel. The motivation for doing so would have been that post-training quantization does not require re-training models and is more easily applied in practice (Nagel, Page 1, Section 1, Paragraph 3, Lines 3-7, “Although many methods exist that do quantization-aware training (Jacob et al., 2018; Louizos et al., 2019) and get excellent results, these methods require a user to spend significant time on re-training models and hyperparameter tuning”; Nagel et al., 2019; Cai et al., 2020; Choukroun et al., 2019; Banner et al., 2019), which can be more easily applied in practice. These types of methods allow for network quantization to happen on-the-fly when deploying models, without the user of the model spending time and energy on quantization”).
Regarding claim 6, the rejection of claim 5 is incorporated.
The proposed combination thus far does not explicitly teach the learning unit performs convolution operations implemented when generating the learned parameter with higher precision than operations implemented by the convolution operation circuit.
Nagel teaches the learning unit performs convolution operations implemented when generating the learned parameter with higher precision than operations implemented by the convolution operation circuit (Nagel, Page 1, Section 1, Paragraph 4, “post-training quantization methods (Nagel et al., 2019; Cai et al., 2020; Choukroun et al., 2019; Banner et al., 2019), which can be more easily applied in practice. These types of methods allow for network quantization to happen on-the-fly when deploying models, without the user of the model spending time and energy on quantization” Because the quantization is not performed until after training, the operations performed by the learning unit would be at full precision, and because the learning unit performs backpropagation of errors, convolutional operations are performed during training, and afterward the quantization would occur, resulting in the operations performed by the neural network execution model, which includes the convolution operation circuit, having a lower precision).
It would have been obvious, to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the neural network generating device taught by the proposed combination to include post-training quantization as taught by Nagel. The motivation for doing so would have been that post-training quantization does not require re-training models and is more easily applied in practice (Nagel, Page 1, Section 1, Paragraph 3, Lines 3-7, “Although many methods exist that do quantization-aware training (Jacob et al., 2018; Louizos et al., 2019) and get excellent results, these methods require a user to spend significant time on re-training models and hyperparameter tuning”; Nagel, Page 1, Section 1, Paragraph 4, “post-training quantization methods Nagel et al., 2019; Cai et al., 2020; Choukroun et al., 2019; Banner et al., 2019), which can be more easily applied in practice. These types of methods allow for network quantization to happen on-the-fly when deploying models, without the user of the model spending time and energy on quantization”).
Regarding claim 12, the rejection of claim 9 is incorporated.
The proposed combination does not explicitly teach learning learned parameters of the generated neural network execution model comprises performing associated operations implemented when generating the learned parameters with higher precision than operations implemented by the neural network execution model.
Nagel teaches learning learned parameters of the generated neural network execution model comprises performing associated operations implemented when generating the learned parameters with higher precision than operations implemented by the neural network execution model (Nagel, Page 1, Section 1, Paragraph 4, “post-training quantization methods (Nagel et al., 2019; Cai et al., 2020; Choukroun et al., 2019; Banner et al., 2019), which can be more easily applied in practice. These types of methods allow for network quantization to happen on-the-fly when deploying models, without the user of the model spending time and energy on quantization” Because the quantization is not performed until after training, the operations performed by the learning step would be at full precision, and afterward the quantization would occur, resulting in the operations performed by the neural network execution model having a lower precision).
It would have been obvious, to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the neural network generating method taught by the proposed combination to include post-training quantization as taught by Nagel. The motivation for doing so would have been that post-training quantization does not require re-training models and is more easily applied in practice (Nagel, Page 1, Section 1, Paragraph 3, Lines 3-7, “Although many methods exist that do quantization-aware training (Jacob et al., 2018; Louizos et al., 2019) and get excellent results, these methods require a user to spend significant time on re-training models and hyperparameter tuning”; Nagel et al., 2019; Cai et al., 2020; Choukroun et al., 2019; Banner et al., 2019), which can be more easily applied in practice. These types of methods allow for network quantization to happen on-the-fly when deploying models, without the user of the model spending time and energy on quantization”).
Regarding claim 16, the rejection of claim 13 is incorporated.
The proposed combination does not explicitly teach making the computer learn learned parameters of the generated neural network execution model comprises performing associated operations implemented when generating the learned parameters with higher precision than operations implemented by the neural network execution model.
Nagel teaches making the computer learn learned parameters of the generated neural network execution model comprises performing associated operations implemented when generating the learned parameters with higher precision than operations implemented by the neural network execution model (Nagel, Page 1, Section 1, Paragraph 4, “post-training quantization methods (Nagel et al., 2019; Cai et al., 2020; Choukroun et al., 2019; Banner et al., 2019), which can be more easily applied in practice. These types of methods allow for network quantization to happen on-the-fly when deploying models, without the user of the model spending time and energy on quantization” Because the quantization is not performed until after training, the operations performed by the learning step would be at full precision, and afterward the quantization would occur, resulting in the operations performed by the neural network execution model having a lower precision).
It would have been obvious, to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the neural network generating method taught by the proposed combination to include post-training quantization as taught by Nagel. The motivation for doing so would have been that post-training quantization does not require re-training models and is more easily applied in practice (Nagel, Page 1, Section 1, Paragraph 3, Lines 3-7, “Although many methods exist that do quantization-aware training (Jacob et al., 2018; Louizos et al., 2019) and get excellent results, these methods require a user to spend significant time on re-training models and hyperparameter tuning”; Nagel et al., 2019; Cai et al., 2020; Choukroun et al., 2019; Banner et al., 2019), which can be more easily applied in practice. These types of methods allow for network quantization to happen on-the-fly when deploying models, without the user of the model spending time and energy on quantization”).
Regarding claim 18, Konishi teaches A neural network generating device that generates a neural network execution model for operating a neural network (Konishi, Paragraph 0005, Lines 1-4, “the present disclosure provides a neural network construction device that contributes to an improvement in the efficiency of obtaining an optimal neural network by narrowing candidate neural networks”), the neural network generating device comprising:
an execution model generation unit comprising first instructions executed by a processor device of the neural network generating device that (Konishi, Paragraph 0190, “Some or all of the structural elements included in each device in the above embodiments may be one system LSI (Large Scale Integration: large scale integrated circuit). The system LSI is a super multifunctional LSI manufactured by integrating a plurality of components onto a single chip. Specifically, the system LSI is a computer system configured of a microprocessor, read-only memory (ROM), random-access memory (RAM), and so on. A computer program is stored in the RAM. The system LSI achieves its function as a result of the microprocessor operating according to the computer program”), when executed, generates the neural network execution model based on hardware information regarding hardware in which the neural network execution model is operating and network information regarding the neural network (Konishi, Paragraph 0049, Lines 1-3, “the first condition may include a resource condition related to a computational resource of an embedded device”; Konishi, Paragraph 0053, Lines 1-4, “ the first condition may include information of at least one of a size of input data input to the neural network or a size of output data output from the neural network”; Konishi, Paragraph 0053, Lines 10-17, “More specifically, the size of the input data may be dimensionality of the input data, the size of the output data may be dimensionality of the output data, and the one or more candidate hyperparameters may include both a total number of layers in the neural network and a total number of nodes in the neural network. Furthermore, the first condition may further include information indicating that the neural network is a convolutional neural network”; The “resource condition” is considered to be “hardware information” and the “embedded device” is considered to be the “hardware in which the neural network execution model is operating”. Further, the “size of the input data”, “size of output data” and the indication of the type of the neural network are considered to be the “network information”; Konishi, Paragraph 0045, Lines 1-10, “A neural network construction device according to the above technique includes: an obtainer which obtains a first condition and a second condition, the first condition being used to determine a candidate hyperparameter that is a candidate of a hyperparameter of a neural network to be constructed … a generator which generates the model of the neural network using the candidate hyperparameter”; The “generator” is considered to be the “execution model generation unit”, the “model of the neural network” is considered to be the “neural network execution model”, and because the generator uses the “candidate hyperparameter” which is determined using the “first condition” which includes both the hardware and network information, thus the neural network execution model is generated “based on” the hardware and network information); and
a learning unit comprising second instructions executed by the processor device of the neural network generating device that (Konishi, Paragraph 0190, “Some or all of the structural elements included in each device in the above embodiments may be one system LSI (Large Scale Integration: large scale integrated circuit). The system LSI is a super multifunctional LSI manufactured by integrating a plurality of components onto a single chip. Specifically, the system LSI is a computer system configured of a microprocessor, read-only memory (ROM), random-access memory (RAM), and so on. A computer program is stored in the RAM. The system LSI achieves its function as a result of the microprocessor operating according to the computer program”), when executed, generates learned parameters of the generated neural network execution model (Konishi, Paragraph 0083, Lines 1-2, “Learning unit 19 performs, using the learning data, learning of the model generated by generator 13”; Konishi, Paragraph 0057, Lines 1-2, “As a result of parameters such as the weight being determined by such learning”).
Konishi does not explicitly teach wherein: the neural network execution model comprises a convolution operation circuit that implements convolution operations and a quantization operation circuit that implements quantization operations; the learning unit performs convolution operations implemented when generating the learned parameter through backpropagation with higher precision than operations implemented by the convolution operation circuit; and the learning unit performs the operations performed when calculating output data through forward propagation based on the convolution operation circuit.
Shibata teaches wherein: the neural network execution model comprises a convolution operation circuit that implements convolution operations and a quantization operation circuit that implements quantization operations (Shibata, Paragraph 0048, Lines 1-8, “An inferential device according to an exemplary embodiment includes a quantization part 101, a convolutional operation part 102, and an input data conversion part 103 (see FIG. 1). The quantization part 101 quantizes a result of a convolutional operation in a convolutional neural network using input data and weights. The convolutional operation part 102 performs convolutional operation using the quantized operation result as input data”; The “quantization part” is considered to be the “quantization operation circuit” and the “convolutional operation part” is considered to be the “convolution operation circuit”);
the learning unit performs convolution operations implemented when generating the learned parameter through backpropagation… and the learning unit performs the operations performed when calculating output data through forward propagation based on the convolution operation circuit (Shibata, Paragraph 0189, Lines 11-17, “the learning device can be configured by adding an error back propagation part that performs error back propagation, a weight update part that performs weight update, and the like to the inferential device 10 described above. The error back propagation part and the weight update part can be realized by a known algorithm, etc., and the explanation is omitted”; A person of ordinary skill in the art would recognize that in order to use backpropagation to generate learned parameters, forward propagation is required to perform calculating output).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the neural network execution model of Konishi to include a convolution operation circuit and a quantization operation circuit and using backpropagation for learning as taught by Shibata. The motivation for doing so would have been that performing quantization allows for convolution operations to be performed using a single shift operation, which reduces the number of multiplications required for convolution operations and reduces the hardware scale required, and backpropagation is easy to implement as it is a well-known and well tested method (Shibata, Paragraph 0022, "an input data to be multiplied in a convolutional operation is rounded to a power of two representation, and the multiplication process of the rounded input data and its weights is performed in a single shift operation. As a result, the number of multiplications required for convolutional operation can be reduced and the hardware scale for performing convolutional operation can be reduced"; Shibata, Paragraph 0049, Lines 1-3, "The above inferential device 100 is configured so that the multiplication process in the convolutional operation can be realized by the shift operation"; Shibata, Paragraph 0189).
The proposed combination does not explicitly teach the learning unit performs convolution operations implemented when generating the learned parameter … with higher precision than operations implemented by the convolution operation circuit.
Nagel teaches the learning unit performs convolution operations implemented when generating the learned parameter … with higher precision than operations implemented by the convolution operation circuit (Nagel, Page 1, Section 1, Paragraph 4, “post-training quantization methods (Nagel et al., 2019; Cai et al., 2020; Choukroun et al., 2019; Banner et al., 2019), which can be more easily applied in practice. These types of methods allow for network quantization to happen on-the-fly when deploying models, without the user of the model spending time and energy on quantization” Because the quantization is not performed until after training, the operations performed by the learning unit would be at full precision, and afterward the quantization would occur, resulting in the operations performed by the neural network execution model having a lower precision).
It would have been obvious, to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the neural network generating device taught by the proposed combination to include post-training quantization as taught by Nagel. The motivation for doing so would have been that post-training quantization does not require re-training models and is more easily applied in practice (Nagel, Page 1, Section 1, Paragraph 3, Lines 3-7, “Although many methods exist that do quantization-aware training (Jacob et al., 2018; Louizos et al., 2019) and get excellent results, these methods require a user to spend significant time on re-training models and hyperparameter tuning”; Nagel et al., 2019; Cai et al., 2020; Choukroun et al., 2019; Banner et al., 2019), which can be more easily applied in practice. These types of methods allow for network quantization to happen on-the-fly when deploying models, without the user of the model spending time and energy on quantization”).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Konishi in view of Shibata, in further view of Yoshiyama et al., U.S. Patent Application Publication No. 20210110260, hereinafter referred to as “Yoshiyama”.
Regarding claim 7, the rejection of claim 5 is incorporated.
The proposed combination thus far does not explicitly teach the learning unit learns quantization parameters that the quantization operation circuit uses for the quantization operations.
Yoshiyama teaches the learning unit learns quantization parameters that the quantization operation circuit uses for the quantization operations (Yoshiyama, Paragraph 55, Lines 7-12, “a learning unit 110 that optimizes parameters that determine a dynamic range by an error back propagation and a stochastic gradient descent in a quantization function of a neural network in which the parameters that determine the dynamic range are arguments”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the learning unit of the proposed combination to include learning quantization parameters as taught by Yoshiyama. The motivation to do so would have been that learning the quantization parameters using error backpropagation using the learning unit reduces processing load and realizes learning with higher accuracy (Yoshiyama, Paragraph 0008, “As described above, according to the present disclosure, it is possible to reduce processing load of operation and to realize learning with higher accuracy”).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Konishi in view of Shibata, in further view of Yoshiyama in further view of Choi et al., PACT: PARAMETERIZED CLIPPING ACTIVATION FOR QUANTIZED NEURAL NETWORKS, 07/17/2018, https://arxiv.org/pdf/1805.06085, hereinafter referred to as “Choi”.
Regarding claim 8, the rejection of claim 7 is incorporated.
The proposed combination does not explicitly teach the learning unit learns a scaling factor for quantization operation output data quantized by the quantization parameters when learning the quantization parameters.
Choi teaches the learning unit learns a scaling factor for quantization operation output data quantized by the quantization parameters when learning the quantization parameters (Choi, Page 1, Abstract, Lines 7-9, “PArameterized Clipping acTivation (PACT), uses an activation clipping parameter α that is optimized during training to find the right quantization scale” The “clipping parameter α” is considered to be the “scaling factor; Choi, Page 5, Section 4.2, Paragraph 2, “the best scope for α was to share α per layer. This choice also reduces hardware complexity because α needs to be multiplied only once after all multiply-accumulate (MAC) operations in reduced-precision in a layer are completed”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the learning unit as taught by the proposed combination to include learning a scaling factor as taught by Choi. The motivation for doing so would have been that the scaling factor enables neural networks to maintain accuracy while working with ultra-low precision weights and activations (Choi, Abstract, Lines 4-7, “This paper proposes a novel quantization scheme for activations during training - that enables neural networks to work well with ultra low precision weights and activations without any significant accuracy degradation”).
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Konishi in view of Shibata, in further view of Esser et al., LEARNED STEP SIZE QUANTIZATION, 05/07/2020, https://arxiv.org/pdf/1902.08153, hereinafter referred to as “Esser”.
Regarding claim 20, Konishi teaches A neural network generating device that generates a neural network execution model for operating a neural network (Konishi, Paragraph 0005, Lines 1-4, “the present disclosure provides a neural network construction device that contributes to an improvement in the efficiency of obtaining an optimal neural network by narrowing candidate neural networks”), the neural network generating device comprising:
an execution model generation unit comprising first instructions executed by a processor device of the neural network generating device that (Konishi, Paragraph 0190, “Some or all of the structural elements included in each device in the above embodiments may be one system LSI (Large Scale Integration: large scale integrated circuit). The system LSI is a super multifunctional LSI manufactured by integrating a plurality of components onto a single chip. Specifically, the system LSI is a computer system configured of a microprocessor, read-only memory (ROM), random-access memory (RAM), and so on. A computer program is stored in the RAM. The system LSI achieves its function as a result of the microprocessor operating according to the computer program”), when executed, generates the neural network execution model based on hardware information regarding hardware in which the neural network execution model is operating and network information regarding the neural network (Konishi, Paragraph 0049, Lines 1-3, “the first condition may include a resource condition related to a computational resource of an embedded device”; Konishi, Paragraph 0053, Lines 1-4, “ the first condition may include information of at least one of a size of input data input to the neural network or a size of output data output from the neural network”; Konishi, Paragraph 0053, Lines 10-17, “More specifically, the size of the input data may be dimensionality of the input data, the size of the output data may be dimensionality of the output data, and the one or more candidate hyperparameters may include both a total number of layers in the neural network and a total number of nodes in the neural network. Furthermore, the first condition may further include information indicating that the neural network is a convolutional neural network”; The “resource condition” is considered to be “hardware information” and the “embedded device” is considered to be the “hardware in which the neural network execution model is operating”. Further, the “size of the input data”, “size of output data” and the indication of the type of the neural network are considered to be the “network information”; Konishi, Paragraph 0045, Lines 1-10, “A neural network construction device according to the above technique includes: an obtainer which obtains a first condition and a second condition, the first condition being used to determine a candidate hyperparameter that is a candidate of a hyperparameter of a neural network to be constructed … a generator which generates the model of the neural network using the candidate hyperparameter”; The “generator” is considered to be the “execution model generation unit”, the “model of the neural network” is considered to be the “neural network execution model”, and because the generator uses the “candidate hyperparameter” which is determined using the “first condition” which includes both the hardware and network information, thus the neural network execution model is generated “based on” the hardware and network information); and
a learning unit comprising second instructions executed by the processor device of the neural network generating device that (Konishi, Paragraph 0190, “Some or all of the structural elements included in each device in the above embodiments may be one system LSI (Large Scale Integration: large scale integrated circuit). The system LSI is a super multifunctional LSI manufactured by integrating a plurality of components onto a single chip. Specifically, the system LSI is a computer system configured of a microprocessor, read-only memory (ROM), random-access memory (RAM), and so on. A computer program is stored in the RAM. The system LSI achieves its function as a result of the microprocessor operating according to the computer program”), when executed, generates learned parameters of the generated neural network execution model (Konishi, Paragraph 0083, Lines 1-2, “Learning unit 19 performs, using the learning data, learning of the model generated by generator 13”; Konishi, Paragraph 0057, Lines 1-2, “As a result of parameters such as the weight being determined by such learning”).
Konishi does not explicitly teach wherein: the neural network execution model comprises a convolution operation circuit that implements convolution operations and a quantization operation circuit that implements quantization operations; the learning unit learns quantization parameters that the quantization operation circuit uses for the quantization operations; the learning unit learns a scaling factor for quantization operation output data quantized by the quantization parameters when learning the quantization parameters; and the quantization operation circuit performs the quantization operation using the quantization parameters incorporating the scaling factor.
Shibata teaches the neural network execution model comprises a convolution operation circuit that implements convolution operations and a quantization operation circuit that implements quantization operations (Shibata, Paragraph 0048, Lines 1-8, “An inferential device according to an exemplary embodiment includes a quantization part 101, a convolutional operation part 102, and an input data conversion part 103 (see FIG. 1). The quantization part 101 quantizes a result of a convolutional operation in a convolutional neural network using input data and weights. The convolutional operation part 102 performs convolutional operation using the quantized operation result as input data”; The “quantization part” is considered to be the “quantization operation circuit” and the “convolutional operation part” is considered to be the “convolution operation circuit”);
the quantization operation circuit performs the quantization operation … (Shibata, Paragraph 0048, Lines 4-6, “The quantization part 101 quantizes a result of a convolutional operation in a convolutional neural network using input data and weights”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the neural network execution model of Konishi to include a convolution operation circuit and a quantization operation circuit as taught by Shibata. The motivation for doing so would have been that performing quantization allows for convolution operations to be performed using a single shift operation, which reduces the number of multiplications required for convolution operations and reduces the hardware scale required (Shibata, Paragraph 0022, "an input data to be multiplied in a convolutional operation is rounded to a power of two representation, and the multiplication process of the rounded input data and its weights is performed in a single shift operation. As a result, the number of multiplications required for convolutional operation can be reduced and the hardware scale for performing convolutional operation can be reduced"; Shibata, Paragraph 0049, Lines 1-3, "The above inferential device 100 is configured so that the multiplication process in the convolutional operation can be realized by the shift operation").
The proposed combination does not explicitly teach the learning unit learns quantization parameters that the quantization operation circuit uses for the quantization operations; the learning unit learns a scaling factor for quantization operation output data quantized by the quantization parameters when learning the quantization parameters; nor quantization being performed using the quantization parameters incorporating the scaling factor.
Esser teaches learns quantization parameters that the quantization operation circuit uses for the quantization operations; the learning unit learns a scaling factor for quantization operation output data quantized by the quantization parameters when learning the quantization parameters (Esser, Page 3, Section 2, Lines 3-6, “Given data to quantize v, quantizer step size s, the number of positive and negative quantization levels
Q
P
and
Q
N
, respectively, we define a quantizer that computes
v
-
, a quantized and integer scaled representation of the data, and
v
^
, a quantized representation of the data at the same scale as v”; Esser, Page 4, Section 2.3, Lines 1-3, “Model quantizers are trained with LSQ by making their step sizes learnable parameters with loss gradient computed using the quantizer gradient described above, while other model parameters can be trained using existing techniques”; Esser, Page 3, Section 2.1, Lines 1-2 and Equation 3, “LSQ provides a means to learn s based on the training loss by introducing the following gradient through the quantizer to the step size parameter:
∂
v
^
∂
s
=
-
v
s
+
[
v
s
]
i
f
-
Q
N
<
v
s
<
Q
P
-
Q
N
i
f
v
s
≤
-
Q
N
Q
P
i
f
v
s
≥
Q
P
(3)”; “s” is considered to be the “scaling factor”) and
quantization being performed using the quantization parameters incorporating the scaling factor (Esser, Page 5, Section 3.2, Lines 1-2, “We trained several networks using LSQ and compare accuracy with other quantized networks and full precision baselines (Table 1)”; see also Esser, Table 1; In order to determine the collection of results in Table 1, quantization must have been performed according to “LSQ” meaning “using the quantization parameters incorporating the scaling factor”).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the learning unit of the proposed combination to include learning quantization parameters including a scaling parameter as taught by Esser and to modify the quantization circuit of the proposed combination to include performing quantization using the parameters and scaling factor as taught by Esser. The motivation to do so would have been that the method of Esser achieves the highest accuracy to date with a variety of architectures and requires only a simple modification of existing training code (Esser, Page 1, Abstract, Lines 4-7, “Learned Step Size Quantization, that achieves the highest accuracy to date on the ImageNet dataset when using models, from a variety of architectures, with weights and activations quantized to 2-, 3- or 4-bits of precision, and that can train 3-bit models that reach full precision baseline accuracy; Esser, Page 1, Abstract, Lines 12-14, “This approach works using different levels of precision as needed for a given system and requires only a simple modification of existing training code”).
Response to Arguments
Applicant’s amendments to Figure 3 with respect to the objections to the drawings have been fully considered, and overcome the objections set forth in the non-final office action dated 10/01/2025, consequently the objections have been withdrawn.
Applicant’s amendments to the claims with respect to the claims’ interpretation under 35 U.S.C. 112(f) and the corresponding 35 U.S.C. 112(b) rejections have been fully considered. The amendments to claims 9-10, 12-14, and 16 overcome the 35 U.S.C. 112(b) rejections set forth in the non-final office action dated 10/01/2025. As a result of the amendments to the remainder of the claims, it has become unclear if the respective limitations are to be interpreted under 35 U.S.C. 112(f), and therefore the proper 35 U.S.C. 112(b) rejection has been applied. For a more in depth explanation, see the updated 35 U.S.C. 112(b) rejections above.
Applicant’s arguments regarding the 35 U.S.C. 101 rejections of the claims have been fully considered but are unpersuasive.
Applicant first argues, on page 12 of the response, that “independent claim 1 is not directed to a mathematical concept”, pointing specifically to the limitations of “generates the neural network execution model based on hardware information regarding hardware in which the neural network execution model is operating and network information regarding the neural network” and “generates learned parameters of the generated neural network execution model”. Examiner respectfully disagrees. According to applicant’s specification paragraphs 0066-0067, the “execution model generation unit” generates a neural network execution model in a “layer mapping step”, a “convolution operation circuit generation step”, a “quantization operation circuit generation step”, and “a DMAC generation step”. These steps are further described in applicant’s specification paragraphs 0068, 0080, 0082, 0089, 0100, 0106, 0111, 0118, 0121; these paragraphs disclose mathematical concepts performed in order to generate the neural network execution model; for example, applicant’s specification paragraph 0082 discloses “data partitioning in convolution operations” and discloses Equations 7 and 8 which are used to partition the operations, in order for the generation of the neural network execution model to be considered to be “based on hardware information regarding hardware in which the neural network execution model is operating and network information regarding the neural network”, this partitioning must occur and requires mathematical calculations, which are considered mathematical concepts. Thus, the limitation “generates the neural network execution model based on hardware information regarding hardware in which the neural network execution model is operating and network information regarding the neural network” is not merely “based on or involves a mathematical concept” but rather recites a mathematical concept as the limitation, when read in light of applicant’s specification, cannot be separated from mathematical concepts. With regard to the limitation “generates learned parameters of the generated neural network execution model”, applicant’s specification paragraph 0181 discloses the learned parameters are generated using “error backpropagation, which is a known technique, or the like”, and further discloses the use of a “loss function” which is used to update the weight and the quantization parameter. This limitation is not merely “based on or involves a mathematical concept” but rather recites a mathematical concept as the limitation, when read in light of applicant’s specification, is entirely mathematical concepts.
Applicant next argues, on page 13, paragraph 1 of the response, that “applicant’s claims are eligible under Prong Two of Step 2A, because any judicial exception recited in Applicant’s claims is integrated into a practical application”. Examiner respectfully disagrees. Applicant further argues, on page 14, lines 3-5 of the response, that “in addition to the “neural network execution model for operating a neural network” limitation and the “execution model generation unit” limitation, claim 1 recites many additional limitations”. With respect to the “neural network execution model for operation a neural network” this has been identified as an additional element that generally links the use of the judicial exception to a particular technological environment or field of use. Further, the “execution model generation unit” is an additional element that merely amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Thus, while these limitations have been identified as additional elements, they do not integrate the judicial exception into a practical application. With respect to the remainder of additional elements, please refer to the updated 35 U.S.C. 101 rejection above. Further, applicant admits that MPEP 2106.04(d)(II) instructs examiners to evaluate integration into a practical application by “(1) identifying whether there are any additional elements recited in the claim beyond the judicial exception(s); and (2) evaluating those additional elements individually and in combination to determine whether they integrate the exception into a practical application…”; and while applicant argued claim 1 recites additional elements, applicant made no argument to dispute the evaluation of those additional elements by examiner individually and in combination to determine whether they integrate the exception into a practical application.
Applicant next argues, on page 14, Prong Two (2), Paragraph 3 of the response, that “any use of a “neural network generating device” in Applicant’s claims is clearly integrated into the practical application of “generate[ing] the neural network execution model based on hardware information regarding hardware in which the neural network execution model is operating and network information regarding the neural network” and “generat[ing] learned parameters of the generated neural network execution model”. Examiner respectfully disagrees. It is important to note that mere physical or tangible implementation of an exception is not in itself an inventive concept and does not guarantee eligibility, see MPEP 2106.05(I)(A); thus, the use of a “neural network generating device” does not integrate the judicial exceptions into a practical application. Further, the limitations “generate[ing] the neural network execution model based on hardware information regarding hardware in which the neural network execution model is operating and network information regarding the neural network” and “generat[ing] learned parameters of the generated neural network execution model” have been identified as abstract ideas, and an inventive concept cannot be furnished by the unpatentable abstract idea itself, see MPEP 2106.05(I), the judicial exception alone cannot provide the improvement, see MPEP 2106.05(a). Applicant further argues the practical application of “generating a neural network execution model that is embeddable in a particular device”. Examiner again points to an inventive concept cannot be furnished by the unpatentable abstract idea itself, see MPEP 2106.05(I), and the judicial exception alone cannot provide the improvement, see MPEP 2106.05(a). An improvement to generating a neural network execution model may be an improvement in an abstract idea but not an improvement in the functioning of a computer, as a computer.
Applicant next argues, on page 14, final paragraph, that “claim 1 integrates the exception into a practical application because the generated neural network execution model is tailored to run efficiently on specific hardware and is implemented in specific circuit structures, particularly convolution and quantization circuits and the generation of the execution model provides a specific solution that improves the efficiency of physical devices”. Examiner respectfully disagrees. With regard to the generated neural network execution model being tailored to run efficiently on specific hardware, this is in reference to limitation “generate[ing] the neural network execution model based on hardware information regarding hardware in which the neural network execution model is operating and network information regarding the neural network” which has been identified as an abstract idea, and thus an inventive concept cannot be furnished by the unpatentable abstract idea itself, see MPEP 2106.05(I), and the judicial exception alone cannot provide the improvement, see MPEP 2106.05(a). With regard to being implemented in “specific circuit structures” while the claim does recite “a convolution operation circuit” and “a quantization operation circuit”, these limitations have been identified as additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. While circuits are claimed, there are no details of a specific configuration of circuits that would amount to more than mere means to apply the judicial exception. With respect to “improv[ing] the efficiency of physical devices”, it is important to note that claiming the improved speed or efficiency inherent with applying the abstract idea on a computer does not integrate a judicial exception into a practical application or provide an inventive concept, see MPEP 2106.05(f).
Applicant's arguments regarding the remainder of the claims rely upon the arguments asserted with respect to the independent claims, and are thus unpersuasive.
Applicant’s arguments regarding the prior art rejections of the claims have been fully considered but are unpersuasive.
Applicant first argues, on page 17, paragraph 2 of the response, that Shibata’s method of converting input data into sub-channels differs from independent claim 1 because it “fails to teach or suggest “a first memory storing input data” and “a second memory for storing convolution operation output data of the convolution operation circuit”. Examiner respectfully disagrees. Shibata clearly discloses “a first memory storing input data” (The “primary input data storage part” indicated with 212 in Figure 7) and “a second memory for storing convolution operation output data of the convolution operation circuit (The “primary operation result memory part” indicated with 213 in Figure 7). For a more in depth mapping of these limitations please refer to the updated 35 U.S.C. 103 rejection seen above. Further, applicant argues Shibata fails to teach or suggest “anything related to “a quantization operation circuit for performing a quantization operation on the convolution operation output data stored in the second memory””. Examiner respectfully disagrees. Shibata teaches performing quantization on the convolution output data (Shibata, Paragraph 0048, Lines 4-6, “The quantization part 101 quantizes a result of a convolutional operation in a convolutional neural network using input data and weights”; see also Shibata, Figure 1), because the convolution output data is stored in the “primary operation result memory part”, considered to be the “second memory”, and the quantization part performs quantization on convolution output data, the quantization part must be performing quantization on the data stored in the second memory.
Applicant next argues, on page 17, paragraph 3 of the response, that Shibata does not teach or suggest that ““convolution operations” are partitioned “based on the generated neural network execution model in accordance with capacities of the first memory and second memory”. Examiner respectfully disagrees. Shibata teaches partitioning the convolutional operations according to “the number of bits of the element that forms the input data of the first layer” (Shibata, Paragraph 0125, Lines 1-4; Shibata, Paragraph 0133, Lines 1-4; Shibata, Paragraph 0134; Performing the convolution operations using “shift operation[s]” and then adding the results, is considered to be partitioned convolution operations, further, since the method creates the subchannels according to “the number of bits of the element that forms the input data of the first layer”, it is considered to be performed based on the “generated neural network execution model”, since the “generated neural network execution model” is generated based on “computational resources” of the device, it is considered to be “in accordance with capacities of the first memory and second memory”). Further, because Shibata performs the partitioning and stores the input in the first memory and the output in the second memory, a person of ordinary skill in the art would recognize that the convolution operations must be performed in “accordance with capacities of the first memory and second memory” in order to be able to store the input and output. The broadest reasonable interpretation of “in accordance with capacities of the first memory and second memory” does not place specific limitations on the partitioning; the established teaching of Shibata that the input to the partitioned convolutions are stored in the first memory and the output of the partitioned convolutions are stored in the second memory covers the limitation of performing the partitioning “in accordance with capacities of the first memory and second memory”.
Applicant's arguments regarding the dependent claims rely upon the arguments asserted with respect to the independent claims, and are thus unpersuasive.
Applicant next argues, on page 18, “New Claim 17” that “none of the cited references teach or suggest the use of the “first memory area and the second memory area” let alone that the memory areas are “in a same memory””. Examiner respectfully disagrees. Shibata clearly discloses “a first memory area” (The “primary input data storage part” indicated with 212 in Figure 7) and “a second memory area (The “primary operation result memory part” indicated with 213 in Figure 7); both of which are a part of the “primary memory” which is considered to be the “same memory”.
Applicant next argues, on page 18, “New Claim 18”, that Nagel does not disclose “the learning unit performs convolution operations implemented when generating the learned parameter through backpropagation with higher precision than operations implemented by the convolution operation circuit””. While this limitation may not be taught by Nagel alone, Examiner disagrees that the limitation is not taught by the cited references. This limitation is taught by a combination of Konishi, Shibata, and Nagel. For a more in depth analysis, see the updated 35 U.S.C. 103 rejection above.
Applicant’s arguments with respect to “New Claim 19”, the limitations of “learn[ing] quantization parameters that the quantization operation circuit uses for the quantization operations … and a scaling factor for quantization operation output data quantized by the quantization parameters when learning the quantization parameters” have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOLLY CLARKE SIPPEL whose telephone number is (571)272-3270. The examiner can normally be reached Monday - Friday, 7:30 a.m. - 4:30 p.m. ET..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571)272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.C.S./ Examiner, Art Unit 2122
/KAKALI CHAKI/ Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights