Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
The present application has a provisional application No. 62/666,269 filed on May 03, 2018.
Claim Objections
Claim 1 objected to because of the following informalities: use of “the number of output classes” in Line4, which lacks antecedent basis. Appropriate correction is required.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-18 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-16 of U.S. Patent No. 11875251-B2. Although the claims at issue are not identical, they are not patentably distinct from each other because Claims 1-16 of US-11875251-B2 teach every limitation of Claim 1-18 of the instance application.
Instance Application
US-11875251-B2
1. A processor-implemented neural network method of an apparatus that includes a processor and a memory storing information, including stored predetermined precision parameters of a layer of a n neural network, about the layer, the method comprising:
obtaining information about the layer in the memory indicative of the number of output classes; determining, based on the obtained information, a precision for the layer based on the number of output classes of the layer, wherein the precision is determined proportionally with respect to the obtained number of output classes; and processing new parameters, with a set precision, for the layer based on the stored parameter.
1. A processor-implemented neural network method of an apparatus that includes a processor and a memory storing information, including stored predetermined precision parameters of a layer of an in-training neural network, about the layer, the method comprising:
generating new parameters, with a set precision, for the layer based on the stored parameters, where the precision is set based on a result of a determination of a number of output classes of the layer based on information about the layer in the memory indicative of the number of output classes;
“processing” and “obtaining” inherently encompasses the “generating” step of the U.S. Patent No. 11875251-B2 Claim 1. As to process the new parameters, it will determine a set precision that is based on the number of layers or output classes meaning it inherently will obtain the information about the layer in the memory indicative of the number of output classes, and it must have generated new parameters ahead of time before the processing.
The current Claim 1 further limits that the precision is determined “proportionally” with respect to the number of output classes. A person ordinary skilled in the art (PHOSITA) at the time of the invention would have found it obvious to determine the precision proportionally based on the number of output classes.
Therefore, the examined claim is a nonstatutory/obvious modification of the reference claim and is not patentably distinct.
2. The method of claim 1, further comprising: generating a trained neural network configured to generate an inference result based on an input provided to the trained neural network by training the layer of an in-training neural network using the new parameters such that a layer of the trained neural network has the set precision, the layer of the trained neural network is corresponding to the layer of the in-training neural network.
Part of 1,
generating a trained neural network configured to generate an inference result based on an input provided to the trained neural network by training the layer of the in-training neural network using the new parameters such that a layer of the trained neural network has the set precision, the layer of the trained neural network is corresponding to the layer of the in-training neural network,
3. The method of claim 2, further comprising: generating loss information of the layer; and performing the training of the layer dependent on the generated loss.
2. The method of claim 1, further comprising:
generating loss information of the layer; and
performing the training of the layer dependent on the generated loss.
4. The method of claim 2,
wherein the neural network further comprises a softmax layer configured to receive results of a forward implementation of the layer, and wherein the training of the neural network is based on gradients of a cross-entropy loss derived from a loss generated by a loss layer connected to the softmax layer during the training.
6. The method of claim 1,
wherein the neural network further comprises a softmax layer configured to receive results of a forward implementation of the layer, and
wherein the training of the neural network is based on gradients of a cross-entropy loss derived from a loss generated by a loss layer connected to the softmax layer during the training.
5. The method of claim 2, wherein the training of the layer includes training the layer by selectively adjusting the new parameters according to respective gradients of a cross-entropy loss derived from a loss, of the neural network, dependent on results of a forward implementation of the layer with the new parameters.
7. The method of claim 1, wherein the training of the layer includes training the layer by selectively adjusting the new parameters according to respective gradients of a cross-entropy loss derived from a loss, of the neural network, dependent on results of a forward implementation of the layer with the new parameters.
6. The method of claim 1,
wherein a set precision of the layer with a first number of output classes is greater than another set precision of the layer with a second number of output classes that are less than the first number of output classes.
Part of 1,
wherein a set precision of the layer with a first number of output classes is greater than another set precision of the layer with a second number of output classes that are less than the first number of output classes,
7. The method of claim 1,
wherein the set precision of the parameters is a bit width of the new parameters.
Part of 1,
wherein the set precision of the parameters is a bit width of the new parameters.
8. The method of claim 1, wherein the layer is a last fully-connected layer of the neural network.
5. The method of claim 1, wherein the layer is a last fully-connected layer of the neural network.
9. A non-transitory computer-readable recording medium storing instructions which when executed by one or more processors causes the one or more processors to perform the method of claim 1.
8. A non-transitory computer-readable recording medium storing instructions which when executed by one or more processors causes the one or more processors to perform the method of claim 1.
10. A neural network apparatus, the apparatus comprising:
a memory storing information, including stored pre-determined precision parameters of a layer of a neural network, about the layer; and a processor configured to:
obtain information about the layer in the memory indicative of the number of output classes; determine, based on the obtained information, a precision for the layer based on the number of output classes of the layer, wherein the precision is determined proportionally with respect to the obtained number of output classes; and process new parameters, with a set precision, for the layer based on the stored parameter.
9. A neural network apparatus, the apparatus comprising:
a memory storing information, including stored pre-determined precision parameters of a layer of an in-training neural network, about the layer; and a processor configured to:
generate new parameter, with a set precision, for the layer based on the stored parameters, where the precision is set based on a result of a determination of a number of output classes of the layer based on information about the layer in the memory indicative of the number of output classes;
“process” and “obtain” inherently encompasses the “generate” step of the U.S. Patent No. 11875251-B2 Claim 9. As to process the new parameters, it will determine a set precision that is based on the number of layers or output classes meaning it inherently will obtain the information about the layer in the memory indicative of the number of output classes, and it must have generated new parameters ahead of time before the processing.
The current Claim 10 further limits that the precision is determined “proportionally” with respect to the number of output classes. A person ordinary skilled in the art (PHOSITA) at the time of the invention would have found it obvious to determine the precision proportionally based on the number of output classes.
Therefore, the examined claim is a nonstatutory/obvious modification of the reference claim and is not patentably distinct.
11. The apparatus of claim 10, wherein the processor is further configured to:
generate a trained neural network configured to generate an inference result based on an input provided to the trained neural network by training the layer of an in-training neural network using the new parameters such that a layer of the trained neural network has the set precision, the layer of the trained neural network is corresponding to the layer of the in-training neural network.
Part of 9,
generate a trained neural network configured to generate an inference result based on an input provided to the trained neural network by training the layer of the in-training neural network using the new parameters such that a layer of the trained neural network has the set precision, the layer of the trained neural network is corresponding to the layer of the in-training neural network,
12. The apparatus of claim 11, wherein the processor is further configured to: generate loss information of the layer; and perform the training of the layer dependent on the generated loss.
10. The apparatus of claim 9, further comprising:
generating loss information of the layer; and
performing the training of the layer dependent on the generated loss.
13. The apparatus of claim 11,
wherein the neural network further comprises a softmax layer configured to receive results of a forward implementation of the layer,
wherein, for the training, the processor is configured to train the neural network based on gradients of a cross-entropy loss derived from a loss generated by a loss layer connected to the softmax layer for the training.
14. The apparatus of claim 9,
wherein the neural network further comprises a softmax layer configured to receive results of a forward implementation of the layer,
wherein, for the training, the processor is configured to train the neural network based on gradients of a cross-entropy loss derived from a loss generated by a loss layer connected to the softmax layer for the training.
14. The apparatus of claim 11, wherein the processor is further configured to: perform the training of the layer through selective adjustment of the new parameters according to respective gradients of a cross-entropy loss derived from a loss, of the neural network, dependent on results of a forward implementation of the layer with the new parameters.
15. The apparatus of claim 9, wherein the processor is further configured to perform the training of the layer through selective adjustment of the new parameters according to respective gradients of a cross-entropy loss derived from a loss, of the neural network, dependent on results of a forward implementation of the layer with the new parameters.
15. The apparatus of claim 10,
wherein a set precision of the layer with a first number of output classes is greater than another set precision of the layer with a second number of output classes that are less than the first number of output classes.
Part of 9,wherein a set precision of the layer with a first number of output classes is greater than another set precision of the layer with a second number of output classes that are less than the first number of output classes
16. The apparatus of claim 10,
wherein the set precision of the parameters is a bit width of the new parameters.
Part of 9,wherein the set precision is a bit width of the new parameters.
17. The apparatus of claim 10,
wherein the layer is a last fully-connected layer in the neural network.
13. The apparatus of claim 9,
wherein the layer is a last fully-connected layer in the neural network.
18. The apparatus of claim 10, wherein the memory, or another memory of the apparatus, stores instructions, which when executed by the processor configure the processor to perform the generation of the new parameters, and the training of the layer.
16. The apparatus of claim 9, wherein the memory, or another memory of the apparatus, stores instructions, which when executed by the processor configure the processor to perform the generation of the new parameters, and the training of the layer.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-18 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 1 is a method claim thus it falls into one of the four categories of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding independent claim 1, following limitations recite a judicial exception:
“determining, based on the obtained information, a precision for the layer based on the number of output classes of the layer, wherein the precision is determined proportionally with respect to the obtained number of output classes”
[Mental Process] – determining precision for a layer based on a number simply involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen
“processing new parameters, with a set precision, for the layer based on the stored parameter”
[Mental Process] – processing parameters with a set precision for a layer involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 1, the claim recites additional elements of
“obtaining information about the layer in the memory indicative of the number of output classes”
Obtaining information from a memory is merely data gathering recited at a high level of generality, thus is insignificant extra-solution activity (See MPEP 2106.05(g)).
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional element [1] is considered an insignificant extra solution activity and at best the equivalent of a mere data gathering recited at a high level of generality and amount to receiving or transmitting data over a network, which is well-understood, routine, conventional activity (See MPEP 2106.05(d), subsection II). This limitation remains insignificant extra-solution activity even upon reconsideration. Even when considered in combination, the additional element represents insignificant extra-solution activity, which cannot provide an inventive concept.
Regarding Claim 3
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 3 is a dependent claim of 2, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 3, following limitations recite a judicial exception:
“generating loss information of the layer”
[Mathematical Calculations] – generating loss information of a layer requires pure mathematical computations which recites to an abstract idea
“performing the training of the layer dependent on the generated loss”
[Mathematical Calculations] – performing the training for the layer using the generated loss requires mathematical computations which recites to an abstract idea
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 3 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 4
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 4 is a dependent claim of 2, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 4, following limitations recite a judicial exception:
“a softmax layer configured to receive results of a forward implementation of the layer”
[Mathematical Calculations] – a softmax layer and a forward implementation are mathematical steps to calculate a mathematical output which recites to an abstract idea
“the training of the neural network is based on gradients of a cross-entropy loss derived from a loss generated by a loss layer connected to the softmax layer during the training”
[Mathematical Calculations] – using gradients of a cross-entropy loss requires mathematical computations which recites to an abstract idea
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 4 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 5
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 5 is a dependent claim of 2, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 5, following limitations recite a judicial exception:
“training the layer by selectively adjusting the new parameters according to respective gradients of a cross-entropy loss derived from a loss, of the neural network, dependent on results of a forward implementation of the layer with the new parameters”
[Mathematical Calculations] – training the layer by adjusting the parameters using gradients of a cross-entropy loss and a forward implementation is mathematical procedures to adjust the parameters which recites to an abstract idea
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 5 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 6
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 6 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 6, following limitations recite a judicial exception:
“a set precision of the layer with a first number of output classes is greater than another set precision of the layer with a second number of output classes that are less than the first number of output classes”
[Mental Process] – setting a precision based on simply comparing two numbers indicating the number of classes involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 6 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 7
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 7 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 7 does not have any abstract idea by itself, thus uses all the limitations of Claim 1.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 7 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 8
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 8 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 8 does not have any abstract idea by itself, thus uses all the limitations of Claim 1.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 8 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 9
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 9 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 9 does not have any abstract idea by itself, thus uses all the limitations of Claim 1.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 9, the claim recites additional elements of
“one or more processors”
The word “processors” is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional element [1] is considered a mere instruction to apply an exception to the generic computer components. (see MPEP 2106.05(f)) This limitation remains a mere instruction to apply an exception even upon reconsideration. Even when considered in combination, the additional element represents a mere instruction to apply an exception, which cannot provide an inventive concept.
Regarding Claim 10
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 10 is an apparatus claim thus it falls into one of the four categories of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding independent claim 10, following limitations recite a judicial exception:
“determine, based on the obtained information, a precision for the layer based on the number of output classes of the layer, wherein the precision is determined proportionally with respect to the obtained number of output classes”
[Mental Process] – determining precision for a layer based on a number simply involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen
“process new parameters, with a set precision, for the layer based on the stored parameter”
[Mental Process] – processing parameters with a set precision for a layer involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 10, the claim recites additional elements of
“a memory storing information, including stored pre-determined precision parameters of a layer of a neural network, about the layer”
A memory is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
“a processor configured to”
A processor is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
“obtain information about the layer in the memory indicative of the number of output classes”
Obtaining information from a memory is merely data gathering recited at a high level of generality, thus is insignificant extra-solution activity (See MPEP 2106.05(g)).
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional elements [1,2] are merely computer components that are just to store and execute code-based instructions which are considered a mere instruction to apply an exception and amount to storing and receiving information in memory, which is well-understood, routine, conventional activity (See MPEP 2106.05(d), subsection II). This limitation remains a mere instruction to apply an exception.
The additional element [3] is considered an insignificant extra solution activity and at best the equivalent of a mere data gathering recited at a high level of generality and amount to receiving or transmitting data over a network, which is well-understood, routine, conventional activity (See MPEP 2106.05(d), subsection II). This limitation remains insignificant extra-solution activity even upon reconsideration. Even when considered in combination, the additional elements represent a mere instruction to apply an exception and an insignificant extra-solution activity, which cannot provide an inventive concept.
Regarding Claim 12
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 12 is a dependent claim of 11, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 12, following limitations recite a judicial exception:
“generate loss information of the layer”
[Mathematical Calculations] – generating loss information of a layer requires pure mathematical computations which recites to an abstract idea
“perform the training of the layer dependent on the generated loss”
[Mathematical Calculations] – performing the training for the layer using the generated loss requires mathematical computations which recites to an abstract idea
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 12 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 13
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 13 is a dependent claim of 11, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 13, following limitations recite a judicial exception:
“a softmax layer configured to receive results of a forward implementation of the layer”
[Mathematical Calculations] – a softmax layer and a forward implementation are mathematical steps to calculate a mathematical output which recites to an abstract idea
“train the neural network is based on gradients of a cross-entropy loss derived from a loss generated by a loss layer connected to the softmax layer for the training”
[Mathematical Calculations] – using gradients of a cross-entropy loss requires mathematical computations which recites to an abstract idea
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
“the processor configured to”
The processor is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional element [1] is merely computer components that are just to store and execute code-based instructions which are considered a mere instruction to apply an exception and amount to storing and receiving information in memory, which is well-understood, routine, conventional activity (See MPEP 2106.05(d), subsection II). This limitation remains a mere instruction to apply an exception. Even when considered in combination, the additional element represents a mere instruction to apply an exception, which cannot provide an inventive concept.
Regarding Claim 14
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 14 is a dependent claim of 11, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 14, following limitations recite a judicial exception:
“perform the training of the layer through selective adjustment of the new parameters according to respective gradients of a cross-entropy loss derived from a loss, of the neural network, dependent on results of a forward implementation of the layer with the new parameters”
[Mathematical Calculations] – training the layer by adjusting the parameters using gradients of a cross-entropy loss and a forward implementation is mathematical procedures to adjust the parameters which recites to an abstract idea
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
“the processor further configured to”
The processor is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional element [1] is merely computer components that are just to store and execute code-based instructions which are considered a mere instruction to apply an exception and amount to storing and receiving information in memory, which is well-understood, routine, conventional activity (See MPEP 2106.05(d), subsection II). This limitation remains a mere instruction to apply an exception. Even when considered in combination, the additional element represents a mere instruction to apply an exception, which cannot provide an inventive concept.
Regarding Claim 15
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 15 is a dependent claim of 10, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 15, following limitations recite a judicial exception:
“a set precision of the layer with a first number of output classes is greater than another set precision of the layer with a second number of output classes that are less than the first number of output classes”
[Mental Process] – setting a precision based on simply comparing two numbers indicating the number of classes involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 15 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 16
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 16 is a dependent claim of 10, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 16 does not have any abstract idea by itself, thus uses all the limitations of Claim 10.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 16 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 17
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 17 is a dependent claim of 10, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 17 does not have any abstract idea by itself, thus uses all the limitations of Claim 10.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 17 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 18
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 18 is a dependent claim of 10, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 18, following limitations recite a judicial exception:
“generation of the new parameters, and the training of the layer”
[Mathematical Calculations] – generating new parameters and the training of the layer requires mathematical computations which recites to an abstract idea.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 18, the claim recites additional elements of
“the memory, or another memory of the apparatus, stores instructions, which when executed by the processor”
The words “memory” and ”processor” are recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional element [1] is considered a mere instruction to apply an exception to the generic computer components. (see MPEP 2106.05(f)) This limitation remains a mere instruction to apply an exception even upon reconsideration. Even when considered in combination, the additional element represents a mere instruction to apply an exception, which cannot provide an inventive concept.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 9, 10 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Zhou et al. (Zhou), Non-Patent Literature listed in IDS filed on 12/05/23, “Adaptive Quantization for Deep Neural Network”, published on 12/04/2017, Pages: 14.
As to independent Claim 1,
Zhou teaches A processor-implemented neural network method of an apparatus that includes a processor and a memory storing information, including stored predetermined precision parameters of a layer of a n neural network, about the layer, the method comprising:
obtaining information about the layer in the memory indicative of the number of output classes (Zhou, Pg3, Right Column, Paragraph4, Lines2-3,"under softmax classifier with a number of classes equal to d", Pg3, Right Column, Equation6
PNG
media_image1.png
41
271
media_image1.png
Greyscale
, and Pg6, Right Column, Subsection Empirical results about measurements, Paragraph2, Lines6-9, "The quantized modelis then tested on the validation set of Imagenet (Krizhevsky, Sutskever, and Hinton 2012), which contains 50000 images in 1000 classes", wherein computing the robust threshold
PNG
media_image2.png
29
56
media_image2.png
Greyscale
, which uses the number of classes, d, on the quantized model inherently indicates that the number of classes must be stored in the memory and has to be obtained before the computation, which is equivalent to the claimed invention);
PNG
media_image3.png
549
383
media_image3.png
Greyscale
determining, based on the obtained information, a precision for the layer based on the number of output classes of the layer, wherein the precision is determined proportionally with respect to the obtained number of output classes(Zhou, Pg6, Left Column, Optimal bit-width for each layer, the procedure, Pg3, Right Column, Lemma1 and Equation6,
PNG
media_image4.png
133
376
media_image4.png
Greyscale
PNG
media_image1.png
41
271
media_image1.png
Greyscale
Pg4, Equation14,
PNG
media_image5.png
171
371
media_image5.png
Greyscale
Pg6, Equation 22,23,
PNG
media_image6.png
196
360
media_image6.png
Greyscale
, wherein the robust threshold
PNG
media_image2.png
29
56
media_image2.png
Greyscale
or the equation6 and Lemma1 mathematically proves that the overall noise tolerance threshold theta is a direct function of, and uniquely determined by the specific number of output classes, d. Any modification to the class count d fundamentally alters the value of theta. Also, in the equation21 which minimizes the total bit-width allocation subject to a precision constraint uses the parameter ti and the total accuracy degradation constant C (from Eq14) that are directly from the noise boundary conditions established by
PNG
media_image2.png
29
56
media_image2.png
Greyscale
from equation6. Lastly, to solve the constrained optimization problem of Equation21, the optimal bit-width allocation rule of the equation 22 has to be met. The equation22 is the mathematical solution that dictates the exact bit-width (bi) for each layer. Because the pivotal variable ti, and the baseline boundary b1 (governed by constraint C) are mathematically bound to
PNG
media_image2.png
29
56
media_image2.png
Greyscale
, the final output of this equation or the determined bit width bi is functionally and inherently dependent on the number of output classes d); and
processing new parameters, with a set precision, for the layer based on the stored parameter (Zhou, Pg1, Right Column, Lines3-5, "By applying quantization on model parameters, these parameters can be stored and computed under lower bit-width", wherein the quantization model of Zhou relies on the bit-width computed from the above algorithms, thus it is functionally equivalent to the claimed invention.)
As to dependent Claim 9,
it is a non-transitory computer readable medium claim that contains similar limitations of Claim 1 and thus rejected under the same rationale.
As to independent Claim 10,
it is an apparatus claim that contains similar limitations of Claim 1 and thus rejected under the same rationale.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 2-5, 11-14, 18 are rejected under 35 U.S.C. 103 as being unpatentable over Zhou as mentioned in Claim 1 in view of Taigman et al. (Taigman), U.S Patent NO. 2015/0125049-A1 listed in IDS filed on 12/05/2023.
As to dependent Claim 2,
Zhou teaches, as mentioned above, all the limitations of Claim 1. Zhou teaches about processing new parameters with the set precision or the bit-width that was determined proportionally based on the number of output classes. However, Zhou does not teach generating a trained neural network configured to generate an inference result based on an input provided to the trained neural network by training the layer of an in-training neural network using the new parameters such that a layer of the trained neural network has the set precision, the layer of the trained neural network is corresponding to the layer of the in-training neural network.
In the same field of endeavor, Taigman teaches about generating a trained neural network configured to generate an inference result based on an input provided to the trained neural network (Taigman, Pg7, Paragraph88, Lines10-12, "The network may be trained for roughly 15 sweeps (epochs) over the whole data", Pg7, Paragraph89, Lines6-8, "DNNs (e.g., DeepFace-1.5M, DeepFace-3.3M, and DeepFace-4.4M) are trained", and Pg, 13, Claim1, "classifying, by the computing system, an identity of the 2D face image based on provision of the 3D-aligned face image to a deep neural network (DNN), the identity of the 2D face image comprising a feature vector", wherein the trained DNN is used to classify the input meaning it will inherently generate an inference result based on the input and the trained DNN corresponds to the network that was being trained.)
As mentioned above, Zhou teaches about the set precision is a bit-width, while Taigman further teaches about training the layer of an in-training neural network using the new parameters such that a layer of the trained neural network has the set precision, the layer of the trained neural network is corresponding to the layer of the in-training neural network (Taigman, Paragraph80, Lines5-10,"The cross-entropy loss for a given input is L=−log pk, where k is the index of the true label for a given input. The loss may be minimized over the parameters by computing the gradient of L with respect to the parameters and by updating the parameters using stochastic gradient descent (SGD). In some embodiments, the gradients may be determined by standard back-propagation of the error", Pg3, Paragraph56, Lines12-14, " In various embodiments, in order to prevent overfitting on the face verification task, training is only enabled for the two topmost layers of a Siamese network", wherein isolating and updating the parameters of specific layers using SGD during the active training phase, which directly correspond to the identical layers within the finalized, fully trained network model. Thus, during the SGD phase, use of the set precision given by Zhou, bit-width, is functionally equivalent to the claimed invention.)
Zhou and Taigman are analogous to the claimed invention as they are from the same field of endeavor of neural network optimization and training. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the dynamic bit-width accuracy model of Zhou with the SGD implemented model to improve the accuracy by Taigman. The motivation is as recited by Taigman (Taigman, pg6, Paragraph80, Line3-5, “The cross-entropy loss for each training sample may be minimized in order to maximize the probability of the correct class” and Lines7-10, “The loss may be minimized over the parameters by computing the gradient of L with respect to the parameters and by updating the parameters using stochastic gradient descent (SGD)”) such that utilizing specific layer-wise bit-width (set precision) determined by Zhou as the parameter constraints and subsequently executing explicit cross-entropy-based SGD routine to update and fine-tune the parameters can successfully minimize network loss and recovering any residual accuracy degradation caused by the quantization process.
As to dependent Claim 3,
The combination of Zhou and Taigman teaches, as mentioned above, all the limitations of Claim 2. It teaches about generating a trained neural network, which was trained with the set precision for the specific layer, to generate an inference result of an input.
Zhou does not teach the following limitations but in the same field of endeavor, Taigman further teaches the method of claim 2, further comprising:
generating loss information of the layer (Taigman, Pg6, Paragraph80, Lines5-6,"The cross-entropy loss for a given input is L=−log pk, where k is the index of the true label for a given input", wherein the cross-entropy loss, L, is the corresponding loss information which computes the error difference between the true label and the input that is functionally equivalent to the claimed invention); and performing the training of the layer dependent on the generated loss(Taigman, Pg6, Paragraph80, Lines6-10,"The loss may be minimized over the parameters by computing the gradient of L with respect to the parameters and by updating the parameters using stochastic gradient descent (SGD). In some embodiments, the gradients may be determined by standard back-propagation of the error", wherein the loss information is used in backward-propagation which the gradient computed from the loss information and the gradient is used to update the parameters of the layer, thus it is functionally equivalent to the claimed invention.)
Zhou and Taigman are analogous to the claimed invention as they are from the same field of endeavor of neural network optimization and training. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the dynamic bit-width accuracy model of Zhou with the SGD implemented model to improve the accuracy by Taigman. The motivation is as recited by Taigman (Taigman, pg6, Paragraph80, Line3-5, “The cross-entropy loss for each training sample may be minimized in order to maximize the probability of the correct class” and Lines7-10, “The loss may be minimized over the parameters by computing the gradient of L with respect to the parameters and by updating the parameters using stochastic gradient descent (SGD)”) such that utilizing specific layer-wise bit-width (set precision) determined by Zhou as the parameter constraints and subsequently executing explicit cross-entropy-based SGD routine to update and fine-tune the parameters can successfully minimize network loss and recovering any residual accuracy degradation caused by the quantization process.
As to dependent Claim 4,
The combination of Zhou and Taigman teaches, as mentioned above, all the limitations of Claim 2. It teaches about generating a trained neural network, which was trained with the set precision for the specific layer, to generate an inference result of an input.
Zhou does not teach the following limitations but in the same field of endeavor, Taigman further teaches the method of claim 2, wherein the neural network further comprises
a softmax layer configured to receive results of a forward implementation of the layer (Taigman, Pg6, Paragraph79, Lines6-9, "In some embodiments, the output of the DNN (e.g., the output of the last fully-connected layer) may be provided to a K-way softmax, where K is the number of classes", wherein the output that is from the last fully-connected layer of DNN indicates that its result is preceded by the layers before the last layer, which is equivalent to the forward implementation, that is provided to the k-max softmax or the corresponding softmax layer, which is inherently equivalent to the claimed invention), and
wherein the training of the neural network is based on gradients of a cross-entropy loss derived from a loss generated by a loss layer connected to the softmax layer during the training (Taigman, Pg6, Paragraph79, Lines6-9, "In some embodiments, the output of the DNN (e.g., the output of the last fully-connected layer) may be provided to a K-way softmax, where K is the number of classes", Pg6, Paragraph79, Lines10-13, "The probability assigned to the i-th class is the output of the softmax function: pi=exp(oi)/Σj exp(oj), where oi denotes the i-th output of the network on a given input" and Paragraph80, Lines5-10, "The cross-entropy loss for a given input is L=−log pk, where k is the index of the true label for a given input. The loss may be minimized over the parameters by computing the gradient of L with respect to the parameters and by updating the parameters using stochastic gradient descent (SGD). In some embodiments, the gradients may be determined by standard back-propagation of the error", wherein the pi or the output from the softmax that is passed to the loss layer or where it computes the gradients of the cross-entropy loss, L, based on this pi, which is functionally equivalent to the claimed invention.)
Zhou and Taigman are analogous to the claimed invention as they are from the same field of endeavor of neural network optimization and training. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the dynamic bit-width accuracy model of Zhou with the SGD implemented model to improve the accuracy by Taigman. The motivation is as recited by Taigman (Taigman, pg6, Paragraph80, Line3-5, “The cross-entropy loss for each training sample may be minimized in order to maximize the probability of the correct class” and Lines7-10, “The loss may be minimized over the parameters by computing the gradient of L with respect to the parameters and by updating the parameters using stochastic gradient descent (SGD)”) such that utilizing specific layer-wise bit-width (set precision) determined by Zhou as the parameter constraints and subsequently executing explicit cross-entropy-based SGD routine to update and fine-tune the parameters can successfully minimize network loss and recovering any residual accuracy degradation caused by the quantization process.
As to dependent Claim 5,
The combination of Zhou and Taigman teaches, as mentioned above, all the limitations of Claim 2. It teaches about generating a trained neural network, which was trained with the set precision for the specific layer, to generate an inference result of an input.
Zhou does not teach the following limitations but in the same field of endeavor, Taigman further teaches the method of claim 2, wherein the training of the layer includes
training the layer by selectively adjusting the new parameters according to respective gradients of a cross-entropy loss derived from a loss, of the neural network, dependent on results of a forward implementation of the layer with the new parameters (Taigman, Pg6, Paragraph79, Lines6-9, "In some embodiments, the output of the DNN (e.g., the output of the last fully-connected layer) may be provided to a K-way softmax, where K is the number of classes", Pg6, Paragraph79, Lines10-13, "The probability assigned to the i-th class is the output of the softmax function: pi=exp(oi)/Σj exp(oj), where oi denotes the i-th output of the network on a given input" and Paragraph80, Lines5-10, "The cross-entropy loss for a given input is L=−log pk, where k is the index of the true label for a given input. The loss may be minimized over the parameters by computing the gradient of L with respect to the parameters and by updating the parameters using stochastic gradient descent (SGD). In some embodiments, the gradients may be determined by standard back-propagation of the error", and Pg7, Paragraph88, Lines10-12, "The network may be trained for roughly 15 sweeps (epochs) over the whole data", wherein as mentioned in Claim 4, training the model uses the forward implementation such that the gradients of computed from the cross-entropy loss, L, is used to update the parameters by the backward-propagation. As it uses the stochastic gradient descent, which inherently indicates that the parameters are updated according to the gradients meaning that each parameter is updated separately or selectively. And the network is trained 15 epochs indicates that once backward-propagation is done updating the parameters, forward propagation will start to compute a new result with the updated parameters, which is functionally equivalent to the claimed invention.)
Zhou and Taigman are analogous to the claimed invention as they are from the same field of endeavor of neural network optimization and training. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the dynamic bit-width accuracy model of Zhou with the SGD implemented model to improve the accuracy by Taigman. The motivation is as recited by Taigman (Taigman, pg6, Paragraph80, Line3-5, “The cross-entropy loss for each training sample may be minimized in order to maximize the probability of the correct class” and Lines7-10, “The loss may be minimized over the parameters by computing the gradient of L with respect to the parameters and by updating the parameters using stochastic gradient descent (SGD)”) such that utilizing specific layer-wise bit-width (set precision) determined by Zhou as the parameter constraints and subsequently executing explicit cross-entropy-based SGD routine to update and fine-tune the parameters can successfully minimize network loss and recovering any residual accuracy degradation caused by the quantization process.
As to dependent Claim 11,
it is an apparatus claim that contains similar limitations of Claim 1 and thus rejected under the same rationale.
As to dependent Claim 12,
it is an apparatus claim that contains similar limitations of Claim 3 and thus rejected under the same rationale.
As to dependent Claim 13,
it is an apparatus claim that contains similar limitations of Claim 4 and thus rejected under the same rationale.
As to dependent Claim 14,
it is an apparatus claim that contains similar limitations of Claim 5 and thus rejected under the same rationale.
As to dependent Claim 18,
Zhou teaches, as mentioned above, all the limitations of Claim 10. Zhou teaches about processing new parameters with the set precision or the bit-width that was determined proportionally based on the number of output classes. However, Zhou does not teach the apparatus of claim 10, wherein the memory, or another memory of the apparatus, stores instructions, which when executed by the processor configure the processor to perform the generation of the new parameters, and the training of the layer.
In the same field of endeavor, Taigman teaches this limitation (Taigman, Pg1, Paragraph23, “In a further embodiment according to the invention, one or more computer-readable non-transitory storage media embody software that is operable when executed to perform a method according to the invention or any of the above mentioned embodiments”, Pg1, Paragraph24, “In a further embodiment according to the invention, a system comprises: one or more processors; and at least one memory coupled to the processors and comprising instructions executable by the processors, the processors operable when executing the instructions to perform a method according to the invention or any of the above mentioned embodiments” and pg6, Paragraph80, Line3-5, “The cross-entropy loss for each training sample may be minimized in order to maximize the probability of the correct class” and Lines7-10, “The loss may be minimized over the parameters by computing the gradient of L with respect to the parameters and by updating the parameters using stochastic gradient descent (SGD)”, wherein the using the SGD to update the parameters to create new parameters is functionally equivalent to the claimed invention while this updating the parameters inherently encompasses the training of the layer as it has to go through forward and backward propagations which are used to train the layer.)
Zhou and Taigman are analogous to the claimed invention as they are from the same field of endeavor of neural network optimization and training. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the dynamic bit-width accuracy model of Zhou with the SGD implemented model to improve the accuracy by Taigman. The motivation is as recited by Taigman (Taigman, pg6, Paragraph80, Line3-5, “The cross-entropy loss for each training sample may be minimized in order to maximize the probability of the correct class” and Lines7-10, “The loss may be minimized over the parameters by computing the gradient of L with respect to the parameters and by updating the parameters using stochastic gradient descent (SGD)”) such that utilizing specific layer-wise bit-width (set precision) determined by Zhou as the parameter constraints and subsequently executing explicit cross-entropy-based SGD routine to update and fine-tune the parameters can successfully minimize network loss and recovering any residual accuracy degradation caused by the quantization process.
Claims 6, 7, 15, 16 are rejected under 35 U.S.C. 103 as being unpatentable over Zhou as mentioned in Claim 1 in view of Yin et al. (Yin), Non-Patent Literature listed in IDS filed on 12/05/2023, “Quantization and Training of Low Bit-Width Convolutional Neural Networks for Object Detection”, Pages: 10.
As to dependent Claim 6,
Zhou teaches, as mentioned above, all the limitations of Claim 1. Zhou teaches about processing new parameters with the set precision or the bit-width that was determined proportionally based on the number of output classes. However, Zhou does not teach the method of claim 1, wherein
a set precision of the layer with a first number of output classes is greater than another set precision of the layer with a second number of output classes that are less than the first number of output classes. In the same field of endeavor, Yin teaches this limitation (Yin, Pg1, Introduction, Lines15-17, "Yet with fully ternarized weights, there is still noticeable drop in performance on larger datasets like ImageNet [4], which suggests the necessity of relatively wider bit-width models with stronger model capacity for challenging tasks", wherein as mentioned in Claim1, Zhou discloses that the number of classes alters the bit-width or the corresponding set precision which inherently indicates that the one precision is bigger than another precision according to the number of classes. Also, Yin discloses that for larger datasets that inherently contains bigger number of classes requires wider bit-width, which is functionally equivalent to the claimed invention.)
Zhou and Yin are analogous to the claimed invention as they are from the same field of endeavor of deep neural network quantization and compression. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the adaptive quantization framework that determines optimal layer-wise bit-width constraints of Zhou with the low bit-width weight quantization and projected gradient training scheme of Yin. The motivation is as recited by Yin (Yin, Pg1, Introduction, Lines5-8, “To address this issue, recent efforts have been made to compress the model size and train neural networks with heavily quantized weights, activations, and gradients [1, 2, 6, 7, 9, 17, 20, 22, 25, 26, 27], which demand less storage and fewer FLOPs for deployment”) such that utilizing the optimal layer-wise bit-widths mathematically determined by Zhou’s framework as the precise parameter constraints required within Yin’s projected SGH training pipeline, thereby enabling the physical computing hardware to seamlessly execute actual weight quantization and iterative parameter optimization in tandem to successfully recover any residual accuracy degradation while fully realizing the storage and computational efficiency benefits.
As to dependent Claim 7,
Zhou teaches, as mentioned above, all the limitations of Claim 1. Zhou teaches about processing new parameters with the set precision or the bit-width that was determined proportionally based on the number of output classes. Zhou teaches how to determine a set precision for the specific layer that would improve the accuracy of the training. Here, Yin further teaches how this set precision can be specifically utilized such that
the set precision of the parameters is a bit width of the new parameters (Yin, Pg4. Subsetion 2.2, Lines1-3, "We used a projected SGD-like algorithm as in [17, 20] for training LBW-Net. At each gradient descent step, the minibatch gradient is evaluated at the quantized weights, and a scaled gradient is subtracted from the full-precision weights instead of the quantized weights per standard projected gradient method" and Yin, Pg2, Subsection2.1, Paragraph1, Lines5-7, "To quantize the full-precision weights into low-precision ones of b bits (b>=2), we constrain the quantized weights to the value set of
PNG
media_image7.png
24
258
media_image7.png
Greyscale
for some integers s in Z, where n = 2^b-2 and 2^s serves as the scaling factor", wherein Yin discloses that the numerical bounds and discrete states available for the weights are directly derived from the set bit-width. Also,
PNG
media_image7.png
24
258
media_image7.png
Greyscale
mathematically demonstrates that the quantization grid used during training is defined by and utilizes the size of the set bit-width(b). Also, it further details how this bit-width constraint actively participates in the back-propagation and gradient descent loop, which utilizes the set precision to bound and modulate the parameter updates during the SGD routine, thus it is functionally equivalent to the claimed invention.)
Zhou and Yin are analogous to the claimed invention as they are from the same field of endeavor of deep neural network quantization and compression. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the adaptive quantization framework that determines optimal layer-wise bit-width constraints of Zhou with the low bit-width weight quantization and projected gradient training scheme of Yin. The motivation is as recited by Yin (Yin, Pg1, Introduction, Lines5-8, “To address this issue, recent efforts have been made to compress the model size and train neural networks with heavily quantized weights, activations, and gradients [1, 2, 6, 7, 9, 17, 20, 22, 25, 26, 27], which demand less storage and fewer FLOPs for deployment”) such that utilizing the optimal layer-wise bit-widths mathematically determined by Zhou’s framework as the precise parameter constraints required within Yin’s projected SGH training pipeline, thereby enabling the physical computing hardware to seamlessly execute actual weight quantization and iterative parameter optimization in tandem to successfully recover any residual accuracy degradation while fully realizing the storage and computational efficiency benefits.
As to dependent Claim 15,
it is an apparatus claim that contains similar limitations of Claim 6 and thus rejected under the same rationale.
As to dependent Claim 16,
it is an apparatus claim that contains similar limitations of Claim 7 and thus rejected under the same rationale.
Claims 8 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Zhou as mentioned in Claim 1 in view of Ishii et al. (Ishii), Non-Patent Literature listed in IDS filed on 12/05/2023, “Evaluation of Quantization Bit Width Optimization for each Neuron for DNN”, Pages: 16.
As to dependent Claim 8,
Zhou teaches, as mentioned above, all the limitations of Claim 1. Zhou teaches about processing new parameters with the set precision or the bit-width that was determined proportionally based on the number of output classes. However, Zhou does not teach the method of claim 1, wherein
the layer is a last fully-connected layer of the neural network. From the same field of endeavor, Ishii teaches this limitation (Ishii, Pg6, Lines1-2, "In addition, optimization for each neuron is limited to the fully connected layer in this article" and Lines23, "Next, optimization of the bit width for each neuron will be described", wherein the layers used during the training is actually the fully-connected(FC) layers.)
Zhou and Ishii are analogous to the claimed invention as they are from the same field of endeavor of deep neural network quantization and resource optimization. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the adaptive quantization framework that determines optimal layer-wise bit-width constraints of Zhou with the fine-grained quantization method that optimizes bit-widths for each individual neuron within a layer of Ishii. The motivation is as recited by Ishii (Ishii, Pg1, Introduction, Lines7-10, “When executing a large-scale CNN in an assembled device with limited computational resources or power resources, it is a challenge to achieve both reduction in energy consumption required for calculation and memory access and high performance”) such that employing Zhou’s class-dependent framework to satisfy Ishii’s explicit algorithmic prerequisite of first-performing quantization for each layer to set an optimum bit width, to establish a mathematically rigorous and structurally sound layer-level baseline before seamlessly executing Ishii’s secondary neuron-level precision tuning to maximize model compression and computing efficiency under strict hardware constraints.
As to dependent Claim 17,
it is an apparatus claim that contains similar limitations of Claim 8 and thus rejected under the same rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Jiao et al. Non-Patent Literature, “Accelerating low bit-width convolutional neural networks with embedded FPGA”, Published on 2017, Pages: 4
Kapur et al. Non-Patent Literature, “Low Precision RNNs: Quantizing RNNs Without Losing Accuracy”, published on 2017, Pages: 5
Chen et al. Non-Patent Literature, “A Low Bit-width Parameter Representation Method for Hardware-oriented Convolution Neural Networks”, published on 2017, Pages: 4
Yin et al. Non-Patent Literature, “A High Energy Efficient Reconfigurable Hybrid Neural Network Processor for Deep Learning Applications”, published on 2017, Pages: 15
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DONG YOON JUNG whose telephone number is (571)270-0198. The examiner can normally be reached 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached at (571) 272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DONG YOON JUNG/ Examiner, Art Unit 2145
/CESAR B PAULA/ Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights