Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the application and claims filed 10/30/2023. Claims 12-26 are pending and have been examined. Claims 12-26 are rejected.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The present application claims foreign priority based on Japanese Application 2021-076487 filed 04/28/2021. The examiner notes that a certified copy (in Japanese) of the above-noted application was retrieved on 10/30/2023. Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/11/2024 and 11/03/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claim 16 objected to because of the following informalities: Claim 16 recites “a switch circuit” however, claim 16 depends on claim 13 which already established “a switch circuit”. Therefore, “a switch circuit” is now introduced twice, once in claim 13 and once in claim 16. Claim 18 also is objected for similar reasons, as it recites “a link circuit” which was already previously introduced in claim 14. Appropriate correction is required.
Claim 12-18, 21, 24, and 26 are objected to because of the following informalities: Claim 12-18, 21, 24, and 26 all recite “plurality of neuron cell circuit” circuit should read as “circuits”. Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 12, 14, 26 are rejected under 35 U.S.C. 102 as being anticipated by Toda et al. (US006026178A), hereinafter “Toda”.
Claim 12
Toda teaches:
A machine learning processing circuit including a plurality of neuron cell circuit, (Abstract, "a neural network arranged in an image processing apparatus"; Col 2, ln 65-67, "the connection pattern (weight coefficients) is adjusted by learning. An algorithm for the learning is called back propagation"; Col 5, ln 1-2, "the neural network shown in FIG. 1 is realized by a hardware arrangement shown in FIG. 2"; -- EN: Toda's neural network is trained by learning (back propagation) and implemented as a hardware processing circuit in FIG. 2, in which each neuron is realized by an adder plus a sigmoid look-up table so these constitute the plurality of neuron cell circuits.)
each of the plurality of neuron cell circuit comprises: an input unit that receives a plurality of input signals, (Col 5, ln 25-27, "The output from the look-up table 205 and the five outputs from the look-up table unit 206 are added to each other by an adder 207"; Col 5, line 15-16, "The five outputs from the look-up tables 203 are added to each other by an adder 204"; FIG. 5 showing inputs 501/502 feeding an ADD block -- EN: Each neuron's input receive a plurality of input signals, e.g., adder 207 receives six input signals and adder 204 receives five input signals)
an adder unit that adds the input signals received by the input unit, and (Col 5, ln 25-27, "The output from the look-up table 205 and the five outputs from the look-up table unit 206 are added to each other by an adder 207"; Col 5, line 15-16, "The five outputs from the look-up tables 203 are added to each other by an adder 204"; -- EN: Adder 207 (or adder 204) is the adder unit that sums the plurality of input signals received at the neuron's input.)
a storage unit that holds output results of a non-linear function corresponding to input values, uses an output signal output by the adder unit, as an input value, and outputs an output result of the non-linear function corresponding to the input value. (Col 5, line 27-28, "The sum data is input to a final sigmoid function look-up table 208, which outputs multi-value data"; Col. 5, line 16-18, "The sum data is input to a sigmoid function look-up table 205, which outputs data converted using the sigmoid function"; Col 4, ln 66-67, "assume that the sigmoid function is also processed using a look-up table"; Col 3, ln 30, “sigmoid f(X) = 1/(1+e⁻ˣ)” -- EN: The sigmoid-function look-up table (a ROM per Toda claim 3) holds the stored output values of the non-linear sigmoid function and, taking the adder's sum as its input/address value, outputs the corresponding non-linear function result.)
Claim 14
Toda teaches:
The machine learning processing circuit according to claim 12, further comprising:
a link circuit that uses some of the plurality of neuron cell circuit as output-end circuits, uses the plurality of neuron cell circuit other than the output-end circuits as intermediate circuits, and connects an output signal of each of the neuron cell circuits included in the intermediate circuits to input units of at least some of other neuron cell circuits. (Col 4, ln 17-19, "All the neurons in the intermediate layer are connected to a neuron in the output layer"; Col 5, ln 25-27, "The output from the look-up table 205 and the five outputs from the look-up table unit 206 are added to each other by an adder 207" -- EN: The output-layer neuron (adder 207/sigmoid LUT 208) is the output-end circuit and the intermediate-layer neurons (adder 204/sigmoid LUT 205 and look-up tables 206a-206e) are the intermediate circuits, and the wiring connecting these intermediate-circuit outputs to the input of adder 207 is the link circuit that connects each intermediate circuit's output signal to the input unit of another (output-end) neuron cell circuit.)
Claim 26
Toda teaches:
An information processing apparatus comprising: (Title/Abstract, "Image Processing Apparatus Using Neural Network" -- EN: Toda's image processing apparatus, which receives picture-element information and outputs processed multi-value information, is an information processing apparatus.)
a machine learning processing circuit including a plurality of neuron cell circuit, each of the plurality of neuron cell circuit including (Abstract, "a neural network arranged in an image processing apparatus"; Col 2, ln 65-67, "the connection pattern (weight coefficients) is adjusted by learning. An algorithm for the learning is called back propagation"; Col 5, ln 1-2, "the neural network shown in FIG. 1 is realized by a hardware arrangement shown in FIG. 2"; -- EN: Toda's neural network is trained by learning (back propagation) and implemented as a hardware processing circuit in FIG. 2, in which each neuron is realized by an adder plus a sigmoid look-up table so these correspond to the plurality of neuron cell circuits.)
an input unit that receives a plurality of input signals, (Col 5, ln 25-27, "The output from the look-up table 205 and the five outputs from the look-up table unit 206 are added to each other by an adder 207"; Col 5, line 15-16, "The five outputs from the look-up tables 203 are added to each other by an adder 204"; FIG. 5 showing inputs 501/502 feeding an ADD block -- EN: Each neuron's input receive a plurality of input signals, e.g., adder 207 receives six input signals and adder 204 receives five input signals)
an adder unit that adds the input signals received by the input unit, and (Col 5, ln 25-27, "The output from the look-up table 205 and the five outputs from the look-up table unit 206 are added to each other by an adder 207"; Col 5, line 15-16, "The five outputs from the look-up tables 203 are added to each other by an adder 204"; -- EN: Adder 207 (or adder 204) is the adder unit that sums the plurality of input signals received at the neuron's input.)
a storage unit that holds output results of a non-linear function corresponding to input values, uses an output signal output by the adder unit, as an input value, and outputs an output result of the non-linear function corresponding to the input value. (Col 5, line 27-28, "The sum data is input to a final sigmoid function look-up table 208, which outputs multi-value data"; Col. 5, line 16-18, "The sum data is input to a sigmoid function look-up table 205, which outputs data converted using the sigmoid function"; Col 4, ln 66-67, "assume that the sigmoid function is also processed using a look-up table"; Col 3, ln 30, “sigmoid f(X) = 1/(1+e⁻ˣ)” -- EN: The sigmoid-function look-up table (a ROM per Toda claim 3) holds the stored output values of the non-linear sigmoid function and, taking the adder's sum as its input/address value, outputs the corresponding non-linear function result.)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Examiner’s Note: Some rejections will include an Examiner’s Note (labeled ‘EN’) to provide additional context or rationale explaining the basis for the rejection.
Claims 13, 15, 16, 22, 23 are rejected under 35 U.S.C. 103 as being unpatentable over US patent Toda et al. (US006026178A), hereinafter “Toda” in view of patent publication Takamaeda et al. (US 20210232899 A1), hereinafter “Takamaeda” further in view of non-patent literature Shirakawa et al. (“Dynamic Optimization of Neural Network Structures Using Probabilistic Modeling”), hereinafter “Shirakawa”
Claim 13
Takamaeda teaches:
a switch circuit that uses some of the plurality of neuron cell circuit as output-end circuits, uses the plurality of neuron cell circuit other than the output-end circuits as intermediate circuits, (Para 272, "through which stepwise transmission and reception of one-bit output data O (input data I) are performed"; Para 274, "the neural network integrated circuits C1 to C4 can transmit and receive one-bit input data I and one-bit output data O to and from each other through switches SW1 to SW4, … connection modes between the neural network integrated circuits C1 to C4 … are switched by switch boxes SB1 to SB4 through the switches SW1 to SW4." -- EN: In the FIG. 20B/CC3 switch-box embodiment, the switch boxes route each integrated circuit's output O to another circuit as input I in a stepwise (multi-stage) network; the circuits whose output feeds other circuits are the intermediate circuits and the final-stage circuit producing the output is the output-end circuit, so the switch circuit designates which circuits serve as output-end vs. intermediate.)
and switches whether or not to connect an output signal of each of the neuron cell circuits included in the intermediate circuits to an input unit of a corresponding one of other neuron cell circuits, (Para 274, “In addition, as illustrated in FIG. 20B, the neural network integrated circuits C1 to C4 can transmit and receive one-bit input data I and one-bit output data O to and from each other through switches SW1 to SW4. … are switched by switch boxes SB1 to SB4 through the switches SW1 to SW4.” – EN: the switch boxes selectively connect an intermediate circuit's output to another circuit's input, which reads on "switches whether or not to connect.”)
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to combine Toda's LUT-based neuron cells with Takamaeda's switch-box interconnect, because Takamaeda teaches that switching the connections lets a single circuit flexibly realize various network topologies and sizes while reducing cost. (Takamaeda [0282]: "the large-scale neural network integrated circuit CC3 can be efficiently realized while significantly reducing the corresponding cost")
Toda in view of Takamaeda does not explicitly teach:
wherein the machine learning processing circuit updates a link relation between the neuron cell circuits through the switch circuit during machine learning.
However, Shirakawa teaches:
wherein the machine learning processing circuit updates a link relation between the neuron cell circuits through the switch circuit during machine learning. (Abstract, “we propose a method to simultaneously optimize the network structure and weight parameters during neural network training.” Page 7, “We decide the existence of the connections between layers in each dense block according to the binary vector M. Namely, we remove the connection when the corresponding bit equals zero.” – EN: Shirakawa optimizes (updates) the inter-layer connection structure simultaneously with the weights during training, which reads on updating the link relation during machine learning.
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to combine Toda's LUT-based neuron cells and Takamaeda's switch-box interconnect with the update link during machine learning of Shirakawa. The motivation for doing so would be to automatically obtain an appropriate, competitive network structure within the training loop. (Shirakawa, Abstract: "the proposed method can find the appropriate and competitive network structures").
Claim 15
Takamaeda teaches:wherein the plurality of neuron cell circuit are sorted into a plurality of neuron cell circuit groups each including a plurality of neuron cell circuit, (Para 274, “n neural network circuits CS11 to CS1n … are integrated to form one neural network integrated circuit C1 … m neural network circuits CS21 to CS2m are integrated to form one neural network integrated circuit C2, neural network circuits CS31 to CS3p … C3 … CS41 to CS4q … C4.” – EN: each integrated circuit C1–C4 is a group containing plural neuron cell circuits.)
the machine learning processing circuit further includes a switch circuit that switches whether or not to connect an output signal of each of the neuron cell circuits included in an i-th (i is a natural number equal to or greater than 1) neuron cell circuit group to an input unit of a corresponding one of the neuron cell circuits included in an (i+1)-th neuron cell circuit group, (Para 274, "the neural network integrated circuits C1 to C4 can transmit and receive one-bit input data I and one-bit output data O to and from each other through switches SW1 to SW4 … the modes of transmission and reception of the input data I and the output data O between the neural network integrated circuits C1 to C4 (that is, connection modes between the neural network integrated circuits C1 to C4) are switched by switch boxes SB1 to SB4"; Para 276, "the switch box SB for controlling the connection mode … and consequently the number of effective neurons NR is formed by connecting selectors M.sub.1 to M.sub.5"). -- EN: under BRI, designating any two switch-connected groups as the i-th group (e.g., C1) and the (i+1)-th group (e.g., C2) and the switch boxes/selectors checks the connection mode between them and "controlling … the number of effective neurons" reads on switching whether or not to connect a group's output to the next group's input.)
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to combine Toda's LUT-based neuron cells with Takamaeda's switch-box interconnect, because Takamaeda teaches that connecting plural integrated-circuit groups through switch boxes lets a large network be built from grouped circuits while reducing cost. (Takamaeda [0282]: "the large-scale neural network integrated circuit CC3 can be efficiently realized while significantly reducing the corresponding cost")
Toda in view of Takamaeda does not explicitly teach:
wherein the machine learning processing circuit updates a link relation between the neuron cell circuits through the switch circuit during machine learning.
However, Shirakawa teaches:
wherein the machine learning processing circuit updates a link relation between the neuron cell circuits through the switch circuit during machine learning. (Abstract, “we propose a method to simultaneously optimize the network structure and weight parameters during neural network training.” Page 7, “We decide the existence of the connections between layers in each dense block according to the binary vector M. Namely, we remove the connection when the corresponding bit equals zero.” – EN: Shirakawa optimizes (updates) the inter-layer connection structure simultaneously with the weights during training, which reads on updating the link relation during machine learning.
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to combine Toda's LUT-based neuron cells and switch-connected groups of Takamaeda with the update link during machine learning of Shirakawa. The motivation for doing so would be to automatically obtain an appropriate, competitive network structure within the training loop. (Shirakawa, Abstract: "the proposed method can find the appropriate and competitive network structures").
Claim 16
Takamaeda further teaches:the plurality of neuron cell circuit are sorted into a plurality of neuron cell circuit groups each including a plurality of neuron cell circuit (Para 274, “n neural network circuits CS11 to CS1n … are integrated to form one neural network integrated circuit C1 … m neural network circuits CS21 to CS2m are integrated to form one neural network integrated circuit C2, neural network circuits CS31 to CS3p … C3 … CS41 to CS4q … C4.” – EN: each integrated circuit C1–C4 is a group containing plural neuron cell circuits.)
the machine learning processing circuit further includes a switch circuit that switches whether or not to connect an output signal of each of the neuron cell circuits included in an i-th (i is a natural number equal to or greater than 1) neuron cell circuit group to an input unit of a corresponding one of the neuron cell circuits included in an (i+1)-th neuron cell circuit group, (Para 274, "the neural network integrated circuits C1 to C4 can transmit and receive one-bit input data I and one-bit output data O to and from each other through switches SW1 to SW4 … the modes of transmission and reception of the input data I and the output data O between the neural network integrated circuits C1 to C4 (that is, connection modes between the neural network integrated circuits C1 to C4) are switched by switch boxes SB1 to SB4"; Para 276, "the switch box SB for controlling the connection mode … and consequently the number of effective neurons NR is formed by connecting selectors M.sub.1 to M.sub.5"). -- EN: under BRI, designating any two switch-connected groups as the i-th group (e.g., C1) and the (i+1)-th group (e.g., C2) and the switch boxes/selectors checks the connection mode between them and "controlling … the number of effective neurons" reads on switching whether or not to connect a group's output to the next group's input.)
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to combine Toda's LUT-based neuron cells with Takamaeda's switch-box interconnect, because Takamaeda teaches that connecting plural integrated-circuit groups through switch boxes lets a large network be built from grouped circuits while reducing cost. (Takamaeda [0282]: "the large-scale neural network integrated circuit CC3 can be efficiently realized while significantly reducing the corresponding cost")
Toda in view of Takamaeda does not explicitly teach:
wherein the machine learning processing circuit updates a link relation between the neuron cell circuits through the switch circuit during machine learning.
However, Shirakawa further teaches:
wherein the machine learning processing circuit updates a link relation between the neuron cell circuits through the switch circuit during machine learning. (Abstract, “we propose a method to simultaneously optimize the network structure and weight parameters during neural network training.” Page 7, “We decide the existence of the connections between layers in each dense block according to the binary vector M. Namely, we remove the connection when the corresponding bit equals zero.” – EN: Shirakawa optimizes (updates) the inter-layer connection structure simultaneously with the weights during training, which reads on updating the link relation during machine learning.
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to combine Toda's LUT-based neuron cells and switch-connected groups of Takamaedas with the update link during machine learning of Shirakawa. The motivation for doing so would be to automatically obtain an appropriate, competitive network structure within the training loop. (Shirakawa, Abstract: "the proposed method can find the appropriate and competitive network structures").
Claim 22
The machine learning processing circuit according to claim 15, (Toda in view of Takamaeda further in view of Shirakawa teaches the machine learning processing circuit according to claim 15 (see rejection of claim 15 above).
Toda further teaches:
(…) are first-type neuron cell circuits each including the storage unit that is of a first type configured to hold output results of a first non-linear function corresponding to input values, use an output signal output by the adder unit, as an input value, and output an output result of the first non-linear function corresponding to the input value, and (Col 5, line 27-28, "The sum data is input to a final sigmoid function look-up table 208, which outputs multi-value data"; Col. 5, line 16-18, "The sum data is input to a sigmoid function look-up table 205, which outputs data converted using the sigmoid function"; Col 4, ln 66-67, "assume that the sigmoid function is also processed using a look-up table"; Col 3, ln 30, “sigmoid f(X) = 1/(1+e⁻ˣ)” -- EN: The sigmoid-function look-up table (a ROM per Toda claim 3) holds the stored output values of the non-linear sigmoid function and, taking the adder's sum as its input/address value, outputs the corresponding non-linear function result.)
Toda does not explicitly teach:
wherein at least some of the neuron cell circuits included in the i-th (i is a natural number equal to or greater than 1) neuron cell circuit group
However, Takamaeda teaches:
wherein at least some of the neuron cell circuits included in the i-th (i is a natural number equal to or greater than 1) neuron cell circuit group (Para 274, "n neural network circuits CS11 to CS1n … are integrated to form one neural network integrated circuit C1"; -- EN: this denotes that the neural network circuits CS integrated into integrated circuit C1 read on neuron cell circuits included in the i-th neuron cell circuit group.)
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to combine Toda's LUT-based neuron cells with Takamaeda's arrangement of neuron cell circuits into integrated-circuit groups, because Takamaeda teaches that integrating plural neuron cell circuits into grouped integrated circuits lets a large-scale neural network be efficiently realized while significantly reducing cost. (Takamaeda [0282]: "the large-scale neural network integrated circuit CC3 can be efficiently realized while significantly reducing the corresponding cost")
Toda in view of Takamaeda does not explicitly teach:
at least one of the neuron cell circuits different from the first-type neuron cell circuits among the neuron cell circuits included in the i-th (i is a natural number equal to or greater than 1) neuron cell circuit group is a second-type neuron cell circuit including the storage unit that is of a second type configured to hold output results of a second non-linear function different from the first non-linear function and corresponding to input values, use an output signal output by the adder unit, as an input value, and output an output result of the second non-linear function corresponding to the input value.
However, Shirakawa teaches:
at least one of the neuron cell circuits different from the first-type neuron cell circuits among the neuron cell circuits included in the i-th (i is a natural number equal to or greater than 1) neuron cell circuit group is a second-type neuron cell circuit including the storage unit that is of a second type configured to hold output results of a second non-linear function different from the first non-linear function and corresponding to input values, use an output signal output by the adder unit, as an input value, and output an output result of the second non-linear function corresponding to the input value. (Page 5, "Different activation functions can be mixed in the same layer. The activation function of i-th unit is ReLU Frelu if mi = 1 and the hyperbolic tangent Ftanh if mi = 0"; EN: this denotes that within a single layer, which under the broadest reasonable interpretation reads on the i-th neuron cell circuit group, at least one unit uses the hyperbolic tangent (second-type) different from the ReLU first-type used by other units. Holding this different non-linear function in Toda’s sigmoid-type look-up table (the storage unit of claim 12) yields the second-type storage unit, so the same group contains a second-type neuron cell circuit holding a second, different non-linear function.)
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to configure Toda's first-type neuron cell circuits, as arranged in Takamaeda's i-th group, so that at least one neuron cell circuit within that same group is instead a second-type neuron cell circuit holding a second, different non-linear function as taught by Shirakawa, in order to improve classification performance, because Shirakawa teaches that mixing activation functions within a layer outperforms a uniform activation. (Shirakawa, page 5, "the obtained networks by AdaptiveActivation have a better classification performance compared to both uniform activations: ReLU and hyperbolic tangent").
Claim 23
The machine learning processing circuit according to claim 16, (Toda in view of Takamaeda further in view of Shirakawa teaches the machine learning processing circuit according to claim 16 (see rejection of claim 16 above).
Toda further teaches:
(…) are first-type neuron cell circuits each including the storage unit that is of a first type configured to hold output results of a first non-linear function corresponding to input values, use an output signal output by the adder unit, as an input value, and output an output result of the first non-linear function corresponding to the input value, and (Col 5, line 27-28, "The sum data is input to a final sigmoid function look-up table 208, which outputs multi-value data"; Col. 5, line 16-18, "The sum data is input to a sigmoid function look-up table 205, which outputs data converted using the sigmoid function"; Col 4, ln 66-67, "assume that the sigmoid function is also processed using a look-up table"; Col 3, ln 30, “sigmoid f(X) = 1/(1+e⁻ˣ)” -- EN: The sigmoid-function look-up table (a ROM per Toda claim 3) holds the stored output values of the non-linear sigmoid function and, taking the adder's sum as its input/address value, outputs the corresponding non-linear function result.)
Toda does not explicitly teach:
wherein at least some of the neuron cell circuits included in the i-th (i is a natural number equal to or greater than 1) neuron cell circuit group
However, Takamaeda teaches:
wherein at least some of the neuron cell circuits included in the i-th (i is a natural number equal to or greater than 1) neuron cell circuit group (Para 274, "n neural network circuits CS11 to CS1n … are integrated to form one neural network integrated circuit C1"; -- EN: this denotes that the neural network circuits CS integrated into integrated circuit C1 read on neuron cell circuits included in the i-th neuron cell circuit group.)
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to combine Toda's LUT-based neuron cells with Takamaeda's arrangement of neuron cell circuits into integrated-circuit groups, because Takamaeda teaches that integrating plural neuron cell circuits into grouped integrated circuits lets a large-scale neural network be efficiently realized while significantly reducing cost. (Takamaeda [0282]: "the large-scale neural network integrated circuit CC3 can be efficiently realized while significantly reducing the corresponding cost")
Toda in view of Takamaeda does not explicitly teach:
at least one of the neuron cell circuits different from the first-type neuron cell circuits among the neuron cell circuits included in the i-th (i is a natural number equal to or greater than 1) neuron cell circuit group is a second-type neuron cell circuit including the storage unit that is of a second type configured to hold output results of a second non-linear function different from the first non-linear function and corresponding to input values, use an output signal output by the adder unit, as an input value, and output an output result of the second non-linear function corresponding to the input value.
However, Shirakawa teaches:
at least one of the neuron cell circuits different from the first-type neuron cell circuits among the neuron cell circuits included in the i-th (i is a natural number equal to or greater than 1) neuron cell circuit group is a second-type neuron cell circuit including the storage unit that is of a second type configured to hold output results of a second non-linear function different from the first non-linear function and corresponding to input values, use an output signal output by the adder unit, as an input value, and output an output result of the second non-linear function corresponding to the input value. (Page 5, "Different activation functions can be mixed in the same layer. The activation function of i-th unit is ReLU Frelu if mi = 1 and the hyperbolic tangent Ftanh if mi = 0"; -- EN: this denotes that within a single layer, which under the broadest reasonable interpretation reads on the i-th neuron cell circuit group, at least one unit uses the hyperbolic tangent (second-type) different from the ReLU first-type used by other units. Holding this different non-linear function in Toda’s sigmoid-type look-up table (the storage unit of claim 12) yields the second-type storage unit, so the same group contains a second-type neuron cell circuit holding a second, different non-linear function.)
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to configure Toda's first-type neuron cell circuits, as arranged in Takamaeda's i-th group, so that at least one neuron cell circuit within that same group is instead a second-type neuron cell circuit holding a second, different non-linear function as taught by Shirakawa, in order to improve classification performance, because Shirakawa teaches that mixing activation functions within a layer outperforms a uniform activation. (Shirakawa, page 5, "the obtained networks by AdaptiveActivation have a better classification performance compared to both uniform activations: ReLU and hyperbolic tangent").
Claim 17 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over US patent Toda et al. (US006026178A), hereinafter “Toda” in view of patent publication Takamaeda et al. (US 20210232899 A1), hereinafter “Takamaeda”
Claim 17
Takamaeda teaches:
The machine learning processing circuit according to claim 12 wherein the plurality of neuron cell circuit are sorted into a plurality of neuron cell circuit groups each including a plurality of neuron cell circuit, (Para 262, "n neural network circuits CS11 to CS1n … are integrated to form one neural network integrated circuit C1 … These neural network circuits CS21 to CS2n form another neural network integrated circuit C2." – EN: this denotes each integrated circuit C1, C2, C3 is a group of plural neuron cell circuits.)
and the machine learning processing circuit further includes a link circuit that connects an output signal of each of the neuron cell circuits included in an i-th (i is a natural number equal to or greater than 1) neuron cell circuit group to input units of at least some of the neuron cell circuits included in an (i+1)-th neuron cell circuit group. (Para 262, “the neural network circuits CS11 to CS1n forming the neural network integrated circuit C1 output one-bit output data O11 to one-bit output data O1n, respectively, and these are commonly input to n neural network circuits CS21 to CS2n in the next stage.” – EN: this denotes the outputs of group C1 (the i-th group) are wired as inputs to group C2 (the (i+1)-th group.)
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine Toda's LUT-based neuron cells with Takamaeda's grouped multi-stage series structure in which the i-th group's outputs are linked as inputs to the (i+1)-th group. The motivation for doing so would be increasing the number of layers. See Takamaeda [0216]: "a two-layer neural network having three inputs and two outputs and a two-layer neural network having two inputs and four outputs are connected in series," and at [0212]: "FIG. 13 is a schematic diagram illustrating an example of increasing the number of layers of the neural network."
Claim 18
Takamaeda teaches:
The machine learning processing circuit according to claim 14 wherein the plurality of neuron cell circuit are sorted into a plurality of neuron cell circuit groups each including a plurality of neuron cell circuit, (Para 262, "n neural network circuits CS11 to CS1n … are integrated to form one neural network integrated circuit C1 … These neural network circuits CS21 to CS2n form another neural network integrated circuit C2." – EN: this denotes each integrated circuit C1, C2, C3 is a group of plural neuron cell circuits.)
and the machine learning processing circuit further includes a link circuit that connects an output signal of each of the neuron cell circuits included in an i-th (i is a natural number equal to or greater than 1) neuron cell circuit group to input units of at least some of the neuron cell circuits included in an (i+1)-th neuron cell circuit group. (Para 262, “the neural network circuits CS11 to CS1n forming the neural network integrated circuit C1 output one-bit output data O11 to one-bit output data O1n, respectively, and these are commonly input to n neural network circuits CS21 to CS2n in the next stage.” – EN: this denotes the outputs of group C1 (the i-th group) are wired as inputs to group C2 (the (i+1)-th group.)
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine Toda's LUT-based neuron cells with Takamaeda's grouped multi-stage series structure in which the i-th group's outputs are linked as inputs to the (i+1)-th group. The motivation for doing so would be increasing the number of layers. See Takamaeda [0216]: "a two-layer neural network having three inputs and two outputs and a two-layer neural network having two inputs and four outputs are connected in series," and at [0212]: "FIG. 13 is a schematic diagram illustrating an example of increasing the number of layers of the neural network."
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over US patent Toda et al. (US006026178A), hereinafter “Toda” in view of non-patent literature Shirakawa et al. (“Dynamic Optimization of Neural Network Structures Using Probabilistic Modeling”), hereinafter “Shirakawa”
Claim 21
Toda teaches:
The machine learning processing circuit according to claim 12, wherein some of the plurality of neuron cell circuit are first-type neuron cell circuits each including the storage unit that is of a first type configured to hold output results of a first non- linear function corresponding to input values, use an output signal output by the adder unit, as an input value, and output an output result of the first non-linear function corresponding to the input value, and (Col 5, line 27-28, "The sum data is input to a final sigmoid function look-up table 208, which outputs multi-value data"; Col. 5, line 16-18, "The sum data is input to a sigmoid function look-up table 205, which outputs data converted using the sigmoid function"; Col 4, ln 66-67, "assume that the sigmoid function is also processed using a look-up table"; Col 3, ln 30, “sigmoid f(X) = 1/(1+e⁻ˣ)” -- EN: Similar to claim 1 above, the sigmoid-function look-up table (a ROM per Toda claim 3) holds the stored output values of the non-linear sigmoid function and, taking the adder's sum as its input/address value, outputs the corresponding non-linear function result.)
Toda does not explicitly teach:
at least one of the neuron cell circuits different from the first-type neuron cell circuits among the plurality of neuron cell circuit is a second-type neuron cell circuit including the storage unit that is of a second type configured to hold output results of a second non-linear function different from the first non-linear function and corresponding to input values, use an output signal output by the adder unit, as an input value, and output an output result of the second non-linear function corresponding to the input value.
However, Shirakawa teaches:
at least one of the neuron cell circuits different from the first-type neuron cell circuits among the plurality of neuron cell circuit is a second-type neuron cell circuit including the storage unit that is of a second type configured to hold output results of a second non-linear function different from the first non-linear function and corresponding to input values, use an output signal output by the adder unit, as an input value, and output an output result of the second non-linear function corresponding to the input value (Page 5, “(II) Selection of Activation Functions … Different activation functions can be mixed in the same layer. The activation function of i-th unit is ReLU Frelu if mi = 1 and the hyperbolic tangent Ftanh if mi = 0.” -- EN: This denotes that while some units apply ReLU as the first non-linear function, at least one other unit applies the hyperbolic tangent, a second non-linear function different from the first, which when combined with Toda's LUT/ROM storage unit reads on a second-type neuron cell circuit whose storage unit holds a second, different non-linear function.)
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to configure Toda's LUT-based neuron cell circuits so that at least one holds a second non-linear function different from the first, as taught by Shirakawa, in order to improve classification performance, because Shirakawa teaches that networks mixing activation functions outperform uniform ones. See Shirakawa, page 5, "In addition, the obtained networks by AdaptiveActivation have a better classification performance compared to both uniform activations: ReLU and hyperbolic tangent. Comparing the training time, we observe that the proposed method needs about twice the computational time for training compared to the fixed structured neural networks.")
Claims 19, 20 are rejected under 35 U.S.C. 103 as being unpatentable over US patent Toda et al. (US006026178A), hereinafter “Toda” in view of patent publication Takamaeda et al. (US 20210232899 A1), hereinafter “Takamaeda” in view of non-patent literature Shirakawa et al. (“Dynamic Optimization of Neural Network Structures Using Probabilistic Modeling”), hereinafter “Shirakawa” further in view of non-patent literature Du et al. (“ShiDianNao: Shifting Vision Processing Closer to the Sensor”), hereinafter “Du”
Claim 19
The machine learning processing circuit according to claim 15, (Toda in view of Takamaeda in view of Shirakawa teaches claim 15 (see the rejection of claim 15 above))
Du teaches:
wherein the number of input signals received by the input unit of each of the neuron cell circuits is set such that there are i and j where a value of the number Ni of input signals received by the input units of the neuron cell circuits included in the i-th (i is a natural number equal to or greater than 1) neuron cell circuit group is smaller than a value of the number Nj of input signals received by the input units of the neuron cell circuits included in a j-th (j is a natural number equal to or greater than 1 where j>i) neuron cell circuit group. (Page 94, Section 3, "Convolutional Layer," "Each local filter has a kernel having Kx × Ky coefficients, and processes a convolutional window capturing Kx ×Ky input neurons in one input feature map” … “Classifier Layer," "In a typical classifier layer, output neurons are fully connected to input neurons with independent synapses" Page 93, “It consists of two convolutional layers (Cl and C3 in Figure 2), two pooling layers (S2 and S4 in Figure 2), and three classifier layers (F5, F6 and F7 in Figure 2).” – EN: In Du's CNN, a neuron in an earlier convolutional-layer group (i) receives only the small Kx×Ky window of inputs, while a neuron in a deeper fully-connected classifier-layer group (j, j>i) receives every input neuron, so the per-neuron input count of the later group Nj is larger than that of the earlier group Ni.)
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to combine Toda's LUT-based neuron cells, Takamaeda's switch-connected groups, and Shirakawa's update-link-during-machine-learning with Du's convolutional-then-classifier layer organization in which a deeper group's neurons receive more input signals than an earlier group's neurons. The motivation for doing so would be to enable the deeper, fully-connected layers to integrate information from all input neurons and compute the final recognition result. (Du, Page 94, Section 3 "Classifier Layer": "After a sequence of other layers, a CNN integrates one or more classifier layers to compute the final result")
Claim 20
The machine learning processing circuit according to claim 16, (Toda in view of Takamaeda in view of Shirakawa teaches claim 16 (see the rejection of claim 16 above))
Du teaches:
wherein the number of input signals received by the input unit of each of the neuron cell circuits is set such that there are i and j where a value of the number Ni of input signals received by the input units of the neuron cell circuits included in the i-th (i is a natural number equal to or greater than 1) neuron cell circuit group is smaller than a value of the number Nj of input signals received by the input units of the neuron cell circuits included in a j-th (j is a natural number equal to or greater than 1 where j>i) neuron cell circuit group. (Page 94, Section 3, "Convolutional Layer," "Each local filter has a kernel having Kx × Ky coefficients, and processes a convolutional window capturing Kx ×Ky input neurons in one input feature map” … “Classifier Layer," "In a typical classifier layer, output neurons are fully connected to input neurons with independent synapses" Page 93, “It consists of two convolutional layers (Cl and C3 in Figure 2), two pooling layers (S2 and S4 in Figure 2), and three classifier layers (F5, F6 and F7 in Figure 2).” – EN: In Du's CNN, a neuron in an earlier convolutional-layer group (i) receives only the small Kx×Ky window of inputs, while a neuron in a deeper fully-connected classifier-layer group (j, j>i) receives every input neuron, so the per-neuron input count of the later group Nj is larger than that of the earlier group Ni.)
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to combine Toda's LUT-based neuron cells, Takamaeda's switch-connected groups, and Shirakawa's update-link-during-machine-learning with Du's convolutional-then-classifier layer organization in which a deeper group's neurons receive more input signals than an earlier group's neurons. The motivation for doing so would be to enable the deeper, fully-connected layers to integrate information from all input neurons and compute the final recognition result. (Du, Page 94, Section 3 "Classifier Layer": "After a sequence of other layers, a CNN integrates one or more classifier layers to compute the final result")
Claims 24 and 25 are rejected under 35 U.S.C. 103 as being unpatentable over US patent Toda et al. (US006026178A), hereinafter “Toda” in view of non-patent literature Du et al. (“ShiDianNao: Shifting Vision Processing Closer to the Sensor”), hereinafter “Du”
Claim 24
Du teaches:
wherein the machine learning processing circuit includes a die provided with the plurality of neuron cell circuit, and a chip is formed. (Page 95, Section 5.1, "our NFU is a 2D mesh of Px ×Py Processing Elements (PEs)"; Page 101, Section 10.1, "ShiDianNao has 8 × 8 (64) Pes … the total area of ShiDianNao is only 3.52× larger than that of DianNao (4.86 mm2 vs. 1.38 mm2)"; Page 92, Abstract, "We present a full design down to the layout at 65 nm, with a modest footprint of 4.86 mm2" -- EN: Du constructs its mesh of processing elements (the plurality of neuron cell circuits) as a single 65 nm CMOS chip occupying a 4.86 mm² die, which is a die provided with the plurality of neuron cell circuits from which a chip is formed. Also see figure 17 below.
PNG
media_image1.png
373
426
media_image1.png
Greyscale
)
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to form Toda's plurality of LUT-based neuron cell circuits on a die as a chip, as taught by Du. The motivation for doing so would be to obtain a compact, energy-efficient hardware implementation of the network. (Du, Page 92, Abstract: "a modest footprint of 4.86 mm2 and consuming only 320 mW, but still about 30× faster than high-end GPUs").
Claim 25
Du teaches:
further comprising: shift register circuit units provided to correspond to the neuron cell circuits, (Page 95, Section 4, “Overall, we have chosen the mapping in Figure 3(d): our processing elements (i) represent neurons, (ii) are organized in a 2D mesh…” – EN: this denotes every neuron in Du is built as a PE, and each PE has its own pair of FIFO buffers. A FIFO that clocks data in and shifts its out in order is a shift register under BRI. See Page 95 of Du where he describes the input feature map moving through the mesh “receive through right-left and up-down shifts the input feature map…”
each of the shift register circuit units being configured to receive an input of data at every predetermined timing, (Page 95, “Processing elements. At each cycle, each PE can perform a multiplication and an addition for a convolutional, classifier, or normalization layer, or just an addition for an average pooling layer, or a comparison for a max pooling layer, etc”; Page 98-99, “Cycle #0: All four PEs respectively read the first input neurons… Cycle #1: PEo,o and PEo, I respectively read their required data (input neurons XI,O and XI,]) from the FIFO-Hs of PEI,o and PEI,I… Cycle #2: Similar to Cycle #1, PEo,o and PEo,1 respectively read…” – EN: this denotes the clock cycle is the “predetermined timing”. And on each cycle every PE pulls in a input neuron and writes it into its FIFOs, so each shift register receives an input of data at every predetermined timing.)
hold data input in a predetermined number of times in the past, and (Page 96, “We enable this by having two FIFOs (horizontal and vertical: FIFO-H and FIFO-V) in each PE to temporarily store the input values it received.”; Page 98, “In addition, each PE collects its received input neuron in its FIFO-H and FIFO-V for future inter-PE data propagation.” Page 99, “Cycle #3: PEo,o and PEI,o respectively read their required data (input neurons XO,I and xI,d from FIFO-Vs of PEo, I and PEI I…” – EN: this denotes Du’s FIFO-H and FIFO-V “temporarily store the input values it received” and a FIFO holds a fixed number of the most-recent inputs, so it holds data “a predetermined number of times in the past”).
output at least part of the held data to a neuron cell circuit in a later stage at the predetermined timing. (Page 96, "FIFO-H buffers data … such data will be propagated to the left neighbor PE for reuse. FIFO-V buffers the data … such data will be propagated to the lower neighbor PE for reuse"; Page 96, (each PE has) “one output for propagating locally-stored neurons to neighbor Pes” – EN: pages 96-98 denote the data sitting in a PE’s FIFO is clocked out to a neighboring PE (another neuron) which consumes it on a later cycle. The receiving neighbor is the neuron cell receiving at least part of the held data.)
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to combine Toda's LUT-based neuron cell circuits with the shift registers/FIFO buffers that hold input data and forward it to neighboring neurons of Du. The motivation for doing so would be to reuse the held input data across neuron cell circuits so that the same data doesn’t need to be repeatedly re-supplied, thus reducing the extra data movement and the associated hardware/bandwidth cost. See Du page 96, “With inter PE data propagation, the internal bandwidth requirement can be drastically reduced (see Figure 7).”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NAYMUR RAHMAN ALI whose telephone number is (571)272-0007. The examiner can normally be reached Mon-Fri. 9:30-6:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached at (571)270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NAYMUR RAHMAN ALI/Examiner, Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights